Image detection method and apparatus

Abstract

The present disclosure provides an image detection method and apparatus, which is related to the field of computer vision. A specific embodiment of the method includes: performing instance segmentation on a detection target image using an image instance segmentation model to obtain each instance in the detection target image; calculating a tentative center of each instance based on point cloud data and each instance; correcting the tentative center of each instance using an instance center correction model to obtain a corrected center of each instance; inputting the corrected center of each instance into a target detection model to output a box body and a category of each instance. This embodiment can solve the problem of poor image detection performance.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to Chinese Patent Application No. 202210158788.3, ​​filed on February 21, 2022, with the publication title "Image Detection Method and Apparatus," and the contents disclosed in the above Chinese patent application are hereby incorporated by reference in their entirety as part or in whole into this application. [Technical Field]

[0002] The present disclosure relates to the field of computer vision, and in particular to image detection methods and apparatus. [Background technology]

[0003] Autonomous vehicles are typically equipped with multiple types of sensors for detecting obstacles, including LIDAR, RGB imaging cameras, and millimeter-wave radar. However, different sensors have different imaging principles and mounting angles, resulting in different output results. For example, a top-mounted multi-line LIDAR outputs a 360-degree point cloud, while an imaging sensor consists of multiple RGB cameras, each with a fixed field of view and responsible for generating RGB images corresponding to a specific region. The data generated by these sensors differs in modality, resulting in different data characteristics. For example, LIDAR is insensitive to light intensity and can be used at night, but it is expensive, and mass-produced autonomous vehicles generally do not allow for high-wire harness products, resulting in sparse point cloud data and low resolution. RGB imaging cameras are low cost, have high resolution, and can clearly capture texture features, but are easily affected by changes in weather and ambient light intensity.

[0004] From an algorithmic perspective, detection results are typically obtained using dedicated algorithms on data corresponding to each modality, and then the detection results of each model are fused. This is commonly referred to in the industry as a "post-fusion method." The corresponding method is called a "pre-fusion method," which uses deep models or algorithms to process all data from different modalities collectively and directly obtain the final detection results. For example, the AVOD algorithm uses deep neural networks to extract features from data from different modalities to obtain feature maps, and then performs fusion at the feature map level. Due to differences in the data from different modalities, it is difficult to align feature maps from different modalities, such as a feature map from a 360-degree point cloud and a feature map corresponding to a front-view camera. However, whether it is a post-fusion method or a front-view fusion method, the fusion effect is poor, resulting in poor image detection performance. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide an image detection method and apparatus to solve the problem of poor image detection performance. In order to achieve the above object, according to one aspect of the embodiment of the present disclosure, performing instance segmentation on the target image using an image instance segmentation model to obtain each instance in the target image; calculating a virtual center of each of the instances based on the point cloud data and each of the instances; correcting the tentative center of each of the instances using an instance center correction model to obtain a corrected center of each of the instances; inputting the corrected center of each instance into a target detection model and outputting a box and category of each instance; Contains An image detection method is provided.

[0006] According to one or more embodiments of the present disclosure, calculating a virtual center of each of the instances based on point cloud data and each of the instances includes: projecting point cloud data onto each of the instances; Merging point cloud data of the same instance appearing in different detection target images according to point cloud data of overlapping portions in each of the instances; calculating point cloud data of the body of each of the instances using a clustering algorithm; calculating a virtual center of each of the instances based on point cloud data of the body of each of the instances; Includes.

[0007] According to one or more embodiments of the present disclosure, calculating point cloud data of the body of each of the instances using a clustering algorithm includes: calculating point cloud data of the main body of each of the instances using a spatial clustering algorithm, and deleting point cloud data other than the main body of each of the instances; Includes.

[0008] According to one or more embodiments of the present disclosure, prior to performing instance segmentation on the detection target image using an image instance segmentation model and obtaining each instance in the detection target image, further comprising: Obtaining a sample image at each angle, marking a position, a category, and a contour of each sample instance on the sample image at each angle, and training using a first model to obtain an image instance segmentation model; Calculating a tentative center of each sample instance, marking a box of each sample instance in the sample point cloud data, and training using a second model to obtain an instance center correction model; using and training a third model to obtain a target detection model; Includes.

[0009] According to one or more embodiments of the present disclosure, training using a first model to obtain an image instance segmentation model includes: training an image instance segmentation model by inputting the sample images of each angle and the position, category, and contour of each sample instance into a first model and training the first model; Including, Among them, the first model is Mask R-CNN, Hybrid Task Cascade, or BlendMaskd.

[0010] According to one or more embodiments of the present disclosure, calculating a tentative center of each sample instance, marking a box of each sample instance in sample point cloud data, and training using a second model to obtain an instance center correction model may include: calculating a tentative center of each of the sample instances based on the sample point cloud data and each of the sample instances; marking a box for each sample instance in the sample point cloud data and calculating the center of the box for each sample instance; training and obtaining an instance center correction model by inputting the virtual center of each sample instance and the center of the box body of each sample instance into a second model and training the second model; Includes.

[0011] According to one or more embodiments of the present disclosure, calculating a tentative center of each of the sample instances based on sample point cloud data and each of the sample instances includes: projecting sample point cloud data onto each of the sample instances; Merging point cloud data of the same sample instance appearing in different sample images according to point cloud data of overlapping portions in each of the sample instances; calculating point cloud data of the body of each sample instance using a clustering algorithm; calculating a tentative center of each sample instance based on point cloud data of the body of each sample instance; Includes.

[0012] According to one or more embodiments of the present disclosure, the second model is PointNet, PointNet++, Cylider3D, or Rand-LA.

[0013] According to one or more embodiments of the present disclosure, training using a third model to obtain a target detection model includes: Inputting attribute data of each point cloud of each sample instance and the box body of each sample instance into a third model and training the third model, thereby training and obtaining a target detection model. Including, The attribute data of each point group includes three-dimensional coordinates, a category, and the center coordinates of the box body to which it belongs.

[0014] According to one or more embodiments of the present disclosure, the third model is PointPillars, VoxelNet, Sparsely Embedded Convolutional Detection, or CenterPoint.

[0015] According to another aspect of the embodiment of the present disclosure, a segmentation module for performing instance segmentation on the target image using an image instance segmentation model to obtain each instance in the target image; a calculation module for calculating a virtual center of each of the instances based on point cloud data and each of the instances; a correction module for correcting the tentative center of each of the instances using an instance center correction model to obtain a corrected center of each of the instances; a detection module for inputting the corrected center of each instance into a target detection model and outputting a box and category for each instance; Contains The present invention provides an image detection device.

[0016] According to one or more embodiments of the present disclosure, the computing module may further include: projecting the point cloud data onto each of the instances; Merging point cloud data of the same instance appearing in different detection target images according to point cloud data of the overlapping portion of each of the instances; calculating point cloud data of the body of each of the instances using a clustering algorithm; Calculating a virtual center of each of the instances based on point cloud data of the main body of each of the instances This is for the purpose.

[0017] According to one or more embodiments of the present disclosure, the computing module may further include: Calculating point cloud data of the main body of each of the instances using a spatial clustering algorithm, and deleting point cloud data other than the main body of each of the instances. This is for the purpose.

[0018] According to one or more embodiments of the present disclosure, there is further provided Obtaining a sample image for each angle, marking a position, a category, and a contour of each sample instance on the sample image for each angle, and training using a first model to obtain an image instance segmentation model; Calculating a tentative center of each sample instance, marking a box of each sample instance in the sample point cloud data, and training using a second model to obtain an instance center correction model; Use a third model and train it to obtain a target detection model Training modules for Includes.

[0019] According to one or more embodiments of the present disclosure, the training module may further include: The sample images at each angle and the positions, categories, and contours of each sample instance are input to a first model for training, thereby training and obtaining an image instance segmentation model. It is for the purpose of Among them, the first model is Mask R-CNN, Hybrid Task Cascade, or BlendMaskd.

[0020] According to one or more embodiments of the present disclosure, the training module may further include: calculating a tentative center of each of the sample instances based on the sample point cloud data and each of the sample instances; Marking a box for each sample instance in the sample point cloud data and calculating the center of the box for each sample instance; The virtual center of each sample instance and the center of the box body of each sample instance are input to a second model and training is performed to train and obtain an instance center correction model. This is for the purpose.

[0021] According to one or more embodiments of the present disclosure, the training module may further include: projecting the sample point cloud data onto each of the sample instances; Merging point cloud data of the same sample instance appearing in different sample images according to point cloud data of the overlapping portions of each of the sample instances; calculating point cloud data of the body of each sample instance using a clustering algorithm; Calculating a tentative center of each sample instance based on point cloud data of the body of each sample instance This is for the purpose.

[0022] According to one or more embodiments of the present disclosure, the second model is PointNet, PointNet++, Cylider3D, or Rand-LA.

[0023] According to one or more embodiments of the present disclosure, the training module may further include: The attribute data of each point cloud of each sample instance and the box body of each sample instance are input to a third model for training, thereby training and obtaining a target detection model. It is for the purpose of The attribute data of each point group includes three-dimensional coordinates, a category, and the center coordinates of the box body to which it belongs.

[0024] According to one or more embodiments of the present disclosure, the third model is PointPillars, VoxelNet, Sparsely Embedded Convolutional Detection, or CenterPoint.

[0025] According to another aspect of the present disclosure, the present invention further comprises: one or more processors; a storage device for storing one or more programs; Including, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of the above embodiments. We provide electronic devices.

[0026] According to another aspect of the present disclosure, the present invention further comprises: A computer program that, when executed by a processor, implements the method according to any one of the above embodiments is stored. A computer-readable medium is provided.

[0027] One embodiment of the above disclosure has the following advantages or beneficial effects. The problem of poor image detection performance in the prior art is solved by using an image instance segmentation model to perform instance segmentation on the detection target image, an instance center correction model to correct the virtual center of each instance, and inputting the corrected center of each instance into the target detection model. In the embodiments of the present disclosure, the image instance segmentation results are used to perform instance-level annotation for each point in the point cloud, and the center of each instance is corrected, and then target detection is performed using a point cloud detection algorithm. Since the point cloud instance information is obtained before being sent to the point cloud detection algorithm, the difficulty of training the detection algorithm is greatly reduced and detection performance is greatly improved.

[0028] Further advantages of the above-described unconventional alternatives will be explained in conjunction with the detailed description below. [Brief explanation of the drawings]

[0029] The drawings are intended to provide a better understanding of the present disclosure and are not to be construed as undue limitations thereon.

[0030] [Figure 1] FIG. 1 is a schematic diagram of the main processes of an image detection method according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a schematic diagram of the main process of an image detection method according to one possible embodiment of the present disclosure. [Figure 3] FIG. 3 is a schematic diagram of the main modules of an image detection device according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is an exemplary system architecture diagram in which embodiments of the present disclosure can be applied. [Figure 5] FIG. 5 is a structural schematic diagram of a computer system of a terminal or server suitable for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0031] The following describes exemplary embodiments of the present disclosure in conjunction with the drawings. The detailed description of the embodiments of the present disclosure is intended to be illustrative and not restrictive. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. For clarity and conciseness, the following description will omit descriptions of known functions and structures.

[0032] In an embodiment of the present disclosure, the instance segmentation results of an image are used to perform instance-level annotation for each point in a point cloud. After the first stage of instance segmentation, each point cloud data is labeled with a category such as car, pedestrian, or bicycle. The first stage of instance segmentation can also identify different instances of the same category, such as 10 cars or 20 pedestrians. Instance segmentation can distinguish between instances of the same category, such as car 1, car 2, car 3, etc., or pedestrian 1, pedestrian 2, pedestrian 3, etc.

[0033] In the embodiment of the present disclosure, after projecting the point cloud data onto the result of the instance segmentation model, the image instance segmentation model is further used to obtain independent information for each instance, for example, to calculate the corresponding instance center for each point cloud data and correct it. Therefore, using this information as input data for the second stage can further reduce the difficulty of learning the algorithm and improve the accuracy of image detection. Therefore, in the embodiment of the present disclosure, by fully utilizing the information in the image, the difficulty of point cloud detection in the second stage is reduced and detection performance is improved.

[0034] The embodiments of the present disclosure break down the absolute "barrier" between postfusion and prefusion by cleverly utilizing the results of image instance segmentation. Then, according to the calibration relationship between the point cloud and the camera, the original point cloud data is projected onto the image instance segmentation results to obtain entity classifications such as cars, pedestrians, and bicycles for each point cloud data. The center of each instance in the radar coordinate system is then calculated, and target detection is performed using a conventional point cloud detection algorithm. Because the point cloud instance information is acquired before being sent to the point cloud detection algorithm, the difficulty of training the detection algorithm is greatly reduced, and detection performance is greatly improved. The image detection method provided by the embodiments of the present disclosure does not fuse the detection results of two modalities. Unlike postfusion methods, the data from the two modalities is processed sequentially, rather than directly and simultaneously, and therefore differs from prefusion methods. Because the data from the two modalities have a sequential connection relationship, it can be considered a two-stage image-point cloud detection method.

[0035] 1 is a schematic diagram of the main process of an image detection method according to an embodiment of the present disclosure. As an embodiment of the present disclosure, as shown in FIG. 1, the image detection method may include the following steps: Step 101: Performing instance segmentation on the detection target image using an image instance segmentation model to obtain each instance in the detection target image.

[0036] In the inference stage, it is necessary to acquire point cloud data for one box body and at least one corresponding RGB image (i.e., an image of the detection target that can be collected by RGB cameras attached at different angles). Note that the number of detection target images can be one or multiple, and is determined by the number of RGB cameras. When multiple cameras are attached at different angles, one detection target image is collected by each camera, and instance segmentation is performed on each of the multiple detection target images. However, when only one camera is attached, instance segmentation is performed on the single image collected by this camera.

[0037] In the embodiment of the present disclosure, first, an image instance segmentation model is used to perform instance segmentation on the detection target image at each angle, and each instance in each detection target image is obtained. The image instance segmentation model identifies which instance each pixel belongs to, for example, the image instance segmentation model identifies which pixels belong to car 1 and which pixels belong to car 2.

[0038] In order to obtain better instance segmentation results, the image instance segmentation model needs to be pre-trained. Optionally, prior to step 101, further comprising: Obtaining a sample image at each angle, marking a position, a category, and a contour of each sample instance on the sample image at each angle, and training using a first model to obtain an image instance segmentation model; Calculating a tentative center of each sample instance, marking a box of each sample instance in the sample point cloud data, and training using a second model to obtain an instance center correction model; using and training a third model to obtain a target detection model; may also include: First, sample images are collected at each angle using RGB cameras attached at each angle, and the labels of the instances to which each pixel in the sample images belongs are marked. Based on the marking results, an image instance segmentation model is trained and obtained. Then, calculate the virtual center of each sample instance and simultaneously combine it with the box body of each sample instance marked in the sample point cloud data to obtain the instance center correction module through training; Finally, we train it to obtain a target detection model. In the embodiments of the present disclosure, the image instance segmentation model, the instance center correction model, and the target detection model need to be pre-trained, so that the target detection can be performed accurately in the subsequent detection steps.

[0039] Optionally, training using a first model to obtain an image instance segmentation model includes: The sample images of each angle and the positions, categories, and contours of each sample instance are input to a first model for training, thereby training and obtaining an image instance segmentation model. Includes. First, sample images are collected at each angle using an RGB camera attached at each angle, and then the label of the instance to which each pixel in the sample image belongs is marked. Then, the sample images of each angle and their mark results are input to a first model for training, and an image instance segmentation model is obtained through iterative training. Optionally, the first model is Mask R-CNN, Hybrid Task Cascade, or BlendMaskd, and better training results can be obtained by training the sample images of each angle with these models.

[0040] Optionally, calculating a tentative center of each sample instance, marking a box of each sample instance in the sample point cloud data, and training using a second model to obtain an instance center correction model includes: calculating a tentative center of each of the sample instances based on the sample point cloud data and each of the sample instances; marking a box for each sample instance in the sample point cloud data and calculating the center of the box for each sample instance; training and obtaining an instance center correction model by inputting the virtual center of each sample instance and the center of the box body of each sample instance into a second model and training the second model; It may also include: In the subsequent step, in order to fully utilize the image information obtained in step 101 to perform point cloud detection, an instance center correction model needs to be pre-trained. When training the instance center correction model, the tentative center of each sample instance is used as input data, the center of the box of each sample instance in the data is marked as the regression target, and these two parts of data are sent to a second model for training. Through iterative training, an instance center correction module is obtained, and this module is used to correct the instance centers, thereby greatly reducing the difficulty of learning the detection algorithm and greatly improving the detection performance.

[0041] Specifically, the entire point cloud data is projected onto the corresponding RGB image, and the result of image instance segmentation is combined to obtain the point cloud instances. Then, the virtual center [c1, c2, c3] of each instance is calculated, and at the same time, a box of each sample instance is marked in the sample point cloud data to calculate the center of the box of each sample instance. The format of the point cloud data input to the second model is [x, y, z, r, c1, c2, c3], and the corresponding true value is [x, y, z, r, c1 * ,c2* ,c3 * ], and the box center of each sample instance input to the second model is in the format [x,y,z,r,c1´,c2´,c3´], which is the regression target, where x,y,z are the 3D coordinates, r is the reflectance, and [c1´,c2´,c3´] are the box centers of the marked sample instances.

[0042] Optionally, calculating a tentative center of each sample instance based on sample point cloud data and each sample instance comprises: projecting sample point cloud data onto each of the sample instances; Merging point cloud data of the same sample instance appearing in different sample images according to point cloud data of overlapping portions in each of the sample instances; calculating point cloud data of the body of each sample instance using a clustering algorithm; calculating a tentative center of each sample instance based on point cloud data of the body of each sample instance; Includes.

[0043] First, the sample point cloud data is projected onto each sample instance, and then, according to the point cloud data of the overlapping parts of each sample instance, the point cloud data of the overlapping parts is merged into point cloud data corresponding to the same sample instance; Then, a clustering algorithm is used to calculate the point cloud data of the body of each sample instance. Finally, the tentative center of each sample instance is calculated. Since the same sample instance may appear in multiple sample images taken at different angles, there may be overlapping point clouds in the multiple sample images taken at different angles. Therefore, it is necessary to first merge the point cloud data of the overlapping parts into point cloud data corresponding to the same sample instance, and then calculate the virtual center of each sample instance.

[0044] Optionally, after calculating the body point cloud data of each sample instance using a clustering algorithm, it is also necessary to delete the ground point cloud data to avoid the ground point cloud data affecting the clustering of the body point cloud data.

[0045] Optionally, calculating point cloud data of the body of each sample instance using a clustering algorithm includes: Calculating point cloud data of the body of each sample instance using a spatial clustering algorithm, and deleting point cloud data other than the body of each instance. A clustering algorithm can cluster the point cloud data, with each cluster representing one sample instance, allowing us to accurately find the body of each sample instance and remove point cloud data other than the body of each instance.

[0046] Optionally, the clustering algorithm may be DBSCAN, FN-DBSCAN, landmark FN-DBSCAN, or OPTICS, etc. These clustering algorithms can accurately find the body of each sample instance.

[0047] Optionally, the second model is PointNet, PointNet++, Cylider3D or Rand-LA, and using these models, an instance center correction model can be trained to accurately correct the instance center, thereby greatly reducing the learning difficulty of the point cloud detection algorithm and greatly improving the point cloud detection performance.

[0048] Although accurate calculation of the center of each sample instance can further reduce the difficulty of training the point cloud detection algorithm, due to the influence of LIDAR camera calibration accuracy and the synchronization time difference between boxes of different sensors, there is usually a certain deviation in the results from point cloud to image. These factors may ultimately lead to inaccuracies in the projection results, especially at the edges of instance segmentation. Therefore, the embodiment of the present disclosure uses a series of post-processing methods for projected point clouds and model correction to obtain better instance centers.

[0049] Optionally, using and training a third model to obtain a target detection model comprises: Inputting attribute data of each point cloud of each sample instance and the box body of each sample instance into a third model and training the third model, thereby training and obtaining a target detection model. and The attribute data of each point group includes three-dimensional coordinates, a category, and the center coordinates of the box body to which it belongs. In the process of training the target detection model, the attribute data of each dimension of each point cloud needs to be input into the third model. For example, the format of the point cloud input into the third model is [x, y, z, r, cls, c1', c2', c3'], where x, y, z are the three-dimensional coordinates, r is the reflectance, cls is the category corresponding to the image instance segmentation, and c1', c2', c3' are the center coordinates of the box to which the point cloud belongs.

[0050] Optionally, the third model is PointPillars, VoxelNet, Sparsely Embedded Convolutional Detection or CenterPoint, and target detection models trained using these models can improve the accuracy of image detection.

[0051] Step 102: Calculating a virtual center of each of the instances based on the point cloud data and each of the instances.

[0052] In step 1, the entire point cloud data collected by the LIDAR is projected onto each instance to calculate a hypothetical center [c1, c2, c3] for each instance. Note that in the embodiment of the present disclosure, the hypothetical center of each instance is calculated in the radar coordinate system, but it can also be calculated in the image coordinate system, as long as the training phase and the inference phase are in the same coordinate system. The embodiment of the present disclosure does not limit the coordinate system.

[0053] Optionally, step 102 comprises: projecting point cloud data onto each of the instances; Merging point cloud data of the same instance appearing in different detection target images according to point cloud data of overlapping portions in each of the instances;

[0054] calculating point cloud data of the body of each of the instances using a clustering algorithm; calculating a virtual center of each of the instances based on point cloud data of the body of each of the instances; It may also include: As in the training phase, First, the entire point cloud data is projected onto each instance, and then, according to the point cloud data of the overlapping parts in each instance, the point cloud data of the overlapping parts is merged into point cloud data corresponding to the same sample instance; Next, a clustering algorithm is used to calculate the point cloud data of the body of each instance. Finally, the tentative center of each sample instance is calculated. Since the same instance may appear in multiple images taken at different angles, there may be overlapping point clouds in the multiple sample images taken at different angles. First, the point cloud data of the overlapping parts must be merged into the point cloud data corresponding to the same instance, and then the virtual center of each instance must be calculated.

[0055] Optionally, after calculating the body point cloud data of each instance using a clustering algorithm, the ground point cloud data also needs to be deleted to avoid the ground point cloud data affecting the clustering of the body point cloud data. Optionally, calculating point cloud data of the body of each of the instances using a clustering algorithm comprises: calculating point cloud data of the main body of each of the instances using a spatial clustering algorithm, and deleting point cloud data other than the main body of each of the instances; Includes. Optionally, the clustering algorithm may be DBSCAN, FN-DBSCAN, landmark FN-DBSCAN, or OPTICS, etc. These clustering algorithms can accurately find the body of each instance.

[0056] Step 103: Correcting the tentative center of each of the instances using an instance center correction model to obtain a corrected center of each of the instances.

[0057] After calculating the tentative center [c1, c2, c3] of each instance, the instance center correction module is used to correct the tentative center of each instance. The output of the instance center correction module is [x, y, z, r, c1', c2', c3'], where [c1', c2', c3'] is the corrected instance center.

[0058] Step 104: inputting the corrected center of each of the instances into a target detection model and outputting a box and category of each of the instances.

[0059] All points in the entire point cloud data are input to the target detection model, with each point in the format [x,y,z,r,c1´,c2´,c3´], and the target detection model outputs the box and category of each instance, i.e., the box and category of the object to be measured.

[0060] From the above-described embodiments, it can be seen that the embodiments of the present disclosure solve the problem of poor image detection performance in the prior art by using an image instance segmentation model to perform instance segmentation on the detection target image, an instance center correction model to correct the virtual center of each instance, and input the corrected center of each instance into the target detection model. In the embodiments of the present disclosure, the image instance segmentation results are used to perform instance-level annotation on each point in the point cloud, and the center of each instance is further corrected. Target detection is then performed using a point cloud detection algorithm. Since the point cloud instance information is obtained before being sent to the point cloud detection algorithm, the difficulty of training the detection algorithm is greatly reduced and detection performance is greatly improved.

[0061] 2 is a schematic diagram of the main process of an image detection method according to one possible embodiment of the present disclosure. As another embodiment of the present disclosure, as shown in FIG. 2, the image detection method may include the following steps:

[0062] Step 201: Obtaining a sample image of each angle, marking the position, category, and contour of each sample instance on the sample image of each angle, and training using a first model to obtain an image instance segmentation model.

[0063] It is possible to obtain sample point cloud data for multiple boxes and RGB images corresponding to the sample point cloud data for each box, where each box can correspond to multiple RGB images (i.e., sample images that can be collected by RGB cameras mounted at different angles).

[0064] Specifically, mark the label of the instance to which each pixel in the sample image belongs, and train to obtain an image instance segmentation model based on the marking result. Optionally, training to obtain an image instance segmentation model using the first model includes: The sample images of each angle and the positions, categories, and contours of each sample instance are input to a first model for training, thereby training and obtaining an image instance segmentation model. Includes. First, sample images are collected at each angle using an RGB camera attached at each angle, and then the label of the instance to which each pixel in the sample image belongs is marked. Then, the sample images at each angle and their mark results are input to the first model for training, and an image instance segmentation model is obtained through iterative training. Optionally, the first model is Mask R-CNN, Hybrid Task Cascade or BlendMaskd, and better training results can be obtained by training sample images of each angle with these models.

[0065] Step 202: Calculating a tentative center of each sample instance, marking a box of each sample instance in the sample point cloud data, and training using a second model to obtain an instance center correction model.

[0066] The tentative center of each sample instance is calculated, and at the same time, it is combined with the box of each sample instance marked in the sample point cloud data and trained to obtain an instance center correction module. When training the instance center correction model, the tentative center of each sample instance is used as input data, and the center of the box of each sample instance in the data is marked as the regression target. These two parts of data are sent to a second model for training, and through iterative training, an instance center correction module is obtained. This module is used to correct the instance centers, significantly reducing the learning difficulty of the detection algorithm and significantly improving detection performance. The point cloud data input to the second model is in the format [x,y,z,r,c1,c2,c3]. The box center of each sample instance input to the second model is in the format [x,y,z,r,c1',c2',c3'], which is the regression target, where x,y,z are 3D coordinates, r is reflectance, and [c1',c2',c3'] are the box centers of the marked sample instances.

[0067] Step 203: Using the third model to train and obtain a target detection model.

[0068] Specifically, the attribute data of each point cloud of each sample instance and the box body of each sample instance are input into a third model for training, thereby training and obtaining a target detection model. The attribute data of each point cloud includes 3D coordinates, a category, and the center coordinates of the box body to which it belongs. In the process of training the target detection model, the attribute data of each dimension of each point cloud needs to be input into the third model. For example, the format of the point cloud input into the third model is [x, y, z, r, cls, c1', c2', c3'], where x, y, and z are 3D coordinates, r is reflectance, cls is the category corresponding to image instance segmentation, and c1', c2', and c3' are the center coordinates of the box body to which the point cloud belongs.

[0069] Step 204: Performing instance segmentation on each detection target image using an image instance segmentation model to obtain each instance in each detection target image.

[0070] In the inference stage, it is necessary to acquire point cloud data of a box body and at least one corresponding RGB image (i.e., the target image that can be collected by RGB cameras mounted at different angles).

[0071] Step 205: Calculating a virtual center of each of the instances based on the point cloud data and each of the instances.

[0072] The entire point cloud data collected by the LIDAR is projected onto each instance, and the tentative center [c1, c2, c3] of each instance is calculated. Specifically, the point cloud data is projected onto each instance, and point cloud data of the same instance appearing in different detection target images is merged according to the point cloud data of the overlapping parts of each instance, and point cloud data of the main body of each instance is calculated using a clustering algorithm, and the tentative center of each instance is calculated based on the point cloud data of the main body of each instance. As in the training stage, First, the entire point cloud data is projected onto each instance, and then, according to the point cloud data of the overlapping parts in each instance, the point cloud data of the overlapping parts is merged into point cloud data corresponding to the same sample instance; Next, a clustering algorithm is used to calculate the point cloud data of the body of each instance. Finally, the tentative center of each sample instance is calculated. Since the same instance may appear in multiple images taken at different angles, there may be overlapping point clouds in the multiple images taken at different angles. First, the point cloud data of the overlapping parts must be merged into the point cloud data corresponding to the same instance, and then the virtual center of each instance must be calculated. Optionally, after calculating the body point cloud data of each instance using a clustering algorithm, the ground point cloud data also needs to be deleted to avoid the ground point cloud data affecting the clustering of the body point cloud data. Optionally, calculating point cloud data of the body of each of the instances using a clustering algorithm comprises: calculating point cloud data of the main body of each of the instances using a spatial clustering algorithm, and deleting point cloud data other than the main body of each of the instances; Includes. Optionally, the clustering algorithm may be DBSCAN, FN-DBSCAN, landmark FN-DBSCAN, or OPTICS, etc. These clustering algorithms can accurately find the body of each instance.

[0073] Step 206: Correcting the tentative center of each of the instances using an instance center correction model to obtain a corrected center of each of the instances.

[0074] After calculating the tentative center [c1, c2, c3] of each instance, the instance center correction model is used to correct the tentative center of each instance. The output of the instance center correction module is [x, y, z, r, c1', c2', c3'], of which [c1', c2', c3'] is the corrected instance center.

[0075] Step 207: inputting the corrected center of each of the instances into a target detection model and outputting a box and category of each of the instances.

[0076] All points in the entire point cloud data are input to the target detection model, with each point in the format [x,y,z,r,c1´,c2´,c3´], and the target detection model outputs the box and category of each instance, i.e., the box and category of the measured object.

[0077] In this embodiment, steps 201-203 are the training stage, and steps 204-207 are the inference stage.

[0078] Furthermore, the specific implementation content of the image detection method according to one possible embodiment of the present disclosure has already been described in detail in the above-mentioned image detection method, so the overlapping content will not be described here.

[0079] 3 is a schematic diagram of the main modules of an image detection device according to an embodiment of the present disclosure. As shown in FIG. 3, the image detection device 300 includes: a segmentation module 301 for performing instance segmentation on target images using an image instance segmentation model to obtain each instance in each target image; a calculation module 302 for calculating a virtual center of each of the instances based on point cloud data and each of the instances; a correction module 303 for correcting the tentative center of each of the instances using an instance center correction model to obtain a corrected center of each of the instances; a detection module 304 for inputting the corrected center of each instance into a target detection model and outputting a box and category for each instance; Includes.

[0080] Optionally, the calculation module 302 further comprises: projecting the point cloud data onto each of the instances; Merging point cloud data of the same instance appearing in different detection target images according to point cloud data of the overlapping portion of each of the instances; calculating point cloud data of the body of each of the instances using a clustering algorithm; Calculate a virtual center of each of the instances based on point cloud data of the body of each of the instances. This is for the purpose.

[0081] Optionally, the calculation module 302 further comprises: Calculating point cloud data of the main body of each of the instances using a spatial clustering algorithm, and deleting point cloud data other than the main body of each of the instances. This is for the purpose.

[0082] Optionally, further comprising: Obtaining a sample image for each angle, marking a position, a category, and a contour of each sample instance on the sample image for each angle, and training using a first model to obtain an image instance segmentation model; Calculating a tentative center of each sample instance, marking a box of each sample instance in the sample point cloud data, and training using a second model to obtain an instance center correction model; Use a third model and train it to obtain a target detection model Includes training modules for:

[0083] Optionally, the training module further comprises: The sample images at each angle and the positions, categories, and contours of each sample instance are input to a first model for training, thereby training and obtaining an image instance segmentation model. It is for the purpose of Among them, the first model is Mask R-CNN, Hybrid Task Cascade, or BlendMaskd.

[0084] Optionally, the training module further comprises: calculating a tentative center of each of the sample instances based on the sample point cloud data and each of the sample instances; Marking a box for each sample instance in the sample point cloud data and calculating the center of the box for each sample instance; The virtual center of each sample instance and the center of the box body of each sample instance are input to a second model and training is performed to train and obtain an instance center correction model. This is for the purpose.

[0085] Optionally, the training module further comprises: projecting the sample point cloud data onto each of the sample instances; Merging point cloud data of the same sample instance appearing in different sample images according to point cloud data of the overlapping portions of each of the sample instances; calculating point cloud data of the body of each sample instance using a clustering algorithm; Calculating a tentative center of each sample instance based on point cloud data of the body of each sample instance This is for the purpose.

[0086] Optionally, the second model is PointNet, PointNet++, Cylider3D, or Rand-LA.

[0087] Optionally, the training module further comprises: The attribute data of each point cloud of each sample instance and the box body of each sample instance are input to a third model for training, thereby training and obtaining a target detection model. It is for the purpose of The attribute data of each point group includes three-dimensional coordinates, a category, and the center coordinates of the box body to which it belongs.

[0088] Optionally, the third model is PointPillars, VoxelNet, Sparsely Embedded Convolutional Detection or CenterPoint.

[0089] Note that the specific implementation of the image detection device according to the present disclosure has already been described in detail in the image detection method described above, so the overlapping content will not be described here.

[0090] FIG. 4 illustrates an exemplary system architecture 400 of an image detection method or apparatus to which embodiments of the present disclosure may be applied.

[0091] 4, system architecture 400 may include terminal devices 401, 402, 403, a network 404, and a server 405. Network 404 is a medium for providing a communication link between terminal devices 401, 402, 403 and server 405. Network 404 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

[0092] Users can use terminal devices 401, 402, and 403 to communicate with server 405 via network 404 to send and receive messages, etc. Various communication client applications can be installed on terminal devices 401, 402, and 403, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and social platform software (just to name a few).

[0093] The terminal devices 401, 402, and 403 may be various electronic devices that have a display and support web browsing, including, but not limited to, smartphones, tablet computers, laptop computers, and desktop computers.

[0094] Server 405 may be a server that provides various services, and may be, for example, a background management server (just an example) that provides support for shopping sites that users browse using terminal devices 401, 402, and 403. The background management server performs processing such as analysis on received data such as product information inquiry requests, and can feed back the processing results to the terminal devices.

[0095] It should be noted that the image detection method provided by the embodiments of the present disclosure is generally executed by the server 405, and correspondingly, such an image detection device is generally provided in the server 405. The image detection method provided by the embodiments of the present disclosure may also be executed by the terminal devices 401, 402, and 403, and correspondingly, such an image detection device may be provided in the terminal devices 401, 402, and 403.

[0096] It should be understood that the number of terminal devices, networks, and servers in FIG. 4 is merely approximate, and that implementations may have any number of terminal devices, networks, and servers as required.

[0097] 5, which shows a structural schematic diagram of a computer system 500 suitable for implementing the terminal device of the embodiment of the present disclosure. The terminal device shown in FIG. 5 is merely an example and does not limit the function and scope of use of the embodiment of the present disclosure.

[0098] 5, computer system 500 includes a central processing unit (CPU) 501 capable of performing various appropriate operations and processes based on programs stored in read-only memory (ROM) 502 or programs loaded from storage portion 508 into random access memory (RAM) 503. RAM 503 also stores various programs and data necessary for the operation of system 500. CPU 501, ROM 502, and RAM 503 are connected to one another via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0099] The following components are connected to the I / O interface 505: Input section 506 including keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; A communication section 509 including a network interface card such as a LAN card or a modem. The communication unit 509 executes communication processing via a network such as the Internet. A driver 510 is also connected to the I / O interface 505 as needed. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is installed in the driver 510 as needed, just as a computer program read from the removable medium 511 is installed in the storage section 508 as needed.

[0100] In particular, based on the embodiments disclosed in the present disclosure, the processes described above with reference to the flowcharts may be implemented as a computer software program. For example, the embodiments disclosed in the present disclosure include a computer program stored on a computer-readable medium, the computer program including program code for performing the methods shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 509 and / or installed from removable media 511. When executed by the central processing unit (CPU) 501, the computer program performs the functions defined in the system of the present disclosure.

[0101] It should be noted that the computer-readable medium referred to in this disclosure may be a computer-readable signal medium, a computer-readable storage medium, or any combination of the above. The computer-readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of the computer-readable storage medium may include, but are not limited to, an electrical connection having one or more conductors, a portable computer magnetic disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), a rewritable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact magnetic disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used in or in conjunction with a command execution system, apparatus, or device. Also, in this disclosure, a computer-readable signal medium can include a data signal propagating in baseband or as part of a carrier wave, having computer-readable program code stored therein. Such a propagated data signal can take various forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transport a program used by or in connection with a command execution system, apparatus, or device.The program code contained in the computer readable medium may be transmitted over any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0102] The flowcharts and block diagrams in the figures illustrate possible architectures, functions, and operations of systems, methods, and computer programs according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program section, or portion of code, which includes one or more executable commands for implementing a given logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks may occur in a different order than noted in the figures. For example, two successively shown blocks may actually be executed essentially in parallel, or may even be executed in the reverse order, as determined by the functionality involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented by a dedicated hardware-based system for performing a given function or operation, or by a combination of dedicated hardware and computer commands.

[0103] The relevant modules described in the embodiments of the present disclosure may be implemented in software or hardware. The described modules may be provided in a processor, and may be described as including, for example, a segmentation module, a calculation module, a correction module, and a detection module. The names of these modules, in some cases, are not intended to be limitations on the modules themselves.

[0104] In another aspect, the present disclosure further provides a computer-readable medium. The computer-readable medium may be included in the device described in the above embodiment, or may exist independently of the device. The computer-readable medium stores one or more programs, and when the one or more programs are executed by one of the devices, the device performs the following: performing instance segmentation on the target image using an image instance segmentation model to obtain each instance in the target image; calculating a virtual center of each of the instances based on the point cloud data and each of the instances; correcting the tentative center of each of the instances using an instance center correction model to obtain a corrected center of each of the instances; inputting the corrected center of each instance into a target detection model, and outputting a box and category of each instance; Realize the method.

[0105] According to the technical solution of the embodiments of the present disclosure, an image instance segmentation model is used to perform instance segmentation on the detection target image, an instance center correction model is used to correct the virtual center of each instance, and the corrected center of each instance is input into the target detection model, thereby solving the problem of poor image detection performance in the prior art. In the embodiments of the present disclosure, the image instance segmentation results are used to perform instance-level annotation for each point in the point cloud, and the center of each instance is further corrected, and then target detection is performed using a point cloud detection algorithm. Because the point cloud instance information is obtained before being sent to the point cloud detection algorithm, the difficulty of training the detection algorithm is greatly reduced and detection performance is greatly improved.

[0106] The above-described embodiments of the present invention do not limit the scope of protection of the present disclosure. It is obvious to those skilled in the art that various modifications, combinations, subcombinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. performing instance segmentation on the target image using an image instance segmentation model to obtain each instance in the target image; Prior to that, further Obtaining a sample image at each angle, marking a position, a category, and a contour of each sample instance on the sample image at each angle, and training using a first model to obtain an image instance segmentation model; Calculating a tentative center of each sample instance, marking a box of each sample instance in the sample point cloud data, and training using a second model to obtain an instance center correction model; training using a third model to obtain a target detection model; acquiring each instance in the target image, calculating a virtual center of each of the instances based on the point cloud data and each of the instances; correcting the tentative center of each of the instances using the instance center correction model to obtain a corrected center of each of the instances; inputting the corrected center of each instance into a target detection model and outputting a box and category of each instance; Contains Image detection methods.

2. Calculating a virtual center of each of the instances based on point cloud data and each of the instances includes: projecting point cloud data onto each of the instances; Merging point cloud data of the same instance appearing in different detection target images according to point cloud data of overlapping portions in each of the instances; calculating point cloud data of the body of each of the instances using a clustering algorithm; calculating a virtual center of each of the instances based on point cloud data of a body of each of the instances; Contains The method of claim 1.

3. calculating point cloud data of the body of each of the instances using a clustering algorithm, calculating point cloud data of the main body of each of the instances using a spatial clustering algorithm, and deleting point cloud data other than the main body of each of the instances; Contains The method of claim 2.

4. Using and training a first model to obtain an image instance segmentation model includes: The sample images of each angle and the position, category, and contour of each sample instance are input to a first model for training, thereby training and obtaining an image instance segmentation model. Including, Wherein the first model is Mask R-CNN, Hybrid Task Cascade or BlendMaskd; The method of claim 1.

5. Calculating a tentative center of each sample instance, marking a box of each sample instance in sample point cloud data, and training using a second model to obtain an instance center correction model includes: calculating a tentative center of each of the sample instances based on the sample point cloud data and each of the sample instances; marking a box for each sample instance in the sample point cloud data and calculating the center of the box for each sample instance; training and obtaining an instance center correction model by inputting the tentative center of each sample instance and the center of the box of each sample instance into a second model and training the second model; Contains The method of claim 1.

6. Calculating a tentative center of each of the sample instances based on sample point cloud data and each of the sample instances includes: projecting sample point cloud data onto each of the sample instances; Merging point cloud data of the same sample instance appearing in different sample images according to point cloud data of overlapping portions in each of the sample instances; calculating point cloud data of the body of each sample instance using a clustering algorithm; calculating a virtual center of each sample instance based on point cloud data of the body of each sample instance; Including, The method of claim 5.

7. Using and training a third model to obtain a target detection model Inputting attribute data of each point cloud of each sample instance and the box body of each sample instance into a third model and training the third model, thereby training and obtaining a target detection model. Including, The attribute data of each point cloud includes three-dimensional coordinates, categories, and the center coordinates of the box body to which it belongs. The method of claim 1.

8. a segmentation module for performing instance segmentation on the target image using an image instance segmentation model to obtain each instance in the target image; a training module, wherein before the segmentation module performs instance segmentation on the detection target image using the image instance segmentation model to obtain each instance in the detection target image, Obtaining a sample image for each angle, marking a position, a category, and a contour of each sample instance on the sample image for each angle, and training using a first model to obtain an image instance segmentation model; Calculating a tentative center of each sample instance, marking a box of each sample instance in the sample point cloud data, and training using a second model to obtain an instance center correction model; a training module for training using the third model to obtain a target detection model; a calculation module for calculating a virtual center of each of the instances based on point cloud data and each of the instances; a correction module for correcting a tentative center of each of the instances using the instance center correction model to obtain a corrected center of each of the instances; a detection module for inputting the corrected center of each instance into a target detection model and outputting a box and category for each instance; Contains Image detection device.

9. one or more processors; a storage device for storing one or more programs; Including, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7. electronic equipment.

10. A computer-readable medium having stored thereon a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images