Image detection method and apparatus

Abstract

The present disclosure provides an image detection method and apparatus, which is related to the field of computer vision. A specific embodiment of the method includes: performing instance segmentation on a detection target image using an image instance segmentation model to obtain each instance in the detection target image; calculating a tentative center of each instance based on point cloud data and each instance; correcting the tentative center of each instance using an instance center correction model to obtain a corrected center of each instance; inputting the corrected center of each instance into a target detection model to output a box body and a category of each instance. This embodiment can solve the problem of poor image detection performance.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to Chinese patent application No. 202210158788.3, ​​filed on February 21, 2022, with the publication title "Image Detection Method and Apparatus," and the contents disclosed in the above Chinese patent application are hereby incorporated in full as part or in whole of this application. [Technical field]

[0002] The present disclosure relates to the field of computer vision, and in particular to an image detection method and apparatus. [Background technology]

[0003] Autonomous vehicles are usually equipped with multiple types of sensors to detect obstacles, such as LIDAR, RGB image cameras, and millimeter-wave radar. However, the imaging principles and mounting angles of different sensors are different, so the output results obtained are also different. For example, a multi-line LIDAR mounted on the top has a 360-degree point cloud as its output, and the image sensor is composed of multiple RGB cameras, each of which has a fixed field of view and is responsible for generating RGB images corresponding to partial regions. Since the data modalities generated by these sensors are different, the data characteristics generated are also different. For example, LIDAR is not sensitive to light intensity and can be used at night, but it is expensive and high-wire harness products are generally not available in mass-produced autonomous vehicles, so the point cloud data obtained is sparse and has low resolution as a result. RGB image cameras, while low cost, high resolution, and obvious texture features, are easily affected by weather changes and ambient light intensity.

[0004] From the perspective of algorithms, detection results are usually obtained by using a dedicated algorithm for the data corresponding to each modality, and then the detection results of each model are fused, which is commonly referred to in the industry as the "post-fusion method". The corresponding one is called the "pre-fusion method", that is, a deep model or algorithm is used to process all the data of different modalities collectively to directly obtain the final detection result. For example, the AVOD algorithm uses a deep neural network to perform feature extraction on the data of different modalities to obtain a feature map, and then performs fusion at the level of the feature map. Due to the difference in different modality data, it is difficult to align the feature maps from different modalities, for example, a feature map from a 360-degree point cloud and a feature map corresponding to a front-view camera. However, whether it is the post-fusion method or the front fusion method, the fusion effect is not good, so the image detection performance is poor. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide an image detection method and apparatus to solve the problem of poor image detection performance. In order to achieve the above object, according to one aspect of the embodiment of the present disclosure, performing instance segmentation on the target image using an image instance segmentation model to obtain each instance in the target image; calculating a virtual center of each of the instances based on the point cloud data and each of the instances; correcting the tentative center of each of the instances using an instance center correction model to obtain a corrected center of each of the instances; inputting the corrected centroid of each of the instances into a target detection model to output a box and a category of each of the instances; Includes A method for detecting an image is provided.

[0006] According to one or more embodiments of the present disclosure, calculating a tentative center of each of the instances based on point cloud data and each of the instances includes: projecting point cloud data onto each of the instances; Merging point cloud data of the same instance appearing in different detection target images according to point cloud data of overlapping portions of each of the instances; calculating point cloud data of a body of each of the instances using a clustering algorithm; calculating a virtual center of each of the instances based on point cloud data of a body of each of the instances; Includes.

[0007] According to one or more embodiments of the present disclosure, calculating point cloud data of the body of each of the instances using a clustering algorithm includes: calculating point cloud data of the body of each of the instances using a spatial clustering algorithm, and deleting point cloud data other than the body of each of the instances; Includes.

[0008] According to one or more embodiments of the present disclosure, prior to performing instance segmentation on the detection target image using an image instance segmentation model to obtain each instance in the detection target image, further comprising: Obtaining a sample image of each angle, marking a position, a category, and a contour of each sample instance on the sample image of each angle, and training using a first model to obtain an image instance segmentation model; Calculating a tentative center of each of the sample instances, marking a box of each of the sample instances in the sample point cloud data, and training using a second model to obtain an instance center correction model; Training the third model to obtain a target detection model; and Includes.

[0009] According to one or more embodiments of the present disclosure, training using a first model to obtain an image instance segmentation model includes: training the sample images of each angle and the positions, categories, and contours of each sample instance of the sample images into a first model for training, thereby obtaining an image instance segmentation model; Including, Among them, the first model is Mask R-CNN, Hybrid Task Cascade, or BlendMaskd.

[0010] According to one or more embodiments of the present disclosure, calculating a tentative center of each sample instance, marking a box body of each sample instance in sample point cloud data, and using a second model to train and obtain an instance center correction model may include: calculating a tentative center of each of the sample instances based on the sample point cloud data and each of the sample instances; marking a box for each of the sample instances in the sample point cloud data and calculating a center of the box for each of the sample instances; training and acquiring an instance center correction model by inputting the tentative center of each sample instance and the center of the box body of each sample instance into a second model and training the second model; Includes.

[0011] According to one or more embodiments of the present disclosure, calculating a tentative center of each of the sample instances based on sample point cloud data and each of the sample instances includes: projecting sample point cloud data onto each of the sample instances; merging point cloud data of the same sample instance appearing in different sample images according to point cloud data of overlapping portions in each of the sample instances; calculating point cloud data of the body of each of the sample instances using a clustering algorithm; Calculating a tentative center of each of the sample instances based on point cloud data of a body of each of the sample instances; Includes.

[0012] According to one or more embodiments of the present disclosure, the second model is PointNet, PointNet++, Cylider3D, or Rand-LA.

[0013] According to one or more embodiments of the present disclosure, training using a third model to obtain a target detection model includes: Inputting attribute data of each point cloud of each of the sample instances and the box body of each of the sample instances into a third model for training, thereby training and obtaining a target detection model. Including, The attribute data of each point group includes three-dimensional coordinates, a category, and the center coordinates of the box body to which it belongs.

[0014] According to one or more embodiments of the present disclosure, the third model is PointPillars, VoxelNet, Sparsely Embedded Convolutional Detection, or CenterPoint.

[0015] According to another aspect of the present disclosure, a segmentation module for performing instance segmentation on the target image using an image instance segmentation model to obtain each instance in the target image; a calculation module for calculating a virtual center of each of the instances based on the point cloud data and each of the instances; a correction module for correcting the tentative center of each of the instances using an instance center correction model to obtain a corrected center of each of the instances; a detection module for inputting the corrected centroid of each of the instances into a target detection model to output a box and a category of each of the instances; Includes The present invention provides an image detection device.

[0016] According to one or more embodiments of the present disclosure, the computing module may further include: projecting the point cloud data onto each of the instances; Merging point cloud data of the same instance appearing in different detection target images according to point cloud data of overlapping portions of each of the instances; calculating point cloud data of the body of each of said instances using a clustering algorithm; Calculating a tentative center of each of the instances based on point cloud data of the main body of each of the instances This is for the purpose.

[0017] According to one or more embodiments of the present disclosure, the computing module may further include: A spatial clustering algorithm is used to calculate point cloud data of the main body of each of the instances, and point cloud data other than the main body of each of the instances is deleted. This is for the purpose.

[0018] According to one or more embodiments of the present disclosure, the method further comprises: Obtaining a sample image for each angle, marking a position, a category, and a contour of each sample instance on the sample image for each angle, and training using a first model to obtain an image instance segmentation model; Calculate a tentative center of each of the sample instances, mark a box of each of the sample instances in the sample point cloud data, and use a second model to train to obtain an instance center correction model; Use a third model to train and obtain a target detection model Training module for Includes.

[0019] According to one or more embodiments of the present disclosure, the training module may further include: The sample images of each angle and the positions, categories, and contours of each sample instance are input to a first model for training, thereby training and obtaining an image instance segmentation model. This is for the purpose of Among them, the first model is Mask R-CNN, Hybrid Task Cascade or BlendMaskd.

[0020] According to one or more embodiments of the present disclosure, the training module may further include: Calculating a tentative center of each of the sample instances based on the sample point cloud data and each of the sample instances; marking a box for each of the sample instances in the sample point cloud data and calculating a center of the box for each of the sample instances; The tentative center of each sample instance and the center of the box body of each sample instance are input to a second model and training is performed to train and obtain an instance center correction model. This is for the purpose.

[0021] According to one or more embodiments of the present disclosure, the training module may further include: projecting the sample point cloud data onto each of the sample instances; Merging point cloud data of the same sample instance appearing in different sample images according to point cloud data of the overlapping portion in each of the sample instances; calculating point cloud data of the body of each of the sample instances using a clustering algorithm; Calculating a tentative center of each of the sample instances based on point cloud data of the body of each of the sample instances This is for the purpose.

[0022] According to one or more embodiments of the present disclosure, the second model is PointNet, PointNet++, Cylider3D, or Rand-LA.

[0023] According to one or more embodiments of the present disclosure, the training module may further include: The attribute data of each point cloud of each of the sample instances and the box body of each of the sample instances are input to a third model for training, thereby training and obtaining a target detection model. This is for the purpose of The attribute data of each point group includes three-dimensional coordinates, a category, and the center coordinates of the box body to which it belongs.

[0024] According to one or more embodiments of the present disclosure, the third model is PointPillars, VoxelNet, Sparsely Embedded Convolutional Detection, or CenterPoint.

[0025] According to another aspect of the embodiment of the present disclosure, one or more processors; a storage device for storing one or more programs; Including, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of the above embodiments. Provides electronic equipment.

[0026] According to another aspect of the embodiment of the present disclosure, A computer program that, when executed by a processor, implements the method according to any one of the above embodiments is stored. A computer readable medium is provided.

[0027] One embodiment of the above disclosure has the following advantages or beneficial effects. The problem of poor image detection performance in the prior art has been solved by using an image instance segmentation model to perform instance segmentation on the detection target image, using an instance center correction model to correct the tentative center of each instance, and inputting the corrected center of each instance into the target detection model. In the embodiment of the present disclosure, the image instance segmentation result is used to perform instance-level annotation for each point in the point cloud, and the center of each instance is further corrected, and then target detection is performed using a point cloud detection algorithm. Since the instance information of the point cloud is obtained before being sent to the point cloud detection algorithm, the difficulty of learning the detection algorithm is greatly reduced and the detection performance is greatly improved.

[0028] Further advantages of the above unconventional options will be explained in conjunction with the detailed description below. [Brief description of the drawings]

[0029] The drawings are for a better understanding of the present disclosure and are not to be construed as construing as undue limitations thereon.

[0030] [Figure 1] FIG. 1 is a schematic diagram of the main processes of an image detection method according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a schematic diagram of the main process of an image detection method according to one possible embodiment of the present disclosure. [Diagram 3] FIG. 3 is a schematic diagram of the main modules of an image detection device according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is an exemplary system architecture diagram in which embodiments of the present disclosure can be applied. [Diagram 5] FIG. 5 is a structural schematic diagram of a terminal or server computer system suitable for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] The following describes exemplary embodiments of the present disclosure in conjunction with the drawings. In particular, the details of the embodiments of the present disclosure are included to aid in understanding, and should be considered merely as examples. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, for clarity and conciseness, the following description omits descriptions of known functions and structures.

[0032] In the embodiment of the present disclosure, the instance segmentation results of the image are used to perform instance-level annotation for each point in the point cloud, and after the first stage of instance segmentation, each point cloud data is labeled with a category such as car, pedestrian, and bicycle, and the first stage of instance segmentation can also identify different instances of the same category, such as 10 cars or 20 pedestrians. Instance segmentation can distinguish between instances of the same category, such as car 1, car 2, car 3, etc., or pedestrian 1, pedestrian 2, pedestrian 3, etc.

[0033] In the embodiment of the present disclosure, after projecting the point cloud data onto the result of the instance segmentation model, the image instance segmentation model is further used to obtain independent information of each instance, for example, for each point cloud data, the corresponding instance center is calculated and corrected, so that using this information as input data for the second stage can further reduce the difficulty of learning the algorithm and improve the accuracy of image detection. Therefore, in the embodiment of the present disclosure, the information in the image is fully utilized to reduce the difficulty of point cloud detection in the second stage and improve the detection performance.

[0034] In the embodiment of the present disclosure, the absolute "barrier" between postfusion and prefusion is broken, and the result of image instance segmentation is skillfully used, and then, according to the calibration relationship between the point cloud and the camera, the original point cloud data is projected onto the result of image instance segmentation, and entity classification such as automobiles, pedestrians, and bicycles is obtained for each point cloud data. Then, the center of each instance in the radar coordinate system is calculated, and target detection is performed using a conventional point cloud detection algorithm. Since the point cloud instance information is obtained before being sent to the point cloud detection algorithm, the difficulty of learning the detection algorithm is greatly reduced, and the detection performance is greatly improved. The image detection method provided by the embodiment of the present disclosure does not perform fusion with the detection results of two modalities, and unlike the postfusion method, the data of the two modalities is not directly processed simultaneously, but is processed sequentially, which is also different from the prefusion method. Since the data of the two modalities have a sequential connection relationship, it can be considered as a two-stage detection method of image and point cloud.

[0035] 1 is a schematic diagram of a main process of an image detection method according to an embodiment of the present disclosure. As an embodiment of the present disclosure, as shown in FIG. 1, the image detection method may include the following steps: Step 101: Performing instance segmentation on a detection target image using an image instance segmentation model to obtain each instance in the detection target image.

[0036] In the inference stage, it is necessary to obtain point cloud data of one box body and at least one corresponding RGB image (i.e., a detection target image that can be collected by RGB cameras installed at different angles). Note that the detection target image can be one or multiple, and is determined by the number of RGB cameras. When multiple cameras are installed at different angles, one detection target image is collected for each camera, and instance segmentation is performed for each of the multiple detection target images, but when only one camera is installed, instance segmentation is performed for the single image collected by this camera.

[0037] In the embodiment of the present disclosure, first, an image instance segmentation model is used to perform instance segmentation on the detection target image of each angle, and each instance in each detection target image is obtained respectively. The image instance segmentation model identifies which instance each pixel belongs to, for example, the image instance segmentation model identifies which pixels belong to car 1 and which pixels belong to car 2.

[0038] In order to obtain a better instance segmentation result, the image instance segmentation model needs to be pre-trained. Optionally, prior to step 101, further comprising: Obtaining a sample image of each angle, marking a position, a category, and a contour of each sample instance on the sample image of each angle, and training using a first model to obtain an image instance segmentation model; Calculating a tentative center of each of the sample instances, marking a box of each of the sample instances in the sample point cloud data, and training using a second model to obtain an instance center correction model; Training the third model to obtain a target detection model; and may also include First, collect sample images at each angle using RGB cameras attached at each angle, mark the label of the instance to which each pixel in the sample images belongs, and obtain an image instance segmentation model through training based on the marking results; Then, calculate the tentative center of each sample instance and at the same time combine it with the box body of each sample instance marked in the sample point cloud data to obtain an instance center correction module through training; Finally, we train it to obtain a target detection model. In the embodiment of the present disclosure, the image instance segmentation model, the instance center correction model, and the target detection model need to be pre-trained, so that the target detection can be performed accurately in the subsequent detection steps.

[0039] Optionally, training using the first model to obtain an image instance segmentation model includes: Inputting the sample images of each angle and the positions, categories, and contours of each sample instance of the sample images into a first model for training, thereby training and obtaining an image instance segmentation model. Includes. First, we collect sample images at each angle using an RGB camera attached at each angle, then mark the label of the instance to which each pixel in the sample image belongs, Then, the sample images of each angle and their mark results are input to a first model for training, and an image instance segmentation model is obtained through iterative training. Optionally, the first model is Mask R-CNN, Hybrid Task Cascade, or BlendMaskd, and better training results can be obtained by training the sample images of each angle with these models.

[0040] Optionally, calculating a tentative center of each of the sample instances and marking a box body of each of the sample instances in the sample point cloud data, and training using a second model to obtain an instance center correction model, includes: calculating a tentative center of each of the sample instances based on the sample point cloud data and each of the sample instances; marking a box for each of the sample instances in the sample point cloud data and calculating a center of the box for each of the sample instances; training and acquiring an instance center correction model by inputting the tentative center of each sample instance and the center of the box body of each sample instance into a second model and training the second model; It may also include. In the subsequent step, in order to fully utilize the image information obtained in step 101 to perform point cloud detection, it is further necessary to pre-train an instance center correction model. When training the instance center correction model, the tentative center of each sample instance is used as input data, the center of the box body of each sample instance in the data is marked as the regression target, and these two parts of data are sent to a second model for training, and an instance center correction module is obtained through iterative training, and this module is used to correct the instance center, thereby greatly reducing the learning difficulty of the detection algorithm and greatly improving the detection performance.

[0041] Specifically, the entire point cloud data is projected onto the corresponding RGB image, and the image instance segmentation results are combined to obtain the point cloud instances. Then, the virtual centers [c1, c2, c3] of each instance are calculated, and at the same time, the boxes of each sample instance are marked in the sample point cloud data, and the centers of the boxes of each sample instance are calculated. The format of the point cloud data input to the second model is [x, y, z, r, c1, c2, c3], and the corresponding true values ​​are [x, y, z, r, c1 * ,c2* ,c3 * ], and the box center of each sample instance input to the second model is in the format [x,y,z,r,c1´,c2´,c3´], which is the regression target, where x,y,z are the 3D coordinates, r is the reflectance, and [c1´,c2´,c3´] are the box centers of the marked sample instances.

[0042] Optionally, calculating a tentative center of each of the sample instances based on sample point cloud data and each of the sample instances comprises: projecting sample point cloud data onto each of the sample instances; merging point cloud data of the same sample instance appearing in different sample images according to point cloud data of overlapping portions in each of the sample instances; calculating point cloud data of the body of each of the sample instances using a clustering algorithm; Calculating a tentative center of each of the sample instances based on point cloud data of a body of each of the sample instances; Includes.

[0043] First, the sample point cloud data is projected onto each sample instance, and then, according to the point cloud data of the overlapping parts of each sample instance, the point cloud data of the overlapping parts is merged into point cloud data corresponding to the same sample instance; Then, a clustering algorithm is used to calculate the point cloud data of the body of each sample instance. Finally, we calculate the tentative center of each sample instance. In addition, since the same sample instance may appear in multiple sample images with different angles, there may be overlapping point clouds in the multiple sample images with different angles. It is necessary to first merge the point cloud data of the overlapping parts into the point cloud data corresponding to the same sample instance, and then calculate the tentative center of each sample instance.

[0044] Optionally, after calculating the body point cloud data of each sample instance using a clustering algorithm, it is also necessary to delete the ground point cloud data to avoid the ground point cloud data affecting the clustering of the body point cloud data.

[0045] Optionally, calculating point cloud data of the body of each sample instance using a clustering algorithm comprises: and calculating point cloud data of the body of each of the sample instances using a spatial clustering algorithm, and removing point cloud data other than the body of each of the instances. A clustering algorithm can cluster the point cloud data, with each cluster representing one sample instance, allowing us to accurately find the body of each sample instance and remove the point cloud data other than the body of each instance.

[0046] Optionally, the clustering algorithm may be DBSCAN, FN-DBSCAN, landmark FN-DBSCAN, or OPTICS, etc. These clustering algorithms can accurately find the body of each sample instance.

[0047] Optionally, the second model is PointNet, PointNet++, Cylider3D or Rand-LA, and using these models, an instance center correction model can be trained to accurately correct the instance center, so that the learning difficulty of the point cloud detection algorithm is greatly reduced and the point cloud detection performance is greatly improved.

[0048] Although the difficulty of learning the point cloud detection algorithm can be further reduced by accurately calculating the center of each sample instance, due to the influence of the calibration accuracy of the LIDAR camera and the problem of synchronization time difference between the boxes of different sensors, there is usually a certain deviation in the results from point cloud to image, and these factors may ultimately lead to inaccuracies in the projection results, especially at the edges of the instance segmentation. Therefore, the embodiment of the present disclosure uses a series of projection point cloud post-processing and model correction methods to obtain better instance centers.

[0049] Optionally, training using a third model to obtain a target detection model includes: Inputting attribute data of each point cloud of each of the sample instances and the box body of each of the sample instances into a third model for training, thereby training and obtaining a target detection model. may include The attribute data of each point group includes three-dimensional coordinates, a category, and the center coordinates of the box body to which it belongs. In the process of training the target detection model, the attribute data of each dimension of each point cloud needs to be input into the third model. For example, the format of the point cloud input into the third model is [x,y,z,r,cls,c1´,c2´,c3´], where x,y,z are the three-dimensional coordinates, r is the reflectance, cls is the category corresponding to the image instance segmentation, and c1´,c2´,c3´ are the center coordinates of the box to which the point cloud belongs.

[0050] Optionally, the third model is PointPillars, VoxelNet, Sparsely Embedded Convolutional Detection or CenterPoint, and the target detection model trained using these models can improve the accuracy of image detection.

[0051] Step 102: Calculating a virtual center of each of the instances based on the point cloud data and each of the instances.

[0052] In step 1, the entire point cloud data collected by the LIDAR is projected onto each instance to calculate the tentative center [c1, c2, c3] of each instance. In the embodiment of the present disclosure, the tentative center of each instance is calculated in the radar coordinate system, but it is also possible to calculate the tentative center in the image coordinate system, as long as the training stage and the inference stage are in the same coordinate system. The embodiment of the present disclosure does not limit the coordinate system.

[0053] Optionally, step 102 includes: projecting point cloud data onto each of the instances; Merging point cloud data of the same instance appearing in different detection target images according to point cloud data of overlapping portions of each of the instances;

[0054] calculating point cloud data of a body of each of the instances using a clustering algorithm; calculating a virtual center of each of the instances based on point cloud data of a body of each of the instances; It may also include. As in the training phase, First, the entire point cloud data is projected onto each instance, and then, according to the point cloud data of the overlapping parts of each instance, the point cloud data of the overlapping parts is merged into point cloud data corresponding to the same sample instance; Then, a clustering algorithm is used to calculate the point cloud data of the body of each instance. Finally, we calculate the tentative center of each sample instance. In addition, since the same instance may appear in multiple images with different angles, there may be overlapping point clouds in multiple sample images with different angles. First, the point cloud data of the overlapping parts must be merged into the point cloud data corresponding to the same instance, and then the tentative center of each instance must be calculated.

[0055] Optionally, after calculating the body point cloud data of each of the instances using a clustering algorithm, the ground point cloud data also needs to be deleted to avoid the ground point cloud data affecting the clustering of the body point cloud data. Optionally, calculating point cloud data of the body of each of the instances using a clustering algorithm comprises: calculating point cloud data of the body of each of the instances using a spatial clustering algorithm, and deleting point cloud data other than the body of each of the instances; Includes. Optionally, the clustering algorithm may be DBSCAN, FN-DBSCAN, landmark FN-DBSCAN, OPTICS, etc. These clustering algorithms can accurately find the body of each instance.

[0056] Step 103: correcting the tentative center of each of the instances using an instance center correction model to obtain a corrected center of each of the instances.

[0057] After calculating the tentative center [c1, c2, c3] of each instance, the instance center correction module is used to correct the tentative center of each instance, and the output of the instance center correction module is [x, y, z, r, c1´, c2´, c3´], of which [c1´, c2´, c3´] is the corrected instance center.

[0058] Step 104: inputting the corrected centroid of each of the instances into a target detection model to output a box and category of each of the instances.

[0059] All points in the entire point cloud data are input to the target detection model, with each point in the format of [x,y,z,r,c1´,c2´,c3´], and the target detection model outputs the box and category of each instance, i.e., the box and category of the measured object.

[0060] From the above-mentioned embodiments, it can be seen that the embodiment of the present disclosure solves the problem of poor image detection performance in the conventional technology by using an image instance segmentation model to perform instance segmentation on the detection target image, correcting the tentative center of each instance using an instance center correction model, and inputting the corrected center of each instance into the target detection model. In the embodiment of the present disclosure, the image instance segmentation result is used to perform instance-level annotation for each point in the point cloud, and the center of each instance is further corrected, and then target detection is performed by the point cloud detection algorithm. Since the instance information of the point cloud is obtained before being sent to the point cloud detection algorithm, the difficulty of learning the detection algorithm is greatly reduced and the detection performance is greatly improved.

[0061] 2 is a schematic diagram of a main process of an image detection method according to one possible embodiment of the present disclosure. As another embodiment of the present disclosure, as shown in FIG. 2, the image detection method may include the following steps:

[0062] Step 201: obtaining a sample image of each angle, marking the position, category and contour of each sample instance on the sample image of each angle, and using a first model for training to obtain an image instance segmentation model;

[0063] The sample point cloud data of the multi-box body and RGB images corresponding to the sample point cloud data of each box body can be obtained. The sample point cloud data of each box body can correspond to multiple RGB images (i.e., sample images that can be collected by RGB cameras mounted at different angles).

[0064] Specifically, mark the label of the instance to which each pixel in the sample image belongs, and obtain an image instance segmentation model by training based on the marking result. Optionally, the step of obtaining an image instance segmentation model by training using the first model includes: Inputting the sample images of each angle and the positions, categories, and contours of each sample instance of the sample images into a first model for training, thereby training and obtaining an image instance segmentation model. Includes. First, we collect sample images at each angle using an RGB camera attached at each angle, then mark the label of the instance to which each pixel in the sample image belongs, Then, the sample images of each angle and their mark results are input to the first model for training, and an image instance segmentation model is obtained through iterative training. Optionally, the first model is Mask R-CNN, Hybrid Task Cascade or BlendMaskd, and better training results can be obtained by training sample images of each angle with these models.

[0065] Step 202: Calculating a tentative center of each of the sample instances, marking a box body of each of the sample instances in the sample point cloud data, and using a second model to train and obtain an instance center correction model.

[0066] At the same time, the tentative center of each sample instance is calculated, and the box body of each sample instance marked in the sample point cloud data is combined with the box body of each sample instance marked in the sample point cloud data, and trained to obtain an instance center correction module. When training the instance center correction model, the tentative center of each sample instance is used as input data, and the center of the box body of each sample instance in the data is marked as the regression target, and these two parts of data are sent to the second model for training, and an instance center correction module is obtained through iterative training, and this module is used to correct the instance center, thereby greatly reducing the difficulty of learning the detection algorithm and greatly improving the detection performance. The format of the point cloud data input to the second model is [x,y,z,r,c1,c2,c3], and the format of the box body center of each sample instance input to the second model is [x,y,z,r,c1´,c2´,c3´], which is the regression target, where x,y,z are the three-dimensional coordinates, r is the reflectance, and [c1´,c2´,c3´] are the box body centers of the marked sample instances.

[0067] Step 203: Using the third model to train and obtain a target detection model.

[0068] Specifically, the attribute data of each point cloud of each sample instance and the box body of each sample instance are input into a third model for training, so as to train and obtain a target detection model. Among them, the attribute data of each point cloud includes three-dimensional coordinates, a category, and the center coordinates of the box body to which it belongs. In the process of training the target detection model, the attribute data of each dimension of each point cloud needs to be input into the third model. For example, the format of the point cloud input into the third model is [x, y, z, r, cls, c1', c2', c3'], in which x, y, z are three-dimensional coordinates, r is reflectance, cls is a category corresponding to image instance segmentation, and c1', c2', c3' are the center coordinates of the box body to which the point cloud belongs.

[0069] Step 204: Performing instance segmentation on each detection target image using an image instance segmentation model to obtain each instance in each detection target image.

[0070] In the inference stage, it is necessary to obtain point cloud data of a box body and at least one corresponding RGB image (i.e., the target image that can be collected by RGB cameras mounted at different angles).

[0071] Step 205: Calculating a tentative center of each of the instances based on the point cloud data and each of the instances.

[0072] The entire point cloud data collected by the LIDAR is projected onto each instance, and the tentative center [c1, c2, c3] of each instance is calculated. Specifically, the point cloud data is projected onto each instance, and the point cloud data of the same instance that appears in different detection target images is merged according to the point cloud data of the overlapping parts of each instance, and the point cloud data of the main body of each instance is calculated using a clustering algorithm, and the tentative center of each instance is calculated based on the point cloud data of the main body of each instance. Similar to the training stage, First, the entire point cloud data is projected onto each instance, and then, according to the point cloud data of the overlapping parts of each instance, the point cloud data of the overlapping parts is merged into point cloud data corresponding to the same sample instance; Then, a clustering algorithm is used to calculate the point cloud data of the body of each instance. Finally, we calculate the tentative center of each sample instance. In addition, since the same instance may appear in multiple images with different angles, there may be overlapping point clouds in the multiple images with different angles. First, the point cloud data of the overlapping parts must be merged into the point cloud data corresponding to the same instance, and then the tentative center of each instance must be calculated. Optionally, after calculating the body point cloud data of each of the instances using a clustering algorithm, the ground point cloud data also needs to be deleted to avoid the ground point cloud data affecting the clustering of the body point cloud data. Optionally, calculating point cloud data of the body of each of the instances using a clustering algorithm comprises: calculating point cloud data of the body of each of the instances using a spatial clustering algorithm, and deleting point cloud data other than the body of each of the instances; Includes. Optionally, the clustering algorithm may be DBSCAN, FN-DBSCAN, landmark FN-DBSCAN, OPTICS, etc. These clustering algorithms can accurately find the body of each instance.

[0073] Step 206: correcting the tentative center of each of the instances using an instance center correction model to obtain a corrected center of each of the instances.

[0074] After calculating the tentative center [c1, c2, c3] of each instance, the instance center correction model is used to correct the tentative center of each instance. The output of the instance center correction module is [x, y, z, r, c1´, c2´, c3´], of which [c1´, c2´, c3´] is the corrected instance center.

[0075] Step 207: inputting the corrected centroid of each of the instances into a target detection model to output a box and category of each of the instances.

[0076] All points in the entire point cloud data are input to the target detection model, with each point in the format of [x,y,z,r,c1´,c2´,c3´], and the target detection model outputs the box and category of each instance, i.e., the box and category of the measured object.

[0077] In this embodiment, steps 201-203 are the training stage, and steps 204-207 are the inference stage.

[0078] In addition, the specific implementation content of the image detection method according to one possible embodiment of the present disclosure has already been described in detail in the above-mentioned image detection method, so the duplicated content will not be described here.

[0079] 3 is a schematic diagram of the main modules of an image sensing device according to an embodiment of the present disclosure. As shown in FIG. 3, the image sensing device 300 includes: a segmentation module 301 for performing instance segmentation on the target images using an image instance segmentation model to obtain each instance in each target image; a calculation module 302 for calculating a virtual center of each of the instances based on the point cloud data and each of the instances; a correction module 303 for correcting the tentative center of each of the instances using an instance center correction model to obtain a corrected center of each of the instances; a detection module 304 for inputting the corrected centroid of each of the instances into a target detection model to output a box and category of each of the instances; Includes.

[0080] Optionally, the calculation module 302 further comprises: projecting the point cloud data onto each of the instances; Merging point cloud data of the same instance appearing in different detection target images according to point cloud data of overlapping portions of each of the instances; calculating point cloud data of the body of each of said instances using a clustering algorithm; Calculate a tentative center of each of the instances based on point cloud data of the body of each of the instances. This is for the purpose.

[0081] Optionally, the calculation module 302 further comprises: A spatial clustering algorithm is used to calculate point cloud data of the main body of each of the instances, and point cloud data other than the main body of each of the instances is deleted. This is for the purpose.

[0082] Optionally, further comprising: Obtaining a sample image for each angle, marking a position, a category, and a contour of each sample instance on the sample image for each angle, and training using a first model to obtain an image instance segmentation model; Calculate a tentative center of each of the sample instances, mark a box of each of the sample instances in the sample point cloud data, and use a second model to train to obtain an instance center correction model; Use a third model to train and obtain a target detection model Includes a training module for

[0083] Optionally, the training module further comprises: The sample images of each angle and the positions, categories, and contours of each sample instance are input to a first model for training, thereby training and obtaining an image instance segmentation model. This is for the purpose of Among them, the first model is Mask R-CNN, Hybrid Task Cascade, or BlendMaskd.

[0084] Optionally, the training module further comprises: Calculating a tentative center of each of the sample instances based on the sample point cloud data and each of the sample instances; marking a box for each of the sample instances in the sample point cloud data and calculating a center of the box for each of the sample instances; The tentative center of each sample instance and the center of the box body of each sample instance are input to a second model and training is performed to train and obtain an instance center correction model. This is for the purpose.

[0085] Optionally, the training module further comprises: projecting the sample point cloud data onto each of the sample instances; Merging point cloud data of the same sample instance appearing in different sample images according to point cloud data of the overlapping portion in each of the sample instances; calculating point cloud data of the body of each of the sample instances using a clustering algorithm; Calculating a tentative center of each of the sample instances based on point cloud data of the body of each of the sample instances. This is for the purpose.

[0086] Optionally, the second model is PointNet, PointNet++, Cylider3D, or Rand-LA.

[0087] Optionally, the training module further comprises: The attribute data of each point cloud of each of the sample instances and the box body of each of the sample instances are input to a third model for training, thereby training and obtaining a target detection model. This is for the purpose of The attribute data of each point group includes three-dimensional coordinates, a category, and the center coordinates of the box body to which it belongs.

[0088] Optionally, the third model is PointPillars, VoxelNet, Sparsely Embedded Convolutional Detection, or CenterPoint.

[0089] Note that the specific implementation of the image detection device according to the present disclosure has already been described in detail in the above-mentioned image detection method, so that duplicated content will not be described here.

[0090] FIG. 4 illustrates an exemplary system architecture 400 of an image detection method or device to which embodiments of the present disclosure may be applied.

[0091] 4, the system architecture 400 may include terminal devices 401, 402, 403, a network 404, and a server 405. The network 404 is a medium for providing a communication link between the terminal devices 401, 402, 403 and the server 405. The network 404 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

[0092] Using terminal devices 401, 402, and 403, users can communicate with server 405 via network 404 to send and receive messages, etc. Various communication client applications can be installed on terminal devices 401, 402, and 403, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, and the like (just to name a few).

[0093] The terminal devices 401, 402, 403 may be various electronic devices equipped with a display and supporting web browsing, including, but not limited to, smartphones, tablet computers, laptop computers, desktop computers, and the like.

[0094] Server 405 may be a server that provides various services, for example, a background management server (just an example) that provides support to shopping sites that users browse using terminal devices 401, 402, and 403. The background management server performs processing such as analysis on received data such as product information inquiry requests, and can feed back the processing results to the terminal devices.

[0095] In addition, the image detection method provided by the embodiment of the present disclosure is generally executed by the server 405, and correspondingly, the image detection device is generally provided in the server 405. The image detection method provided by the embodiment of the present disclosure may also be executed by the terminal devices 401, 402, and 403, and correspondingly, the image detection device may be provided in the terminal devices 401, 402, and 403.

[0096] It should be understood that the number of terminal devices, networks and servers in FIG. 4 is merely schematic, and that there may be any number of terminal devices, networks and servers as required for implementation.

[0097] Reference is now made to Fig. 5, which illustrates a structural schematic diagram of a computer system 500 suitable for implementing the terminal device of the embodiment of the present disclosure. The terminal device illustrated in Fig. 5 is merely an example, and does not limit the function and scope of use of the embodiment of the present disclosure.

[0098] 5, computer system 500 includes a central processing unit (CPU) 501 capable of performing various suitable operations and processes based on a program stored in a read-only memory (ROM) 502 or loaded from a storage portion 508 into a random access memory (RAM) 503. RAM 503 further stores various programs and data necessary for the operation of system 500. CPU 501, ROM 502 and RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to bus 504.

[0099] The following items are connected to the I / O interface 505: An input section 506 including a keyboard, mouse, etc.; An output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; A storage section 508 including a hard disk, etc.; A communication section 509 including a network interface card such as a LAN card or a modem. A communication unit 509 executes communication processing via a network such as the Internet. A driver 510 is also connected to the I / O interface 505 as necessary. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is installed in the driver 510 as necessary so that a computer program read from the removable medium 511 is installed in the storage portion 508 as necessary.

[0100] In particular, based on the embodiments disclosed in the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, the embodiments disclosed in the present disclosure include a computer program stored on a computer-readable medium, the computer program including program codes for performing the methods shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 509 and / or installed from a removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, it performs the above-mentioned functions limited to the system of the present disclosure.

[0101] It should be noted that the computer readable medium referred to in this disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the above. The computer readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of computer readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer magnetic disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), a rewritable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact magnetic disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this disclosure, the computer readable storage medium may be any tangible medium that contains or stores a program, which may be used in or in combination with a command execution system, apparatus, or device. Also, in the present disclosure, a computer-readable signal medium may include a data signal propagating in baseband or as part of a carrier wave, in which computer-readable program code is stored. Such a propagating data signal may take various forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may transmit, propagate, or transmit a program used by or in conjunction with a command execution system, apparatus, or device.The program code contained in the computer readable medium may be transmitted over any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0102] The flowcharts and block diagrams in the drawings illustrate possible architectures, functions and operations of systems, methods and computer programs according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, a program section or a portion of code, which includes one or more executable commands for implementing a given logic function. It should also be noted that in some alternative implementations, the functions noted in the blocks may occur in a different order than those noted in the drawings. For example, two consecutively shown blocks may actually be essentially executed in parallel, or may even be executed in the opposite order, as determined by the associated functionality. It should also be noted that each block in the block diagrams or flowcharts, and combinations of blocks in the block diagrams or flowcharts, may be implemented by a dedicated hardware-based system for implementing a given function or operation, or by a combination of dedicated hardware and computer commands.

[0103] The relevant modules described in the embodiments of the present disclosure may be implemented in a software manner or a hardware manner. The described modules may be provided in a processor, and may be described as including, for example, a segmentation module, a calculation module, a correction module, and a detection module. In particular, the names of the modules are not intended to be limitations on the modules themselves.

[0104] In another aspect, the present disclosure further provides a computer-readable medium. The computer-readable medium may be included in the device described in the above embodiment, or may be independent of the device. The computer-readable medium stores one or more programs, and when the one or more programs are executed by one of the devices, the device performs the following: performing instance segmentation on the target image using an image instance segmentation model to obtain each instance in the target image; Calculating a virtual center of each of the instances based on the point cloud data and each of the instances; correcting the tentative center of each of the instances using an instance center correction model to obtain a corrected center of each of the instances; inputting the corrected centroid of each of the instances into a target detection model, and outputting a box and a category of each of the instances; Realize the method.

[0105] According to the technical solution of the embodiment of the present disclosure, the image instance segmentation model is used to perform instance segmentation on the detection target image, the instance center correction model is used to correct the tentative center of each instance, and the corrected center of each instance is input into the target detection model, thereby solving the problem of poor image detection performance in the prior art. In the embodiment of the present disclosure, the image instance segmentation result is used to perform instance-level annotation for each point in the point cloud, and the center of each instance is further corrected, and then the target is detected by the point cloud detection algorithm. Since the instance information of the point cloud is obtained before being sent to the point cloud detection algorithm, the difficulty of learning the detection algorithm is greatly reduced and the detection performance is greatly improved.

[0106] The above-mentioned embodiments for carrying out the invention do not limit the scope of protection of the present disclosure. It is obvious to those skilled in the art that various modifications, combinations, subcombinations and substitutions may occur according to design requirements and other factors. Any modifications, equivalent replacements, improvements, etc. made within the scope of the idea and principle of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. performing instance segmentation on the target image using an image instance segmentation model to obtain each instance in the target image; calculating a virtual center of each of the instances based on the point cloud data and each of the instances; correcting the tentative center of each of the instances using an instance center correction model to obtain a corrected center of each of the instances; inputting the corrected centroid of each of the instances into a target detection model to output a box and a category of each of the instances; Includes Image detection methods.

2. Calculating a virtual center of each of the instances based on point cloud data and each of the instances includes: projecting point cloud data onto each of the instances; Merging point cloud data of the same instance appearing in different detection target images according to point cloud data of overlapping portions of each of the instances; calculating point cloud data of a body of each of the instances using a clustering algorithm; Calculating a virtual center of each of the instances based on point cloud data of a body of each of the instances; Includes The method of claim 1.

3. Calculating point cloud data of the body of each of the instances using a clustering algorithm includes: calculating point cloud data of the body of each of the instances using a spatial clustering algorithm, and deleting point cloud data other than the body of each of the instances; Includes The method of claim 2.

4. performing instance segmentation on the detection target image using an image instance segmentation model to obtain each instance in the detection target image, further comprising: Obtaining a sample image of each angle, marking a position, a category, and a contour of each sample instance on the sample image of each angle, and training using a first model to obtain an image instance segmentation model; Calculating a tentative center of each of the sample instances, marking a box of each of the sample instances in the sample point cloud data, and training using a second model to obtain an instance center correction model; Training the third model to obtain a target detection model; and Includes The method of claim 1.

5. Training using a first model to obtain an image instance segmentation model includes: training the sample images of each angle and the positions, categories, and contours of each sample instance of the sample images into a first model for training, thereby obtaining an image instance segmentation model; Including, Wherein, the first model is Mask R-CNN, Hybrid Task Cascade, or BlendMaskd; The method according to claim 4.

6. Calculating a tentative center of each sample instance, marking a box body of each sample instance in sample point cloud data, and training using a second model to obtain an instance center correction model, calculating a tentative center of each of the sample instances based on the sample point cloud data and each of the sample instances; marking a box for each of the sample instances in the sample point cloud data and calculating a center of the box for each of the sample instances; training and acquiring an instance center correction model by inputting the tentative center of each sample instance and the center of the box of each sample instance into a second model and training the second model; Includes The method according to claim 4.

7. Calculating a tentative center of each of the sample instances based on the sample point cloud data and each of the sample instances includes: projecting sample point cloud data onto each of the sample instances; merging point cloud data of the same sample instance appearing in different sample images according to point cloud data of overlapping portions in each of the sample instances; calculating point cloud data of the body of each of the sample instances using a clustering algorithm; Calculating a tentative center of each of the sample instances based on point cloud data of a body of each of the sample instances; Including, The method according to claim 6.

8. Training using a third model to obtain a target detection model; Inputting attribute data of each point cloud of each of the sample instances and the box body of each of the sample instances into a third model for training, thereby training and obtaining a target detection model. Including, The attribute data of each point group includes three-dimensional coordinates, a category, and the center coordinates of the box body to which it belongs. The method according to claim 4.

9. a segmentation module for performing instance segmentation on the target image using an image instance segmentation model to obtain each instance in the target image; a calculation module for calculating a virtual center of each of the instances based on the point cloud data and each of the instances; a correction module for correcting the tentative center of each of the instances using an instance center correction model to obtain a corrected center of each of the instances; a detection module for inputting the corrected centroid of each of the instances into a target detection model to output a box and a category of each of the instances; Includes Image detection device.

10. one or more processors; a storage device for storing one or more programs; Including, When said one or more programs are executed by said one or more processors, said one or more processors implement the method according to any one of claims 1 to 8. electronic equipment.

11. A computer readable medium having stored thereon a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images