Multi-object detection method and data platform based on multi-scale features

Abstract

The application discloses a multi-object detection method and data platform based on multi-scale features, and the method comprises the following steps: inputting an object image of multiple objects into a pre-trained neural network model; performing feature processing operation on multiple feature maps of the object image by using an instance segmentation network model to obtain the category, visible bounding box and corresponding mask of each object; performing segmentation operation on an initial feature map in the multiple feature maps according to the visible bounding box and corresponding mask of each object to obtain a final feature map; predicting the coordinate difference of the visible bounding box and a complete box of each object on the final feature map and calculating the coordinate data of the complete box; and obtaining the position information of each object on the object image according to the coordinate data of the complete box. The application can realize simultaneous detection of multiple networks, has very good generalization ability and robustness, and has the most advanced performance, and has important theoretical research value and broad application prospect in the object detection direction.

Description

Technical Field

[0001] This invention relates to the field of computer vision object detection technology, and in particular to a multi-object detection method and data platform based on multi-scale features. Background Technology

[0002] Object detection is an important research area in computer vision. Its applications in fields such as autonomous driving, robotics, navigation, surveillance, and healthcare are very extensive, making its research value extremely significant.

[0003] Current object detection algorithms based on monocular images can only detect the unoccluded portions (i.e., the visible parts) of objects, especially occluded objects, and cannot accurately detect their complete 2D bounding boxes (referred to as complete 2D boxes in this invention). However, detecting complete 2D boxes is crucial for accurate object localization. Research on object perception using brain visual neural networks has found that the brain's visual cortex can accurately construct the complete 2D box of an object based on image features of the visible portion, enabling accurate detection and localization of objects in the physical world. Therefore, there is an urgent need for a detection method based on a network model to achieve accurate detection and localization of multiple objects with multi-scale features. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] Therefore, the purpose of this invention is to propose a multi-object detection method and data platform based on multi-scale features, which can simultaneously detect the visible 2D box, mask, and complete 2D box of an object, and achieve very good generalization ability, robustness, and state-of-the-art performance. It has important theoretical research value and broad application prospects in the field of object detection.

[0006] To achieve the above objectives, this invention proposes a multi-object detection method based on multi-scale features, comprising:

[0007] The object images of multiple objects are input into a pre-trained neural network model, which includes an instance segmentation network model and an object bounding box detection network model.

[0008] The instance segmentation network model is used to perform feature processing operations on multiple feature maps of the object image to obtain the category, visible bounding box, and corresponding mask of each object; the object complete box detection network model is used to segment the initial feature map in the multiple feature maps according to the visible bounding box and corresponding mask of each object to obtain the final feature map;

[0009] The object complete bounding box detection network model predicts the coordinate difference between the visible bounding box and the complete bounding box of each object on the final feature map and calculates the coordinate data of the complete bounding box. Based on the coordinate data of the complete bounding box, the position information of each object on the object image is obtained.

[0010] The multi-object detection method based on multi-scale features according to embodiments of the present invention may also have the following additional technical features:

[0011] Furthermore, in one embodiment of the present invention, the step of using an instance segmentation network model to perform feature processing operations on multiple feature maps of an object image to obtain the category, visible bounding box, and corresponding mask of each object in the object image includes: using a first preset network to extract features from object images with the same pixels to obtain the initial feature map; performing segmentation processing on the initial feature map according to a second preset network to obtain a first feature map; and performing feature processing on the first feature map based on a third preset network to obtain the category, visible bounding box, and corresponding mask of each object in the object image; wherein, the feature processing includes pooling operation, object classification, bounding box regression, and mask prediction.

[0012] Furthermore, in one embodiment of the present invention, the step of extracting features from object images with the same pixels using a first preset network to obtain the initial feature map includes: acquiring multiple object images with different pixels; performing pixel transformation operations on the multiple object images with different pixels to obtain object images with the same pixels; and extracting features from the object images with the same pixels using a residual network ResNet and a feature map pyramid network FPN to obtain the initial feature map.

[0013] Furthermore, in one embodiment of the present invention, the step of segmenting the initial feature map according to the second preset network to obtain the first feature map includes: obtaining a Region Proposal Network (RPN) and generating multiple proposal boxes of the RPN; and using the multiple proposal boxes to perform image segmentation on the initial feature map to obtain the first feature map corresponding to the proposal box.

[0014] Furthermore, in one embodiment of the present invention, the step of performing feature processing on the first feature map based on a third preset network to obtain the category of each object in the object image, the visible bounding box of each object, and the corresponding mask includes: performing region pooling operation on the first feature map to obtain a second feature map; performing object classification and bounding box regression on the second feature map using a fully connected layer to obtain the category and visible bounding box of each object in the object image; and performing mask prediction on the second feature map using an object mask prediction network to obtain the corresponding mask.

[0015] Furthermore, in one embodiment of the present invention, the step of predicting the coordinate difference between the visible bounding boxes and the complete boxes of each object on the final feature map based on the object complete box detection network model and calculating the coordinate data of the complete boxes includes: obtaining the pooling and fully connected layers of the object complete box detection network model; inputting the final feature map into the pooling and fully connected layers, predicting the center coordinate difference and height-width scaling factor between the visible bounding boxes and the complete boxes of each object on the final feature map, and calculating the coordinate data of the complete boxes based on the center coordinate difference and height-width scaling factor.

[0016] Furthermore, in one embodiment of the present invention, before obtaining the final feature map, the method further includes: performing pixel processing operations on the pixel values ​​of the final feature map according to the masks corresponding to the visible bounding boxes of each object.

[0017] Furthermore, in one embodiment of the present invention, the pixel processing operation includes setting the pixel value on the final feature map corresponding to the coordinate point with a first mask value of true to an initial value state, and setting the pixel value on the final feature map corresponding to the coordinate point with a second mask value of false to 0.

[0018] To achieve the above objectives, another aspect of the present invention proposes a data platform, comprising:

[0019] The dataset acquisition module is used to acquire the training and testing datasets for the instance segmentation and bounding box detection network.

[0020] The dataset annotation module is used to annotate instances in the dataset; wherein, the instance annotation includes object category, truncation degree, occlusion degree, bounding box, and object instance mask;

[0021] The training and testing module is used to obtain the number of complete boxes corresponding to the number of object instance masks, and to obtain the coordinate data of the object's visible bounding box based on the coordinates of the object instance masks. The module trains the instance segmentation and complete box detection network model based on the coordinate data of the object's visible bounding box and the corresponding complete box coordinate data, and performs dataset testing based on the trained instance segmentation and complete box detection network model.

[0022] To achieve the above objectives, another aspect of the present invention provides a computer device.

[0023] It includes a processor and a memory; wherein the processor runs a program corresponding to the executable program code stored in the memory to implement a multi-object detection method based on multi-scale features.

[0024] The multi-object detection method and data platform based on multi-scale features of this invention can realize simultaneous detection by multiple networks, achieve accurate detection and localization of multiple objects with multi-scale features, and achieve very good generalization ability and robustness as well as state-of-the-art performance. It has important theoretical research value and broad application prospects in the field of object detection.

[0025] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0026] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0027] Figure 1 This is a flowchart of a multi-object detection method based on multi-scale features according to an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram of a neural network structure for detecting complete 2D bounding boxes based on object image features according to an embodiment of the present invention;

[0029] Figure 3 This is a visualization diagram of image annotations for the KITTI Mask dataset according to an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of the data platform structure according to an embodiment of the present invention;

[0031] Figure 5 This is a schematic diagram of the visual detection results according to an embodiment of the present invention;

[0032] Figure 6 A computer device according to an embodiment of the present invention. Detailed Implementation

[0033] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0034] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0035] The following description, with reference to the accompanying drawings, illustrates a multi-object detection method and data platform based on multi-scale features proposed according to embodiments of the present invention.

[0036] Figure 1 This is a flowchart of a multi-object detection method based on multi-scale features according to an embodiment of the present invention.

[0037] like Figure 1 As shown, the method includes, but is not limited to, the following steps:

[0038] S1, input the object images of multiple objects into a pre-trained neural network model, which includes an instance segmentation network model and an object bounding box detection network model.

[0039] Understandable Figure 2 This is a schematic diagram of the complete 2D bounding box detection neural network structure based on object image features proposed in this invention. Figure 2 As shown, this network mainly consists of two sub-networks: one is an object instance segmentation network, and the other is an object complete 2D bounding box detection network. The object instance segmentation network provides the object complete 2D bounding box detection network with image features, bounding boxes, and masks of the visible part of the object.

[0040] As an example, the object instance segmentation network is responsible for object classification, visible 2D box regression, and mask prediction. This sub-network structure includes an object feature map extraction part based on ResNet and Feature Map Pyramid Network (FPN), a Region Proposal Network (RPN) part, a Region of Interest Pooling (RoI) part, an object classification and visible 2D box regression part based on fully connected layers (FC layers), and an object mask prediction network (Mask branch).

[0041] S2, the instance segmentation network model is used to perform feature processing operations on multiple feature maps of the object image to obtain the category, visible bounding box and corresponding mask of each object; the object complete box detection network model performs segmentation operation on the initial feature maps of multiple feature maps according to the visible bounding box and corresponding mask of each object to obtain the final feature map.

[0042] As an example, the subnetwork of the instance segmentation network model first transforms object images of different pixel sizes (e.g., 800*600, 1242*375, etc.) into 1024*1024 pixel images. Then, it uses a residual network (ResNet) and a feature map pyramid network (FPN) to extract features from the image. Next, it uses a region proposal network (RPN) to generate proposal boxes and segments the corresponding feature maps within the proposal boxes on the image feature maps. Then, the feature maps within each proposal box are processed by region of interest pooling (RoI pooling) to generate fixed-size (e.g., 7*7) feature maps. Finally, the pooled feature maps are shared to fully connected layers (FC layers) for object classification and visible 2D box regression, and simultaneously shared to the object mask prediction network (Mask branch) for mask prediction.

[0043] S3: Based on the object complete box detection network model, predict the coordinate difference between the visible bounding box and the complete box of each object on the final feature map and calculate the coordinate data of the complete box. Based on the coordinate data of the complete box, obtain the position information of each object on the object image.

[0044] Specifically, the complete 2D bounding box detection network is responsible for detecting complete 2D bounding boxes of objects. This sub-network first segments the object feature maps (box_mask feature maps) within the visible 2D bounding box (box) and the corresponding mask detected by the object instance segmentation network above on the object feature maps. The specific operation steps are as follows: (1) Segment the feature maps within the corresponding box on the image feature maps based on the visible 2D bounding box (box); (2) Based on the mask corresponding to the visible 2D bounding box (box), operate on the pixel values ​​of the feature maps within the visible 2D bounding box (box), that is, keep the pixel values ​​on the feature maps corresponding to the coordinate points with a mask value of true unchanged, and set the pixel values ​​on the feature maps corresponding to the coordinate points with a mask value of false to 0; then input the generated feature maps (box_mask feature maps) into the pooling and fully connected layers (Pooling & FC). Layers are used to estimate the difference in center coordinates and the scaling factor of height and width between the visible 2D box and the complete 2D box, i.e., complete box regression.

[0045] In other words, the coordinates of the complete 2D bounding box are obtained by regression, which enables the prediction of the position information of object images with multiple objects. The position information of each object on the object image can be obtained based on the coordinate data of the complete 2D bounding box.

[0046] This invention presents a multi-object detection method based on multi-scale features, utilizing a neural network for detecting complete 2D bounding boxes based on object image features. This network uses Mask R-CNN for accurate object instance segmentation, and then detects the complete 2D bounding boxes of objects based on the features of the segmented semantic parts, achieving accurate detection and localization of objects in monocular images. To train and test the proposed network, a dataset for object instance segmentation and complete 2D bounding box detection was created.

[0047] The multi-object detection method based on multi-scale features according to embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0048] Specifically, this invention creates a dataset (referred to as the KITTI Mask dataset) for training and testing network for object instance segmentation and complete 2D bounding box detection by annotating the KITTI dataset with object masks. This dataset includes 7481 images, with the object category being "Car". It builds upon the KITTI dataset, which already annotates the car category, truncation level, occlusion level, and complete 2D bounding boxes, by adding annotated car instance masks (using polygons format, represented by polygon vertices). The number of instance masks equals the number of complete 2D bounding boxes, and they correspond one-to-one. The coordinates of the visible 2D bounding boxes of the car can be obtained from the coordinates of the car instance masks. This dataset can be used not only for training and testing the complete 2D bounding box detection neural network based on object image features proposed in this invention, but also for training and testing object detection and instance segmentation algorithms. Figure 3 The image shown is a visualization of the labeled examples from this dataset, demonstrating that both the car's mask and its complete 2D bounding box are labeled simultaneously. This dataset fills a gap in current datasets that simultaneously label object masks and complete 2D bounding boxes.

[0049] The multi-object detection method based on multi-scale features according to embodiments of the present invention can realize simultaneous detection by multiple networks, achieve accurate detection and localization of multiple objects with multi-scale features; and achieves very good generalization ability and robustness as well as state-of-the-art performance, which has important theoretical research value and broad application prospects in the field of object detection.

[0050] To achieve the above embodiments, such as Figure 4As shown, this embodiment also provides a data platform 10, characterized in that it includes: a dataset acquisition module 100, a dataset annotation module 200, and a training and testing module 300.

[0051] The dataset acquisition module 100 is used to acquire the training and testing datasets for the instance segmentation and bounding box detection network;

[0052] The dataset annotation module 200 is used to annotate instances in the dataset; the instance annotation includes object category, truncation degree, occlusion degree, bounding box, and object instance mask;

[0053] The training and testing module 300 is used to obtain the number of complete boxes corresponding to the number of object instance masks, and to obtain the coordinate data of the object's visible bounding box based on the coordinates of the object instance masks. Based on the coordinate data of the object's visible bounding box and the corresponding coordinate data of the complete box, the instance segmentation and complete box detection network model is trained, and the trained instance segmentation and complete box detection network model is tested on the dataset.

[0054] Specifically, this invention establishes a training and testing experimental platform for a complete 2D bounding box detection neural network based on object image features.

[0055] As an example, the KITTI Mask dataset created in this invention, used in the experiment, includes 7481 images, divided into 4500 training images, 1450 validation images, and 1531 test images, with the object category being "car". The evaluation of the complete 2D box detection results is based on three difficulty levels: easy, medium, and hard. The Intersection over Union (IoU) threshold for the complete 2D boxes is set to 0.7. The three difficulty levels (easy, medium, and hard) are determined based on the minimum height (40, 25, and 25 pixels) of the ground truth and the detected complete 2D boxes, the maximum occlusion level (0, 1, and 2) of the ground truth, and the maximum truncation level (0.15, 0.3, and 0.5) of the ground truth.

[0056] In the training experiments, the residual network (ResNet), feature map pyramid network (FPN), and region proposal network (RPN) in the complete 2D bounding box detection neural network based on object image features proposed in this invention were pre-trained using the MS-COCO dataset. Fine-tuning was performed using the KITTI Mask dataset created in this invention. The network was trained for 56,250 iterations at a constant learning rate of 0.0001. The hardware used was a 1080TI GPU and a 1.7GHz CPU.

[0057] The test results obtained from the experiment are shown in Table 1. AP 2D This represents the average accuracy of the complete 2D box. It is evident that the Complete 2D Box Estimation Network proposed in this invention, based on object image features, achieves a higher average accuracy for complete 2D boxes across three difficulty levels (easy, medium, and hard) compared to current advanced monocular image-based object detection algorithms such as Faster RCNN, Mask RCNN, RRC, and SHJU-HW.

[0058] Table 1

[0059]

[0060] Figure 5 The test experiment provides some visual detection results of images, showing that the complete 2D bounding box detection neural network based on object image features proposed in this invention can not only achieve accurate instance segmentation (solid polygons in the figure), but also achieve accurate detection of complete 2D bounding boxes of objects (dashed boxes in the figure).

[0061] Understandably, other object detection algorithms based on monocular images can only detect a single visible 2D box, a single complete 2D box, or both the visible 2D box and the mask. However, the complete 2D box detection neural network based on object image features proposed in this invention can simultaneously detect the visible 2D box, the mask, and the complete 2D box, and has excellent generalization ability and robustness, achieving state-of-the-art performance. It has significant theoretical research value and broad application prospects in the field of object detection.

[0062] According to the detection platform of the present invention, the dataset of the detection platform fills the gap in the current datasets that simultaneously annotate object masks and complete 2D bounding boxes. Through the multi-object detection method based on multi-scale features of the detection platform, multiple networks can be used for simultaneous detection, and the accurate detection and localization of multiple objects with multi-scale features can be achieved. Furthermore, it achieves very good generalization ability and robustness as well as state-of-the-art performance, and has important theoretical research value and broad application prospects in the field of object detection.

[0063] To implement the methods of the above embodiments, the present invention also provides a computer device, such as... Figure 6As shown, the computer device 600 includes a memory 601 and a processor 602; wherein, the processor 602 reads the executable program code stored in the memory 601 to run a program corresponding to the executable program code, so as to implement the various steps of the multi-object detection method based on multi-scale features described above.

[0064] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0065] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0066] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A multi-object detection method based on multi-scale features, characterized in that, Includes the following steps: The object images of multiple objects are input into a pre-trained neural network model, which includes an instance segmentation network model and an object bounding box detection network model. The instance segmentation network model is used to perform feature processing operations on various feature maps of the object image to obtain the visible part information of each object. The visible part information includes the object category, visible bounding box and corresponding mask. Based on the visible bounding boxes and corresponding masks of each object, image features corresponding to the visible parts of each object are extracted from the multiple feature maps to construct a final feature map corresponding to each object after removing background interference. Constructing the final feature map corresponding to each object after removing background interference includes: segmenting the feature map within the corresponding box based on the initial feature map of the visible bounding boxes in the multiple feature maps; and based on the mask corresponding to the visible bounding boxes, keeping the pixel values ​​of the feature maps within the boxes corresponding to coordinate points with true mask values ​​at their initial values, and setting the pixel values ​​of the feature maps within the boxes corresponding to coordinate points with false mask values ​​to zero, to generate the final feature map. Based on the object complete bounding box detection network model, and based on the final feature map, the geometric transformation parameters between the visible bounding box and the corresponding complete bounding box of each object are regressed and predicted; and based on the geometric transformation parameters and the coordinate data of the visible bounding box, the coordinate data of the complete box is calculated, thereby obtaining the complete position information of each object on the object image; The geometric transformation parameters include the difference in center coordinates between the visible bounding box and the complete bounding box, and the scaling factor of the height and width of the complete bounding box relative to the visible bounding box.

2. The method according to claim 1, characterized in that, The method of using an instance segmentation network model to perform feature processing operations on multiple feature maps of an object image to obtain the category, visible bounding box, and corresponding mask of each object in the object image includes: The initial feature map is obtained by extracting features from object images with the same pixels using a first preset network. The initial feature map is segmented according to the second preset network to obtain the first feature map; The first feature map is processed based on a third preset network to obtain the category of each object in the object image, the visible bounding box of each object, and the corresponding mask; wherein, the feature processing includes pooling operation, object classification, bounding box regression, and mask prediction.

3. The method according to claim 2, characterized in that, The step of extracting features from object images with the same pixels using a first preset network to obtain the initial feature map includes: Acquire object images with multiple different pixel values; Perform pixel conversion operations on the multiple object images with different pixels to obtain object images with the same pixels; The initial feature map is obtained by extracting features from the object image with the same pixels using the ResNet residual network and the Feature Map Pyramid Network (FPN).

4. The method according to claim 2, characterized in that, The step of segmenting the initial feature map according to the second preset network to obtain the first feature map includes: Obtain the Regional Proposal Network (RPN) and generate multiple proposal boxes for the RPN; The initial feature map is segmented using the multiple suggestion boxes to obtain the first feature map corresponding to each suggestion box.

5. The method according to claim 2, characterized in that, The feature processing of the first feature map based on the third preset network to obtain the category of each object in the object image, the visible bounding box of each object, and the corresponding mask includes: The first feature map is processed by region-of-interest pooling to obtain the second feature map; A fully connected layer is used to perform object classification and bounding box regression on the second feature map to obtain the category and visible bounding box of each object in the object image; and, The second feature map is used to perform mask prediction using an object mask prediction network to obtain the corresponding mask.

6. The method according to claim 1, characterized in that, Based on the object bounding box detection network model and the final feature map, the geometric transformation parameters between the visible bounding box and the corresponding complete bounding box of each object are regressed and predicted, including: Obtain the pooling and fully connected layers of the object bounding box detection network model; The final feature map is input into a pooling and fully connected layer to predict the center coordinate difference and height-width scaling factor between the visible bounding box and the complete box of each object on the final feature map, and the coordinate data of the complete box is calculated based on the center coordinate difference and height-width scaling factor.

7. A data platform, characterized in that, include: The image input and model loading module is used to input multi-object images into a pre-trained neural network model, which includes an instance segmentation network model and an object bounding box detection network model. The instance segmentation and visual information extraction module is used to perform feature processing operations on multiple feature maps of the object image using the instance segmentation network model to obtain the visual part information of each object. The visual part information includes the object category, the visual bounding box and the corresponding mask. The mask-based feature extraction and final feature map construction module is used to extract image features corresponding to the visible parts of each object from the multiple feature maps based on the visible bounding boxes and corresponding masks of each object, so as to construct a final feature map corresponding to each object and free from background interference. The construction of the final feature map includes: segmenting the feature map within the corresponding box based on the initial feature map of the visible bounding box in the multiple feature maps, and based on the mask corresponding to the visible bounding box, keeping the pixel values ​​of the feature map within the box corresponding to the coordinate point with a true mask value as the initial value, and setting the pixel values ​​of the feature map within the box corresponding to the coordinate point with a false mask value to zero, so as to generate the final feature map. The complete bounding box prediction and geometric transformation module is used to predict the geometric transformation parameters between the visible bounding box and the corresponding complete bounding box of each object based on the final feature map according to the object complete bounding box detection network model; and calculate the coordinate data of the complete bounding box according to the geometric transformation parameters and the coordinate data of the visible bounding box, thereby obtaining the complete position information of each object on the object image; the geometric transformation parameters include the center coordinate difference between the visible bounding box and the complete bounding box and the height and width scaling factor of the complete bounding box relative to the visible bounding box.

8. A computer device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the multi-object detection method based on multi-scale features as described in any one of claims 1-6.

Images