A mask pattern generation method and apparatus
By extracting and fusing feature images in box detection and using an attention enhancement mechanism to remove gaps, a mask image that removes the influence of gaps is generated, which solves the problem of box detection errors and improves detection accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU HIKROBOT TECH CO LTD
- Filing Date
- 2024-07-05
- Publication Date
- 2026-07-03
AI Technical Summary
During the cabinet inspection process, errors occurred due to gaps on the cabinet surface, especially gaps in the cabinet doors, which caused the cabinet to be detected as two separate cabinets.
By acquiring the target image, extracting feature images of multiple sizes, and fusing them, an attention enhancement mechanism is used to remove gaps, generating a box mask image that removes the influence of gaps, thereby improving detection accuracy.
This reduces cabinet detection errors caused by gaps and improves the network's positioning accuracy for cabinets containing gaps.
Smart Images
Figure CN118887667B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of information technology, and in particular to a method and apparatus for generating mask images. Background Technology
[0002] Currently, cabinet detection is widely used in target recognition. During cabinet recognition, a cabinet mask map can also be generated. However, in practical applications, gaps often exist on the surface of the cabinet. For example, a cabinet with two doors may be detected incorrectly due to the gap between the doors, resulting in one cabinet being detected as two. Summary of the Invention
[0003] The purpose of this application is to provide a mask image generation method and apparatus to solve the problem of box detection errors caused by gaps. The specific technical solution is as follows:
[0004] A first aspect of this application provides a mask image generation method, the method comprising:
[0005] Acquire a target image, wherein the target image includes a box, and at least one box has a gap on its surface;
[0006] Feature images are extracted from the target image to obtain feature images of multiple sizes;
[0007] The feature images of the multiple sizes are fused to obtain a first fused feature image and a second fused feature image, wherein the size of the first fused feature image is larger than that of the second fused feature image;
[0008] Based on the first fused feature image and the second fused feature image, gap removal is performed to generate a box mask image of the target image with the gaps removed.
[0009] In one possible implementation, the extraction of feature images from the target image to obtain feature images of multiple sizes includes:
[0010] The target image is filtered for background to obtain a foreground image;
[0011] The foreground image is resized to obtain a foreground image of a preset size;
[0012] Feature images are extracted from the foreground image of the preset size to obtain feature images of the multiple sizes.
[0013] In one possible implementation, the step of performing gap removal based on the first fused feature image and the second fused feature image to generate a box mask image of the target image with the gaps removed includes:
[0014] Based on the first fused feature image, gap removal is performed on the second fused feature image through attention enhancement to obtain the removal result;
[0015] The rejection results are used to identify the boxes to obtain the first box information;
[0016] Based on the information of the first box, a mask image of the box is generated.
[0017] In one possible implementation, the step of performing gap removal on the second fused feature image based on the first fused feature image using attention enhancement to obtain the removal result includes:
[0018] The first fused feature image is downsampled to obtain a third fused feature image of the same size as the second fused feature image;
[0019] The third fused feature image and the second fused feature image are fused together to remove the gap features in the second fused feature image, thus obtaining the removal result.
[0020] In one possible implementation, generating a mask image of the housing based on the first housing information includes:
[0021] Based on the first fused feature image and the second fused feature image, extract the encoded features and the top-level features;
[0022] Based on the first box information, the encoded features, and the top-level features, a mask image prediction is performed to obtain the mask image of the box.
[0023] In one possible implementation, the step of performing gap removal based on the first fused feature image and the second fused feature image to generate a box mask image of the target image with the gaps removed includes:
[0024] The box is identified based on the second fused feature image to obtain the second box information;
[0025] Based on the first fused feature image and the second box information, a box mask image for the target image is generated to remove the effect of gaps.
[0026] In one possible implementation, after fusing the feature images of the plurality of sizes to obtain a first fused feature image and a second fused feature image, the method further includes:
[0027] The first fused feature image is subjected to binary classification semantic segmentation to obtain a slit mask image.
[0028] In one possible implementation, after performing binary semantic segmentation on the first fused feature image to obtain a slit mask image, the method further includes:
[0029] Identify the slot location information based on the slot mask image;
[0030] For any target gap, based on the gap location information, identify two sub-boxes that satisfy a preset relative positional relationship with the target gap;
[0031] The two identified sub-boxes are merged to obtain the merged corrected mask image.
[0032] In one possible implementation, merging the two identified sub-boxes to obtain the merged corrected mask image includes:
[0033] When the target gap and the two sub-boxes intersect, the two sub-boxes are merged to obtain the corrected mask image;
[0034] For any target gap, based on the gap location information, identifying two sub-boxes that satisfy a preset relative positional relationship with the target gap includes:
[0035] For any target gap, based on the gap location information, identify the two boxes located in the middle of the target gap to obtain the two sub-boxes.
[0036] A second aspect of this application provides a mask image generation apparatus, the apparatus comprising:
[0037] An image acquisition module is used to acquire a target image, wherein the target image includes a box, and at least one box has a gap on its surface;
[0038] The feature extraction module is used to extract feature images from the target image to obtain feature images of multiple sizes;
[0039] An image fusion module is used to fuse the feature images of the multiple sizes to obtain a first fused feature image and a second fused feature image, wherein the size of the first fused feature image is larger than that of the second fused feature image;
[0040] The gap removal module is used to remove gaps based on the first fused feature image and the second fused feature image, and generate a box mask image of the target image with the gaps removed.
[0041] In one possible implementation, the feature extraction module is specifically used to perform background filtering on the target image to obtain a foreground image; to resize the foreground image to obtain a foreground image of a preset size; and to extract feature images from the foreground image of the preset size to obtain feature images of the multiple sizes.
[0042] In one possible implementation, the gap removal module includes:
[0043] The result generation submodule is used to perform gap removal on the second fused feature image based on the first fused feature image by using attention enhancement, and obtain the removal result.
[0044] The box identification submodule is used to identify the boxes in the rejection results to obtain the first box information;
[0045] The mask generation submodule is used to generate a mask image of the box based on the first box information.
[0046] In one possible implementation, the result generation submodule is specifically used to downsample the first fused feature image to obtain a third fused feature image of the size corresponding to the second fused feature image; and to perform feature fusion between the third fused feature image and the second fused feature image to remove gap features in the second fused feature image to obtain the removal result.
[0047] In one possible implementation, the mask generation submodule is specifically used to extract encoded features and top-level features based on the first fused feature image and the second fused feature image; and to perform mask prediction based on the first box information, the encoded features and the top-level features to obtain the mask image of the box.
[0048] In one possible implementation, the gap removal module is specifically used to identify the box based on the second fused feature image to obtain second box information; and to generate a box mask image of the target image with gaps removed based on the first fused feature image and the second box information.
[0049] In one possible implementation, the device further includes:
[0050] The binary classification module is used to perform binary semantic segmentation on the first fused feature image to obtain a slit mask image.
[0051] In one possible implementation, the device further includes:
[0052] The position recognition module is used to identify the position information of the slit based on the slit mask image;
[0053] The sub-box identification module is used to identify two sub-boxes that satisfy a preset relative positional relationship with any target gap based on the gap position information.
[0054] The box merging module is used to merge two identified sub-boxes to obtain a merged corrected mask image.
[0055] In one possible implementation, the box merging module is specifically used to merge the two sub-boxes when the target gap and the two sub-boxes intersect, to obtain the corrected mask image;
[0056] The sub-box identification module is specifically used to identify two boxes located in the middle of any target gap based on the gap location information, thereby obtaining the two sub-boxes.
[0057] In another aspect of this application, an electronic device is provided, comprising:
[0058] Memory, used to store computer programs;
[0059] The processor, when executing a program stored in memory, implements any of the mask generation methods described above.
[0060] In another aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements any of the mask generation methods described above.
[0061] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the mask generation methods described above.
[0062] Beneficial effects of the embodiments in this application:
[0063] This application provides a mask generation method and apparatus. The mask generation method includes: acquiring a target image, wherein the target image includes a box, and at least one box has a gap on its surface; extracting feature images from the target image to obtain feature images of multiple sizes; fusing the feature images of multiple sizes to obtain a first fused feature image and a second fused feature image, wherein the size of the first fused feature image is larger than that of the second fused feature image; and removing gaps based on the first fused feature image and the second fused feature image to generate a box mask of the target image with the gaps removed. This solution allows for feature image extraction after acquiring a target image containing a box, and the fusion of the extracted feature images to obtain a first fused feature image and a second fused feature image. The gaps are then removed and a mask is generated using the obtained first fused feature image and the second fused feature image, resulting in a box mask with the gaps removed. This reduces box detection errors caused by gaps and improves the network's positioning accuracy for boxes containing gaps.
[0064] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description
[0065] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.
[0066] Figure 1 A schematic flowchart of a mask image generation method provided in an embodiment of this application;
[0067] Figure 2 This is a schematic flowchart of a feature image extraction method provided in an embodiment of this application;
[0068] Figure 3 A schematic flowchart for generating a box mask image provided in an embodiment of this application;
[0069] Figure 4 A flowchart illustrating an example of a mask image generation method provided in this application embodiment;
[0070] Figure 5 This is another schematic flowchart illustrating the mask image generation method provided in the embodiments of this application;
[0071] Figure 6 This is another schematic flowchart of the mask image generation method provided in the embodiments of this application;
[0072] Figure 7 This is another schematic flowchart of the mask image generation method provided in the embodiments of this application;
[0073] Figure 8 A schematic diagram of the attention module provided in an embodiment of this application.
[0074] Figure 9 A schematic flowchart of a box mask correction method provided in an embodiment of this application;
[0075] Figure 10 Another schematic flowchart of the box mask pattern correction method provided in the embodiments of this application;
[0076] Figure 11 A schematic diagram of the mask image generation apparatus provided in the embodiments of this application;
[0077] Figure 12 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation
[0078] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.
[0079] To address the problem of incorrect detection of enclosures due to gaps on the enclosure surface, this application provides a mask image generation method and apparatus.
[0080] A first aspect of this application provides a method for generating a mask image, see [link to previous section]. Figure 1 , Figure 1 A schematic flowchart of a mask image generation method provided in this application embodiment, the method including:
[0081] Step S11: Obtain the target image;
[0082] Step S12: Extract feature images from the target image to obtain feature images of multiple sizes;
[0083] Step S13: Fuse feature images of multiple sizes to obtain a first fused feature image and a second fused feature image, wherein the size of the first fused feature image is larger than that of the second fused feature image;
[0084] Step S14: Perform gap removal based on the first fused feature image and the second fused feature image to generate a box mask image of the target image with gaps removed.
[0085] Corresponding to step S11 above, the target image referred to in this embodiment includes a box, and at least one box has a gap on its surface. Specifically, the box can refer to a cabinet, a box, etc. The gap on the surface of the box can refer to the gap between two cabinet doors, or cracks on the surface of the box, etc.
[0086] Corresponding to step S12 above, feature images are extracted from the target image to obtain feature images of multiple sizes. Feature image extraction can be performed using a pre-created network module. In one example, this pre-created network module can be a Backbone module. By inputting the target image into the Backbone module for feature image extraction, feature maps of different scales can be obtained. The Backbone module is mainly responsible for receiving input data, performing data preprocessing and feature extraction to obtain the corresponding feature maps of the input image, and then passing them to the next layer. In this embodiment, when the Backbone module is used for feature image extraction, multiple feature images can be extracted, such as a first feature image, a second feature image, a third feature image, and a fourth feature image, which can be represented as res2, res3, res4, and res5, respectively. Specifically, the Backbone module can be a ResNet50 (a network structure). res2, res3, res4, and res5 can be extracted by different network layers, specifically by the first layer, the second layer, the third layer, and the fourth layer, respectively.
[0087] Corresponding to step S13 above, when fusing feature images of multiple sizes to obtain a first fused feature image and a second fused feature image, the first fused feature image can include more detailed features because its size is larger than the second fused feature image. Because the first fused feature image is larger than the second fused feature image, it can include more gap features, while the second fused feature image may include fewer gap features or none at all. Specifically, when fusing feature images of multiple sizes, feature images at each scale can be input into an FPN (Feature Map Pyramid Network) for fusion to obtain a first fused feature image and a second fused feature image. In this embodiment, when the target image includes gaps, the first fused feature image can target the gap detection task and has the ability to identify gap features. The second feature image is responsible for the detection and segmentation of the box, and this second feature image is easily affected by gaps, leading to false detections. Therefore, according to the solution of this application embodiment, when the surface of the box in the target image contains gaps, the first fused feature image and the second fused feature map can be fused by an attention mechanism, so that the second feature map has the ability to eliminate gaps while detecting the box, thereby improving the accuracy of detection.
[0088] Corresponding to step S14 above, as mentioned above, the size of the first fused feature image is larger than that of the second fused feature image. The first fused feature image includes more gap features. Therefore, gap removal can be performed based on the first and second fused feature images. Specifically, in one case, the second fused feature image can be directly used to identify the box by referring to the first fused feature image. Specifically, the first feature image responsible for locating the box gaps is fused to the second feature image responsible for box localization. Through attention mechanism learning, the weight of non-gap features is increased, so that the network has the function of gap perception and gap removal while locating the box surface, resulting in a box mask image of the target image with gaps removed. In another case, after identifying the box using the second fused feature image, the box instance segmentation result is sent to the post-processing module. In the post-processing module, the box surface gaps predicted by the first fused feature image are used to post-process the instance segmentation result to eliminate the influence of gaps, resulting in a corrected result, and ultimately improving the network's localization accuracy for boxes containing gaps.
[0089] As can be seen, the mask generation method of this application can extract feature images after obtaining the target image containing the box, and obtain a first fused feature image and a second fused feature image by fusing the extracted feature images. Then, the first fused feature image and the second fused feature image are used to remove gaps and generate a mask image, thereby obtaining a box mask image with the gaps removed, which improves the positioning accuracy of the network for boxes containing gaps.
[0090] In one possible implementation, see Figure 2 , Figure 2 This is a schematic flowchart of a feature image extraction method provided in an embodiment of this application. Step S12 extracts feature images from the target image to obtain feature images of multiple sizes, including:
[0091] Step S121: Filter the background of the target image to obtain the foreground image;
[0092] Step S122: Adjust the size of the foreground image to obtain a foreground image of a preset size;
[0093] Step S123: Extract feature images from the foreground image of a preset size to obtain feature images of multiple sizes.
[0094] In this embodiment, the target image can be an image captured by a camera or other capture device. When filtering the background of the target image, background interference such as pallets can be filtered out, retaining only the foreground image. When resizing the foreground image, preprocessing operations such as resizing can be used to adjust the size to obtain a foreground image of a preset size. Specifically, this preset size can be set according to actual conditions; in one example, it can be 640*640, 320*320, etc. Feature image extraction from the preset-sized foreground image can be performed using a pre-created network module, such as the Backbone module. The Backbone module used in this embodiment can extract feature images to obtain res2, res3, res4, and res5. res2, res3, res4, and res5 can be feature images of different sizes, specifically extracted by different layers, possessing different sizes and receptive fields, thus fully covering information at different levels.
[0095] In one possible implementation, see Figure 3 , Figure 3 This is a schematic flowchart of generating a box mask image provided in an embodiment of this application. Step S14 involves gap removal based on the first fused feature image and the second fused feature image to generate a box mask image of the target image with the gaps removed, including:
[0096] Step S141: Based on the first fused feature image, perform gap removal on the second fused feature image through attention enhancement to obtain the removal result;
[0097] Step S142: Identify the boxes in the rejection results to obtain the information of the first box;
[0098] Step S143: Generate a mask image of the box based on the information of the first box.
[0099] In one possible implementation, based on the first fused feature image, gap removal is performed on the second fused feature image using attention enhancement to obtain the removal result. This includes: downsampling the first fused feature image to obtain a third fused feature image of the same size as the second fused feature image; fusing the third fused feature image and the second fused feature image to remove gap features from the second fused feature image, thus obtaining the removal result. Specifically, the first fused feature image can be downsampled, passed through a softmax layer (normalization layer), and the resulting score_map (detection result) can be weighted and multiplied by the features of the second fused feature image to achieve feature fusion. See also Figure 4 After the target image is processed by the backbone network, four feature images are obtained: a first feature image, a second feature image, a third feature image, and a fourth feature image. These four feature images are then fused to obtain a first fused feature image and a second fused feature image. The first fused feature image is then subjected to binary classification to obtain a slit mask image. The first fused feature image can also be downsampled and normalized to obtain the detection result, which is then used... The second fused feature image is used for calculation, and then processed by the detection head module and the feature extraction module respectively. The detection head module obtains box information and attention features, which are encoded by the feature extraction module and then input into the segmentation module to obtain a box mask image. This figure is a schematic diagram of the entire network architecture. The dashed box represents the box gap attention module. Through the attention mechanism of this module, the second fused feature image can have gap perception capabilities, so as to eliminate gaps on the box surface based on the perceived gap features. This allows the network to correctly learn the complete box's location information in subsequent target detection tasks, and is less susceptible to false detections due to gaps on the box surface. The Backbone module can extract feature images: res2, res3, res4, and res5. The extracted feature images are fused using an FPN (Feature Map Pyramid Network) to obtain p2 and p3. p2 corresponds to the first fused feature image in this embodiment, and p3 corresponds to the second fused feature image in this embodiment. The fused image can be downsampled to obtain box information and attention features. See [link to relevant documentation]. Figure 5The input image undergoes image preprocessing and is then fused through a backbone network to obtain a first fused feature image and a second fused feature image. The first fused feature image is used by the semantic segmentation module to output a gap segmentation result. The second fused feature image and the first fused feature image are input together into an attention module, then through an object detection module to output an object detection result. The output of the object detection module can be further processed by a feature extraction module and then by an instance segmentation module to output an instance segmentation result. After processing by the attention module, further processing can be performed by the object detection module and the instance segmentation module to finally output the instance segmentation result. During the instance segmentation process, feature extraction can be performed by a tower module (feature extraction module) to obtain encoded features. After object detection by the object detection module, the object detection result can also be output to generate a mask map. Figure 4 In the process, after processing by the attention module corresponding to the dashed box area, encoded features, as well as box features such as Boxes, Classes, and Attention, can be obtained. During instance segmentation, segmentation can be performed using the ins-seg (instance segmentation module), ultimately generating a box mask image.
[0100] In one possible implementation, gap removal is performed based on the first fused feature image and the second fused feature image to generate a box mask map of the target image with the gaps removed. This includes: identifying boxes based on the second fused feature image to obtain second box information; and generating a box mask map of the target image with the gaps removed based on the first fused feature image and the second box information. When identifying boxes based on the second fused feature image to obtain the second box information, the second fused feature image can be input into a target detection module to obtain the detection bounding box of the box and its corresponding confidence level, etc. See also... Figure 6The input image, after image preprocessing, is input into the backbone network and then fused to obtain a first fused feature image and a second fused feature image. The first fused feature image is processed by the semantic segmentation module to output the gap segmentation result, and then input into the post-processing module. The second fused feature image is processed by the feature extraction module and then input into the instance segmentation module, and the second fused feature image is also input into the instance segmentation module after the target detection module. The input result of the instance segmentation module is processed by the post-processing module to output the instance segmentation result. After feature image fusion by the FPN module, the first fused feature image (p2) and the second fused feature image (p3) are obtained. For the first fused feature image, semantic segmentation can be performed by the semantic segmentation module to obtain and output the gap segmentation result. For the second fused feature image, it can be processed by the target detection module and the instance segmentation module to obtain the second box information. Finally, post-processing is performed based on the second box information and the gap stitching result to generate a box mask map of the target image with the gap removal effect. In this embodiment of the invention, the gap elimination process can be as follows: Figure 6 and Figure 7 There are two process methods. Figure 7 In the process, the input image, after image preprocessing, is input into the backbone network and then fused to obtain the first fused feature image, which is then output as a gap segmentation result by the semantic segmentation module. Similarly, the input image, after image preprocessing, is input into the backbone network and then fused to obtain the second fused feature image. These images are then processed by the feature extraction module and input into the instance segmentation module, and by the object detection module and input into the instance segmentation module, respectively, before being output as instance segmentation results by the post-processing module. The box gap segmentation result is obtained by a separate network. Figure 6 The process is in Figure 7 Based on the existing workflow, instead of training a separate network to learn the process of perceiving gaps on the box surface, a shared network feature extractor is used. This feature extractor outputs gap segmentation and box localization results, which are then processed by a post-processing module to obtain and output the final, correct instance segmentation result. See also... Figure 8This semantic segmentation module can consist of multiple blocks (network modules), specifically, it can include: Network Module 1, Network Module 2, Network Module 3, Network Module 4, and Network Module 5. The input features are processed through these five network modules to obtain the final input features. The calculation results from each of the five network modules can be output as convolutional layers, normalization layers, and activation function layers. Each block can consist of three layers: a convolutional layer, a normalization layer, and a ReLU layer. In one example, when segmenting the seams on the surface of a box, the scale of the first fused feature image can be [batch, 192, 160, 160]. After passing through the semantic segmentation module, the output feature dimension can be [batch, num_class, 160, 160], where num_class can be 2. During the segmentation process, the seams on the box surface can be set as the foreground, and the rest as the background. Specifically, a binary classification semantic segmentation task can be performed based on the cross-entropy loss function.
[0101] In one possible implementation, after fusing feature images of multiple sizes to obtain a first fused feature image and a second fused feature image, the method further includes: performing binary semantic segmentation on the first fused feature image to obtain a slit mask map. See [example example]. Figure 4 After generating the first fused feature image (p2), binary classification can be performed using seg-head (a semantic segmentation network) to obtain the slit mask image.
[0102] In one possible implementation, generating a mask image of the box based on the first box information includes: extracting encoded features and top-level features based on a first fused feature image and a second fused feature image; and predicting the mask image based on the first box information, encoded features, and top-level features to obtain the box mask image. Specifically, the first box information, encoded features, and top-level features can be input into the instance segmentation module to obtain corresponding feature maps. Then, the feature maps are scaled to a uniform size before predicting the mask image to obtain the box mask image. In one example, the input to the instance segmentation module can be the bottom-level encoded features, the top-level attentions (interest features), and box information. Specifically, the corresponding feature map can be obtained from the ROI (region of interest) provided by the box, based on the encoded features of the fused attentions. Finally, the feature map is scaled to a uniform size before predicting the instance segmentation mask.
[0103] In one possible implementation, after performing binary semantic segmentation on the first fused feature image to obtain the slit mask map, see [link to relevant documentation]. Figure 9 The above methods also include:
[0104] Step S91: Identify the gap location information based on the gap mask image;
[0105] Step S92: For any target gap, based on the gap location information, identify two sub-boxes that satisfy a preset relative positional relationship with the target gap;
[0106] Step S93: Merge the two identified sub-boxes to obtain the merged corrected mask image.
[0107] In one possible implementation, the two identified sub-boxes are merged to obtain a merged corrected mask image, including: when the target gap and the two sub-boxes intersect, the two sub-boxes are merged to obtain a corrected mask image; for any target gap, based on the gap position information, two sub-boxes that satisfy a preset relative position relationship with the target gap are identified, including: for any target gap, based on the gap position information, two boxes located in the middle position of the target gap are identified to obtain two sub-boxes.
[0108] For a specific example, see Figure 10 The steps for generating the corrected mask image may include:
[0109] The first step is to obtain a binary mask image of the gaps on the surface of the box, and then obtain individual gaps by traversing connected components.
[0110] The second step is to obtain the center point (cx, cy) of the gap region based on the mask image of a single gap, where cx and cy represent the x-coordinate and y-coordinate of the center point, respectively.
[0111] The third step involves finding the two nearest boxes (boxes) to the obtained gap center point. These boxes represent the instance segmentation results of the boxes, and the minimum bounding rectangle is calculated for each instance mask image. First, it's determined whether the gap is in the middle of a single box. If it is, the instance prediction is correct, and no post-processing is needed; the next gap is then obtained for processing. If the gap is not in the middle of a single box, proceed to the fourth step for further processing.
[0112] The fourth step is to determine whether the gap is between two boxes. If not, the instance mask merging condition is not met, and no post-processing is required. Continue to obtain the next gap for processing. If the gap is between two boxes, proceed to the fifth step for further processing.
[0113] The fifth step is to determine whether there is an intersection between the gap and the masks of the two boxes. If not, the instance mask merging condition is not met, and no post-processing is required. Continue to obtain the next gap for processing. If there is an intersection, the merging condition is met, the three masks are merged, the prediction result is updated, the gap mask and the two instance segmentation masks are merged, and the merged mask is updated in the instance segmentation prediction result, thereby removing false detection boxes caused by the influence of the gap on the box surface.
[0114] The proposed solution can simultaneously perform object detection and instance segmentation tasks on the box and build a network for locating gaps on the box surface. This allows the network to segment gaps on the box surface in real time, and then take the corresponding features as a scoremap, which is fused with the features used for object detection. Through this attention mechanism, the network can achieve gap perception and gap elimination while locating the box surface, ultimately improving the network's accuracy in locating boxes containing gaps.
[0115] A second aspect of this application provides a mask image generation apparatus, see [link to previous section]. Figure 11 , Figure 11 This is a schematic diagram of a mask image generation apparatus provided in an embodiment of this application. The apparatus includes:
[0116] Image acquisition module 1101 is used to acquire a target image, wherein the target image includes a box, and at least one box has a gap on its surface;
[0117] The feature extraction module 1102 is used to extract feature images from the target image to obtain feature images of multiple sizes;
[0118] Image fusion module 1103 is used to fuse feature images of multiple sizes to obtain a first fused feature image and a second fused feature image, wherein the size of the first fused feature image is larger than that of the second fused feature image;
[0119] The gap removal module 1104 is used to remove gaps based on the first fused feature image and the second fused feature image, and generate a box mask image of the target image with the gaps removed.
[0120] In one possible implementation, the feature extraction module is specifically used to perform background filtering on the target image to obtain a foreground image; to resize the foreground image to obtain a foreground image of a preset size; and to extract feature images from the foreground image of the preset size to obtain feature images of multiple sizes.
[0121] In one possible implementation, the gap removal module includes:
[0122] The result generation submodule is used to perform gap removal on the second fusion feature image based on the first fusion feature image through attention enhancement, and obtain the removal result.
[0123] The box identification submodule is used to identify the boxes in the rejection results and obtain the first box information;
[0124] The mask generation submodule is used to generate a mask image of the box based on the information of the first box.
[0125] In one possible implementation, the result generation submodule is specifically used to downsample the first fused feature image to obtain a third fused feature image of the size corresponding to the second fused feature image; and to perform feature fusion between the third fused feature image and the second fused feature image to remove gap features in the second fused feature image to obtain the removal result.
[0126] In one possible implementation, the mask generation submodule is specifically used to extract encoded features and top-level features based on the first fused feature image and the second fused feature image; and to perform mask prediction based on the first box information, encoded features and top-level features to obtain the mask image of the box.
[0127] In one possible implementation, the gap removal module is specifically used to identify the box based on the second fused feature image to obtain the second box information; and to generate a box mask image of the target image with gaps removed based on the first fused feature image and the second box information.
[0128] In one possible implementation, the device further includes:
[0129] The binary classification module is used to perform binary semantic segmentation on the first fused feature image to obtain a slit mask image.
[0130] In one possible implementation, the device further includes:
[0131] The location recognition module is used to identify the location information of the gap based on the gap mask image;
[0132] The sub-box identification module is used to identify two sub-boxes that satisfy a preset relative positional relationship with any target gap based on the gap location information.
[0133] The box merging module is used to merge two identified sub-boxes to obtain a merged corrected mask image.
[0134] In one possible implementation, the box merging module is specifically used to merge the two sub-boxes when the target gap and the two sub-boxes intersect, to obtain a corrected mask image.
[0135] The sub-box identification module is specifically used to identify two boxes located in the middle of any target gap based on the gap location information, thus obtaining two sub-boxes.
[0136] As can be seen, the mask generation device of this application can extract feature images after acquiring a target image containing the box, and obtain a first fused feature image and a second fused feature image by fusing the extracted feature images. Then, gap removal and mask generation can be performed using the obtained first fused feature image and second fused feature image to obtain a box mask image with gaps removed.
[0137] This application also provides an electronic device, such as... Figure 12 As shown, it includes:
[0138] Memory 1201 is used to store computer programs;
[0139] When processor 1202 executes the program stored in memory 1201, it performs the following steps:
[0140] Acquire a target image, wherein the target image includes a box, and at least one box has a gap on its surface;
[0141] Feature images are extracted from the target image to obtain feature images of multiple sizes;
[0142] Multiple feature images of different sizes are fused to obtain a first fused feature image and a second fused feature image, wherein the size of the first fused feature image is larger than that of the second fused feature image;
[0143] Based on the first fused feature image and the second fused feature image, gap removal is performed to generate a box mask image of the target image that removes the influence of gaps.
[0144] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 1202, the communication interface, and the memory 1201 communicating with each other via the communication bus.
[0145] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.
[0146] The communication interface is used for communication between the aforementioned electronic devices and other devices.
[0147] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.
[0148] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0149] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the mask generation methods described above.
[0150] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the mask generation methods described above.
[0151] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.
[0152] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0153] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, storage media, and computer program products are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0154] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.
Claims
1. A method of mask pattern generation, characterized by, The method includes: Acquire a target image, wherein the target image includes a box, and at least one box has a gap on its surface; Feature images are extracted from the target image to obtain feature images of multiple sizes; The feature images of the multiple sizes are fused to obtain a first fused feature image and a second fused feature image, wherein the size of the first fused feature image is larger than that of the second fused feature image; Based on the first fused feature image and the second fused feature image, gap removal is performed to generate a box mask image of the target image to remove the influence of gaps, including: The first fused feature image is downsampled to obtain a third fused feature image of the same size as the second fused feature image; The third fused feature image and the second fused feature image are fused together to remove the gap features in the second fused feature image, resulting in the removal result; The rejection results are used to identify the boxes to obtain the first box information; Based on the first fused feature image and the second fused feature image, extract the bottom-level coding features and the top-level interest features; Based on the first box information, the coding features of the bottom layer, and the interest features of the top layer, a mask image prediction is performed to obtain the mask image of the box.
2. The method of claim 1, wherein, The step of extracting feature images from the target image to obtain feature images of multiple sizes includes: The target image is filtered for background to obtain a foreground image; The foreground image is resized to obtain a foreground image of a preset size; Feature images are extracted from the foreground image of the preset size to obtain feature images of the multiple sizes.
3. The method according to claim 1, characterized in that, The step of performing gap removal based on the first fused feature image and the second fused feature image to generate a box mask image of the target image with the gaps removed includes: The box is identified based on the second fused feature image to obtain the second box information; Based on the first fused feature image and the second box information, a box mask image for the target image is generated to remove the effect of gaps.
4. The method of claim 1, wherein, After fusing the feature images of the multiple sizes to obtain a first fused feature image and a second fused feature image, the method further includes: The first fused feature image is subjected to binary classification semantic segmentation to obtain a slit mask image.
5. The method of claim 4, wherein, After performing binary classification semantic segmentation on the first fused feature image to obtain a slit mask image, the method further includes: Identify the slot location information based on the slot mask image; For any target gap, based on the gap location information, identify two sub-boxes that satisfy a preset relative positional relationship with the target gap; The two identified sub-boxes are merged to obtain the merged corrected mask image.
6. The method of claim 5, wherein, For any target gap, based on the gap location information, identifying two sub-boxes that satisfy a preset relative positional relationship with the target gap includes: For any target gap, based on the gap location information, identify the two boxes located in the middle of the target gap to obtain the two sub-boxes; And / or, the merging of the two identified sub-boxes to obtain the merged corrected mask image includes: When the target gap and the two sub-boxes intersect, the two sub-boxes are merged to obtain the corrected mask image.
7. A mask pattern generation apparatus, characterized by comprising: The device includes: An image acquisition module is used to acquire a target image, wherein the target image includes a box, and at least one box has a gap on its surface; The feature extraction module is used to extract feature images from the target image to obtain feature images of multiple sizes; An image fusion module is used to fuse the feature images of the multiple sizes to obtain a first fused feature image and a second fused feature image, wherein the size of the first fused feature image is larger than that of the second fused feature image; The gap removal module is used to remove gaps based on the first fused feature image and the second fused feature image, and generate a box mask image of the target image with the gaps removed. The gap removal module is specifically used to downsample the first fused feature image to obtain a third fused feature image of the same size as the second fused feature image; perform feature fusion between the third fused feature image and the second fused feature image to remove gap features from the second fused feature image, obtaining a removal result; identify boxes based on the removal result to obtain first box information; extract the bottom layer coding features and the top layer interest features based on the first fused feature image and the second fused feature image; and perform mask image prediction based on the first box information, the bottom layer coding features, and the top layer interest features to obtain the mask of the box.