Image region extraction and segmentation method based on deep learning and visual algorithm
By combining deep learning and traditional computer vision algorithms with image area extraction and segmentation methods, the image segmentation problem in complex backgrounds and multi-screen scenes is solved, and higher segmentation accuracy and stability are achieved.
Patent Information
- Application Number
- CN202411778248.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-05
- Publication Date
- 2025-03-21
AI Technical Summary
The prior art has the unsatisfactory image segmentation effect under complex backgrounds, molar interference, multi-screen scenes and angle changes, and the deep learning method is inaccurate in segmentation results when there is insufficient data, making it difficult to identify the main screen in multiple screens.
Image area extraction and segmentation methods based on deep learning and vision algorithms are adopted, combined with deep learning models (such as DocSegNet) and traditional computer vision algorithms, and image segmentation is realized through data preprocessing, edge detection, encoder and decoder modules.
Improves the accuracy and stability of image segmentation, can accurately locate and eliminate molar interference, accurately identify the main screen in multiple screens, and enhances robustness and adaptability.
Smart Images
Figure CN119672055A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing and analysis, and more specifically, to an image region extraction and segmentation method based on deep learning and visual algorithms. Background Art
[0002] At present, traditional image segmentation methods, such as algorithms based on edge detection, threshold segmentation, and color space transformation, can detect and segment paper and display screens in simple backgrounds, but these methods are usually difficult to cope with complex situations in actual scenes taken by users. For example, in complex backgrounds, traditional algorithms may not be able to distinguish between paper or display screens and their surroundings, resulting in inaccurate segmentation. In addition, there may be moiré interference (i.e., stripes caused by the interaction between the camera sensor and the display pixel array) in the display screen images taken by users, and traditional methods are not very effective in eliminating these interferences.
[0003] On the other hand, existing deep learning-based segmentation methods perform well when processing complex image scenes, especially when supported by a large amount of labeled data, and can achieve high segmentation accuracy. However, these methods still have some limitations in practical applications. For example, when a user takes an image containing multiple display screens, it may be difficult for a deep learning model to accurately identify the main screen, especially when the contents displayed on each screen are similar or there are occlusions between the screens. In addition, existing methods often handle images taken from multiple angles poorly, and the segmentation accuracy drops significantly when the screen is tilted or deformed.
[0004] Therefore, the main problems existing in the prior art include: 1) For images of paper or display screens taken by users, the existing image segmentation methods do not provide ideal segmentation results in complex backgrounds, moiré interference, multi-screen scenes, and angle changes; 2) When processing complex images, traditional methods are easily affected by background noise and display moiré, making it difficult to accurately extract the target area; 3) Existing deep learning methods are prone to inaccurate segmentation results when the data set is insufficient, and it is difficult to identify and extract the main screen when there are multiple screens.
[0005] To this end, this solution proposes an image region extraction and segmentation method based on deep learning and visual algorithms to address the above-mentioned existing defects. Summary of the invention
[0006] 1. Technical problems to be solved: In view of the problems existing in the prior art, the purpose of the present invention is to provide an image region extraction and segmentation method based on deep learning and visual algorithms. By combining deep learning with traditional computer vision algorithms, the existing defects are solved, the accuracy and stability of image segmentation under complex backgrounds are improved, and better results are achieved in multi-screen detection and moiré interference elimination.
[0007] 2. Technical solution: To solve the above problems, the present invention adopts the following technical solutions.
[0008] A method for extracting and segmenting image regions based on deep learning and visual algorithms, comprising the following steps: SA1. Data collection and data preprocessing: Collect information such as paper, computer screens, and mobile phone screenshots taken by users and manually exclude pictures that are not in the above three categories as raw data; SA2. Build a dataset: Manually mark the polygon foreground edges, process and fill them into masks as mask targets, and combine the masks and images to construct training sets, test sets, and validation sets. SA3. Building a deep learning model: Construct a deep learning network DocSegNet consisting of an edge extraction network and a deep learning network, and initialize its network parameters; SA4. Training DocSegNet: Perform supervised training on the entire DocSegNet, take all test sets and training sets, perform cross-validation training, select the network model with the smallest average loss, and save the model.
[0009] A further improvement is that the input of the model in the SA3 step is H*W*3, and the output is H*W*1.
[0010] A further improvement is that the network layer hierarchy design of the model includes the following steps: SB1, edge detection: Process the input image x through the edge module (sobelnet) and extract the edge information, denoted as hx; SB2, edge information encoding: input the edge feature fusion image hx into the convolution for encoding; SB3, encoder feature decoding: progressively upsample through the decoder module and fuse the corresponding encoder features using skip connections; SB4, output side features: The side output layers (side1 to side6) are used to generate feature maps at different scales, denoted as d1 to d6, and are uniformly upsampled to the scale of d1; SB5, final output: concatenate the side features and generate the final output d0 through outconv.
[0011] A further improvement is that the edge information encoding comprises the following steps: Stage 1: Input the edge feature fusion image hx into the first convolution layer to generate feature representation hx1, and then downsample through the pool12 maximum pooling layer; Stage2: hx1 is further passed into the second convolution layer to generate feature representation hx2, and then downsampled through the pool23 maximum pooling layer; Stage3: Similarly, generate feature hx3 and downsample through pool34; Stage4: Generate feature hx4 and further downsample through pool45; stage5: generate feature hx5 and downsample through pool56; Stage6: Finally, hx5 is further convolved to generate feature representation hx6, which is upsampled to the scale of hx5, denoted as hx6up.
[0012] A further improvement is that the encoder feature decoding comprises the following steps: stage5d: fuses hx6up with hx5 to generate feature representation hx5d, and upsamples to the scale of hx4; stage4d: fuses the upsampled hx5d with hx4 to generate hx4d, and upsamples to the scale of hx3; stage3d: repeat the process to generate hx3d and upsample to the scale of hx2; stage2d: generates hx2d and upsamples to the scale of hx1; stage1d: generates the final decoder output hx1d.
[0013] A further improvement is that the loss function calculation formula of the sample is: ; in: N is the number of samples, d i (j) represents the predicted value of the current scale model, labels v (j) Represents the actual annotation mask data label.
[0014] A further improvement is that the summation formula of the network layer hierarchy design is: ; That is, the final loss is: .
[0015] A further improvement is that the segmentation evaluation formula of the graphic area is: ; Among them: output is the mask image predicted by the model, target is the manually annotated labels, and smooth is set to 10 by default to prevent calculation problems. ^ -5.
[0016] 3. Beneficial effects: Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects: (1) Improved image segmentation accuracy: By combining deep learning (U2Netp model) with traditional computer vision algorithms, this method can more accurately extract the paper and display areas, especially in scenes with complex backgrounds and multi-angle shooting, significantly improving the segmentation accuracy.
[0017] (2) Solving the problems of moiré and multi-screen detection: Traditional algorithms find it difficult to effectively eliminate moiré interference or identify the main screen among multiple screens. The present invention combines deep learning prediction with region growing to accurately locate and eliminate moiré interference, while accurately identifying the main screen in a multi-screen scenario.
[0018] (3) Enhanced robustness and adaptability: The present invention effectively removes the noise caused by insufficient training data through morphological operations and region growing techniques, so that a relatively stable segmentation effect can be obtained under the condition of limited data. In addition, the use of convex hull fitting and maximum closure calculation makes the technology more adaptable to screen deformation and image angle changes.
[0019] (4) Multiple verifications to improve accuracy: By calculating the IOU (intersection over union) of the deep learning segmentation results and the traditional algorithm results, the foreground area predicted by deep learning is intelligently selected when the segmentation results do not match, further improving the reliability and accuracy of the final output results.
[0020] It should be noted that the structures not introduced in the present invention are the same as the prior art or can be implemented by using the prior art, and are not described in detail here because they do not involve the design points and improvement directions of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 It is the input original graph of the deep model; Figure 2 is the output prediction map of the deep model; Figure 3 It is a combination Figure 2 The effect of perspective transformation; Figure 4 It is a schematic diagram of the overall process of the present invention; Figure 5 It is the input original computer shooting situation diagram of the whole process; Figure 6 This is the final transformation effect diagram of the entire process. DETAILED DESCRIPTION
[0022] To facilitate understanding of the present invention, the present invention will be described more comprehensively below with reference to the relevant drawings. Several embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Example 1
[0023] The technical solution adopted by the present invention to solve its technical problems is: by combining deep learning with traditional computer vision algorithms, these problems are solved, the accuracy and stability of image segmentation under complex backgrounds are improved, and better results are achieved in multi-screen detection and moiré interference elimination. The following are the specific steps: Data collection and data preprocessing: Collect information such as paper, computer screens, and mobile phone screenshots taken by users and manually exclude pictures that are not in the above three categories as raw data; Build the dataset: Manually mark the polygon foreground edge, and then process and fill it into a mask as the mask target. Combine the mask and the image to build the training set, test set, and validation set; Building a deep learning model: Construct a deep learning network DocSegNet consisting of an edge extraction network and a deep learning network, and initialize its network parameters; Training DocSegNet: Perform supervised training on the entire DocSegNet, take all test sets and training sets, perform cross-validation training, select the network model with the smallest average loss, and save the model.
[0024] In summary: Dataset construction: collect user data in real user scenarios, pre-process and put it into the sample library; Obtain the extracted foreground results through manual annotation; The input of the model is H*W*3 and the output is H*W*1. Example 2
[0025] The hierarchical structure of the network layer design is as follows: 1. Edge detection: Process the input image x through the edge module (sobelnet) and extract the edge information, denoted as hx.
[0026] 2. Encoder part: Stage 1: Input the edge feature fusion image hx into the first convolution layer to generate feature representation hx1, and then downsample through the pool12 maximum pooling layer; Stage2: hx1 is further passed into the second convolution layer to generate feature representation hx2, and then downsampled through the pool23 maximum pooling layer; Stage3: Similarly, generate feature hx3 and downsample through pool34; Stage4: Generate feature hx4 and further downsample through pool45; stage5: generate feature hx5 and downsample through pool56; Stage6: Finally, hx5 is further convolved to generate feature representation hx6, which is upsampled to the scale of hx5, denoted as hx6up.
[0027] 3. Decoder part: The decoder module is gradually upsampled and the corresponding encoder features are fused using skip connections.
[0028] stage5d: fuses hx6up with hx5 to generate feature representation hx5d, and upsamples to the scale of hx4; stage4d: fuses the upsampled hx5d with hx4 to generate hx4d, and upsamples to the scale of hx3; stage3d: repeat the process to generate hx3d and upsample to the scale of hx2; stage2d: generates hx2d and upsamples to the scale of hx1; stage1d: generates the final decoder output hx1d.
[0029] 4. Output side features: The side output layers (side1 to side6) are used to generate feature maps at different scales, denoted as d1 to d6 respectively, and are uniformly upsampled to the scale of d1.
[0030] 5. Final output: The side features are concatenated and the final output d0 is generated through outconv. Example 3
[0031] Use the following loss function calculation formula to calculate the loss of this group of samples: ; in: N is the number of samples, d i (j) represents the predicted value of the current scale model, labels v (j) Represents the actual annotation mask data label. Example 4
[0032] Since this model is designed with multiple layers (d0 to d6), it is necessary to sum each layer: ; That is, the final loss is: . Example 5
[0033] The segmentation evaluation formula of the graphic area is: ; Among them: output is the mask image predicted by the model, target is the manually annotated labels, and smooth is set to 10 by default to prevent calculation problems. ^ -5. Example 6
[0034] The present invention adopts data enhancement technology in the training process to improve the generalization of the model and more closely restore the document complexity in the real scene, including but not limited to random rotation, random cropping, random flipping, random zooming in and out, random RGB color conversion, etc., as well as random pixel discarding, simulated denoising, random perspective transformation, etc. Example 7
[0035] In the model deployment and prediction process of the present invention, the input is a 320×320 RGB three-channel image. Figure 1 As shown in , the model outputs a single-channel 0-1 binary mask image of the same size. The mask image is enlarged to the same resolution as the original image, and a pixel-by-pixel AND operation is performed with the original image to extract the foreground area, as shown in Figure 2 In order to reduce the underfitting caused by insufficient data, the image is post-processed through region growing technology and morphology. At the same time, for a more intuitive visualization effect, the vertices of the four largest edges of the foreground area are further extracted, and the final effect is generated through perspective transformation as shown in the figure below. Figure 3 shown. Example 8
[0036] The specific input for the hierarchical structure of the network layer design in Example 2 is: 1. Stage 1: RSU7 module •Input dimension: 3 channels, output 64 channels, channel adjustment through RSU7 layer.
[0037] •Main parameters of the internal convolutional layer: •Input channels: 16, output channels: 64, convolution kernel size: 3\times3.
[0038] •The activation function is ReLU.
[0039] 2. Pool12: Maximum pooling layer •Internal parameters: kernel size 2\times2, step size 2, ceil_mode=True.
[0040] •Function: Downsampling reduces the resolution of the feature map by half.
[0041] 3. Stage 2: RSU6 module •The input channels are 64, the output channels are 64, and the dimension reduction channels are 16.
[0042] •The internal main convolution parameters are similar to RSU7, and the activation function is ReLU.
[0043] 4.pool23: Maximum pooling layer •Parameters: kernel size 2\times2, step size 2, ceil_mode=True.
[0044] 5.stage3: RSU5 module •The input channels are 64, the output channels are 64, and the dimension reduction channels are 16.
[0045] 6.pool34: Maximum pooling layer •Parameters are the same as above.
[0046] 7.stage4: RSU4 module •The input channels are 64, the output channels are 64, and the dimension reduction channels are 16.
[0047] 8.pool45: Maximum pooling layer •Parameters are the same as above.
[0048] 9.stage5: RSU4F module •The input channels are 64, the output channels are 64, and the dimension reduction channels are 16.
[0049] 10.pool56: Maximum pooling layer •Parameters are the same as above.
[0050] 11.stage6: RSU4F module (no further pooling) •The input channels are 64, the output channels are 64, and the dimension reduction channels are 16.
[0051] Decoding stage The decoding part is symmetrical with the encoding part, using skip connections and upsampling operations.
[0052] 12.stage5d: RSU4F decoding module • 128 input channels (including connections from HX6UP and HX5), 64 output channels.
[0053] •The convolution kernel and step size parameters are consistent with the encoding part.
[0054] 13.hx5dup: upsampling •HX5D is upsampled to HX4 size to complete the high-resolution information.
[0055] 14. stage4d, stage3d, stage2d, and stage1d •These layers are all layer-by-layer decoding RSU modules, each of which has upsampling connections and skip connections corresponding to the encoding stage.
[0056] • Channels are gradually restored and information is integrated through convolution kernels.
[0057] The RSU modules are all blocks constructed by different numbers of convolutions, BatchNorm and activation functions. The convolution kernel size is 3, the step size is 1*dirate, and dilation. Then the layers are normalized and finally the activation layer is activated with Relu as the activation function.
[0058] The above-described embodiments only express a certain implementation mode of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the patent scope of the present invention. It should be pointed out that, for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.
Claims
1. A method for image region extraction and segmentation based on deep learning and visual algorithm, characterized in that: The following steps are involved: SA1. Data collection and data preprocessing: Collect information such as paper, computer screens, and mobile phone screenshots taken by users and manually exclude pictures that are not in the above three categories as raw data; SA2. Build a dataset: Manually mark the polygon foreground edges, process and fill them into masks as mask targets, and combine the masks and images to construct training sets, test sets, and validation sets. SA3. Building a deep learning model: Construct a deep learning network DocSegNet consisting of an edge extraction network and a deep learning network, and initialize its network parameters; SA4. Training DocSegNet: Perform supervised training on the entire DocSegNet, take all test sets and training sets, perform cross-validation training, select the network model with the smallest average loss, and save the model.
2. The image region extraction and segmentation method based on deep learning and visual algorithm according to claim 1, characterized in that: The input of the model in the SA3 step is H*W*3, and the output is H*W*1.
3. The image region extraction and segmentation method based on deep learning and visual algorithm according to claim 1, characterized in that: The network layer hierarchy design of the model includes the following steps: SB1, edge detection: Process the input image x through the edge module (sobelnet) and extract the edge information, denoted as hx; SB2, edge information encoding: input the edge feature fusion image hx into the convolution for encoding; SB3, encoder feature decoding: progressively upsample through the decoder module and fuse the corresponding encoder features using skip connections; SB4, output side features: The side output layers (side1 to side6) are used to generate feature maps at different scales, denoted as d1 to d6, and are uniformly upsampled to the scale of d1; SB5, final output: concatenate the side features and generate the final output d0 through outconv.
4. The image region extraction and segmentation method based on deep learning and visual algorithm according to claim 3, characterized in that: The edge information encoding comprises the following steps: Stage 1: Input the edge feature fusion image hx into the first convolution layer to generate feature representation hx1, and then downsample through the pool12 maximum pooling layer; Stage2: hx1 is further passed into the second convolution layer to generate feature representation hx2, and then downsampled through the pool23 maximum pooling layer; Stage3: Similarly, generate feature hx3 and downsample through pool34; Stage4: Generate feature hx4 and further downsample through pool45; stage5: generate feature hx5 and downsample through pool56; Stage6: Finally, hx5 is further convolved to generate feature representation hx6, which is upsampled to the scale of hx5, denoted as hx6up.
5. The image region extraction and segmentation method based on deep learning and visual algorithm according to claim 3, characterized in that: The encoder feature decoding comprises the following steps: stage5d: fuses hx6up with hx5 to generate feature representation hx5d, and upsamples to the scale of hx4; stage4d: fuses the upsampled hx5d with hx4 to generate hx4d, and upsamples to the scale of hx3; stage3d: repeat the process to generate hx3d and upsample to the scale of hx2; stage2d: generates hx2d and upsamples to the scale of hx1; stage1d: generates the final decoder output hx1d.
6. The image region extraction and segmentation method based on deep learning and visual algorithm according to claim 3, characterized in that: The loss function calculation formula of the sample is: ； in: N is the number of samples, d i (j) represents the predicted value of the current scale model, labels v (j) Represents the actual annotation mask data label.
7. The image region extraction and segmentation method based on deep learning and visual algorithm according to claim 3, characterized in that: The summation formula for the network layer hierarchy design is: ； That is, the final loss is: 。 8. The image region extraction and segmentation method based on deep learning and visual algorithm according to claim 1, characterized in that: The segmentation evaluation formula of the graphic area is: ； Among them: output is the mask image predicted by the model, target is the manually annotated labels, and smooth is set to 10 by default to prevent calculation problems. ^ -5.