Image segmentation method and device, storage medium and program product

By introducing multi-level and double-hop connection structures into the U-Net network, the problem of low accuracy of image segmentation methods is solved, and higher image segmentation accuracy and understanding of semantic information are achieved.

CN118968066APending Publication Date: 2024-11-15PICC INFORMATION TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411096823.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing image segmentation methods are less accurate in the field of computer vision, especially when processing document photos, it is easy to missegment the background and document parts, resulting in identification errors.

Method used

Multiple levels and double jump connections are introduced into the encoding and decoding parts of the U-Net network. Through the feature fusion between levels and codec structure, the model's ability to express image information is enhanced, combined with shallow and deep features, and improved the Accuracy of image segmentation.

Benefits of technology

It significantly improves the accuracy of image segmentation, reduces the problem of mis-boxing and incomplete box selection, enhances the understanding of image semantic information, and improves the ability to deal with complex backgrounds.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968066A_ABST
    Figure CN118968066A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides an image segmentation method and device, a storage medium and a program product, and the method comprises the steps: obtaining a to-be-processed image, inputting the to-be-processed image into an image segmentation model for image segmentation, and obtaining an image segmentation result, a coding part and a decoding part of the image segmentation model comprise a plurality of levels, each hierarchy comprises a first jump connection and a second jump connection, and the first jump connection is used for fusing a feature map output by the last convolution in the corresponding hierarchy in the coding part and a feature map obtained by inter-hierarchy decoding of the previous hierarchy in the decoding part; the second jump connection is used for fusing the feature map output by the last convolutional layer in the corresponding hierarchies in the coding part and the feature map output by the first convolutional layer in the corresponding hierarchies in the decoding part, and inputting the fused second feature map to the next processing layer of the first convolutional layer. According to the method provided by the embodiment, the accuracy of image segmentation can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to an image segmentation method, device, storage medium and program product. Background Art

[0002] Image segmentation is a core task in the field of computer vision, which aims to divide a digital image into multiple regions or objects with semantic information.

[0003] In the related art, a neural network model obtained by U-Net network training can be used to perform image segmentation.

[0004] However, in the process of implementing the present application, the inventors found that there are at least the following problems in the prior art: the accuracy of image segmentation in the prior art is low. Summary of the invention

[0005] The embodiments of the present application provide an image segmentation method, device, storage medium and program product to improve the accuracy of image segmentation.

[0006] In a first aspect, an embodiment of the present application provides an image segmentation method, comprising:

[0007] Get the image to be processed;

[0008] The image to be processed is input into an image segmentation model for image segmentation to obtain an image segmentation result; the image segmentation model includes an encoding part and a decoding part; the encoding part and the decoding part include multiple levels; each level includes a first jump connection and a second jump connection; the first jump connection is used to fuse the feature map output by the last convolution in the corresponding level in the encoding part and the feature map obtained by inter-level decoding of the previous level in the decoding part, so as to input the fused first feature map into the first convolution layer of the corresponding level in the decoding part; the second jump connection is used to fuse the feature map output by the last convolution layer in the corresponding level in the encoding part and the feature map output by the first convolution layer in the corresponding level in the decoding part, and input the fused second feature map into the next processing layer of the first convolution layer.

[0009] In a possible design, fusing the feature map output by the last convolutional layer in the corresponding level in the encoding part and the feature map output by the first convolutional layer in the corresponding level in the decoding part, and inputting the fused second feature map to the next processing layer of the first convolutional layer, includes:

[0010] The feature map output by the last convolutional layer in the corresponding level in the encoding part and the feature map output by the first convolutional layer in the corresponding level in the decoding part are concatenated, and the concatenated second feature map is input to the second convolutional layer adjacent to the first convolutional layer.

[0011] In one possible design, each level in the decoding part includes a codec within the level, and the codec within the level includes a pooling layer, a convolution layer, and a deconvolution layer connected in sequence;

[0012] The pooling layer is used to perform intra-layer encoding on the input feature map, and input the encoded feature map into the convolution layer;

[0013] The convolution layer is used to convolve the encoded feature map, and input the convolved feature map into the deconvolution layer;

[0014] The deconvolution layer is used to perform intra-layer decoding on the convolved feature map to obtain a decoded feature map, so as to perform inter-layer decoding on the decoded feature map and output it to the first convolution layer of the next layer.

[0015] In a possible design, fusing the feature map output by the last convolutional layer in the corresponding level in the encoding part and the feature map output by the first convolutional layer in the corresponding level in the decoding part, and inputting the fused second feature map to the next processing layer of the first convolutional layer, includes:

[0016] The feature map output by the last convolutional layer in the corresponding level in the encoding part and the feature map output by the first convolutional layer in the corresponding level in the decoding part are concatenated, and the concatenated second feature map is input into the codec in the level adjacent to the first convolutional layer.

[0017] In a possible design, before inputting the image to be processed into the image segmentation model to obtain the image segmentation result, the method further includes:

[0018] The neural network model is trained according to the training set to obtain the image segmentation model; the neural network model includes an encoding part and a decoding part; the encoding part and the decoding part include multiple levels; wherein, in each iterative training cycle, for each level, the feature map output by the level is restored to size to obtain the restored feature map corresponding to the level, and the loss value between the restored feature map and the binary mask map is calculated; the final loss value is determined according to the loss values ​​of the multiple levels, and the parameters of the neural network model are adjusted according to the final loss value.

[0019] In a possible design, determining the final loss value according to the loss values ​​of each level includes:

[0020] Determine the intersection area IOU of the predicted bounding box and the true bounding box according to the restored feature map and the binary mask map;

[0021] Determine weights corresponding to the loss values ​​of the multiple levels according to the IOUs of the multiple levels;

[0022] According to the weights corresponding to the loss values ​​of the multiple levels, the loss values ​​of the multiple levels are weightedly summed to obtain a final loss value.

[0023] In a possible design, before training the neural network model according to the training set to obtain the image segmentation model, the method further includes:

[0024] Acquire multiple historical images;

[0025] For each historical image, marking the vertices and boundary center points of the historical image to obtain a marked historical image;

[0026] Perform image enhancement on each annotated historical image to obtain a new image;

[0027] A training set is constructed according to the plurality of annotated historical images and the plurality of newly added images.

[0028] In a second aspect, an embodiment of the present application provides an image segmentation device, including:

[0029] An acquisition module is used to acquire the image to be processed;

[0030] A segmentation module is used to input the image to be processed into an image segmentation model for image segmentation to obtain an image segmentation result; the image segmentation model includes an encoding part and a decoding part; the encoding part and the decoding part include multiple levels; each level includes a first jump connection and a second jump connection; the first jump connection is used to fuse the feature map output by the last convolution in the corresponding level in the encoding part and the feature map obtained by inter-level decoding of the previous level in the decoding part, so as to input the fused first feature map into the first convolution layer of the corresponding level in the decoding part; the second jump connection is used to fuse the feature map output by the last convolution layer in the corresponding level in the encoding part and the feature map output by the first convolution layer in the corresponding level in the decoding part, and input the fused second feature map into the next processing layer of the first convolution layer.

[0031] In a third aspect, an embodiment of the present application provides an image segmentation device, including: at least one processor and a memory;

[0032] The memory stores computer-executable instructions;

[0033] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method described in the first aspect and various possible designs of the first aspect.

[0034] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method described in the first aspect and various possible designs of the first aspect are implemented.

[0035] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect and various possible designs of the first aspect.

[0036] The image segmentation method, device, storage medium and program product provided in this embodiment include obtaining an image to be processed, inputting the image to be processed into an image segmentation model for image segmentation, and obtaining an image segmentation result. The image segmentation model includes an encoding part and a decoding part. The encoding part and the decoding part include multiple levels. Each level includes a first jump connection and a second jump connection. The first jump connection is used to fuse the feature map output by the last convolution in the corresponding level in the encoding part and the feature map obtained by inter-level decoding of the previous level in the decoding part, so as to input the fused first feature map into the first convolution layer of the corresponding level in the decoding part. The second jump connection is used to fuse the feature map output by the last convolution layer in the corresponding level in the encoding part and the feature map output by the first convolution layer in the corresponding level in the decoding part, and input the fused second feature map into the next processing layer of the first convolution layer. The method provided in this embodiment greatly enriches the model's ability to express image information by setting two jump connections between the encoding part and the decoding part in each layer. It not only provides more high-resolution spatial information, but also contains richer and more specific convolutional features. The network can combine more features between shallow and deep layers, so that the network can better process features at different scales and enhance the understanding of image semantic information, thereby improving the accuracy of image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0038] Figure 1A schematic diagram of an application scenario of the image segmentation method provided in an embodiment of the present application;

[0039] Figure 2 A flowchart of an image segmentation method provided in an embodiment of the present application;

[0040] Figure 3 A schematic diagram of the structure of the image segmentation model provided in the embodiment of the present application;

[0041] Figure 4 A flowchart of a method for training an image segmentation model provided in an embodiment of the present application;

[0042] Figure 5 A schematic diagram of the structure of an image segmentation device provided in an embodiment of the present application;

[0043] Figure 6 A schematic diagram of the hardware structure of the image segmentation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0046] It should be noted that the image segmentation method, device, storage medium and program product provided in the present application can be used in the field of image processing, and can also be used in any field other than the field of image processing. The application field of the image segmentation method, device, storage medium and program product provided in the present application is not limited.

[0047] In the insurance industry, image segmentation has a very wide range of applications. Many original images are disturbed by factors such as the background, which leads to large errors in OCR (Optical Character Recognition) and image quality inspection. If the part that needs to be recognized can be segmented out in advance and then processed, the error of subsequent processing can be greatly reduced, the accuracy and stability of the overall recognition can be improved, and it can be ensured that it can still work normally under various background conditions, improve the reliability of the overall service, reduce the risk of errors and misjudgments, and thus improve user experience and business efficiency. In summary, image segmentation is a very important step in image processing, which can improve the performance and adaptability of the overall service in complex backgrounds.

[0048] In the related art, a neural network model, such as a U-Net network, can be used for image segmentation. However, the accuracy of the model obtained by training the existing U-Net network for segmenting ID photos is low. It usually mistakenly segments the parts with similar background and ID as ID, resulting in a large amount of background in the selected part or misidentifying the position without ID as ID for the selected part. In addition, some only segmented the part of the ID, resulting in incomplete selection and inability to see the entire ID image after segmentation. All of these will cause unnecessary problems in the final identification of the ID, thereby increasing the workload.

[0049] In order to solve the above technical problems, the inventors of the present application have found that at least one jump connection can be added to each level of the encoding part and the decoding part of the U-Net network, which greatly enriches the model's ability to express image information, not only provides more high-resolution spatial information, but also contains richer and more specific convolutional features. The network can combine more features between shallow and deep layers, so that the network can better process features at different scales and enhance the understanding of image semantic information, thereby improving the accuracy of image segmentation. Based on this, the embodiment of the present application provides an image segmentation method.

[0050] Figure 1 Schematic diagram of application scenarios of the image segmentation method provided in the embodiment of the present application. Figure 1 As shown, the terminal device may be installed with a business system for receiving an image to be processed uploaded by a user to the business system, inputting the image to be processed into an image segmentation model for image segmentation, and obtaining an image segmentation result.

[0051] In the specific implementation process, when the user needs to handle business, he can upload the document image and other related images to be processed to the business system of the terminal device, and the terminal device inputs the received image to be processed into the image segmentation model for image segmentation to obtain the image segmentation result. The image segmentation model includes an encoding part and a decoding part; the encoding part and the decoding part include multiple levels; each level includes a first jump connection and a second jump connection; in the process of the image segmentation model processing the image to be processed, in the decoding process, the feature map output by the last convolution in the corresponding level of the encoding part and the feature map obtained by inter-level decoding of the previous level in the decoding part are fused through the first jump connection, so as to input the fused first feature map into the first convolution layer of the corresponding level in the decoding part; and the feature map output by the last convolution layer in the corresponding level of the encoding part and the feature map output by the first convolution layer in the corresponding level of the decoding part are fused through the second jump connection, and the fused second feature map is input into the next processing layer of the first convolution layer. The image segmentation method provided in the embodiment of the present application greatly enriches the model's ability to express image information by setting two jump connections between the encoding part and the decoding part in each layer. It not only provides more high-resolution spatial information, but also contains richer and more specific convolutional features. The network can combine more features between shallow and deep layers, so that the network can better process features at different scales and enhance the understanding of image semantic information, thereby improving the accuracy of image segmentation.

[0052] It should be noted that Figure 1 The scene diagram shown is only an example. The image segmentation method and scene described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0053] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0054] Figure 2 The following is a flow chart of the image segmentation method provided in the embodiment of the present application. Figure 2 As shown, the method includes:

[0055] 201. Obtain an image to be processed.

[0056] The execution subject of this embodiment may be a terminal device or a server.

[0057] The image to be processed may be an ID photo or other image.

[0058] 202. Input the image to be processed into an image segmentation model for image segmentation to obtain an image segmentation result; the image segmentation model includes an encoding part and a decoding part; the encoding part and the decoding part include multiple levels; each level includes a first jump connection and a second jump connection; the first jump connection is used to fuse the feature map output by the last convolution in the corresponding level in the encoding part and the feature map obtained by inter-level decoding of the previous level in the decoding part, so as to input the fused first feature map into the first convolution layer of the corresponding level in the decoding part; the second jump connection is used to fuse the feature map output by the last convolution layer in the corresponding level in the encoding part and the feature map output by the first convolution layer in the corresponding level in the decoding part, and input the fused second feature map into the next processing layer of the first convolution layer.

[0059] In this embodiment, the image segmentation model can be obtained by training a U-Net network model or a DC-UNet network model.

[0060] Specifically, the two skip connections introduce the feature maps of the encoding part into different convolution operations in the corresponding layers of the decoding part, which can enrich the ability to express image information and contain richer and more specific convolution features.

[0061] In this embodiment, when the first skip connection and the second skip connection fuse the two feature maps, the two feature maps may be concatenated.

[0062] In some embodiments, the step of fusing the feature map output by the last convolutional layer in the corresponding level in the encoding part and the feature map output by the first convolutional layer in the corresponding level in the decoding part, and inputting the fused second feature map to the next processing layer of the first convolutional layer, may include: splicing the feature map output by the last convolutional layer in the corresponding level in the encoding part and the feature map output by the first convolutional layer in the corresponding level in the decoding part, and inputting the spliced ​​second feature map to a second convolutional layer adjacent to the first convolutional layer.

[0063] Specifically, in the decoding part of the U-Net network, each level may include two adjacent convolutional layers, through which the feature map upsampled in the previous level is subjected to two consecutive convolutional processes, and then the feature map obtained after the two convolution processes is upsampled and output to the two adjacent convolutional layers in the next level.

[0064] In some embodiments, each level in the decoding part includes a codec within the level, and the codec within the level includes a pooling layer, a convolution layer and a deconvolution layer connected in sequence; the pooling layer is used to perform intra-level encoding on the input feature map, and input the encoded feature map to the convolution layer; the convolution layer is used to convolve the encoded feature map, and input the convolutional feature map to the deconvolution layer; the deconvolution layer is used to perform intra-level decoding on the convolutional feature map to obtain a decoded feature map, so as to perform inter-level decoding on the decoded feature map and output it to the first convolution layer of the next level.

[0065] Specifically, by adding codecs at each level in the decoding process, better information transmission and reconstruction can be achieved, thereby improving the accuracy of image segmentation.

[0066] In some embodiments, fusing the feature map output by the last convolutional layer in the corresponding level in the encoding part and the feature map output by the first convolutional layer in the corresponding level in the decoding part, and inputting the fused second feature map to the next processing layer of the first convolutional layer, may include: splicing the feature map output by the last convolutional layer in the corresponding level in the encoding part and the feature map output by the first convolutional layer in the corresponding level in the decoding part, and inputting the spliced ​​second feature map to the codec in the level adjacent to the first convolutional layer.

[0067] For example, Figure 3 As shown, the image segmentation model includes an encoding part on the left and a decoding part on the right. The encoding part is used to extract features of the image to be processed, and the decoding part is used to upsample the feature map extracted by the encoding part to obtain a segmentation result with the same size as the original image. For each level of the decoding part, taking the second level from top to bottom as an example, the feature map 311 of the third level is upsampled to obtain the first feature map (not shown), and the first feature map is fused (e.g., concatenated) with the feature map 312 output by the last convolution layer in the second level of the encoding part through the first skip connection 301 in the second level to obtain a feature map 313, which is input into the first convolution layer of the decoding part in the second level, and the feature map 312 is fused (e.g., concatenated) with the second feature map (not shown) output by the first convolution layer through the second skip connection 302 to obtain a feature map 314, which is input into the codec in the level (corresponding to the yellow arrow), and the codec in the level includes three processing layers, a pooling layer, a convolution layer, and a deconvolution layer (corresponding to the red arrow, the blue arrow, and the green arrow in the illustration of the yellow arrow in the lower right corner, respectively). The codec in the level of the second level outputs a feature map 315, which is output to the first level after upsampling.

[0068] The image segmentation method provided in this embodiment greatly enriches the model's ability to express image information by setting two jump connections between the encoding part and the decoding part in each layer. It not only provides more high-resolution spatial information, but also contains richer and more specific convolutional features. The network can combine more features between shallow and deep layers, so that the network can better process features at different scales and enhance the understanding of image semantic information, thereby improving the accuracy of image segmentation.

[0069] Figure 4 The flowchart of the training method of the image segmentation model provided in the embodiment of the present application is as follows. Figure 4 As shown, the method includes:

[0070] 401. Acquire multiple historical images; for each historical image, mark the vertices and boundary center points of the historical image to obtain a marked historical image.

[0071] 402. Perform image enhancement on each annotated historical image to obtain a newly added image.

[0072] 403. Construct a training set according to the multiple annotated historical images and the multiple newly added images.

[0073] In this embodiment, in one possible implementation, a training set may be constructed based only on the annotated historical images obtained in step 401. In another possible implementation, a training set of augmented samples may be obtained according to steps 401-403. In yet another possible implementation, only the newly added images obtained in step 402 may be added to the training set.

[0074] 404. Train the neural network model according to the training set to obtain the image segmentation model; the neural network model includes an encoding part and a decoding part; the encoding part and the decoding part include multiple levels. In each iterative training cycle, for each level, the feature map output by the level is restored to size, the restored feature map corresponding to the level is obtained, and the loss value between the restored feature map and the binary mask map is calculated; the final loss value is determined according to the loss values ​​of the multiple levels, and the parameters of the neural network model are adjusted according to the final loss value.

[0075] Specifically, labelme can be used to annotate images with ID cards. The eight-point annotation method is used, that is, only four vertices and four boundary center points or special points are marked during the labeling process to achieve segmentation. This can effectively reduce the learning difficulty of the deep learning network and improve the accuracy of the model. The annotated data set is divided into a training set and a validation set for training; the annotated data images are subjected to resolution adjustment, sharpening, Gaussian blur, flipping and other image enhancements, and the model is trained based on the training set after image enhancement processing to make the trained model more robust.

[0076] In the process of training the neural network model, UNet with a double-layer skip connection structure is introduced, and the codec and weighted loss are introduced at each level in the decoding process to form a DC-UNet network. The network is trained, and the total Dice_Loss loss function is calculated and then back-propagation is performed to update the model parameters. After each round of training, the model accuracy is verified on the validation set, and the optimal network model is saved. After the model training is completed, the model can be tested based on the validation set, and the generated model can be used directly to segment the document image. Compared with the previous direct segmentation using the UNet model, it will be found that the image segmentation method provided by this embodiment greatly reduces the problems of wrong frame selection, less frame, and more frame after image segmentation.

[0077] In some embodiments, determining the final loss value based on the loss values ​​of each level may include: determining the IOU of the intersection area of ​​the predicted bounding box and the true bounding box based on the restored feature map and the binary mask map; determining the weights corresponding to the loss values ​​of multiple levels based on the IOUs of the multiple levels; and performing weighted summation of the loss values ​​of multiple levels based on the weights corresponding to the loss values ​​of the multiple levels to obtain the final loss value.

[0078] Specifically, the feature map size is restored by upsampling directly after each decoder layer, the number of channels of the feature map is restored through the fully connected layer, the intersection over union (IOU) and loss of the predicted bounding box and the true bounding box of the mask image are calculated, the loss weight is set according to the IOU result of each layer, and the loss of each layer is weighted to obtain the final loss, so that the model can learn and adjust parameters more effectively, improve the decoding results, and thus significantly improve the accuracy of the segmentation model.

[0079] The image segmentation method provided in this embodiment obtains the restored feature map of the restored size of each level of the decoding part and calculates the corresponding loss value, taking into account the loss value calculation of more levels, enabling the model to learn and adjust parameters more effectively, improve the decoding results, and thus significantly improve the accuracy of the segmentation model.

[0080] Figure 5This is a schematic diagram of the structure of the image segmentation device provided in the embodiment of the present application. Figure 5 As shown, the image segmentation device 50 includes: an acquisition module 501 and a segmentation module 502 .

[0081] The acquisition module 501 is used to acquire the image to be processed.

[0082] The segmentation module 502 is used to input the image to be processed into the image segmentation model for image segmentation to obtain an image segmentation result; the image segmentation model includes an encoding part and a decoding part; the encoding part and the decoding part include multiple levels; each level includes a first jump connection and a second jump connection; the first jump connection is used to fuse the feature map output by the last convolution in the corresponding level in the encoding part and the feature map obtained by inter-level decoding of the previous level in the decoding part, so as to input the fused first feature map into the first convolution layer of the corresponding level in the decoding part; the second jump connection is used to fuse the feature map output by the last convolution layer in the corresponding level in the encoding part and the feature map output by the first convolution layer in the corresponding level in the decoding part, and input the fused second feature map into the next processing layer of the first convolution layer.

[0083] The image segmentation device provided in the embodiment of the present application greatly enriches the model's ability to express image information by setting two jump connections between the encoding part and the decoding part in each layer. It not only provides more high-resolution spatial information, but also contains richer and more specific convolutional features. The network can combine more features between shallow and deep layers, so that the network can better process features at different scales and enhance the understanding of image semantic information, thereby improving the accuracy of image segmentation.

[0084] In some embodiments, fusing the feature map output by the last convolutional layer in the corresponding level in the encoding part with the feature map output by the first convolutional layer in the corresponding level in the decoding part, and inputting the fused second feature map to the next processing layer of the first convolutional layer, comprises:

[0085] The feature map output by the last convolutional layer in the corresponding level in the encoding part and the feature map output by the first convolutional layer in the corresponding level in the decoding part are concatenated, and the concatenated second feature map is input to the second convolutional layer adjacent to the first convolutional layer.

[0086] In some embodiments, each level in the decoding part includes a codec within the level, and the codec within the level includes a pooling layer, a convolution layer, and a deconvolution layer connected in sequence;

[0087] The pooling layer is used to perform intra-layer encoding on the input feature map, and input the encoded feature map into the convolution layer;

[0088] The convolution layer is used to convolve the encoded feature map, and input the convolved feature map into the deconvolution layer;

[0089] The deconvolution layer is used to perform intra-layer decoding on the convolved feature map to obtain a decoded feature map, so as to perform inter-layer decoding on the decoded feature map and output it to the first convolution layer of the next layer.

[0090] In some embodiments, fusing the feature map output by the last convolutional layer in the corresponding level in the encoding part with the feature map output by the first convolutional layer in the corresponding level in the decoding part, and inputting the fused second feature map to the next processing layer of the first convolutional layer, comprises:

[0091] The feature map output by the last convolutional layer in the corresponding level in the encoding part and the feature map output by the first convolutional layer in the corresponding level in the decoding part are concatenated, and the concatenated second feature map is input into the codec in the level adjacent to the first convolutional layer.

[0092] In some embodiments, the device 50 further includes a training module (not shown), the training module being configured to:

[0093] The neural network model is trained according to the training set to obtain the image segmentation model; the neural network model includes an encoding part and a decoding part; the encoding part and the decoding part include multiple levels; wherein, in each iterative training cycle, for each level, the feature map output by the level is restored to size to obtain the restored feature map corresponding to the level, and the loss value between the restored feature map and the binary mask map is calculated; the final loss value is determined according to the loss values ​​of the multiple levels, and the parameters of the neural network model are adjusted according to the final loss value.

[0094] In some embodiments, when determining the final loss value according to the loss values ​​of each layer, the training module is specifically used to:

[0095] Determine the intersection area IOU of the predicted bounding box and the true bounding box according to the restored feature map and the binary mask map;

[0096] Determine weights corresponding to the loss values ​​of the multiple levels according to the IOUs of the multiple levels;

[0097] According to the weights corresponding to the loss values ​​of the multiple levels, the loss values ​​of the multiple levels are weightedly summed to obtain a final loss value.

[0098] In some embodiments, the training module is further configured to:

[0099] Acquire multiple historical images;

[0100] For each historical image, marking the vertices and boundary center points of the historical image to obtain a marked historical image;

[0101] Perform image enhancement on each annotated historical image to obtain a new image;

[0102] A training set is constructed according to the plurality of annotated historical images and the plurality of newly added images.

[0103] The image segmentation device provided in the embodiment of the present application can be used to execute the above-mentioned method embodiment, and its implementation principle and technical effect are similar, which will not be repeated in this embodiment.

[0104] Figure 6 A schematic diagram of the hardware structure of an image segmentation device provided in an embodiment of the present application. The device may be a computer, a message sending and receiving device, a tablet device, a medical device, etc.

[0105] The device 60 may include one or more of the following components: a processing component 601 , a memory 602 , a power component 603 , a multimedia component 604 , an audio component 605 , an input / output (I / O) interface 606 , a sensor component 607 , and a communication component 608 .

[0106] The processing component 601 generally controls the overall operation of the device 60, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 601 may include one or more processors 609 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 601 may include one or more modules to facilitate the interaction between the processing component 601 and other components. For example, the processing component 601 may include a multimedia module to facilitate the interaction between the multimedia component 604 and the processing component 601.

[0107] The memory 602 is configured to store various types of data to support operations on the device 60. Examples of such data include instructions for any application or method operating on the device 60, contact data, phone book data, messages, pictures, videos, etc. The memory 602 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0108] The power supply component 603 provides power to the various components of the device 60. The power supply component 603 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 60.

[0109] The multimedia component 604 includes a screen that provides an output interface between the device 60 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 604 includes a front camera and / or a rear camera. When the device 60 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0110] The audio component 605 is configured to output and / or input audio signals. For example, the audio component 605 includes a microphone (MIC), and when the device 60 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 602 or sent via the communication component 608. In some embodiments, the audio component 605 also includes a speaker for outputting audio signals.

[0111] I / O interface 606 provides an interface between processing component 601 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0112] The sensor assembly 607 includes one or more sensors for providing various aspects of status assessment for the device 60. For example, the sensor assembly 607 can detect the open / closed state of the device 60, the relative positioning of components, such as the display and keypad of the device 60, and the sensor assembly 607 can also detect the position change of the device 60 or a component of the device 60, the presence or absence of user contact with the device 60, the orientation or acceleration / deceleration of the device 60, and the temperature change of the device 60. The sensor assembly 607 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 607 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 607 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0113] The communication component 608 is configured to facilitate wired or wireless communication between the device 60 and other devices. The device 60 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 608 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 608 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0114] In an exemplary embodiment, the device 60 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0115] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 602 including instructions, and the instructions can be executed by a processor 609 of the device 60 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0116] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0117] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0118] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0119] An embodiment of the present application further provides a computer program product, including a computer program, and when the computer program is executed by a processor, the image segmentation method performed by the above image segmentation device is implemented.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image segmentation method, It is characterized in that include: Get the image to be processed; Inputting the image to be processed into an image segmentation model to perform image segmentation and obtain an image segmentation result; The image segmentation model includes an encoding part and a decoding part; the encoding part and the decoding part include multiple levels; each level includes a first jump connection and a second jump connection; the first jump connection is used to fuse the feature map output by the last convolution in the corresponding level in the encoding part and the feature map obtained by inter-level decoding of the previous level in the decoding part, so as to input the fused first feature map into the first convolution layer of the corresponding level in the decoding part; The second jump connection is used to fuse the feature map output by the last convolutional layer in the corresponding level in the encoding part and the feature map output by the first convolutional layer in the corresponding level in the decoding part, and input the fused second feature map into the next processing layer of the first convolutional layer.

2. The method according to claim 1, It is characterized in that The step of fusing the feature map output by the last convolution layer in the corresponding level in the encoding part and the feature map output by the first convolution layer in the corresponding level in the decoding part, and inputting the fused second feature map to the next processing layer of the first convolution layer, comprises: The feature map output by the last convolutional layer in the corresponding level in the encoding part and the feature map output by the first convolutional layer in the corresponding level in the decoding part are concatenated, and the concatenated second feature map is input to the second convolutional layer adjacent to the first convolutional layer.

3. The method according to claim 1, It is characterized in that Each level in the decoding part includes a codec in the level, and the codec in the level includes a pooling layer, a convolution layer, and a deconvolution layer connected in sequence; The pooling layer is used to perform intra-layer encoding on the input feature map, and input the encoded feature map into the convolution layer; The convolution layer is used to convolve the encoded feature map, and input the convolved feature map into the deconvolution layer; The deconvolution layer is used to perform intra-layer decoding on the convolved feature map to obtain a decoded feature map, so as to perform inter-layer decoding on the decoded feature map and output it to the first convolution layer of the next layer.

4. The method according to claim 3, It is characterized in that The step of fusing the feature map output by the last convolution layer in the corresponding level in the encoding part and the feature map output by the first convolution layer in the corresponding level in the decoding part, and inputting the fused second feature map to the next processing layer of the first convolution layer, comprises: The feature map output by the last convolutional layer in the corresponding level in the encoding part and the feature map output by the first convolutional layer in the corresponding level in the decoding part are concatenated, and the concatenated second feature map is input into the codec in the level adjacent to the first convolutional layer.

5. The method according to any one of claims 1 to 4, It is characterized in that Before inputting the image to be processed into the image segmentation model to obtain the image segmentation result, the method further includes: The neural network model is trained according to the training set to obtain the image segmentation model; the neural network model includes an encoding part and a decoding part; the encoding part and the decoding part include multiple levels; wherein, in each iterative training cycle, for each level, the feature map output by the level is restored to size to obtain the restored feature map corresponding to the level, and the loss value between the restored feature map and the binary mask map is calculated; the final loss value is determined according to the loss values ​​of the multiple levels, and the parameters of the neural network model are adjusted according to the final loss value.

6. The method according to claim 5, It is characterized in that Determining the final loss value according to the loss values ​​of each level includes: Determine the intersection area IOU of the predicted bounding box and the true bounding box according to the restored feature map and the binary mask map; Determine weights corresponding to the loss values ​​of the multiple levels according to the IOUs of the multiple levels; According to the weights corresponding to the loss values ​​of the multiple levels, the loss values ​​of the multiple levels are weightedly summed to obtain a final loss value.

7. The method according to claim 5, It is characterized in that Before the neural network model is trained according to the training set to obtain the image segmentation model, the method further includes: Acquire multiple historical images; For each historical image, marking the vertices and boundary center points of the historical image to obtain a marked historical image; Perform image enhancement on each annotated historical image to obtain a new image; A training set is constructed according to the plurality of annotated historical images and the plurality of newly added images.

8. An image segmentation device, It is characterized in that include: An acquisition module, used for acquiring an image to be processed; A segmentation module, used for inputting the image to be processed into an image segmentation model to perform image segmentation and obtain an image segmentation result; The image segmentation model includes an encoding part and a decoding part; the encoding part and the decoding part include multiple levels; each level includes a first jump connection and a second jump connection; the first jump connection is used to fuse the feature map output by the last convolution in the corresponding level in the encoding part and the feature map obtained by inter-level decoding of the previous level in the decoding part, so as to input the fused first feature map into the first convolution layer of the corresponding level in the decoding part; The second jump connection is used to fuse the feature map output by the last convolutional layer in the corresponding level in the encoding part and the feature map output by the first convolutional layer in the corresponding level in the decoding part, and input the fused second feature map into the next processing layer of the first convolutional layer.

9. An image segmentation device, It is characterized in that include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the image segmentation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the image segmentation method according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the image segmentation method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Medical image segmentation model and method, storage medium and electronic equipment

    CN110335276A

  • Certificate photo standardization processing method and device

    CN112633252A

  • Image processing method, system and device and computer medium

    CN114359557A

  • Method and system for identifying and inputting property right information

    CN116129456A

  • Apple leaf disease segmentation and grading system based on attention and feature fusion

    CN118334053A