Image segmentation model training method, image segmentation method, device and equipment

Abstract

The invention relates to an image segmentation model training method, an image segmentation method, devices and equipment. The image segmentation model training method comprises the following steps: acquiring a background sample image and a foreground sample image; based on alpha channel information of the foreground sample image, performing synthesis processing on the background sample image and the foreground sample image to obtain a synthesized sample image; the synthetic sample image comprises a foreground labeling mask; inputting the synthetic sample image and the foreground sample image into an initial image segmentation model, and predicting a foreground mask in the synthetic sample image through the initial image segmentation model to obtain a foreground prediction mask; and according to the difference between the foreground prediction mask and the foreground labeling mask, adjusting parameters of the initial image segmentation model until the difference satisfies a preset difference condition, and obtaining a target image segmentation model. By adopting the method, the generalization ability of the model can be improved under the condition of a single sample.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a training method of an image segmentation model, an image segmentation method, an image segmentation device, a computer device and a computer readable storage medium. BACKGROUND

[0002] Image segmentation is one of the core tasks of computer vision, aiming to divide an image into regions with semantic meaning and accurately extract target objects. In manufacturing and industrial pipeline production, industrial-level part segmentation plays an important role in target recognition, defect detection and quality analysis. Traditional methods usually rely on large-scale pixel-level semantic labeling to ensure the accuracy of part segmentation. However, the traditional way not only has high labeling cost, but also consumes a lot of resources in the training process of the corresponding model.

[0003] In view of the above problem of high labeling cost, most current methods rely on limited public datasets for training, but these data usually cannot cover the complex and diverse scenarios in industrial production, resulting in poor generalization ability of the model. Based on the above analysis, how to improve the generalization ability of the model under single sample condition becomes a problem to be solved. SUMMARY

[0004] Therefore, it is necessary to provide a training method of an image segmentation model, an image segmentation method, an image segmentation device, a computer device, a computer readable storage medium and a computer program product, which can improve the generalization ability of the model under single sample condition.

[0005] In a first aspect, the present application provides a training method of an image segmentation model, comprising:

[0006] obtaining a background sample image and a foreground sample image;

[0007] based on the alpha channel information of the foreground sample image, performing synthesis processing on the background sample image and the foreground sample image to obtain a synthesized sample image; the synthesized sample image comprises a foreground annotation mask;

[0008] inputting the synthesized sample image and the foreground sample image into an initial image segmentation model, and predicting the foreground mask in the synthesized sample image through the initial image segmentation model to obtain a foreground prediction mask;

[0009] adjusting the parameters of the initial image segmentation model according to the difference between the foreground prediction mask and the foreground annotation mask until the difference meets a preset difference condition, and obtaining a target image segmentation model.

[0010] In a second aspect, the present application provides an image segmentation method, comprising:

[0011] Obtaining an image to be segmented;

[0012] Inputting the image to be segmented into the target image segmentation model obtained by the image segmentation model training method, to obtain a target foreground mask in the image to be segmented.

[0013] In a third aspect, the present application provides an image segmentation model training device, comprising:

[0014] An image acquisition module, configured to acquire a background sample image and a foreground sample image;

[0015] An image synthesis module, configured to perform synthesis processing on the background sample image and the foreground sample image based on alpha channel information of the foreground sample image, to obtain a synthesized sample image; the synthesized sample image comprises a foreground annotation mask;

[0016] A foreground mask prediction module, configured to input the synthesized sample image and the foreground sample image into an initial image segmentation model, and perform prediction on the foreground mask in the synthesized sample image by the initial image segmentation model, to obtain a foreground prediction mask;

[0017] A model parameter adjustment module, configured to adjust parameters of the initial image segmentation model according to a difference between the foreground prediction mask and the foreground annotation mask, until the difference meets a preset difference condition, to obtain a target image segmentation model.

[0018] In a fourth aspect, the present application provides a computer device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps in the above method when executing the computer program.

[0019] In a fifth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps in the above method.

[0020] In a sixth aspect, the present application provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in the above method.

[0021] The image segmentation model training method, the image segmentation method, the device, the computer device, the computer readable storage medium, and the computer program product can synthesize the background sample image and the foreground sample image based on the alpha channel information of the foreground sample image to obtain a synthesized sample image, train the image segmentation model based on the synthesized sample image and the foreground sample image, and obtain a target image segmentation model that can segment a foreground mask of an image. That is, the foreground sample image and the background sample image are synthesized based on the alpha channel information of the image to obtain a synthesized sample image, so that a large amount of model training data that is suitable for various scenes can be obtained. The image segmentation model trained based on the training data has strong generalization ability, so that the generalization ability of the model can be improved under a single sample condition. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 An application environment diagram of an image segmentation model training method provided by an embodiment of the present application;

[0023] Figure 2 A flowchart of an image segmentation model training method provided by an embodiment of the present application;

[0024] Figure 3 A foreground sample image diagram provided by an embodiment of the present application;

[0025] Figure 4 A background sample image diagram provided by an embodiment of the present application;

[0026] Figure 5 Another foreground sample image diagram provided by an embodiment of the present application;

[0027] Figure 6 Another background sample image diagram provided by an embodiment of the present application;

[0028] Figure 7 A synthesized sample image diagram provided by an embodiment of the present application;

[0029] Figure 8 Another flowchart of an image segmentation model training method provided by an embodiment of the present application;

[0030] Figure 9 A to-be-segmented image diagram provided by an embodiment of the present application;

[0031] Figure 10 A target foreground mask diagram provided by an embodiment of the present application;

[0032] Figure 11 A structural block diagram of an image segmentation model training device provided by an embodiment of the present application;

[0033] Figure 12 An internal structure diagram of a computer device provided in an embodiment of the present application;

[0034] Figure 13 An internal structure diagram of another computer device provided in an embodiment of the present application;

[0035] Figure 14 A diagram of the internal structure of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0037] The training method of the image segmentation model provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through a communication network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented as an independent server or a server cluster consisting of multiple servers.

[0038] like Figure 2 As shown, the embodiment of the present application provides a training method for an image segmentation model, and the method is applied to Figure 1 The terminal 102 or server 104 in the example is used for explanation. It is understood that the computer device may include at least one of a terminal and a server. The method includes the following steps:

[0039] S202: Obtain a background sample image and a foreground sample image.

[0040] A background sample image refers to a sample image that only includes the background. For example, a background sample image can be an image of the production environment of any industrial part. In other words, a background sample image only includes the background and does not include the foreground. A foreground sample image refers to a sample image that only includes the foreground. For example, a foreground sample image can be an image of any industrial part. In other words, a foreground sample image only includes the foreground and does not include the background, and the background can be represented by either completely black or completely white.

[0041] It should be noted that the foreground sample image includes alpha channel (A) information. The alpha channel refers to the transparency and translucency of the image. The alpha channel information represents the pixel transparency information. The alpha channel is usually stored together with the color channel (such as red, green, and blue, that is, RGB) to form an RGBA format. The alpha value of the alpha channel represents the transparency of the pixel, for example, if the alpha value is 0 (or 0%) indicates that the pixel is completely transparent (invisible); if the alpha value is 255 (or 100%) indicates that the pixel is completely opaque; if the alpha value is 128 (or 50%) indicates that the pixel is translucent, and the like. The alpha value is similar to the color channel value, that is, if the pixel depth of the image is 8 bits, the range of the alpha value is 0 to 255, if the pixel depth of the image is 16 bits, the range of the alpha value is 0 to 65535, and the range of the alpha value corresponding to other pixel depths is similar. For example, the foreground sample image is as shown in Figure 3 The background sample image is as shown in Figure 4

[0042] It is easy to understand that the more the number of foreground sample images and background sample images corresponding, the stronger the generalization ability of the target image segmentation model obtained by training. For example, 2k foreground sample images and 20k background sample images can be obtained to train the image segmentation model.

[0043] In S204, the background sample image and the foreground sample image are synthesized based on the alpha channel information of the foreground sample image to obtain a synthesized sample image. The synthesized sample image includes a foreground annotation mask.

[0044] The alpha channel information can be obtained from the foreground sample image, and the background sample image and the foreground sample image are weightedly fused based on the alpha channel information to obtain the synthesized sample image. That is, the alpha channel information of the foreground sample image can be used as a weighting weight to weightedly fuse the background sample image and the foreground sample image.

[0045] In an exemplary embodiment, the background sample image is an RGB three-channel image, and the foreground sample image is an RGBA four-channel image. The alpha channel (A) information can be obtained from the foreground sample image, and the RGB three-channel information of the background sample image and the RGB three-channel information of the foreground sample image are weightedly fused based on the alpha channel information to obtain the synthesized sample image.

[0046] For example, after obtaining the synthesized sample image, the foreground mask in the synthesized sample image can be annotated to obtain a foreground annotation mask. The foreground mask in the synthesized sample image can be annotated by manual annotation or by an annotation algorithm, which is not limited here. ​

[0047] S206, input the synthesized sample image and the foreground sample image into an initial image segmentation model, and predict the foreground mask in the synthesized sample image through the initial image segmentation model to obtain a foreground prediction mask.

[0048] The initial image segmentation model refers to an image segmentation model to be trained. The initial image segmentation model may be, for example, a convolutional neural network or a Transformer architecture, etc. The foreground prediction mask refers to a foreground mask predicted by the initial image segmentation model.

[0049] For example, the initial image segmentation model can respectively extract features from the synthesized sample image and the foreground sample image to obtain synthesized sample image features and foreground sample image features, fuse the synthesized sample image features and the foreground sample image features to obtain target synthesized features, and predict the foreground mask based on the target synthesized features to obtain the foreground prediction mask.

[0050] S208, adjust parameters of the initial image segmentation model according to a difference between the foreground prediction mask and the foreground annotation mask until the difference meets a preset difference condition, and obtain a target image segmentation model.

[0051] After obtaining the foreground prediction mask, the parameters of the initial image segmentation model are adjusted based on the difference between the foreground prediction mask and the foreground annotation mask until the difference meets a preset difference condition, for example, the difference between the foreground prediction mask and the foreground annotation mask is less than a difference threshold, and the model at this time is taken as the target image segmentation model. The difference between the foreground prediction mask and the foreground annotation mask can be represented by at least one of the mean square error, the cross-entropy, the dice coefficient or the regularization value between the foreground prediction mask and the foreground annotation mask.

[0052] As can be seen, in the embodiments of the present application, the background sample image and the foreground sample image are synthesized based on the alpha channel information of the foreground sample image to obtain a synthesized sample image, and the image segmentation model is trained based on the synthesized sample image and the foreground sample image to obtain a target image segmentation model capable of segmenting the foreground mask of the image, that is, the foreground sample image and the background sample image are synthesized based on the alpha channel information of the image to obtain a synthesized sample image, so that a large amount of model training data suitable for various scenes can be obtained, and the image segmentation model trained based on the training data has strong generalization ability, thereby improving the generalization ability of the model under single sample conditions.

[0053] In some embodiments, the synthesis of the background sample image and the foreground sample image based on the alpha channel information of the foreground sample image in S204 to obtain the synthesized sample image comprises:

[0054] The other channel information of the foreground sample image except the alpha channel information is acquired; and the other channel information of the foreground sample image and the background sample image is weightedly fused based on the alpha channel information of the foreground sample image to obtain a synthesized sample image.

[0055] Generally, the background sample image does not include the alpha channel, which means that the other channel information except the alpha channel is weightedly fused in each pixel corresponding to the foreground sample image and the background sample image to obtain the synthesized sample image.

[0056] In an exemplary embodiment, if the foreground sample image is an RGBA four-channel image and the background sample image is an RGB three-channel image, the other channel information of the foreground sample image except the alpha channel information, i.e. the RGB channel information, can be acquired, and the RGB channel information of the pixel in the background sample image and the RGB channel information of the corresponding pixel in the foreground sample image are weightedly fused based on the alpha channel information of the foreground sample image to obtain the information of the corresponding pixel in the synthesized sample image.

[0057] In an exemplary embodiment, the foreground sample image can be subjected to deformation processing such as rotation and scaling to obtain a deformed foreground sample image, and the other channel information of the deformed foreground sample image and the background sample image is weightedly fused based on the alpha channel information of the deformed foreground sample image to obtain the synthesized sample image. It is easy to understand that the foreground sample image can be subjected to deformation processing of different degrees to obtain a plurality of deformed foreground sample images, which means that the foreground sample image is subjected to data enhancement processing. Exemplarily, assuming that the four-channel foreground sample image is , the background sample image is , the foreground sample image is subjected to deformation processing such as rotation and scaling to obtain a deformed foreground sample image , and the synthesized sample image is which can be represented by the following formula (1).

[0058] Formula (1)

[0059] wherein [,, ] represents the size information and channel information of the image. Assuming that the image information is 64*32*4, which means that the height of the image is 64, the width of the image is 32, and the number of image channels is 4, and the index of the image channel corresponds to 0 to 3, if the image is an RGBA image, the index and the channel can correspond to R-0, B-1, G-2, and A-3. “:” represents taking all, “:3” represents taking the first three channels, and “[3]” represents taking the channel corresponding to the index 3, i.e. the A channel. Therefore, M in the above formula (1) represents the alpha channel information of the deformed foreground sample image, RGB channel information of the foreground sample image after the deformation processing. That is, in formula (1), is weighted sum of the RGB channel information of the foreground sample image after the deformation processing and the background sample image based on M, to obtain the synthesized sample image.

[0060] Exemplarily, if the foreground sample image is as shown in Figure 5 , and the background sample image is as shown in Figure 6 , then the corresponding synthesized sample image is as shown in Figure 7 .

[0061] It can be seen that, in the embodiment, the other channel information of the background sample image and the foreground sample image is weighted and fused based on the alpha channel information of the foreground sample image to obtain the synthesized sample image, a large number of synthesized sample images of multiple types and large quantities can be obtained based on a single sample, that is, a large amount of training data conforming to the real industrial production scene is obtained, and the model is trained based on the synthesized sample image, which can improve the generalization ability of the model and make the model applicable to more actual industrial production scenes.

[0062] In some embodiments, the foreground mask in the synthesized sample image is predicted by the initial image segmentation model in S206 to obtain the foreground prediction mask, including:

[0063] The synthesized sample image feature is obtained by feature extraction of the synthesized sample image by the initial image segmentation model, and the foreground sample image feature is obtained by feature extraction of the foreground sample image; the target synthesis feature is obtained by distribution alignment processing of the synthesized sample image feature to the foreground sample image feature; the target convolution feature is obtained by convolution processing of the target synthesis feature according to the target convolution kernel; and the foreground prediction mask is obtained by prediction of the target convolution feature.

[0064] The synthesized sample image feature refers to the feature of the synthesized sample image, and the foreground sample image feature refers to the feature of the foreground sample image.

[0065] Exemplarily, in terms of feature extraction, the initial image segmentation model can be used for deep feature extraction of the synthesized sample image and the foreground sample image respectively through a feature extraction network of the initial image segmentation model, to correspondingly obtain the synthesized sample image feature and the foreground sample image feature, and the feature spaces of the synthesized sample image feature and the foreground sample image feature are kept aligned. The feature extraction network can include multiple convolution layers, for example, 3, 4 or 5 convolution layers, so as to realize efficient and larger receptive field feature extraction.

[0066] Distribution alignment refers to the process of making two distributions identical or similar. Aligning the synthetic sample image features with the foreground sample image features involves aligning the distribution of the synthetic sample image features to be consistent with the distribution of the foreground sample image features. In other words, the distribution of the target synthetic features obtained through the distribution alignment is consistent with the distribution of the foreground sample image features. For example, the synthetic sample image features can be normalized to obtain normalized synthetic sample image features, and the normalized synthetic sample image features are processed based on the pixel mean and variance of the foreground sample image features to obtain the target synthetic features.

[0067] In an exemplary embodiment, the feature extraction network of the initial image segmentation model extracts features from the synthetic sample image and the foreground sample image, obtaining corresponding synthetic sample image features and foreground sample image features. The synthetic sample image features are aligned with the distribution of the foreground sample image features to obtain a target synthetic feature. A target convolution kernel corresponding to the foreground sample image information is obtained, and the target synthetic feature is convolved with the target convolution kernel to obtain a target convolution feature. The target convolution feature is then predicted based on the prediction network to obtain a foreground prediction mask. The target convolution kernel can be input by the user or determined based on information about the foreground sample image.

[0068] It can be seen that in this embodiment, the initial image segmentation model is used to extract features of the synthetic sample image and the foreground sample image respectively, and the distribution of the corresponding extracted synthetic sample image features is aligned with the distribution of the foreground sample image features. After the convolution operation is performed based on the aligned target synthetic features, the foreground prediction mask is predicted, which can improve the foreground mask segmentation accuracy of the image segmentation model.

[0069] In some embodiments, performing distribution alignment processing on the synthetic sample image features and the foreground sample image features to obtain target synthetic features includes:

[0070] Normalization processing is performed on the synthetic sample image features to obtain normalized synthetic sample image features; and target synthetic features are obtained according to the pixel mean and variance of the normalized synthetic sample image features and the foreground sample image features.

[0071] In an exemplary embodiment, the synthetic sample image features may be normalized to obtain normalized synthetic sample image features, and the normalized synthetic sample image features may be synthesized based on the pixel mean and variance of the foreground sample image features to obtain target synthetic features.

[0072] For example, assume that the synthetic sample image feature is f b , the foreground sample image feature is f t , calculate the pixel mean m of the synthetic sample image featuresb and variance v b and a pixel mean m t and variance v t wherein m b = mean(f b ), m t = mean(f t ), v b = std(f b ), v t = std(f t ), mean() denotes calculating mean value, and std() denotes calculating variance.

[0073] Exemplarily, in the synthesized sample image feature, the difference between each pixel value and the pixel mean value can be calculated, and the ratio of the difference and the variance can be taken as the value corresponding to the corresponding pixel in the normalized synthesized sample image feature. The normalized synthesized sample image feature can be obtained by the following formula (2).

[0074] Formula (2)

[0075] Exemplarily, the product of the pixel value variance of the foreground sample image feature and the normalized synthesized sample image feature, and the sum of the pixel value mean of the foreground sample image feature, can be taken as the target synthesized feature. That is, the target synthesized feature can be obtained by the following formula (3).

[0076] Formula (3)

[0077] It can be seen that in the embodiment, the target synthesized feature is obtained based on the normalized sample image feature obtained by normalizing the synthesized sample image feature, the pixel mean and variance of the foreground sample image feature, which can better balance the difference in distribution between different images, improve the dynamic perception accuracy of the model, and enable the model to adaptively adjust the corresponding feature distribution according to the distribution of the input image, thereby improving the robustness of segmentation.

[0078] In some embodiments, before the target convolution feature is obtained by performing convolution processing on the target synthesized feature according to the target convolution kernel, the above method further comprises:

[0079] pooling the foreground sample image to obtain a pooled foreground sample image; and determining the target convolution kernel according to information of the pooled foreground sample image.

[0080] The information of the pooled foreground sample image may include, for example, the number of channels, dimensions, and spatial attention, and the like. According to the information of the pooled foreground sample image, the target convolution kernel is determined to enable the foreground and the background to better fit.

[0081] In general, different foreground sample image information corresponds to different target convolution kernel sizes. For example, a convolution kernel consistent with the information of the foreground sample image is set as the target convolution kernel of the target synthesized feature.

[0082] As can be seen, in this embodiment, the target convolution kernel is determined according to the information of the pooled foreground sample image, which enables dynamic determination of the target convolution kernel, thereby enabling dynamic convolution of the target synthesized feature, improving the expression capability of the model, and thus improving the image segmentation accuracy.

[0083] In some embodiments, the parameters of the initial image segmentation model are adjusted according to the difference between the foreground prediction mask and the foreground annotation mask in S208 until the difference meets the preset difference condition, and a target image segmentation model is obtained, including:

[0084] The parameters of the initial image segmentation model are adjusted according to the mean square error loss between the foreground prediction mask and the foreground annotation mask until the mean square error loss is less than a loss threshold, and a target image segmentation model is obtained.

[0085] In actual application scenarios, the difference between the foreground prediction mask and the foreground annotation mask can be characterized by the mean square error between the foreground prediction mask and the foreground annotation mask. The mean square error loss between the foreground prediction mask and the foreground annotation mask can be calculated, the parameters of the initial image segmentation model are adjusted according to the mean square error loss, and the model at this time is taken as the target image segmentation model. The loss threshold can be set according to actual application scenarios, for example, the loss threshold is 0.1, 0.01, or 0.2, and the like.

[0086] For example, assume that the foreground annotation mask is M and the foreground prediction mask is The mean square error between the foreground prediction mask and the foreground annotation mask can be represented as When calculating specifically, it is the average value of the square of the difference between the pixels at the corresponding positions in the foreground prediction mask and the foreground annotation mask.

[0087] As can be seen, in this embodiment, the parameters of the initial image segmentation model are adjusted according to the mean square error loss between the foreground prediction mask and the foreground annotation mask until the training condition is met, and a target image segmentation model is obtained, which can improve the accuracy of the image segmentation model.

[0088] In one example, the flowchart of the image segmentation model training method is as shown in Figure 8As shown, based on the alpha channel information of the foreground sample image, the background sample image and the foreground sample image are synthesized to obtain a synthesized sample image, and the foreground mask in the synthesized sample image is labeled. The foreground mask in the synthesized sample image is essentially the alpha channel information corresponding to the foreground. The synthesized sample image and the foreground sample image are input into the initial image segmentation model, and the initial image segmentation model sequentially performs feature extraction, dynamic normalization, template convolution, and segmentation processing to obtain a foreground prediction mask. The foreground prediction mask can be understood as the segmentation result of the initial image segmentation model on the synthesized sample image.

[0089] a. Feature extraction

[0090] The initial image segmentation model extracts features from the synthesized sample image to obtain synthesized sample image features, and extracts features from the foreground sample image to obtain foreground sample image features.

[0091] b. Dynamic normalization

[0092] The synthesized sample image features are normalized to obtain normalized synthesized sample image features. According to the pixel mean and variance of the normalized synthesized sample image features and the foreground sample image features, a target synthesis feature is obtained.

[0093] c. Template convolution

[0094] The target convolution kernel is used to convolve the target synthesis feature to obtain a target convolution feature. The target convolution kernel can be determined according to the information of the pooled foreground sample image. Dynamic convolution of different target synthesis features can be achieved.

[0095] d. Segmentation

[0096] The target convolution feature is decoded and predicted to obtain a foreground prediction mask (i.e., a segmentation result).

[0097] In the training method of the above image segmentation model, by synthesizing single sample image data including only foreground or background, a large number of synthesized sample images in various industrial scenes can be obtained, thereby obtaining a large amount of training data to train the model, and an image segmentation model adapted to different industrial scenes can be obtained under single sample conditions, thereby improving the generalization ability of the image segmentation model. Without any manual semantic annotation, a large amount of training data conforming to the real scene can be obtained, and the model is trained based on the image segmentation process, so that the model can achieve high-precision segmentation tasks.

[0098] In one exemplary embodiment, an image segmentation method is also provided, comprising the following steps:

[0099] Obtain an image to be segmented; input the image to be segmented into the target image segmentation model obtained by the above-mentioned image segmentation model training method to obtain a target foreground mask in the image to be segmented.

[0100] The image to be segmented refers to the image used for foreground segmentation. Typically, the image to be segmented includes both foreground and background. The image to be segmented can be any industrial scene. Segmenting the image to be segmented based on the target image segmentation model accurately obtains the target foreground mask in the image to be segmented.

[0101] In one example, the image to be segmented is Figure 9 As shown, the target foreground mask obtained based on the target image segmentation model is as follows Figure 10 shown.

[0102] It can be seen that in this embodiment, by segmenting the image to be segmented using the target image segmentation model, high-precision segmentation of the image to be segmented in various industrial scenarios can be achieved.

[0103] It should be understood that, although the steps in the flowcharts of the above-mentioned embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above-mentioned embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0104] Based on the same inventive concept, the present application also provides an image segmentation model training device. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more of the following embodiments of the image segmentation model training device can be found in the above-mentioned limitations of the image segmentation model training method and will not be repeated here.

[0105] like Figure 11 As shown, the embodiment of the present application provides an image segmentation model training device 1100, comprising:

[0106] An image acquisition module 1102 is configured to acquire a background sample image and a foreground sample image;

[0107] The image synthesis module 1104 is configured to synthesize the background sample image and the foreground sample image based on the alpha channel information of the foreground sample image to obtain a synthesized sample image, and the synthesized sample image comprises a foreground annotation mask.

[0108] The foreground mask prediction module 1106 is configured to input the synthesized sample image and the foreground sample image into an initial image segmentation model, and predict a foreground prediction mask in the synthesized sample image by using the initial image segmentation model.

[0109] The model parameter adjustment module 1108 is configured to adjust parameters of the initial image segmentation model according to a difference between the foreground prediction mask and the foreground annotation mask until the difference meets a preset difference condition, and obtain a target image segmentation model.

[0110] In some embodiments, in the process of synthesizing the background sample image and the foreground sample image based on the alpha channel information of the foreground sample image to obtain the synthesized sample image, the image synthesis module 1104 is specifically configured to:

[0111] obtain other channel information of the foreground sample image except the alpha channel information;

[0112] perform weighted fusion on the other channel information of the background sample image and the foreground sample image based on the alpha channel information of the foreground sample image to obtain the synthesized sample image.

[0113] In some embodiments, in the process of predicting the foreground prediction mask by using the initial image segmentation model on the foreground mask in the synthesized sample image, the foreground mask prediction module 1106 is specifically configured to:

[0114] perform feature extraction on the synthesized sample image by using the initial image segmentation model to obtain synthesized sample image features, and perform feature extraction on the foreground sample image to obtain foreground sample image features;

[0115] perform distribution alignment processing on the synthesized sample image features to the foreground sample image features to obtain target synthesized features;

[0116] perform convolution processing on the target synthesized features according to a target convolution kernel to obtain target convolution features;

[0117] predict the target convolution features to obtain the foreground prediction mask.

[0118] In some embodiments, in the process of performing distribution alignment processing on the synthesized sample image features to the foreground sample image features to obtain the target synthesized features, the foreground mask prediction module 1106 is specifically configured to:

[0119] The synthetic sample image features are normalized to obtain normalized synthetic sample image features.

[0120] The target synthetic features are obtained according to the normalized synthetic sample image features, the pixel mean value and the variance of the foreground sample image features.

[0121] In some embodiments, the device described above further includes a convolution kernel determination module, which is configured to: perform pooling processing on the foreground sample image to obtain a pooled foreground sample image before performing convolution processing on the target synthetic features according to the target convolution kernel to obtain target convolution features; and determine the target convolution kernel according to information of the pooled foreground sample image.

[0122] In some embodiments, the model parameter adjustment module 1108 is specifically configured to:

[0123] The parameters of the initial image segmentation model are adjusted according to the mean square error loss between the foreground prediction mask and the foreground annotation mask until the mean square error loss is less than a loss threshold, and the target image segmentation model is obtained.

[0124] Embodiments of the present application also provide an image segmentation device, which includes:

[0125] An image acquisition module is configured to acquire a to-be-segmented image.

[0126] An image segmentation module is configured to input the to-be-segmented image into a target image segmentation model obtained by the image segmentation model training method to obtain a target foreground mask in the to-be-segmented image.

[0127] The modules in the image segmentation model training device or the image segmentation device described above can be all or partially realized by software, hardware and combinations thereof. The modules described above can be embedded in or independent of a processor in a computer device in a hardware form, or stored in a memory in a computer device in a software form, so as to be called and executed by a processor to perform the operations corresponding to the modules.

[0128] In some embodiments, a computer device is provided, which can be a server, and an internal structure diagram of the computer device can be as shown in Figure 12As shown in the figure. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control ability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image segmentation model training related data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to realize the steps in the above-mentioned image segmentation model training method.

[0129] In some embodiments, a computer device is provided, which can be a terminal, and its internal structure diagram can be as shown in the figure. Figure 13 As shown in the figure. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control ability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image segmentation model training related data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to realize the steps in the above-mentioned image segmentation model training method. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen; the input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0130] Those skilled in the art can understand that, Figure 12 Or Figure 13The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0131] In some embodiments, a computer device is provided, which includes a memory and a processor. The memory stores a computer program. The processor implements the steps in the above method embodiments when executing the computer program.

[0132] In some embodiments, as Figure 14 shown, an internal structure diagram of a computer readable storage medium is provided. The computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0133] In some embodiments, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0134] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0135] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0136] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0137] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for training an image segmentation model, characterized in that: include: Obtain background sample image and foreground sample image; Based on the alpha channel information of the foreground sample image, synthesizing the background sample image and the foreground sample image to obtain a synthesized sample image; The synthetic sample image includes a foreground annotation mask; Inputting the synthesized sample image and the foreground sample image into an initial image segmentation model, and predicting the foreground mask in the synthesized sample image by the initial image segmentation model to obtain a foreground prediction mask; According to the difference between the foreground prediction mask and the foreground annotation mask, the parameters of the initial image segmentation model are adjusted until the difference meets a preset difference condition, thereby obtaining a target image segmentation model.

2. The method according to claim 1, characterized in that The synthesizing process of the background sample image and the foreground sample image based on the alpha channel information of the foreground sample image to obtain a synthesized sample image includes: Acquire other channel information except the alpha channel information in the foreground sample image; Based on the alpha channel information of the foreground sample image, other channel information of the background sample image and the foreground sample image are weightedly fused to obtain a synthesized sample image.

3. The method according to claim 1, characterized in that The step of predicting the foreground mask in the synthesized sample image by using the initial image segmentation model to obtain the foreground prediction mask includes: Performing feature extraction on the synthetic sample image using the initial image segmentation model to obtain synthetic sample image features, and performing feature extraction on the foreground sample image to obtain foreground sample image features; Performing distribution alignment processing on the synthetic sample image features and the foreground sample image features to obtain target synthetic features; Performing convolution processing on the target synthetic feature according to the target convolution kernel to obtain a target convolution feature; The target convolution feature is predicted to obtain a foreground prediction mask.

4. The method according to claim 3, characterized in that The step of performing distribution alignment processing on the synthetic sample image features and the foreground sample image features to obtain target synthetic features includes: Normalizing the synthesized sample image features to obtain normalized synthesized sample image features; A target synthetic feature is obtained according to the normalized synthetic sample image feature and the pixel mean and variance of the foreground sample image feature.

5. The method according to claim 3, characterized in that Before performing convolution processing on the target synthetic feature according to the target convolution kernel to obtain the target convolution feature, the method further includes: Performing pooling processing on the foreground sample image to obtain a pooled foreground sample image; A target convolution kernel is determined according to information of the pooled foreground sample image.

6. The method according to claim 1, characterized in that The step of adjusting the parameters of the initial image segmentation model according to the difference between the foreground prediction mask and the foreground annotation mask until the difference satisfies a preset difference condition to obtain a target image segmentation model includes: According to the mean square error loss between the foreground prediction mask and the foreground annotation mask, the parameters of the initial image segmentation model are adjusted until the mean square error loss is less than a loss threshold, thereby obtaining a target image segmentation model.

7. An image segmentation method, characterized in that: include: Obtain the image to be segmented; The image to be segmented is input into a target image segmentation model obtained by the image segmentation model training method according to any one of claims 1 to 6 to obtain a target foreground mask in the image to be segmented.

8. An image segmentation model training device, characterized in that: include: An image acquisition module, used for acquiring background sample images and foreground sample images; an image synthesis module, configured to synthesize the background sample image and the foreground sample image based on alpha channel information of the foreground sample image to obtain a synthesized sample image; The synthetic sample image includes a foreground annotation mask; a foreground mask prediction module, configured to input the synthesized sample image and the foreground sample image into an initial image segmentation model, and predict the foreground mask in the synthesized sample image using the initial image segmentation model to obtain a foreground prediction mask; The model parameter adjustment module is used to adjust the parameters of the initial image segmentation model according to the difference between the foreground prediction mask and the foreground annotation mask until the difference meets a preset difference condition, thereby obtaining a target image segmentation model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images