Training method and device of image segmentation model, electronic equipment and medium

Abstract

The present disclosure relates to a method and device for training an image segmentation model, electronic equipment and medium, wherein the method comprises: obtaining an image set and an initial image segmentation model; the image set comprises an original image and a ground truth image of the original image; the ground truth image indicates the object category of each pixel point in the original image; performing blur processing on a target region in the original image to obtain a blurred image corresponding to the original image; constructing training data according to each blurred image and the ground truth image of the original image corresponding to the blurred image; and training the initial image segmentation model using the training data; wherein the blurred image is obtained by performing blur processing on the target region in the original image, and then the training data is constructed; and the training data is determined based on continuous multiple frames of images and corresponding labeled data, thereby reducing the training cost of the image segmentation model and improving the training efficiency of the image segmentation model.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a training method, apparatus, electronic device, and medium for an image segmentation model. Background Technology

[0002] Currently, the training methods for image segmentation models involve acquiring multiple consecutive frames of images; fusing the initial frame image with other frames to obtain a blurred image; and then combining the blurred image with the labeled data of the multiple consecutive frames to construct training data for training the image segmentation model.

[0003] Among these challenges, acquiring multiple consecutive frames of images and their corresponding labeled data is difficult, costly, and time-consuming, resulting in high training costs and poor training efficiency for image segmentation models. Summary of the Invention

[0004] This disclosure provides a training method, apparatus, electronic device, and medium for an image segmentation model.

[0005] According to a first aspect of the present disclosure, a method for training an image segmentation model is provided. The method includes: acquiring an image set and an initial image segmentation model; the image set includes an original image and a ground truth image of the original image; the ground truth image indicates the object category of each pixel in the original image; blurring a target region in the original image to obtain a blurred image corresponding to the original image; constructing training data based on each blurred image and the ground truth image of the original image corresponding to the blurred image; and training the initial image segmentation model using the training data.

[0006] In one embodiment of this disclosure, the target region is a region corresponding to at least one target object category in the original image; the method further includes: determining the object category of each pixel in the original image based on the ground truth image of the original image; selecting at least one target object category from multiple object categories; and for each target object category, determining the region composed of pixels in the original image that have the target object category as the target region.

[0007] In one embodiment of this disclosure, selecting at least one target object category from multiple object categories includes: obtaining at least one candidate object category from multiple object categories; the object corresponding to the candidate object category is a movable object; and selecting the target object category from the at least one candidate object category.

[0008] In one embodiment of this disclosure, after determining the region composed of pixels of the target object category in the original image as the target region for each target object category, the method further includes: obtaining a pixel displacement number threshold; performing pixel diffusion processing on the boundary of the target region in the original image according to the pixel displacement number threshold to obtain a processed region in the original image; and updating the target region according to the processed region.

[0009] In one embodiment of this disclosure, blurring a target region in the original image to obtain a blurred image corresponding to the original image includes: determining a blur kernel for the target region; blurring the target region based on the blur kernel to obtain a blurred region corresponding to the target region; and replacing the target region in the original image with the blurred region corresponding to the target region to obtain the blurred image.

[0010] In one embodiment of this disclosure, determining the blur kernel of the target region includes: determining the blur kernel size and rotation angle; obtaining an identity matrix having the blur kernel size; determining the blur kernel of the target region based on the identity matrix, the rotation angle, a radial transformation matrix with a first variable, and the size information of the target region; the first variable being the coordinates of the rotation center point; the coordinates of the rotation center point being determined based on the identity matrix.

[0011] In one embodiment of this disclosure, determining the fuzz kernel size and rotation angle includes: obtaining a preset size range and an angle range; randomly selecting a size from the preset size range as the fuzz kernel size; and randomly selecting an angle from the angle range as the rotation angle.

[0012] In one embodiment of this disclosure, training the initial image segmentation model using the training data includes: inputting the blurred image into the image segmentation model to obtain a predicted value image output by the image segmentation model; the predicted value image indicating the predicted object category of each pixel in the blurred image; determining the value of the loss function based on the predicted value image, the ground truth image of the original image corresponding to the blurred image, and the loss function of the image segmentation model; and adjusting the parameters of the image segmentation model based on the value of the loss function to achieve training.

[0013] According to a second aspect of the present disclosure, an image segmentation method is also provided, the method comprising: acquiring an image to be processed; inputting the image to be processed into a preset image segmentation model to acquire a predicted value image output by the image segmentation model; the predicted value image indicating the predicted object category of each pixel in the image to be processed; the image segmentation model being trained by combining a sample original image, a ground truth image of the sample original image, and a sample blurred image corresponding to the sample original image; the sample blurred image being obtained by performing region blurring processing on the sample original image; and determining an image segmentation result of the image to be processed based on the image to be processed and the predicted value image.

[0014] According to a third aspect of the present disclosure, a training apparatus for an image segmentation model is also provided. The apparatus includes: an acquisition module for acquiring an image set and an initial image segmentation model; the image set includes an original image and a ground truth image of the original image; the ground truth image indicates the object category of each pixel in the original image; a blurring module for blurring a target region in the original image to obtain a blurred image corresponding to the original image; a construction module for constructing training data based on each blurred image and the ground truth image of the original image corresponding to the blurred image; and a training processing module for training the initial image segmentation model using the training data.

[0015] In one embodiment of this disclosure, the target region is a region corresponding to at least one target object category in the original image; the device further includes: a first determining module, a selecting module, and a second determining module; the first determining module is used to determine the object category of each pixel in the original image based on the ground truth image of the original image; the selecting module is used to select at least one target object category from multiple object categories; the second determining module is used to determine the region composed of pixels of the target object category in the original image as the target region for each target object category.

[0016] In one embodiment of this disclosure, the selection module is specifically used to: obtain at least one candidate object category from a plurality of object categories; the object corresponding to the candidate object category is a movable object; and select the target object category from the at least one candidate object category.

[0017] In one embodiment of this disclosure, the apparatus further includes: a diffusion processing module and an update processing module; the acquisition module is further configured to acquire a pixel displacement number threshold; the diffusion processing module is configured to perform pixel diffusion processing on the boundary of the target region in the original image according to the pixel displacement number threshold to obtain a processed region in the original image; the update processing module is configured to update the target region according to the processed region.

[0018] In one embodiment of this disclosure, the blurring module includes a determining unit, a blurring unit, and a replacement processing unit; the determining unit is used to determine the blur kernel of the target region; the blurring unit is used to perform blurring processing on the target region based on the blur kernel to obtain a blurred region corresponding to the target region; the replacement processing unit is used to replace the target region in the original image with the blurred region corresponding to the target region to obtain the blurred image.

[0019] In one embodiment of this disclosure, the determining unit is specifically used to: determine the blur kernel size and rotation angle; obtain an identity matrix having the blur kernel size; determine the blur kernel of the target region based on the identity matrix, the rotation angle, a radial transformation matrix with a first variable, and the size information of the target region; the first variable is the coordinates of the rotation center point; the coordinates of the rotation center point are determined based on the identity matrix.

[0020] In one embodiment of this disclosure, the determining unit is further configured to: obtain a preset size range and an angle range; randomly select a size from the preset size range as the fuzzy kernel size; and randomly select an angle from the angle range as the rotation angle.

[0021] In one embodiment of this disclosure, the training processing module is specifically used to: input the blurred image into the image segmentation model to obtain a predicted value image output by the image segmentation model; the predicted value image indicates the predicted object category of each pixel in the blurred image; determine the value of the loss function based on the predicted value image, the ground truth image of the original image corresponding to the blurred image, and the loss function of the image segmentation model; and perform parameter adjustment processing on the image segmentation model based on the value of the loss function to achieve training.

[0022] According to a fourth aspect of the present disclosure, an image segmentation apparatus is also provided, the apparatus comprising: an acquisition module for acquiring an image to be processed; an input module for inputting the image to be processed into a preset image segmentation model to acquire a predicted value image output by the image segmentation model; the predicted value image indicating the predicted object category of each pixel in the image to be processed; the image segmentation model being trained by combining a sample original image, a ground truth image of the sample original image, and a sample blurred image corresponding to the sample original image; the sample blurred image being obtained by performing region blurring processing on the sample original image; and a determination module for determining the image segmentation result of the image to be processed based on the image to be processed and the predicted value image.

[0023] According to a fifth aspect of the present disclosure, an electronic device is also provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: implement the step of a training method for an image segmentation model as described above; or implement the step of an image segmentation method as described above.

[0024] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium is also provided, wherein when instructions in the storage medium are executed by a processor, the processor is able to perform the training method of the image segmentation model as described above; or, perform the image segmentation method as described above.

[0025] According to a seventh aspect of the present disclosure, a chip is also provided, including one or more interface circuits and one or more processors; the interface circuits are configured to receive signals from a memory of an electronic device and send the signals to the processors, the signals including computer instructions stored in the memory, wherein when the processor executes the computer instructions, the electronic device performs the image segmentation model training method as described above; or performs the image segmentation method as described above.

[0026] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0027] This process involves acquiring an image set and an initial image segmentation model. The image set includes the original image and its ground truth image. The ground truth image indicates the object category of each pixel in the original image. The target region in the original image is blurred to obtain a blurred image corresponding to the original image. Training data is constructed based on each blurred image and its corresponding ground truth image. The initial image segmentation model is then trained using this training data. Specifically, the blurred image is obtained by blurring the target region in the original image, thus constructing the training data. This avoids determining the training data based on multiple consecutive frames of images and their corresponding annotations, thereby reducing the training cost and improving the training efficiency of the image segmentation model.

[0028] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0030] Figure 1 A flowchart illustrating a training method for an image segmentation model according to an embodiment of this disclosure;

[0031] Figure 2 A flowchart illustrating a training method for an image segmentation model according to another embodiment of this disclosure;

[0032] Figure 3 This is a flowchart of an image segmentation method according to an embodiment of the present disclosure;

[0033] Figure 4 This is a schematic diagram of the image to be processed;

[0034] Figure 5 This is a schematic diagram of the image of the first predicted value;

[0035] Figure 6 This is a schematic diagram of the second predicted value image;

[0036] Figure 7 This is a schematic diagram of the structure of a training device for an image segmentation model according to an embodiment of the present disclosure;

[0037] Figure 8 This is a schematic diagram of the structure of an image segmentation apparatus according to an embodiment of the present disclosure;

[0038] Figure 9 This is a structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure;

[0039] Figure 10 This is a schematic diagram of the structure of a chip according to an embodiment of the present disclosure. Detailed Implementation

[0040] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0041] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0042] Currently, the training methods for image segmentation models involve acquiring multiple consecutive frames of images; fusing the initial frame image with other frames to obtain a blurred image; and then combining the blurred image with the labeled data of the multiple consecutive frames to construct training data for training the image segmentation model.

[0043] Among these challenges, acquiring multiple consecutive frames of images and their corresponding labeled data is difficult, costly, and time-consuming, resulting in high training costs and poor training efficiency for image segmentation models.

[0044] Figure 1 This is a flowchart illustrating a training method for an image segmentation model according to an embodiment of the present disclosure. It should be noted that the training method for the image segmentation model in this embodiment can be applied to an image segmentation model training device, which can be configured in an electronic device or chip to enable the electronic device or chip to perform the image segmentation model training function.

[0045] Among them, electronic devices can be any device with computing capabilities, such as personal computers (PCs), mobile terminals, servers, vehicle controllers, etc. Mobile terminals can be, for example, in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, and other hardware devices with various operating systems, touch screens, and / or displays.

[0046] Furthermore, the training device for the image segmentation model can also be software within an electronic device. Software, for example, includes model training software. The following embodiments will use an electronic device as an example for illustration.

[0047] like Figure 1 As shown, the method includes the following steps:

[0048] Step 101: Obtain the image set and the initial image segmentation model; the image set includes the original image and the ground truth image of the original image; the ground truth image indicates the object category of each pixel in the original image.

[0049] In this embodiment of the disclosure, the number of pixels in the ground truth image of the original image is the same as the number of pixels in the original image. Each pixel in the ground truth image of the original image corresponds one-to-one with the pixels in the original image. The numerical value of each pixel in the ground truth image of the original image is used to indicate the object category of the corresponding pixel in the original image.

[0050] In the original ground truth image, the pixel values ​​can be, for example, 0 or 1. Different pixel values ​​represent different object categories.

[0051] The object category refers to the category of objects in the original image. Examples of object categories include people, vehicles, animals, and plants.

[0052] The initial image segmentation model can be any of the following: Fully Convolutional Networks (FCN); DeepLab model combining deep convolutional neural networks and probabilistic graphical models; Semantic segmentation model (Pyramid Scene Parsing Network, PSPNet); or SegNet, a deep network for image semantic segmentation.

[0053] Step 102: Blur the target region in the original image to obtain the blurred image corresponding to the original image.

[0054] In this embodiment of the disclosure, the target region can be the entire region or a local region of the original image. Wherein, if the target region is a local region of the original image, the target region can be any region in the original image, or the target region can be a region composed of pixels of at least one category in the original image.

[0055] In this embodiment of the disclosure, the electronic device may perform step 102 as follows: determine the blur kernel of the target region; perform blur processing on the target region based on the blur kernel to obtain the blurred region corresponding to the target region; and replace the target region in the original image with the blurred region corresponding to the target region to obtain the blurred image.

[0056] The process by which the electronic device determines the blur kernel of the target region can be, for example, as follows: determining the size of the blur kernel and the rotation angle; obtaining an identity matrix with the size of the blur kernel; determining the blur kernel of the target region based on the identity matrix, the rotation angle, the radiative transformation matrix with a first variable set, and the size information of the target region; the first variable being the coordinates of the rotation center point; the coordinates of the rotation center point being determined based on the identity matrix.

[0057] The process by which the electronic device determines the size of the fuzzy kernel and the rotation angle can be, for example, obtaining a preset size range and an angle range; randomly selecting a size from the preset size range as the fuzzy kernel size; and randomly selecting an angle from the angle range as the rotation angle.

[0058] For example, an electronic device can randomly select a blur kernel size k and a rotation angle θ for each target region; where k ranges from (1, K) and θ ranges from (1, N). Based on the blur kernel size k, a k-dimensional identity matrix can be obtained; combined with the k-dimensional identity matrix, the coordinates of the rotation center point (x0, y0) are determined. Based on the rotation angle θ, the coordinates of the rotation center point (x0, y0), and the radiative transformation matrix with the rotation center point coordinates set as variables, the radiative transformation matrix M can be determined. Combining the k-dimensional identity matrix, the radiative transformation matrix M, and the size of the target region, the blur kernel Amb can be determined. The target region is blurred using the blur kernel Amb to obtain the blurred region. The pixels of the target region in the original image are zeroed out, and then the blurred region is overlaid on the target region; after all target regions in the original image have been processed, the blurred image is obtained.

[0059] The formula for the radiative transformation matrix, which includes the coordinates of the rotation center point, can be shown as follows.

[0060]

[0061] Alternatively, electronic devices can determine the blur kernel of the target region using other methods. These other methods include, for example, combining the image's spectral characteristics or motion blur properties to estimate the blur kernel; or, the electronic device can randomly select a blur kernel from a set of selectable blur kernels.

[0062] In the embodiments disclosed herein, it should be noted that for each target region in the original image, different blur kernels are used during the blurring process, which may result in different blurred regions. In other words, multiple blurred regions may be determined for each target region.

[0063] In the case where there is only one target region in the original image, if multiple blurred regions are determined based on the target region, multiple blurred images may be obtained; all of these blurred images can be used to construct training data. Alternatively, if there are multiple target regions in the original image, and multiple blurred regions are determined based on each target region, these blurred regions can be randomly combined to obtain multiple blurred images; all of these blurred images can be used to construct training data.

[0064] For example, suppose there are two target regions in the original image. Based on the first target region, three blurred regions are determined; based on the second target region, five blurred regions are determined. Therefore, for this original image, 3 × 5 blurred images, or 15 blurred images, can be determined, greatly expanding the number of images in the training data and further improving the accuracy of the trained image segmentation model.

[0065] Step 103: Construct training data based on each blurred image and the ground truth image of the original image corresponding to the blurred image.

[0066] In this embodiment of the disclosure, the training data may include multiple blurred images and ground truth images corresponding to the blurred images.

[0067] Step 104: Use training data to train the initial image segmentation model.

[0068] In this embodiment of the disclosure, the electronic device may perform step 104 as follows: input the blurred image into the image segmentation model to obtain the predicted value image output by the image segmentation model; the predicted value image indicates the predicted object category of each pixel in the blurred image; determine the value of the loss function based on the predicted value image, the ground truth image of the original image corresponding to the blurred image, and the loss function of the image segmentation model; and perform parameter adjustment processing on the image segmentation model based on the value of the loss function to achieve training.

[0069] In the image segmentation model training method of this disclosure embodiment, an image set and an initial image segmentation model are obtained. The image set includes an original image and a ground truth image of the original image. The ground truth image indicates the object category of each pixel in the original image. The target region in the original image is blurred to obtain a blurred image corresponding to the original image. Training data is constructed based on each blurred image and the ground truth image of the original image corresponding to the blurred image. The initial image segmentation model is trained using the training data. The blurred image is obtained by blurring the target region in the original image, and then the training data is constructed. This avoids determining the training data based on multiple consecutive frames of images and corresponding annotation data, thereby reducing the training cost of the image segmentation model and improving the training efficiency of the image segmentation model.

[0070] Figure 2 This is a flowchart illustrating a training method for an image segmentation model according to another embodiment of the present disclosure. It should be noted that the training method for the image segmentation model in this embodiment can be applied to an image segmentation model training device, which can be configured in an electronic device or chip to enable the electronic device or chip to perform the training function of the image segmentation model.

[0071] Among them, electronic devices can be any device with computing capabilities, such as personal computers (PCs), mobile terminals, servers, vehicle controllers, etc. Mobile terminals can be, for example, in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, and other hardware devices with various operating systems, touch screens, and / or displays.

[0072] Furthermore, the training device for the image segmentation model can also be software within an electronic device. Software, for example, includes model training software. The following embodiments will use an electronic device as an example for illustration.

[0073] like Figure 2 As shown, the method includes the following steps:

[0074] Step 201: Obtain the image set and the initial image segmentation model; the image set includes the original image and the ground truth image of the original image; the ground truth image indicates the object category of each pixel in the original image.

[0075] Step 202: Determine the object category of each pixel in the original image based on the ground truth image of the original image.

[0076] Step 203: Select at least one target object category from multiple object categories.

[0077] In this embodiment of the disclosure, the process of the electronic device performing step 203 may be, for example, obtaining at least one candidate object category from a plurality of object categories; the object corresponding to the candidate object category is a movable object; and selecting a target object category from at least one candidate object category.

[0078] In this context, a movable object refers to an object that possesses the ability to move; or, an object that can be moved under the influence of external forces. Examples of objects possessing the ability to move include people, vehicles, and animals. Examples of objects that can be moved under the influence of external forces include plants.

[0079] Using the object category of movable objects as the target object category to determine the target region can enhance the segmentation accuracy of the trained image segmentation model for the region where movable objects are located.

[0080] In addition, when selecting a target object category, electronic devices can also select the target object category from the object categories of objects with motion capabilities.

[0081] Step 204: For each target object category, the region consisting of pixels in the original image that have the target object category is determined as the target region.

[0082] In this embodiment of the disclosure, since there may be multiple objects with the same object category in the original image, that is, for each target object category, there may be multiple objects with that target object category in the original image. Accordingly, for each target object category, the electronic device may determine one or more target regions. For example, taking a person as the target object category, there may be one person or multiple people in the original image.

[0083] In this embodiment of the disclosure, since the position may be displaced when the object corresponding to the target object category moves, in order to further obtain accurate training data and improve the accuracy of the trained image segmentation model, the electronic device may also perform the following process after determining the target region: obtain a pixel displacement number threshold; perform pixel diffusion processing on the boundary of the target region in the original image according to the pixel displacement number threshold to obtain the processed region in the original image; and update the target region according to the processed region.

[0084] The pixel displacement threshold indicates the maximum number of pixels that can be displaced. Pixel diffusion processing of the target region boundary in the original image means that for each pixel on the boundary of the target region, if the boundary is vertical, for each pixel on the boundary, the pixels outside the boundary in the same row that are displaced by the threshold number of pixels are identified as pixels in the target region; if the boundary is horizontal, for each pixel on the boundary, the pixels outside the boundary in the same column that are displaced by the threshold number of pixels are identified as pixels in the target region. This pixel diffusion processing expands the size of the target region, ensuring that the extracted target region can cover the motion-affected area.

[0085] Step 205: Blur the target region in the original image to obtain the blurred image corresponding to the original image.

[0086] Step 206: Construct training data based on each blurred image and the ground truth image of the original image corresponding to the blurred image.

[0087] Step 207: Use training data to train the initial image segmentation model.

[0088] In the image segmentation model training method of this disclosure embodiment, an image set and an initial image segmentation model are acquired. The image set includes an original image and a ground truth image of the original image. The ground truth image indicates the object category of each pixel in the original image. Based on the ground truth image of the original image, the object category of each pixel in the original image is determined. At least one target object category is selected from multiple object categories. For each target object category, the region composed of pixels with the target object category in the original image is determined as the target region. The target region in the original image is blurred to obtain a blurred image corresponding to the original image. Training data is constructed based on each blurred image and the ground truth image of the original image corresponding to the blurred image. The initial image segmentation model is trained using the training data. The blurred image is obtained by blurring the target region in the original image, and then the training data is constructed. This avoids determining the training data based on multiple consecutive frames of images and corresponding annotation data, thereby reducing the training cost of the image segmentation model and improving the training efficiency of the image segmentation model.

[0089] Figure 3 This is a flowchart of an image segmentation method according to an embodiment of the present disclosure. It should be noted that the image segmentation method of this embodiment can be applied to an image segmentation apparatus, which can be configured in an electronic device or chip to enable the electronic device or chip to perform image segmentation functions.

[0090] Among them, electronic devices can be any device with computing capabilities, such as personal computers (PCs), mobile terminals, servers, vehicle controllers, etc. Mobile terminals can be, for example, in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, and other hardware devices with various operating systems, touch screens, and / or displays.

[0091] In addition, the image segmentation device can also be software in an electronic device. Software, for example, includes image processing software. The following embodiments will use an electronic device as an example for explanation.

[0092] like Figure 3 As shown, the method includes the following steps:

[0093] Step 301: Obtain the image to be processed.

[0094] Step 302: Input the image to be processed into a preset image segmentation model and obtain the predicted value image output by the image segmentation model; the predicted value image indicates the predicted object category of each pixel in the image to be processed; the image segmentation model is obtained by training the original sample image, the ground truth image of the original sample image, and the blurred sample image corresponding to the original sample image; the blurred sample image is obtained by performing regional blurring processing on the original sample image.

[0095] In this embodiment of the disclosure, the number of pixels in the predicted value image is the same as the number of pixels in the image to be processed. Each pixel in the predicted value image corresponds one-to-one with a pixel in the image to be processed. The numerical value of each pixel in the predicted value image is used to indicate the object category of the corresponding pixel in the image to be processed.

[0096] The process of obtaining the blurred sample image can be referred to Figures 1 to 2 The descriptions in the embodiments are not detailed here.

[0097] Examples are provided. Figure 4 The image shown is a schematic diagram of the image to be processed. Figure 5 This is a schematic diagram of the first predicted image. The first predicted image is the predicted image obtained after processing the image to be processed by combining other image segmentation strategies, such as the image segmentation model in the background technique. Figure 6 This is a schematic diagram of the second predicted image. The second predicted image is the predicted image obtained after processing the image to be processed using the image segmentation model of this disclosure. After... Figure 5 and Figure 6 The comparison, Figure 6 The image segmentation effect is good, and it can effectively distinguish the hand area of ​​the person.

[0098] Step 303: Determine the image segmentation result of the image to be processed based on the image to be processed and the predicted image.

[0099] In this embodiment of the disclosure, the electronic device may perform step 303 as follows: for each object category involved in the image to be processed, the region composed of pixels of that object category in the image to be processed is determined as a region to be segmented; based on multiple regions to be segmented, the image to be processed is segmented to obtain the image segmentation result of the image to be processed.

[0100] In the image segmentation method of this embodiment, the following steps are taken: First, an image to be processed is acquired. Then, the image to be processed is input into a preset image segmentation model to obtain a predicted value image output by the image segmentation model. The predicted value image indicates the predicted object category of each pixel in the image to be processed. The image segmentation model is trained by combining a sample original image, a ground truth image of the sample original image, and a sample blurred image corresponding to the sample original image. The sample blurred image is obtained by performing region blurring processing on the sample original image. Based on the image to be processed and the predicted value image, the image segmentation result of the image to be processed is determined. The sample blurred image used during image segmentation model training is obtained by performing region blurring processing on the sample original image, thereby reducing the cost of acquiring the sample blurred image. This allows for the acquisition of a large number of sample blurred images for model training at low cost, thereby improving the accuracy of the image segmentation model and ultimately improving the accuracy of image segmentation.

[0101] Figure 7 This is a schematic diagram of the structure of a training device for an image segmentation model according to an embodiment of the present disclosure.

[0102] like Figure 7 As shown, the training device for the image segmentation model may include: an acquisition module 701, a blur processing module 702, a construction module 703, and a training processing module 704.

[0103] The acquisition module 701 is used to acquire an image set and an initial image segmentation model; the image set includes an original image and a ground truth image of the original image; the ground truth image indicates the object category of each pixel in the original image; the blurring module 702 is used to blur the target region in the original image to obtain a blurred image corresponding to the original image; the construction module 703 is used to construct training data based on each blurred image and the ground truth image of the original image corresponding to the blurred image; the training processing module 704 is used to train the initial image segmentation model using the training data.

[0104] In one embodiment of this disclosure, the target region is a region corresponding to at least one target object category in the original image; the device further includes: a first determining module, a selecting module, and a second determining module; the first determining module is used to determine the object category of each pixel in the original image based on the ground truth image of the original image; the selecting module is used to select at least one target object category from multiple object categories; the second determining module is used to determine the region composed of pixels of the target object category in the original image as the target region for each target object category.

[0105] In one embodiment of this disclosure, the selection module is specifically used to: obtain at least one candidate object category from a plurality of object categories; the object corresponding to the candidate object category is a movable object; and select the target object category from the at least one candidate object category.

[0106] In one embodiment of this disclosure, the apparatus further includes: a diffusion processing module and an update processing module; the acquisition module 701 is further configured to acquire a pixel displacement number threshold; the diffusion processing module is configured to perform pixel diffusion processing on the boundary of the target region in the original image according to the pixel displacement number threshold to obtain the processed region in the original image; the update processing module is configured to perform update processing on the target region according to the processed region.

[0107] In one embodiment of this disclosure, the blurring module 702 includes a determining unit, a blurring unit, and a replacement processing unit; the determining unit is used to determine the blur kernel of the target region; the blurring unit is used to perform blurring processing on the target region based on the blur kernel to obtain a blurred region corresponding to the target region; the replacement processing unit is used to replace the target region in the original image with the blurred region corresponding to the target region to obtain the blurred image.

[0108] In one embodiment of this disclosure, the determining unit is specifically used to: determine the blur kernel size and rotation angle; obtain an identity matrix having the blur kernel size; determine the blur kernel of the target region based on the identity matrix, the rotation angle, a radial transformation matrix with a first variable, and the size information of the target region; the first variable is the coordinates of the rotation center point; the coordinates of the rotation center point are determined based on the identity matrix.

[0109] In one embodiment of this disclosure, the determining unit is further configured to: obtain a preset size range and an angle range; randomly select a size from the preset size range as the fuzzy kernel size; and randomly select an angle from the angle range as the rotation angle.

[0110] In one embodiment of this disclosure, the training processing module 704 is specifically configured to: input the blurred image into the image segmentation model to obtain a predicted value image output by the image segmentation model; the predicted value image indicates the predicted object category of each pixel in the blurred image; determine the value of the loss function based on the predicted value image, the ground truth image of the original image corresponding to the blurred image, and the loss function of the image segmentation model; and perform parameter adjustment processing on the image segmentation model based on the value of the loss function to achieve training.

[0111] In the training apparatus of the image segmentation model of this embodiment, an image set and an initial image segmentation model are acquired. The image set includes an original image and a ground truth image of the original image. The ground truth image indicates the object category of each pixel in the original image. The target region in the original image is blurred to obtain a blurred image corresponding to the original image. Training data is constructed based on each blurred image and the ground truth image of the original image corresponding to the blurred image. The initial image segmentation model is trained using the training data. The blurred image is obtained by blurring the target region in the original image, and then the training data is constructed. This avoids determining the training data based on multiple consecutive frames of images and corresponding annotation data, thereby reducing the training cost of the image segmentation model and improving the training efficiency of the image segmentation model.

[0112] Figure 8 This is a schematic diagram of the structure of an image segmentation apparatus according to an embodiment of the present disclosure. Figure 8 As shown, the image segmentation device may include: an acquisition module 801, an input module 802, and a determination module 803.

[0113] The system includes an acquisition module 801 for acquiring an image to be processed; an input module 802 for inputting the image to be processed into a preset image segmentation model to acquire a predicted value image output by the image segmentation model; the predicted value image indicates the predicted object category of each pixel in the image to be processed; the image segmentation model is trained by combining a sample original image, a ground truth image of the sample original image, and a sample blurred image corresponding to the sample original image; the sample blurred image is obtained by performing region blurring processing on the sample original image; and a determination module 803 for determining the image segmentation result of the image to be processed based on the image to be processed and the predicted value image.

[0114] In the image segmentation apparatus of this embodiment, an image to be processed is acquired; the image to be processed is input into a preset image segmentation model to acquire a predicted value image output by the image segmentation model; the predicted value image indicates the predicted object category of each pixel in the image to be processed; the image segmentation model is trained by combining a sample original image, a ground truth image of the sample original image, and a sample blurred image corresponding to the sample original image; the sample blurred image is obtained by performing region blurring processing on the sample original image; the image segmentation result of the image to be processed is determined based on the image to be processed and the predicted value image; wherein, the sample blurred image used in the training of the image segmentation model is obtained by performing region blurring processing on the sample original image, thereby reducing the acquisition cost of the sample blurred image, thereby enabling the acquisition of a large number of sample blurred images for model training at low cost, thereby improving the accuracy of the image segmentation model, and thus improving the accuracy of image segmentation.

[0115] According to a fifth aspect of the present disclosure, an electronic device is also provided, comprising: a processor; and a memory for storing processor-executable instructions, wherein the processor is configured to: implement a training method for an image segmentation model or an image segmentation method as described above.

[0116] To implement the above embodiments, this disclosure also proposes a non-transitory computer-readable storage medium.

[0117] When the instructions in the storage medium are executed by the processor, the processor is able to execute the training method or image segmentation method of the image segmentation model as described above.

[0118] To implement the above embodiments, this disclosure also provides a computer program product.

[0119] When the computer program product is executed by the processor of the electronic device, it enables the electronic device to perform the above-described method.

[0120] Figure 9 This is a structural block diagram of an electronic device according to an exemplary embodiment. Figure 9 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0121] like Figure 9As shown, the electronic device 1000 includes a processor 111, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 112 or a program loaded from memory 116 into random access memory (RAM) 113. The RAM 113 also stores various programs and data required for the operation of the electronic device 1000. The processor 111, ROM 112, and RAM 113 are interconnected via a bus 114. An input / output (I / O) interface 115 is also connected to the bus 114.

[0122] The following components are connected to I / O interface 115: memory 116 including hard disks, etc.; and communication section 117 including network interface cards such as local area network (LAN) cards, modems, etc., communication section 117 performs communication processing via a network such as the Internet; and driver 118 is also connected to I / O interface 115 as needed.

[0123] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 117. When the computer program is executed by processor 111, it performs the functions defined in the methods of this disclosure.

[0124] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory including instructions, which can be executed by the processor 111 of the electronic device 1000 to perform the above-described method. Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0125] In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0126] Figure 10 This is a schematic diagram of the structure of a chip according to an embodiment of this disclosure. Figure 10 As shown, the chip includes a processor 1001 and an interface circuit 1002. The number of processors 1001 and the number of interface circuits 1002 can be one or more.

[0127] Optionally, the chip also includes a memory 1003 for storing necessary computer programs and data; an interface circuit 1002 for receiving signals from the memory 1003 and sending signals to the processor 1001, the signals including computer instructions stored in the memory 1003, and when the processor 1001 executes the computer instructions, causing the electronic device to execute the training method or image segmentation method of the image segmentation model described in the above embodiments of this disclosure.

[0128] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0129] Furthermore, the term “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as advantageous compared to other aspects or designs. Rather, the use of the term “exemplary” is intended to present the concept in a concrete manner. As used herein, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from the context, “X applies A or B” is intended to mean any of the natural inclusive arrangements. That is, “X applies A or B” satisfies any of the foregoing instances if X applies A; X applies B; or both X applies A and B. Additionally, unless otherwise specified or clear from the context to refer to the singular form, the articles “a” and “an” as used in this application and the appended claims are generally understood to mean “one or more.”

[0130] Similarly, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding the specification and drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terminology used to describe such components is intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if structurally not equivalent to the disclosed structure. Furthermore, although specific features of this disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desired and advantageous to any given or particular application. Moreover, with regard to the terms “comprising,” “owning,” “having,” “having,” or variations thereof as used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term “including.”

[0131] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0132] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A training method for an image segmentation model, characterized in that, The method includes: Obtain an image set and an initial image segmentation model; the image set includes original images and ground truth images of the original images; the ground truth images indicate the object category of each pixel in the original images; The target region in the original image is blurred to obtain the blurred image corresponding to the original image; Training data is constructed based on each blurred image and the ground truth image of the original image corresponding to the blurred image; The initial image segmentation model is trained using the training data. The step of training the initial image segmentation model using the training data includes: inputting the blurred image into the image segmentation model to obtain a predicted value image output by the image segmentation model; the predicted value image indicating the predicted object category of each pixel in the blurred image; determining the value of the loss function based on the predicted value image, the ground truth image of the original image corresponding to the blurred image, and the loss function of the image segmentation model; and adjusting the parameters of the image segmentation model based on the value of the loss function to achieve training.

2. The method according to claim 1, characterized in that, The target region is a region corresponding to at least one target object category in the original image; the method further includes: Based on the ground truth image of the original image, determine the object category of each pixel in the original image; Select at least one target object category from multiple object categories; For each target object category, the region consisting of pixels in the original image that belong to the target object category is determined as the target region.

3. The method according to claim 2, characterized in that, The step of selecting at least one target object category from multiple object categories includes: Obtain at least one candidate object category from multiple object categories; the object corresponding to the candidate object category is a movable object; Select the target object category from at least one candidate object category.

4. The method according to claim 2, characterized in that, After determining the region consisting of pixels belonging to the target object category in the original image as the target region for each target object category, the method further includes: Obtain the threshold for the number of pixel displacements; According to the pixel displacement number threshold, pixel diffusion processing is performed on the boundary of the target region in the original image to obtain the processed region in the original image. The target region is updated based on the processed region.

5. The method according to claim 1, characterized in that, The step of blurring the target region in the original image to obtain the blurred image corresponding to the original image includes: Determine the fuzzy kernel of the target region; The target region is blurred based on the blur kernel to obtain the blurred region corresponding to the target region. The target region in the original image is replaced by the blurred region corresponding to the target region to obtain the blurred image.

6. The method according to claim 5, characterized in that, The fuzzy kernel for determining the target region includes: Determine the fuzzy kernel size and rotation angle; Obtain an identity matrix with the stated fuzzy kernel size; The blur kernel of the target region is determined based on the identity matrix, the rotation angle, the radiative transformation matrix with a first variable, and the size information of the target region; the first variable is the coordinates of the rotation center point; the coordinates of the rotation center point are determined based on the identity matrix.

7. The method according to claim 6, characterized in that, Determining the fuzzy kernel size and rotation angle includes: Obtain the preset size range and angle range; Randomly select a size from the preset size range as the fuzzy kernel size; Randomly select an angle from the range of angles as the rotation angle.

8. An image segmentation method, characterized in that, The method includes: Obtain the image to be processed; The image to be processed is input into a preset image segmentation model to obtain a predicted value image output by the image segmentation model; the predicted value image indicates the predicted object category of each pixel in the image to be processed; the image segmentation model is trained by combining the original sample image, the ground truth image of the original sample image, and the blurred sample image corresponding to the original sample image; the blurred sample image is obtained by performing region blurring processing on the original sample image. Based on the image to be processed and the predicted image, determine the image segmentation result of the image to be processed; The training method of the image segmentation model includes: inputting the blurred sample image into the image segmentation model to obtain the sample prediction image output by the image segmentation model; the sample prediction image indicates the predicted object category of each pixel in the blurred sample image; determining the value of the loss function based on the sample prediction image, the ground truth image of the original sample image, and the loss function of the image segmentation model; and adjusting the parameters of the image segmentation model based on the value of the loss function to achieve training.

9. A training device for an image segmentation model, characterized in that, The device includes: An acquisition module is used to acquire an image set and an initial image segmentation model; the image set includes original images and ground truth images of the original images; the ground truth images indicate the object category of each pixel in the original images; The blurring module is used to blur the target region in the original image to obtain the blurred image corresponding to the original image. The construction module is used to construct training data based on each blurred image and the ground truth image of the original image corresponding to the blurred image; The training processing module is used to train the initial image segmentation model using the training data; The training processing module is specifically used to: input the blurred image into the image segmentation model to obtain the predicted value image output by the image segmentation model; the predicted value image indicates the predicted object category of each pixel in the blurred image; determine the value of the loss function based on the predicted value image, the ground truth image of the original image corresponding to the blurred image, and the loss function of the image segmentation model; and perform parameter adjustment processing on the image segmentation model based on the value of the loss function to achieve training.

10. An image segmentation apparatus, characterized in that, The device includes: The acquisition module is used to acquire the image to be processed; The input module is used to input the image to be processed into a preset image segmentation model to obtain the predicted value image output by the image segmentation model; the predicted value image indicates the predicted object category of each pixel in the image to be processed; the image segmentation model is trained by combining the original sample image, the ground truth image of the original sample image, and the blurred sample image corresponding to the original sample image; the blurred sample image is obtained by performing region blurring processing on the original sample image. The determining module is used to determine the image segmentation result of the image to be processed based on the image to be processed and the predicted image. The training method of the image segmentation model includes: inputting the blurred sample image into the image segmentation model to obtain the sample prediction image output by the image segmentation model; the sample prediction image indicates the predicted object category of each pixel in the blurred sample image; determining the value of the loss function based on the sample prediction image, the ground truth image of the original sample image, and the loss function of the image segmentation model; and adjusting the parameters of the image segmentation model based on the value of the loss function to achieve training.

11. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured as follows: The steps of implementing the training method of the image segmentation model as described in any one of claims 1 to 7; or, implementing the image segmentation method as described in claim 8.

12. A non-transitory computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor, the processor is able to perform a training method for an image segmentation model as described in any one of claims 1 to 7; or, perform an image segmentation method as described in claim 8.

13. A chip, characterized in that, The device includes one or more interface circuits and one or more processors; the interface circuits are used to receive signals from the memory of the electronic device and send the signals to the processors, the signals including computer instructions stored in the memory, which, when executed by the processor, cause the electronic device to perform the training method of the image segmentation model according to any one of claims 1 to 7; or, to perform the image segmentation method according to claim 8.

Images