An image target foreground intelligent extraction method based on a semantic segmentation model
Patent Information
- Application Number
- CN202611024345.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-10
- Publication Date
- 2026-10-20
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and more specifically to an intelligent method for extracting image target foreground based on a semantic segmentation model. Background Technology
[0002] Image foreground extraction refers to separating the target object in an image from the background. It is a key foundational technology in applications such as image editing, video compositing, virtual reality, and object recognition. Traditional foreground extraction methods typically rely on low-level features such as color, texture, and edges, or use techniques such as thresholding and background modeling to achieve foreground-background separation.
[0003] However, the aforementioned methods primarily rely on low-level image features, making it difficult to understand the semantic information represented by different regions within the image. In scenarios where the target and background colors are similar, textures are complex, or multiple targets exist, problems such as mis-extraction of the foreground and inaccurate edges often occur, resulting in poor extraction accuracy. Furthermore, related technologies lack effective judgment on the reliability of extraction results when determining foreground regions. When low-confidence regions or isolated mis-segmented regions exist in the semantic segmentation results, it is difficult to effectively filter them, further affecting the accuracy and intelligence of image target foreground extraction. Therefore, there is an urgent need to provide an intelligent image target foreground extraction method that can understand image semantic information and effectively judge the extraction results. Summary of the Invention
[0004] The purpose of this invention is to provide an intelligent image target foreground extraction method based on a semantic segmentation model. This method aims to solve the technical problems of traditional foreground extraction methods, which rely on low-level image features, have difficulty understanding the semantic information of the image, are prone to foreground mis-extraction and inaccurate edges in scenarios where the target and background colors are similar, the texture is complex, or there are multiple targets, and lack effective judgment on the reliability of the extraction results. This invention aims to improve the accuracy and intelligence of image target foreground extraction in complex scenarios.
[0005] To achieve the above objectives, this invention provides an intelligent image target foreground extraction method based on a semantic segmentation model, applied to a foreground extraction device. The foreground extraction device is equipped with a pre-trained semantic segmentation model. The method includes: acquiring an image to be processed; inputting the image to be processed into the semantic segmentation model to obtain semantic segmentation results corresponding to each pixel in the image to be processed, wherein the semantic segmentation results include semantic categories and confidence levels corresponding to the semantic categories; generating a set of determination results based on preset foreground determination rules and the semantic segmentation results; determining a target foreground region in the image to be processed based on the set of determination results, and extracting a target foreground image from the image to be processed based on the target foreground region.
[0006] Optionally, the determination result set includes category determination results. The step of generating the determination result set based on the semantic segmentation results according to the preset foreground determination rules includes: obtaining a preset target category set; determining whether the semantic category of each pixel belongs to the target category set for each pixel, and determining the pixels whose semantic category belongs to the target category set as foreground pixels, thereby obtaining the category determination result.
[0007] Optionally, the step of determining whether the semantic category of each pixel belongs to the target category set, and determining the pixels whose semantic category belongs to the target category set as foreground pixels, to obtain the category determination result, includes: when the target category set contains a single target category, determining the pixels whose semantic category is the same as the target category as foreground pixels; or, when the target category set contains multiple target categories, determining the pixels whose semantic category belongs to any one of the multiple target categories as foreground pixels, to obtain the category determination result.
[0008] Optionally, the set of determination results further includes confidence determination results; the step of generating the set of determination results based on the semantic segmentation results according to the preset foreground determination rules further includes: obtaining the confidence level corresponding to the semantic category of each pixel; and determining the confidence level of each pixel according to the comparison result between the confidence level and the preset confidence threshold to obtain the confidence determination results.
[0009] Optionally, the set of determination results further includes spatial consistency determination results; the step of generating the set of determination results based on the semantic segmentation results according to the preset foreground determination rules further includes: aggregating adjacent foreground pixels into foreground connected regions according to the category determination results of each pixel; and performing spatial consistency determination on the foreground connected regions according to the regional features of the foreground connected regions to obtain the spatial consistency determination results.
[0010] Optionally, the region feature includes the region area. The step of performing spatial consistency determination on the foreground connected region based on the region feature of the foreground connected region to obtain the spatial consistency determination result includes: when the region area of the foreground connected region is greater than or equal to a first area threshold, determining that the foreground connected region passes the spatial consistency determination, and determining the foreground weight corresponding to the foreground connected region as a first foreground weight; when the region area is greater than or equal to a second area threshold and less than the first area threshold, determining that the foreground connected region passes the spatial consistency determination, and determining the foreground weight corresponding to the foreground connected region as a second foreground weight; wherein, the first area threshold is greater than the second area threshold, the first foreground weight is greater than the second foreground weight, and the foreground weight is used to characterize the priority of the foreground connected region belonging to the target foreground region.
[0011] Optionally, the method is applied to target foreground extraction from a video stream, the video stream comprising multiple image frames arranged in temporal order, the image to be processed being the current image frame in the video stream, and the method further comprising: when the same foreground connected region passes the spatial consistency determination in multiple consecutive image frames, controlling the temporal stability corresponding to the foreground connected region to gradually increase; when the same foreground connected region fails the spatial consistency determination in multiple consecutive image frames, controlling the temporal stability to gradually decrease, and when the temporal stability is less than a preset removal threshold, removing the foreground connected region from the target foreground region.
[0012] Optionally, the step of determining the target foreground region in the image to be processed based on the determination result set, and extracting the target foreground image from the image to be processed based on the target foreground region, includes: generating a foreground mask corresponding to the image to be processed based on the determination result set, wherein the foreground mask is used to characterize whether each pixel belongs to the foreground or the background; performing morphological processing and edge optimization on the foreground mask to obtain an optimized foreground mask, and determining the region characterized as the foreground in the optimized foreground mask as the target foreground region; and extracting the target foreground image based on the target foreground region and the image to be processed.
[0013] Optionally, the step of extracting the target foreground image based on the target foreground region and the image to be processed includes: when the image to be processed contains a single target foreground, extracting a single target foreground image from the image to be processed based on the target foreground region; when the image to be processed contains multiple target foregrounds, extracting the target foreground image corresponding to each target foreground from the image to be processed based on each target foreground region; wherein the target foreground image carries a transparent background channel, and the transparent background channel is used to characterize the background region in the target foreground image.
[0014] Optionally, the method further includes: acquiring a target background image; and combining the extracted target foreground image with the target background image based on the position information of the target foreground image in the image to be processed to obtain a target composite image.
[0015] Compared with existing technologies, this invention processes the image by deploying a pre-trained semantic segmentation model, obtaining semantic segmentation results for each pixel, including semantic category and confidence level. This allows foreground extraction to be based on an understanding of the image's semantic information. Compared with traditional methods that rely on low-level features such as color, texture, and edges, it can more accurately distinguish between the target and the background, effectively improving the problem of mis-extraction of the foreground in scenarios where the target and background colors are similar or the textures are complex. Furthermore, based on preset foreground judgment rules, the semantic segmentation results are used to determine the category, confidence level, and spatial consistency, generating a multi-dimensional set of judgment results. Based on this, the target foreground region is determined. This allows for filtering of low-confidence regions using confidence level judgment and removal of isolated mis-segmented regions using spatial consistency judgment, thereby improving the reliability of the extraction results. Further, by performing morphological processing and edge optimization on the foreground mask, the edges of the determined target foreground region are made more accurate. It also supports the separate extraction of single or multiple target foregrounds and the output of foreground images with transparent background channels, thus effectively improving the accuracy, completeness, and intelligence of image target foreground extraction in complex scenes. Attached Figure Description
[0016] Figure 1 A schematic diagram of the foreground extraction device for the intelligent image target foreground extraction method based on semantic segmentation model provided by the present invention; Figure 2 A flowchart illustrating the intelligent image target foreground extraction method based on a semantic segmentation model provided by this invention; Figure 3 This is a schematic diagram of the process for generating category determination results provided by the present invention; Figure 4 This is a schematic diagram of the process for generating spatial consistency determination results provided by the present invention; Figure 5 This is a schematic diagram of the process for extracting the target foreground image provided by the present invention. Detailed Implementation
[0017] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0018] The terms "first," "second," etc., used in the specification and claims of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0019] To more clearly illustrate the technical solution of the present invention, the present invention will be described in detail below with reference to specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.
[0020] This embodiment introduces a semantic segmentation model into the image target foreground extraction process, enabling foreground extraction to be based on an understanding of the image's semantic information. By performing multi-dimensional judgments on the semantic segmentation results, the target foreground region is determined. This allows for accurate differentiation between the target and the background in scenarios where the target and background colors are similar, the texture is complex, or there are multiple targets, thereby improving the accuracy and intelligence of image target foreground extraction.
[0021] See Figure 1The image target foreground intelligent extraction method based on semantic segmentation model described in this embodiment is applied to a foreground extraction device 100, which can be an electronic device with image processing capabilities such as a server, personal computer, or embedded image processing device. The foreground extraction device 100 includes an input interface module 110, a model deployment module 120, an output processing module 130, a storage module 140, and a control module 150. The model deployment module 120 deploys a pre-trained semantic segmentation model 10. The input interface module 110 acquires the image to be processed and transmits it to the model deployment module 120; the model deployment module 120 performs semantic segmentation processing on the image to be processed using the semantic segmentation model 10; the output processing module 130 determines the target foreground region based on the judgment result and extracts the target foreground image; the storage module 140 stores data such as the image to be processed, semantic segmentation results, judgment result set, and preset foreground judgment rules; the control module 150 is connected to each of the above modules and coordinates the operation of each module. It is understood that the above division of modules is only illustrative, and those skilled in the art can merge or split them according to actual needs. This embodiment does not limit this.
[0022] See Figure 2 The image target foreground intelligent extraction method based on semantic segmentation model described in this embodiment includes the following steps.
[0023] Step S10: Obtain the image to be processed, input the image to be processed into the semantic segmentation model, and obtain the semantic segmentation result corresponding to each pixel in the image to be processed. The semantic segmentation result includes the semantic category and the confidence level corresponding to the semantic category.
[0024] The image to be processed can be an image captured in real time by a camera device, or an existing image read from local storage space or the network. Its format can be common image formats such as JPEG, PNG, and BMP. After acquiring the image to be processed through the input interface module 110, the foreground extraction device 100 performs preprocessing such as scaling and normalization to meet the input requirements of the semantic segmentation model 10. For example, it scales the image to be processed to the input resolution required by the model and normalizes the pixel values to a preset range.
[0025] Specifically, the semantic segmentation model 10 is a pre-trained neural network model for pixel-by-pixel semantic classification of images. It can be implemented using known semantic segmentation network structures in the art, such as fully convolutional networks or encoder-decoder structures. For example, the encoder extracts multi-level features from the image to be processed, and the decoder gradually recovers the spatial resolution of the features, ultimately outputting the semantic category of each pixel in the image to be processed. The training process of the semantic segmentation model 10 is as follows: sample images containing the target object are pre-collected, and each pixel in the sample images is labeled with its semantic category to obtain a training sample set with pixel-level annotations. The training sample set is then input into the semantic segmentation network to be trained. The difference between the semantic category output by the network and the labeled semantic category is used as the loss to iteratively update the network parameters until the loss converges or a preset number of training rounds is reached, resulting in the pre-trained semantic segmentation model 10, which is then deployed in the model deployment module 120. It is understood that the semantic segmentation model 10 is a general model available in the art and is not the focus of this embodiment. Those skilled in the art can select appropriate network structures and training methods according to actual needs, which will not be elaborated here.
[0026] Further, after inputting the preprocessed image to be processed into the semantic segmentation model 10, the semantic segmentation model 10 performs pixel-by-pixel semantic classification on each pixel in the image to be processed, and outputs the semantic segmentation result corresponding to each pixel. The semantic segmentation result includes a semantic category and a confidence score corresponding to the semantic category. The semantic category is used to characterize the type of object to which the pixel belongs, such as a person, vehicle, animal, or background. The confidence score is used to characterize the reliability of the pixel being classified into the semantic category, and its value is usually a value between 0 and 1. The larger the value, the higher the probability that the pixel belongs to the semantic category. For example, the semantic segmentation model 10 outputs a set of probability values corresponding to each candidate semantic category for each pixel, takes the candidate semantic category with the highest probability value as the semantic category of the pixel, and uses the highest probability value as the confidence score corresponding to the semantic category. The obtained semantic segmentation result can be stored in the storage module 140 by the control module 150 for subsequent steps.
[0027] Step S20: Based on the preset foreground determination rules, generate a determination result set according to the semantic segmentation results.
[0028] The preset foreground determination rule is a rule pre-configured in the foreground extraction device 100 for determining whether each pixel or region belongs to the foreground, and it can be stored in the storage module 140. The determination result set is a set of determination results obtained after determining based on the semantic segmentation results, which may include one or more of the following: results obtained by determining the semantic category of pixels, results obtained by determining confidence level, and results obtained by determining the spatial distribution of foreground regions.
[0029] Specifically, the foreground extraction device 100 determines the semantic segmentation result obtained in step S10 according to a preset foreground determination rule. For example, it determines whether a pixel belongs to the foreground of interest based on the semantic category of each pixel, determines whether the determination result of the pixel is reliable based on the confidence level of each pixel, and determines whether the region is a valid foreground region based on the spatial distribution of the foreground region. The results obtained from the above determinations are summarized into a determination result set for use in subsequent determination of the target foreground region. It is understood that the specific determination content included in the preset foreground determination rule can be flexibly configured according to the actual application scenario. This embodiment does not limit this, and its specific implementation will be described in detail in subsequent embodiments.
[0030] Step S30: Based on the determination result set, determine the target foreground region in the image to be processed, and extract the target foreground image from the image to be processed based on the target foreground region.
[0031] The target foreground region is the region in the image to be processed that is determined to belong to the foreground, and the target foreground image is the image separated from the image to be processed based on the target foreground region. The foreground extraction device 100, through the output processing module 130, determines the pixels or regions in the image to be processed that are determined to belong to the foreground according to the determination result set obtained in step S20 as the target foreground region, thereby distinguishing the target foreground from the background in the image to be processed.
[0032] For example, after determining the target foreground region, the output processing module 130 retains the image content of the pixels located within the target foreground region in the image to be processed, based on the position of the target foreground region in the image to be processed, and performs corresponding processing on the background region outside the target foreground region, thereby extracting the target foreground image from the image to be processed and outputting it. The extracted target foreground image can be used for subsequent applications such as image editing, video synthesis, and target recognition. Understandably, through the above steps, the foreground extraction device 100, based on understanding the semantic information of the image and combining multi-dimensional judgment of the semantic segmentation results, determines the target foreground region, enabling accurate extraction of the target foreground image in complex scenes and improving the accuracy and intelligence of image target foreground extraction.
[0033] This embodiment introduces a preset set of target categories and determines the category of each pixel accordingly. Based on the semantic information of the image, it can initially distinguish the foreground pixels belonging to the target foreground from the image to be processed, providing an accurate category basis for subsequent determination of the target foreground region, thereby improving the accuracy of foreground extraction.
[0034] See Figure 3 In this embodiment, the set of determination results includes category determination results. The process by which the foreground extraction device 100 generates the set of determination results includes the following steps.
[0035] Step S201: Obtain a preset set of target categories.
[0036] The target category set is a set of semantic categories pre-configured according to actual foreground extraction needs, used to represent the semantic categories corresponding to the foreground of interest. It can be pre-stored in the storage module 140. Specifically, the semantic categories included in the target category set correspond to the semantic categories that the semantic segmentation model 10 can output. It can be flexibly configured according to the application scenario. For example, in a portrait extraction scenario, the target category set can be configured to include people; in a traffic monitoring scenario, the target category set can be configured to include vehicles; and in a multi-target extraction scenario, the target category set can be configured to include both people and vehicles. It can be understood that the target category set can be pre-input and stored by the user through the input interface module 110, or it can be automatically read by the foreground extraction device 100 according to the preset extraction task. This embodiment does not limit this.
[0037] Step S202: For each pixel, determine whether the semantic category of the pixel belongs to the target category set, and determine the pixel whose semantic category belongs to the target category set as a foreground pixel, and obtain the category determination result.
[0038] Specifically, the foreground extraction device 100 sequentially traverses each pixel in the image to be processed, comparing the semantic category corresponding to each pixel with the semantic categories in the target category set. When the semantic category of a pixel belongs to the target category set, the pixel is determined to be a foreground pixel; otherwise, it is determined to be a background pixel. Further, the category determination result can be represented as a label matrix with the same size as the image to be processed. Each element in the label matrix corresponds one-to-one with each pixel in the image to be processed, used to characterize whether the corresponding pixel is a foreground or background pixel. For example, the element corresponding to the foreground pixel can be marked as 1, and the element corresponding to the background pixel can be marked as 0, thereby obtaining the category determination result characterizing the foreground / background affiliation of each pixel, for subsequent use in determining the target foreground region.
[0039] This embodiment provides different methods for determining foreground pixels in different scenarios where the target category set contains a single target category or multiple target categories. This enables the method to be adapted to both the extraction of a single target foreground and the simultaneous extraction of multiple target foregrounds, thereby improving the applicability of the method to different foreground extraction needs.
[0040] When the target category set contains a single target category, pixels with the same semantic category as the target category are determined as foreground pixels; or, when the target category set contains multiple target categories, pixels with a semantic category belonging to any of the multiple target categories are determined as foreground pixels, thus obtaining the category determination result.
[0041] Specifically, when the target foreground is of a single type, the target category set is configured to contain only one target category. In this case, the foreground extraction device 100 determines pixels in the image to be processed that have the same semantic category as the target category as foreground pixels. For example, in a scenario where only human images need to be extracted, the target category set contains only the target category "human". The foreground extraction device 100 then determines pixels in the image to be processed that have the semantic category "human" as foreground pixels and determines the remaining pixels as background pixels.
[0042] Furthermore, when the target foreground has multiple categories, the target category set is configured to include multiple target categories. In this case, the foreground extraction device 100 determines all pixels in the image to be processed whose semantic category belongs to any of the multiple target categories as foreground pixels. For example, in a scenario where both people and vehicles need to be extracted simultaneously, the target category set includes two target categories: people and vehicles. The foreground extraction device 100 then determines all pixels in the image to be processed whose semantic category is people and whose semantic category is vehicles as foreground pixels. It can be understood that by configuring the number of target categories included in the target category set, the method can flexibly adapt to the foreground extraction needs of single or multiple targets.
[0043] This embodiment further introduces confidence judgment results into the judgment result set to filter pixels with low confidence in the semantic segmentation results, thereby avoiding directly using pixels with low semantic classification reliability as foreground, thus improving the reliability of the extraction results.
[0044] The set of determination results in this embodiment also includes confidence determination results. The process of the foreground extraction device 100 generating the set of determination results also includes the following steps.
[0045] Obtain the confidence level corresponding to the semantic category of each pixel.
[0046] Specifically, the foreground extraction device 100 reads the confidence score corresponding to the semantic category of each pixel from the semantic segmentation result. The confidence score is used to characterize the reliability of classifying the pixel into the corresponding semantic category, and its value is usually between 0 and 1. Understandably, the confidence score can be output during the pixel-by-pixel semantic classification process of the semantic segmentation model 10, and stored together with the corresponding semantic category in the storage module 140.
[0047] Based on the comparison result between the confidence level and the preset confidence threshold, the confidence level of each pixel is determined to obtain the confidence level determination result.
[0048] Specifically, the foreground extraction device 100 presets a confidence threshold and compares the confidence of each pixel with the preset confidence threshold. When the confidence of a pixel is greater than or equal to the preset confidence threshold, the pixel is determined to have passed the confidence assessment, indicating that the semantic category of the pixel is reliable. When the confidence of a pixel is less than the preset confidence threshold, the pixel is determined to have failed the confidence assessment, indicating that the semantic category of the pixel is less reliable. For example, the preset confidence threshold can be set to a value between 0.6 and 0.9, such as 0.8, depending on the accuracy requirements of the actual application scenario. Furthermore, for pixels that fail the confidence assessment, they can be excluded from the subsequent determination of the target foreground region, or further verified, thereby avoiding the impact of low-confidence misclassified pixels on the accuracy of the target foreground region.
[0049] This embodiment aggregates adjacent foreground pixels into foreground connected regions and performs spatial consistency determination on the foreground connected regions. This can eliminate isolated and scattered mis-segmented pixels at the region level, thereby improving the accuracy and completeness of the spatial distribution of the determined target foreground region.
[0050] See Figure 4 In this embodiment, the set of determination results also includes spatial consistency determination results. The process of the foreground extraction device 100 generating the set of determination results also includes the following steps.
[0051] Step 501: Based on the category determination results of each pixel, aggregate adjacent foreground pixels into a foreground connected region.
[0052] Specifically, the foreground extraction device 100 determines each foreground pixel in the image to be processed based on the category determination result, and performs connected component analysis on each foreground pixel. Pixels that are adjacent in position and are all determined to be foreground pixels are aggregated into the same foreground connected component, thereby obtaining one or more foreground connected components in the image to be processed. The adjacency can be determined according to the four-neighbor criterion, that is, foreground pixels located above, below, left, and right of a foreground pixel are considered adjacent to it; or it can be determined according to the eight-neighbor criterion, that is, foreground pixels located above, below, left, right, and four diagonal directions of a foreground pixel are considered adjacent to it. It is understood that the above connected component analysis is a known image processing technique in the art, and those skilled in the art can implement it using known methods such as scanning markers, which will not be elaborated here.
[0053] Step 502: Based on the regional characteristics of the foreground connected region, perform spatial consistency determination on the foreground connected region to obtain the spatial consistency determination result.
[0054] Specifically, the regional features are features used to characterize the spatial distribution of foreground connected regions, which may include one or more of the regional area, shape, and position of the foreground connected regions. The foreground extraction device 100 determines whether each foreground connected region is a valid foreground region based on its regional features, thereby obtaining a spatial consistency determination result. For example, for foreground connected regions with excessively small areas, since they are usually composed of isolated mis-segmented pixels, their spatial consistency determination can be deemed unsuccessful; for foreground connected regions with sufficient area, their spatial consistency determination can be deemed successful. Furthermore, through the above spatial consistency determination, foreground connected regions composed of scattered mis-segmented pixels can be eliminated, making the subsequently determined target foreground region more spatially continuous and complete.
[0055] This embodiment sets a first area threshold and a second area threshold, and determines corresponding foreground weights for foreground connected regions with different areas. This allows for the differentiation of the priority of each foreground connected region belonging to the target foreground region while retaining the effective foreground connected regions, thereby further improving the accuracy of the target foreground region determination.
[0056] In this embodiment, the regional features include the regional area, and the process by which the foreground extraction device 100 determines the spatial consistency of the foreground connected region includes the following steps.
[0057] When the area of the foreground connected region is greater than or equal to the first area threshold, the foreground connected region is determined to pass the spatial consistency determination, and the foreground weight corresponding to the foreground connected region is determined to be the first foreground weight.
[0058] Specifically, the area of the region can be represented by the number of foreground pixels contained in the foreground connected region; the more foreground pixels, the larger the area. The foreground extraction device 100 presets a first area threshold. When the area of a certain foreground connected region is greater than or equal to the first area threshold, it indicates that the area of the foreground connected region is large and the probability of it belonging to the target foreground region is high. Then, the foreground connected region is determined to pass the spatial consistency determination, and its corresponding foreground weight is determined as the first foreground weight.
[0059] When the area of the region is greater than or equal to the second area threshold and less than the first area threshold, the foreground connected region is determined to pass the spatial consistency determination, and the foreground weight corresponding to the foreground connected region is determined to be the second foreground weight.
[0060] Specifically, the foreground extraction device 100 also pre-sets a second area threshold. When the area of a certain foreground connected region is greater than or equal to the second area threshold and less than the first area threshold, it indicates that the area of the foreground connected region is moderate and still belongs to a valid foreground connected region. Therefore, the foreground connected region is determined to have passed the spatial consistency determination, and its corresponding foreground weight is determined as the second foreground weight. For example, for a foreground connected region with an area smaller than the second area threshold, since its area is too small, it is usually composed of isolated mis-segmented pixels, and its spatial consistency determination can be determined to have failed, and it can be removed when determining the target foreground region.
[0061] Wherein, the first area threshold is greater than the second area threshold, the first foreground weight is greater than the second foreground weight, and the foreground weight is used to characterize the priority of the foreground connected region belonging to the target foreground region.
[0062] Specifically, the first area threshold is greater than the second area threshold, such that a larger foreground connected region corresponds to a larger first foreground weight, and a moderately sized foreground connected region corresponds to a smaller second foreground weight. The foreground weight is used to characterize the priority of the corresponding foreground connected region belonging to the target foreground region. The larger the foreground weight, the higher the priority of the corresponding foreground connected region belonging to the target foreground region. Further, when determining the target foreground region, the foreground extraction device 100 can prioritize determining the foreground connected region with the larger foreground weight as the target foreground region based on the foreground weight corresponding to each foreground connected region, thereby more accurately determining the target foreground region of interest when multiple foreground connected regions exist. It is understood that the specific values of the first area threshold, the second area threshold, the first foreground weight, and the second foreground weight can all be set by those skilled in the art according to the actual application scenario, and this embodiment does not limit this.
[0063] This embodiment addresses the target foreground extraction scenario in video streams. By statistically analyzing the spatial consistency of the same foreground connected region across multiple consecutive image frames, it dynamically adjusts its temporal stability and eliminates temporally unstable foreground connected regions. This approach can suppress flickering and false detections during video foreground extraction, thereby improving the stability of foreground extraction in video scenarios.
[0064] The method described in this embodiment is applied to target foreground extraction from a video stream, the video stream comprising multiple image frames arranged in chronological order, the image to be processed being the current image frame in the video stream, and the method further comprising the following steps.
[0065] When the same foreground connected region passes the spatial consistency determination in multiple consecutive image frames, the temporal stability corresponding to the foreground connected region gradually increases.
[0066] Specifically, the foreground extraction device 100 determines the foreground connected regions and spatial consistency for each image frame arranged in time sequence in the video stream, and matches the foreground connected regions in adjacent image frames by means of position, degree of overlap, etc., to determine the same foreground connected region corresponding to each other in different image frames. When the same foreground connected region passes the spatial consistency determination in multiple consecutive image frames, it indicates that the foreground connected region is temporally stable. Then, the temporal stability of the foreground connected region is gradually increased. For example, whenever the foreground connected region passes the spatial consistency determination in an image frame, its temporal stability is increased by a preset step size.
[0067] When the same foreground connected region fails the spatial consistency determination in multiple consecutive image frames, the temporal stability is controlled to gradually decrease, and when the temporal stability is less than a preset removal threshold, the foreground connected region is removed from the target foreground region.
[0068] Specifically, when the same foreground connected region fails the spatial consistency determination in multiple consecutive image frames, it indicates that the foreground connected region is temporally unstable and may be an occasional missegmentation region. Therefore, the temporal stability of the corresponding foreground connected region is gradually reduced. For example, whenever the foreground connected region fails the spatial consistency determination in an image frame, its temporal stability is reduced by a preset step size. Further, the foreground extraction device 100 compares the temporal stability of each foreground connected region with a preset rejection threshold. When the temporal stability of a certain foreground connected region is less than the preset rejection threshold, the foreground connected region is removed from the target foreground region, thereby preventing temporally unstable missegmentation regions from being output as the target foreground. It is understood that the specific values of the preset step size and the preset rejection threshold can be set by those skilled in the art according to the actual application scenario, and this embodiment does not limit this.
[0069] This embodiment generates a foreground mask based on the set of determination results, performs morphological processing and edge optimization on the foreground mask, and then determines the target foreground region. This can fill the holes inside the foreground region and smooth the edges of the foreground region, making the extracted target foreground image edges more accurate and the region more complete.
[0070] See Figure 5 In this embodiment, the process by which the foreground extraction device 100 determines the target foreground region and extracts the target foreground image includes the following steps.
[0071] Step S801: Based on the set of determination results, generate a foreground mask corresponding to the image to be processed. The foreground mask is used to characterize whether each pixel belongs to the foreground or the background.
[0072] Specifically, the foreground mask is a binary image with the same size as the image to be processed, and each pixel in the foreground mask corresponds one-to-one with each pixel in the image to be processed. The foreground extraction device 100 determines the pixels belonging to the foreground and the pixels belonging to the background in the image to be processed based on the determination result set, and generates the foreground mask accordingly. For example, the value of the position corresponding to the foreground pixel in the foreground mask can be set to 255, and the value of the position corresponding to the background pixel can be set to 0, thereby using the foreground mask to represent whether each pixel in the image to be processed belongs to the foreground or the background.
[0073] Step S802: Perform morphological processing and edge optimization on the foreground mask to obtain an optimized foreground mask, and determine the region in the optimized foreground mask that is characterized as the foreground as the target foreground region.
[0074] Specifically, the morphological processing may include one or more of erosion, dilation, opening, and closing operations. The foreground extraction device 100, by performing morphological processing on the foreground mask, can remove isolated noise points, fill holes within the foreground region, and connect adjacent foreground regions. Further, the edge optimization is used to smooth the edges between the foreground and background regions in the foreground mask. For example, it can employ feathering, filtering, or other methods known in the art to make the edge transition of the foreground region more natural. After performing morphological processing and edge optimization on the foreground mask, the foreground extraction device 100 obtains an optimized foreground mask and determines the region in the optimized foreground mask that represents the foreground as the target foreground region. It is understood that the specific methods of the above morphological processing and edge optimization are all image processing techniques known in the art, and those skilled in the art can select them according to actual needs; they will not be elaborated further here.
[0075] Step S803: Extract the target foreground image based on the target foreground region and the image to be processed.
[0076] Specifically, the foreground extraction device 100, based on the position of the target foreground region in the image to be processed, retains the image content of pixels located within the target foreground region in the image to be processed, and performs corresponding processing on the background region outside the target foreground region, thereby extracting the target foreground image from the image to be processed. For example, the image to be processed and the optimized foreground mask can be processed pixel-by-pixel, retaining the image content corresponding to the region represented as the foreground in the foreground mask, thereby obtaining the target foreground image.
[0077] This embodiment provides methods for extracting foreground images for different scenarios where the image to be processed contains a single foreground or multiple foreground objects. The extracted foreground images also carry a transparent background channel, which facilitates subsequent image editing and compositing applications and improves the practicality of the method.
[0078] When the image to be processed contains a single target foreground, the single target foreground image is extracted from the image to be processed based on the target foreground region.
[0079] Specifically, when the target foreground in the image to be processed is a single entity, the foreground extraction device 100 extracts the image content corresponding to the target foreground region from the image to be processed based on the determined target foreground region, thereby obtaining a single target foreground image. For example, in a portrait extraction scenario containing only one person, the foreground extraction device 100 extracts the target foreground image corresponding to that person from the image to be processed based on the target foreground region corresponding to that person.
[0080] When the image to be processed contains multiple target foregrounds, the target foreground images corresponding to each target foreground are extracted from the image to be processed according to each target foreground region.
[0081] Specifically, when there are multiple target foregrounds of interest in the image to be processed, the foreground extraction device 100 extracts the image content corresponding to each target foreground region from the image to be processed, thereby obtaining target foreground images corresponding to each target foreground. For example, when the image to be processed contains multiple mutually separate target foregrounds, the connected foreground regions corresponding to each target foreground can be used as the corresponding target foreground regions, and target foreground images corresponding to each target foreground can be extracted accordingly.
[0082] The target foreground image carries a transparent background channel, which is used to characterize the background region in the target foreground image.
[0083] Specifically, the target foreground image carries a transparent background channel, which is used to characterize the background area in the target foreground image other than the target foreground. For example, the extracted target foreground image can be saved as an image format carrying a transparency channel, setting the transparency of the target foreground area in the target foreground image to opaque and the transparency of the background area to transparent, thereby making the background area in the target foreground image transparent, facilitating subsequent image editing or compositing by overlaying the target foreground image onto other images.
[0084] This embodiment achieves background replacement of the target foreground by synthesizing the extracted target foreground image with the target background image, thus expanding the application of the method in scenarios such as image editing and video synthesis.
[0085] The method described in this embodiment also includes the following steps.
[0086] Obtain the target background image.
[0087] Specifically, the target background image is an image used to replace the original background of the image to be processed. The foreground extraction device 100 can obtain the target background image through the input interface module 110. The target background image can be an image pre-stored in the storage module 140 or an image input by the user in real time.
[0088] Based on the position information of the target foreground image in the image to be processed, the extracted target foreground image and the target background image are combined to obtain a target composite image.
[0089] Specifically, the location information is used to characterize the position of the target foreground image in the image to be processed. The foreground extraction device 100 determines the superposition position of the target foreground image on the target background image based on the location information, and superimposes the target foreground image onto the corresponding position of the target background image to obtain the target composite image. For example, when the target foreground image carries a transparent background channel, the image content of the area where the target foreground is located in the target foreground image can be superimposed onto the target background image based on the transparent background channel, while retaining the image content in the target background image at the position corresponding to the background area of the target foreground image, thereby achieving a natural fusion of the target foreground and the target background to obtain the target composite image with background replacement completed. It can be understood that through the above steps, the method can flexibly achieve background replacement based on accurate extraction of the target foreground, meeting diverse application needs such as image editing and video compositing.
[0090] The above description is merely an exemplary embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention specification and drawings under the technical concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.
Claims
1. An intelligent image target foreground extraction method based on a semantic segmentation model, applied to a foreground extraction device, characterized in that, The foreground extraction device is equipped with a pre-trained semantic segmentation model, and the image target foreground intelligent extraction method based on the semantic segmentation model includes: A to-be-processed image is acquired, and the image to be processed is input into the semantic segmentation model to obtain the semantic segmentation result corresponding to each pixel in the image to be processed. The semantic segmentation result includes the semantic category and the confidence level corresponding to the semantic category. Based on the preset foreground determination rules, a set of determination results is generated according to the semantic segmentation results; Based on the set of determination results, the target foreground region in the image to be processed is determined, and the target foreground image is extracted from the image to be processed based on the target foreground region.
2. The image target foreground intelligent extraction method based on semantic segmentation model as described in claim 1, characterized in that, The set of determination results includes category determination results. The step of generating the set of determination results based on the semantic segmentation results according to the preset foreground determination rules includes: Get the preset target category set; For each pixel, determine whether the semantic category of the pixel belongs to the target category set, and determine the pixel whose semantic category belongs to the target category set as a foreground pixel to obtain the category determination result.
3. The image target foreground intelligent extraction method based on semantic segmentation model as described in claim 2, characterized in that, The step of determining whether the semantic category of each pixel belongs to the target category set, and classifying pixels whose semantic category belongs to the target category set as foreground pixels, to obtain the category determination result, includes: When the target category set contains a single target category, pixels with the same semantic category as the target category are determined as foreground pixels; or, when the target category set contains multiple target categories, pixels with a semantic category belonging to any of the multiple target categories are determined as foreground pixels, thus obtaining the category determination result.
4. The image target foreground intelligent extraction method based on semantic segmentation model as described in claim 2, characterized in that, The set of determination results also includes confidence determination results; the step of generating the set of determination results based on the semantic segmentation results according to the preset foreground determination rule further includes: Obtain the confidence level corresponding to the semantic category of each pixel; Based on the comparison result between the confidence level and the preset confidence threshold, the confidence level of each pixel is determined to obtain the confidence level determination result.
5. The image target foreground intelligent extraction method based on semantic segmentation model as described in claim 2, characterized in that, The set of determination results also includes spatial consistency determination results; The step of generating a set of determination results based on the semantic segmentation results according to the preset foreground determination rule further includes: Based on the category determination results of each pixel, adjacent foreground pixels are aggregated into a foreground connected region; Based on the regional characteristics of the foreground connected region, spatial consistency is determined for the foreground connected region to obtain the spatial consistency determination result.
6. The image target foreground intelligent extraction method based on semantic segmentation model as described in claim 5, characterized in that, The region features include region area. The step of performing spatial consistency determination on the foreground connected region based on the region features of the foreground connected region to obtain the spatial consistency determination result includes: When the area of the foreground connected region is greater than or equal to the first area threshold, the foreground connected region is determined to pass the spatial consistency determination, and the foreground weight corresponding to the foreground connected region is determined to be the first foreground weight. When the area of the region is greater than or equal to the second area threshold and less than the first area threshold, it is determined that the foreground connected region passes the spatial consistency determination, and the foreground weight corresponding to the foreground connected region is determined as the second foreground weight. Wherein, the first area threshold is greater than the second area threshold, the first foreground weight is greater than the second foreground weight, and the foreground weight is used to characterize the priority of the foreground connected region belonging to the target foreground region.
7. The image target foreground intelligent extraction method based on semantic segmentation model as described in claim 5, characterized in that, The method is applied to target foreground extraction from a video stream, the video stream comprising multiple image frames arranged in temporal order, the image to be processed being the current image frame in the video stream, and the image target foreground intelligent extraction method based on a semantic segmentation model further includes: When the same foreground connected region passes the spatial consistency determination in multiple consecutive image frames, the temporal stability of the foreground connected region is gradually increased. When the same foreground connected region fails the spatial consistency determination in multiple consecutive image frames, the temporal stability is controlled to gradually decrease, and when the temporal stability is less than a preset removal threshold, the foreground connected region is removed from the target foreground region.
8. The image target foreground intelligent extraction method based on semantic segmentation model as described in claim 2, characterized in that, The step of determining the target foreground region in the image to be processed based on the determination result set, and extracting the target foreground image from the image to be processed based on the target foreground region, includes: Based on the set of determination results, a foreground mask corresponding to the image to be processed is generated. The foreground mask is used to characterize whether each pixel belongs to the foreground or the background. The foreground mask is subjected to morphological processing and edge optimization to obtain an optimized foreground mask, and the region characterized as the foreground in the optimized foreground mask is determined as the target foreground region; The target foreground image is extracted based on the target foreground region and the image to be processed.
9. The image target foreground intelligent extraction method based on semantic segmentation model as described in claim 8, characterized in that, The step of extracting the target foreground image based on the target foreground region and the image to be processed includes: When the image to be processed contains a single target foreground, the single target foreground image is extracted from the image to be processed based on the target foreground region; When the image to be processed contains multiple target foregrounds, the target foreground image corresponding to each target foreground is extracted from the image to be processed according to each target foreground region; The target foreground image carries a transparent background channel, which is used to characterize the background region in the target foreground image.
10. The image target foreground intelligent extraction method based on semantic segmentation model as described in any one of claims 1 to 9, characterized in that, The image target foreground intelligent extraction method based on semantic segmentation model also includes: Obtain the target background image; Based on the position information of the target foreground image in the image to be processed, the extracted target foreground image and the target background image are combined to obtain a target composite image.