Spatial structure-based foreground image segmentation method, device, equipment and medium

Abstract

The application discloses a foreground image segmentation method and device based on a space structure, equipment and a medium, wherein the method comprises the following steps: constructing a target three-dimensional model through multiple shooting images; obtaining an observation image corresponding to an observation visual angle from the target three-dimensional model, and determining a candidate region from the observation image; projecting the candidate region to a three-dimensional space to obtain a corresponding model extraction visual cone, and performing intersection operation on multiple model extraction visual cones to determine a target region; rendering a target image of a target foreground to be segmented based on the target region to obtain a rendered image, determining a target pixel of the target image according to the rendered image, and determining a set of the target pixel as an image of the target foreground. According to the technical scheme of the embodiment, the target region can be obtained to represent the three-dimensional structure of the target foreground, and the target foreground of the target image can be segmented from the space based on the three-dimensional structure of the target foreground, so that the segmentation efficiency of the target foreground can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of 3D modeling technology, specifically to a foreground image segmentation method, apparatus, device, and medium based on spatial structure. Background Technology

[0002] Image segmentation aims to divide the content of an image into multiple regions based on some kind of consistency, such as segmenting the foreground and background of an image based on semantic consistency. Currently, the main method of image segmentation is to segment 2D images based on the similarity relationship between pixels. This makes the accuracy of image segmentation highly dependent on the definition of pixel similarity. When it is necessary to perform foreground segmentation on a series of related images, such as segmenting images of the same scene taken from multiple angles, the correlation information between the images is not easy to extract, the definition of pixel similarity is very difficult, and the accuracy of foreground segmentation cannot be guaranteed. Summary of the Invention

[0003] This application provides a foreground image segmentation method, apparatus, device, and medium based on spatial structure. It can at least ensure that the solution of this application can segment images in three-dimensional space, determine the region of the target foreground through the three-dimensional spatial structure of the image, and segment the foreground image simply and effectively.

[0004] In a first aspect, embodiments of this application provide a foreground image segmentation method based on spatial structure, the method comprising:

[0005] A target 3D model is constructed from multiple captured images, wherein the captured images include the target foreground, and the target 3D model includes the 3D projection of the target foreground in each captured image;

[0006] Multiple preset observation viewpoints are obtained, and observation images corresponding to each observation viewpoint are obtained from the target 3D model. Candidate regions are determined from each observation image, and the candidate regions are used to indicate the projection of the target foreground onto the corresponding observation image.

[0007] Each candidate region is back-projected into three-dimensional space to obtain its corresponding model extraction frustum, and the spatial region obtained by performing an intersection operation on multiple model extraction frustums is determined as the target region.

[0008] A target image of the target foreground to be segmented is obtained, and the target image is rendered based on the target region to obtain a rendered image. The target pixels of the target image are determined according to the rendered image, and the set of the target pixels is determined as the image of the target foreground.

[0009] According to some embodiments of the present invention, determining candidate regions from the various observed images includes:

[0010] Obtain a bounding box in the observed image, the bounding box being used to select the projection of the target foreground in the observed image;

[0011] The area corresponding to the selected border is determined as the candidate area.

[0012] According to some embodiments of the present invention, the target three-dimensional model is a radiation field model, and the construction of the target three-dimensional model from multiple captured images includes:

[0013] A radiation field is constructed based on multiple captured images, wherein the color integral of each radiation field pixel along the observation direction of the radiation field is determined based on the opacity of the radiation field pixel;

[0014] The geometric structure formed by regions in the radiation field whose opacity is greater than a preset opacity threshold is defined as the radiation field model.

[0015] According to some embodiments of the present invention, rendering the target image based on the target region to obtain a rendered image includes:

[0016] Determine the target image's observation direction in the radiation field model;

[0017] Based on the target observation direction, determine the path of the color integral of each image pixel of the target image in the radiation field model;

[0018] The image pixels whose paths of color integration intersect with the target region are determined as rendering pixels, and the set of rendering pixels is determined as the rendering image.

[0019] According to some embodiments of the present invention, determining the target pixels of the target image based on the rendered image includes:

[0020] The segmentation mask corresponding to the rendered image is determined based on the pixel parameter value of each rendered pixel;

[0021] The target pixels of the target image are determined based on the segmentation mask.

[0022] According to some embodiments of the present invention, determining the spatial region obtained by performing intersection operations on the visual cones extracted from multiple models as the target region includes:

[0023] When multiple models are extracted and their visual cones are intersected to obtain multiple spatial regions, the multiple spatial regions are expanded according to preset expansion parameters.

[0024] A connectivity operation is performed on the multiple expanded spatial regions, and the connected regions obtained from the connectivity operation are determined as the target region.

[0025] According to some embodiments of the present invention, determining the connected region obtained by the connectivity operation as the target region includes:

[0026] If the connected region includes an isolated region, the isolated region is removed from the connected region.

[0027] The connected region obtained after deleting the isolated region is determined as the target region.

[0028] Secondly, embodiments of this application provide a foreground image segmentation apparatus based on spatial structure, including at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, which, when executed by the at least one control processor, enables the at least one control processor to perform the foreground image segmentation method based on spatial structure as described in the first aspect.

[0029] Thirdly, embodiments of this application provide an electronic device including the foreground image segmentation apparatus based on spatial structure described in the second aspect.

[0030] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for performing the spatial structure-based foreground image segmentation method as described in the first aspect.

[0031] This application has at least the following beneficial effects: A target 3D model is constructed from multiple captured images, wherein the captured images include a target foreground, and the target 3D model includes 3D projections of the target foreground from each captured image; multiple preset observation viewpoints are obtained, and observation images corresponding to each observation viewpoint are obtained from the target 3D model; candidate regions are determined from each observation image, and the candidate regions are used to indicate the projection of the target foreground onto the corresponding observation image; each candidate region is back-projected into 3D space to obtain its corresponding model extraction frustum, and the spatial region obtained by intersecting multiple model extraction frustums is determined as the target region; a target image of the target foreground to be segmented is obtained, and the target image is rendered based on the target region to obtain a rendered image; target pixels of the target image are determined based on the rendered image, and the set of target pixels is determined as the image of the target foreground. According to the technical solution of this embodiment, the target region can be obtained through the target 3D model to characterize the 3D structure of the target foreground, and then the target foreground can be segmented from the target image in space based on the 3D structure of the target foreground, which can effectively improve the segmentation efficiency of the target foreground. Attached Figure Description

[0032] Figure 1 This is a flowchart of a foreground image segmentation method based on spatial structure proposed in an embodiment of this application;

[0033] Figure 2 This is a flowchart illustrating the process of determining candidate regions by selecting bounding boxes in a foreground image segmentation method based on spatial structure proposed in another embodiment of this application.

[0034] Figure 3 This is a flowchart illustrating the construction of a radiation field model in a foreground image segmentation method based on spatial structure proposed in another embodiment of this application;

[0035] Figure 4 This is a flowchart illustrating image rendering in a foreground image segmentation method based on spatial structure proposed in another embodiment of this application;

[0036] Figure 5 This is a flowchart illustrating the foreground image segmentation method based on spatial structure proposed in another embodiment of this application, which demonstrates foreground segmentation achieved through a segmentation mask.

[0037] Figure 6 This is a flowchart illustrating the dilation and erosion of spatial regions in a foreground image segmentation method based on spatial structure proposed in another embodiment of this application.

[0038] Figure 7 This is a flowchart illustrating the process of deleting isolated regions in a foreground image segmentation method based on spatial structure, as proposed in another embodiment of this application.

[0039] Figure 8 This is a structural diagram of a foreground image segmentation device based on spatial structure proposed in another embodiment of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0041] In some embodiments, although functional modules are divided in the system diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0042] In existing technologies, the main method of image segmentation is to segment 2D images based on the similarity between pixels. This makes the accuracy of image segmentation highly dependent on the definition of pixel similarity. When it is necessary to perform foreground segmentation on a series of related images, such as segmenting images of the same scene taken from multiple angles, it is not easy to extract the correlation information between the images. The definition of pixel similarity is very difficult, and the accuracy of foreground segmentation cannot be guaranteed.

[0043] To at least address the aforementioned problems, this application discloses a foreground image segmentation method, apparatus, device, and medium based on spatial structure. The method involves constructing a target 3D model from multiple captured images, where each captured image includes a target foreground, and the target 3D model includes 3D projections of the target foreground from each captured image. Multiple preset viewing angles are acquired, and observation images corresponding to each viewing angle are obtained from the target 3D model. Candidate regions are determined from each observation image, indicating the projection of the target foreground onto the corresponding observation image. Each candidate region is back-projected into 3D space to obtain its corresponding model extraction frustum. The spatial region obtained by intersecting multiple model extraction frustums is determined as the target region. A target image of the target foreground to be segmented is acquired, and the target image is rendered based on the target region to obtain a rendered image. Target pixels of the target image are determined based on the rendered image, and the set of target pixels is determined as the image of the target foreground. According to the technical solution of this embodiment, the target region can be obtained through the target three-dimensional model to characterize the three-dimensional structure of the target foreground, and then the target foreground can be segmented from the target image in space based on the three-dimensional structure of the target foreground, which can effectively improve the segmentation efficiency of the target foreground.

[0044] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0045] refer to Figure 1 , Figure 1 This is a flowchart of a foreground image segmentation method based on spatial structure proposed in an embodiment of this application. In some embodiments, the foreground image segmentation method based on spatial structure includes, but is not limited to, the following steps:

[0046] Step S110: Construct a target 3D model using multiple captured images. The captured images include the target foreground, and the target 3D model includes the 3D projection of the target foreground from each captured image.

[0047] In some embodiments, the captured images can be multiple images taken from different angles in the same scene. For example, if there is an apple on a table, multiple photos can be taken from different angles with the apple as the foreground. In this embodiment, the shooting angle of the captured images is not limited, as long as there is the same target foreground.

[0048] In some embodiments, as described above, the captured images are two-dimensional images. After obtaining multiple captured images, a target three-dimensional model can be constructed using the radiation field algorithm or the Structure from Motion (SFM) algorithm. The radiation field model refers to a radiation field model generated from multiple unsegmented captured images. The radiation field model includes multiple object models, such as the foreground object model of the target (e.g., the apple example mentioned above), and some mid-ground or background object models, such as the background structures present in the original images, like walls or the ground where the apple is placed. The radiation field model is a novel 3D modeling method that models space using a series of 2D images of a scene, offering advantages such as independence from depth data and realistic rendering. SFM is a technique that obtains camera parameters by analyzing image sequences and then performs 3D reconstruction. Those skilled in the art are motivated to choose the appropriate algorithm for constructing the 3D model based on actual needs; this embodiment does not impose any limitations on this.

[0049] It should be noted that by constructing a target 3D model, the contents of each part of the 2D captured image can be represented in the form of 3D structural information. For example, in the radiation field model, the opacity parameter can be used to represent the geometric structure of the scene. For example, if the opacity parameter of a certain position is large, it can be determined that the position corresponding to that point is occluded, or that the surface of an object passes through it. In this embodiment, by constructing a target 3D model, the 3D structural information of the target 3D model can be used to characterize the target foreground. The approximate area of ​​the target foreground can be determined from the target image to be segmented based on the 3D structural information of the target foreground. There is no need to define pixel consistency, and foreground segmentation can be achieved in a simple and fast way.

[0050] Step S120: Obtain multiple preset observation viewpoints, obtain the observation images corresponding to each observation viewpoint from the target 3D model, determine candidate regions from each observation image, and use the candidate regions to indicate the projection of the target foreground onto the corresponding observation image.

[0051] In some embodiments, the observation viewpoint can be obtained by acquiring user input through a graphical user interface, or multiple observation viewpoints can be pre-set in the corresponding algorithm (such as the radiation field algorithm or SFM algorithm mentioned above), or multiple observation viewpoints can be randomly and automatically selected from the available observation viewpoints after obtaining the target 3D model. The method of selecting the observation viewpoint can be adjusted according to actual needs.

[0052] In some embodiments, the observation viewpoint may include information such as observation direction, observation distance, and observation angle. The observation direction and observation distance are used to determine the specific location of the observation point outside the target 3D model. The observation direction represents the specific direction of the observation point relative to the center of the target 3D model, and the observation distance represents the specific distance of the observation point relative to the center of the target 3D model. The specific location of the observation point and the observation angle are used to generate the observation image. The observation angle represents the field of view angle range of the observation image acquired at the observation point.

[0053] In some embodiments, an observation image corresponding to the observation viewpoint is obtained based on the target 3D model. The observation image is a projection image of the target 3D model on the observation viewpoint, that is, the target 3D model is a 3D model. The target 3D model is rendered in 2D according to the observation viewpoint to obtain a 2D image representation of the target 3D model. The observation image can be understood as a photograph or observation picture of the target 3D model observed by a camera or human eye at the observation viewpoint. The observation image can be a photographed image corresponding to the target 3D model or a real-time rendered image generated by the target 3D model according to the observation viewpoint.

[0054] It should be noted that in this embodiment, multiple observation perspectives are obtained, which can be executed sequentially. Taking the input of observation perspectives in the graphical user interface of the rendering engine as an example, after obtaining the target 3D model, the user can select a perspective in the rendering engine to obtain a 2D observation image. After determining the candidate region including the target foreground in the observation image, the back projection operation in step S130 is performed to obtain the model extraction frustum corresponding to the observation perspective. Then, the above operation is repeated multiple times to obtain multiple candidate regions corresponding to multiple observation perspectives, and multiple model extraction frustums corresponding to multiple candidate regions. Executing them sequentially can reduce the high resource consumption caused by the rendering engine generating multiple model extraction frustums simultaneously. Of course, if resources are sufficient, multiple observation images can also be acquired simultaneously, and after determining the candidate regions of each observation image, back projection can be performed to obtain the model extraction frustum. This embodiment does not impose any limitations on this.

[0055] Step S130: Back-project each candidate region to the three-dimensional space to obtain the corresponding model extraction frustum, and perform an intersection operation on multiple model extraction frustums to determine the spatial region as the target region.

[0056] Those skilled in the art will understand that images obtained by a camera typically have the optical feature of the camera's optical center. In this embodiment, the back projection is the projection of the candidate region back into three-dimensional space after the candidate region has been determined and projected back into three-dimensional space. The projection of the candidate region and the camera's optical center form the visual cone.

[0057] It should be noted that although different images are taken from different angles, and the position and angle of the target foreground in the two-dimensional image are somewhat different, there is a large gap between the three-dimensional spatial structure of the target foreground and the background. Therefore, when the target foreground is selected as the candidate region, the model extracted by back projection is actually a model used to represent the target foreground. However, since one model may not be able to select the entire target foreground, this embodiment performs an intersection operation on multiple model extracted model models. This ensures that the entire three-dimensional structure of the target foreground is included, and also filters out the background part selected in the candidate region, so as to ensure that the target region is the representation of the target foreground in three-dimensional space.

[0058] It should be noted that in this embodiment, after back-projecting the candidate region into 3D space, the 3D coordinate information of the candidate region in the 3D modeling space of the target 3D model includes multiple 3D coordinate points. These multiple 3D coordinate points are connected to form the 3D spatial representation of the candidate region. The view frustum corresponding to the candidate region is defined as a set of points in space, which are required to fall within the candidate region after projection. Specifically, the model extraction view frustum is obtained by back-projecting the candidate region. After determining the 3D coordinate information of the candidate region in the target 3D model, the observation point corresponding to the observation view is obtained. The range of the model extraction view frustum is determined based on the 3D coordinate information of the observation point corresponding to the observation view in the target 3D model and the 3D coordinate information of the candidate region in the target 3D model. Specifically, with the observation point corresponding to the observation view as the origin, multiple rays are drawn from the observation point corresponding to the observation view and the multiple 3D coordinate points corresponding to the candidate selection border. These multiple rays are the multiple edges of the model extraction view frustum, and the cone-shaped range enclosed by the multiple edges is the range corresponding to the model extraction view frustum.

[0059] Step S140: Obtain the target image of the target foreground to be segmented, render the target image based on the target region to obtain a rendered image, determine the target pixels of the target image based on the rendered image, and determine the set of target pixels as the target foreground image.

[0060] In some embodiments, since the candidate region can be selected from the graphical interface of the rendering engine, the target image can be input to the rendering engine for input-output (IO) to obtain the corresponding viewpoint. Since the target region is the representation of the target foreground in three-dimensional space, after the target region in the target image is rendered by the rendering engine, the resulting rendered image is essentially the approximate region of the target region in the target image, that is, the approximate region of the target foreground in the target image. Then, according to the preset filtering rules, the target pixels are selected from the rendered image. For example, the pixel brightness in the rendered image exceeds the threshold as the target pixel. The interference of the background part is excluded, and the target pixel is selected from the target image. The image composed of the resulting set is the image of the target foreground, thus realizing the segmentation of the target foreground in the target image.

[0061] Additionally, in one embodiment, reference is made to Figure 2 , Figure 1 In step S120 of the illustrated embodiment, determining candidate regions from various observed images also includes, but is not limited to, the following steps:

[0062] Step S210: Obtain the bounding box in the observation image. The bounding box is used to select the projection of the target foreground in the observation image.

[0063] Step S220: Determine the area corresponding to the selected border as the candidate area.

[0064] In some embodiments, according to Figure 1 As described in the embodiment, the observation image can be obtained by selecting a viewpoint from the rendering engine. The observation image can be displayed in the graphical interface of the rendering engine. The user can then select the area where the target foreground is actually located, forming a selection border. The rendering engine can obtain the information of the selection border through the graphical interface, surround the projection in the observation image with the polygon corresponding to the selection border, and then determine the internal range corresponding to the polygon information as the candidate region.

[0065] It should be noted that when users select points in the graphical interface, they can either drag a box or select multiple points in the observed image, and the rendering engine will connect the adjacent selected points to form a polygonal border. This polygonal border is then used as the selection border. The specific selection method can be determined according to actual needs, and this embodiment does not impose any restrictions on it.

[0066] It should be noted that, since the technical solution of this embodiment requires multiple candidate regions, one observation image can be displayed in the graphical interface each time. After the selection of the bounding box is completed, the rendering engine automatically performs the subsequent back projection operation, and the graphical interface displays the next observation image for the user to select the bounding box. This process is repeated multiple times to obtain multiple candidate regions and the corresponding model extraction frustum.

[0067] In some embodiments, when the selection border is determined through a graphical user interface, the user can also rotate or scale the observed image through the graphical interface to better determine the area where the target foreground is located.

[0068] In another embodiment, the target three-dimensional model is a radiation field model, referring to... Figure 3 , Figure 1 In step S110 of the illustrated embodiment, constructing a target 3D model using multiple captured images also includes, but is not limited to, the following steps:

[0069] Step S310: Construct a radiation field based on multiple captured images, wherein the color integral of each radiation field pixel along the observation direction of the radiation field is determined based on the opacity of the radiation field pixel.

[0070] Step S320: The geometric structure formed by the regions in the radiation field whose opacity is greater than a preset opacity threshold is determined as the radiation field model.

[0071] It should be noted that, Figure 1 The illustrated examples provide an explanation of the definition of a radiation field, which will not be repeated here.

[0072] It should be noted that the core of the radiation field model is to construct a vector field. The radiation field algorithm can be simply expressed as (o,c)=F(p,v), where p represents the coordinate point of the image pixel in space with a dimension of 3, v represents the direction of the vector with a dimension of 2, o represents the opacity of point p in space, o is usually a specific numerical value, c represents the color of point p as observed from the v direction, with RGB components, and F is the radiation field function.

[0073] It should be noted that in the radiation field model of this embodiment, the color observed by a point x in space along a certain direction d is the color integral along the ray emitted from point x in space as the origin along the direction d. This color integral satisfies the following expression:

[0074]

[0075] It should be noted that the above color integral expression introduces opacity 'o'. Therefore, when constructing the radiation field based on the captured image, the occlusion effect of opacity can be considered. Since the opacity parameter in the radiation field model usually reflects the set structure of the scene, a large 'o' value at a certain position 'p' usually indicates that the surface of an object passes through that position. Therefore, after constructing the radiation field model, by filtering points in space through the opacity threshold, the geometric structure reflected by opacity can be used to perform three-dimensional representation of the content of the captured image, distinguishing the foreground and background by spatial distance. This ensures that the obtained target area can effectively represent the three-dimensional structural information of the target foreground. Then, by rendering the target area, the target foreground is projected onto the target image to determine the approximate area of ​​the target foreground in the target image, effectively improving the efficiency of foreground segmentation.

[0076] Additionally, in one embodiment, reference is made to Figure 4 , Figure 1 In step S140 of the illustrated embodiment, rendering the target image based on the target region to obtain a rendered image also includes, but is not limited to, the following steps:

[0077] Step S410: Determine the target observation direction of the target image in the radiation field model;

[0078] Step S420: Based on the target observation direction, determine the path of the color integral of each image pixel in the radiation field model of the target image;

[0079] Step S430: The image pixels where the path of color integration intersects with the target area are determined as rendering pixels, and the set of rendering pixels is determined as the rendering image.

[0080] In some embodiments, in order to extract the target foreground in the radiation field model, taking the radiation field rendering engine for image rendering as an example, this embodiment performs I / O on the target image in the radiation field rendering engine, determines the viewpoint of the target image in the radiation field model as the target observation direction, and then uses the radiation field rendering engine to render the target region of the observation image in the target observation direction. Since the target region is a three-dimensional representation of the target foreground, the rendered image obtained after rendering is actually a part of the target foreground in the observation image in the target observation direction, thereby realizing the segmentation of the target foreground.

[0081] It should be noted that since the radiation field model is constructed based on the color integration along a ray emitted from a point x in space as the origin along the d direction, the path of the color integration of an image pixel is actually the ray emitted along the d direction from the camera's optical center as the origin and passing through point x in space, as described above. In other words, the rendering in this embodiment controls the path of the color integration of the image pixel in the radiation field model within the target region. Specifically, since the target image is an image captured by the camera, this ray is actually a ray emitted from the camera's optical center as the origin towards point x in space. If this ray, after being extended, intersects with the target region, then the image pixel is the pixel corresponding to the target region, and this pixel is determined as the rendering pixel, thereby realizing the rendering of the target region in the target image.

[0082] Additionally, in one embodiment, reference is made to Figure 5 , Figure 1 In step S140 of the illustrated embodiment, determining the target pixels of the target image based on the rendered image also includes, but is not limited to, the following steps:

[0083] Step S510: Determine the segmentation mask corresponding to the rendered image based on the pixel parameter value of each rendered pixel;

[0084] Step S520: Determine the target pixels of the target image based on the segmentation mask.

[0085] It should be noted that after obtaining the rendered image according to the above embodiments, most of the geometric structure of the background image has been deleted, which is reflected in the rendered image as points without color values. Based on this, this embodiment can further optimize the rendered image, such as filtering based on pixel parameter values. Taking pixel brightness as an example, pixels in the rendered image with deleted background images whose pixel brightness exceeds a certain threshold are determined as pixels corresponding to the target foreground. Of course, other pixel parameter values ​​can also be used for filtering, such as opacity, etc. This embodiment does not limit this much.

[0086] It should be noted that, in order to identify the target pixels, this embodiment can generate a corresponding mask based on these pixels after determining the pixels corresponding to the target foreground through the pixel parameter values. For example, the segmentation mask of pixels with a brightness greater than the threshold is set to 1, and otherwise it is 0. After obtaining the segmentation mask, the segmentation mask is applied to the IO of the target image, so that the target pixels can be segmented from the target image, and then the set of target pixels is determined as the image of the target foreground.

[0087] It should be noted that after obtaining the segmentation mask, those skilled in the art are motivated to add constraints to optimize the segmentation mask according to actual needs, but no further restrictions will be imposed here.

[0088] Additionally, in one embodiment, reference is made to Figure 6 , Figure 1 In step S130 of the illustrated embodiment, the spatial region obtained by performing intersection operations on the view cones extracted from multiple models is determined as the target region, and the steps include, but are not limited to, the following:

[0089] Step S610: When multiple models are extracted and their visual cones are intersected to obtain multiple spatial regions, the multiple spatial regions are dilated according to preset dilation parameters;

[0090] Step S620: Perform a connectivity operation on the multiple expanded spatial regions, and determine the connected region obtained from the connectivity operation as the target region.

[0091] It should be noted that this embodiment uses a radiation field for 3D modeling. Radiation field models usually have a lot of noise. The noise is not related to the actual foreground and background. It is very likely that areas that should be connected are separated by noise. In order to reduce the impact of noise, this embodiment performs dilation and erosion on the spatial region obtained by intersection operation. By setting the dilation parameters, multiple spatial regions are dilated, so that multiple spatial regions are connected to each other after dilation, forming a connected region, thereby improving the accuracy of the target region.

[0092] It should be noted that the expansion parameter can be the size of the spatial region to be expanded, and the spatial region can be expanded by a certain size according to the expansion parameter; the expansion parameter can also be a distance threshold between two spatial regions. When the distance between two spatial regions is less than the distance threshold, it can be determined that the two spatial regions are actually connected regions, and they can be expanded to be connected. The specific connection operation is a technology well known to those skilled in the art, and will not be elaborated here.

[0093] Additionally, in one embodiment, reference is made to Figure 7 , Figure 6 Step S620 of the illustrated embodiment also includes, but is not limited to, the following steps:

[0094] Step S710: When the connected region includes an isolated region, delete the isolated region from the connected region;

[0095] Step S720: The connected region obtained after deleting the isolated region is determined as the target region.

[0096] It should be noted that, according to the description of the above embodiments, after obtaining the connected region by expansion erosion in the spatial region, there may still be certain isolated regions in the connected region, such as noise that expansion erosion failed to eliminate. Based on this, in order to remove isolated regions and achieve noise filtering, this embodiment identifies the isolated regions in the connected region. If a certain region is a small, scattered region, it can be considered as an isolated region isolated from the connected region. Then, the isolated region can be determined as noise, and the connected region obtained after removing it is the target region.

[0097] like Figure 8 As shown, Figure 8 This is a structural diagram of a foreground image segmentation device based on spatial structure according to an embodiment of the present invention. The present invention also provides a foreground image segmentation device based on spatial structure, comprising:

[0098] The processor 801 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0099] The memory 802 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 802 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801 using the spatial structure-based foreground image segmentation method of the embodiments of this application.

[0100] The 803 input / output interface is used to implement information input and output.

[0101] The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0102] Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804);

[0103] The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0104] This application also provides an electronic device, including the foreground image segmentation device based on spatial structure as described above.

[0105] This application embodiment also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described foreground image segmentation method based on spatial structure.

[0106] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate, and may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0107] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0108] The above provides a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

[0109] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network nodes. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0110] Those skilled in the art will understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer-readable storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer-readable storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0111] The above is a detailed description of the preferred embodiments of this application. However, this application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A foreground image segmentation method based on spatial structure, characterized in that, The method includes: A target 3D model is constructed from multiple captured images, wherein the captured images include the target foreground, and the target 3D model includes the 3D projection of the target foreground in each captured image; Multiple preset observation viewpoints are obtained, and observation images corresponding to each observation viewpoint are obtained from the target 3D model. Candidate regions are determined from each observation image, and the candidate regions are used to indicate the projection of the target foreground onto the corresponding observation image. Each candidate region is back-projected into three-dimensional space to obtain its corresponding model extraction frustum, and the spatial region obtained by performing an intersection operation on multiple model extraction frustums is determined as the target region. A target image of the target foreground to be segmented is obtained; the target image is rendered based on the target region to obtain a rendered image; the target pixels of the target image are determined according to the rendered image; and the set of the target pixels is determined as the image of the target foreground. The target 3D model is a radiation field model. The process of constructing the target 3D model using multiple captured images includes: A radiation field is constructed based on multiple captured images, wherein the color integral of each radiation field pixel along the observation direction of the radiation field is determined based on the opacity of the radiation field pixel; The geometric structure formed by regions in the radiation field whose opacity is greater than a preset opacity threshold is defined as the radiation field model; The process of rendering the target image based on the target region to obtain a rendered image includes: Determine the target image's observation direction in the radiation field model; Based on the target observation direction, determine the path of the color integral of each image pixel of the target image in the radiation field model; The image pixels whose paths of color integration intersect with the target region are determined as rendering pixels, and the set of rendering pixels is determined as the rendering image.

2. The foreground image segmentation method based on spatial structure according to claim 1, characterized in that, The step of determining candidate regions from each of the observed images includes: Obtain a bounding box in the observed image, the bounding box being used to select the projection of the target foreground in the observed image; The area corresponding to the selected border is determined as the candidate area.

3. The foreground image segmentation method based on spatial structure according to claim 1, characterized in that, Determining the target pixels of the target image based on the rendered image includes: The segmentation mask corresponding to the rendered image is determined based on the pixel parameter value of each rendered pixel; The target pixels of the target image are determined based on the segmentation mask.

4. The foreground image segmentation method based on spatial structure according to claim 1, characterized in that, The step of determining the spatial region as the target region by performing intersection operations on the extracted visual cones from multiple models includes: When multiple models are extracted and their visual cones are intersected to obtain multiple spatial regions, the multiple spatial regions are expanded according to preset expansion parameters. A connectivity operation is performed on the multiple expanded spatial regions, and the connected regions obtained from the connectivity operation are determined as the target region.

5. The foreground image segmentation method based on spatial structure according to claim 4, characterized in that, The step of determining the connected region obtained from the connectivity operation as the target region includes: If the connected region includes an isolated region, the isolated region is removed from the connected region. The connected region obtained after deleting the isolated region is determined as the target region.

6. A foreground image segmentation device based on spatial structure, characterized in that, It includes at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, which, when executed by the at least one control processor, enable the at least one control processor to perform the spatial structure-based foreground image segmentation method as described in any one of claims 1 to 5.

7. An electronic device, characterized in that, Includes the foreground image segmentation apparatus based on spatial structure as described in claim 6.

8. A computer-readable storage medium storing computer-executable instructions for performing the spatial structure-based foreground image segmentation method as described in any one of claims 1 to 5.

Images