Image segmentation method and device and image segmentation model training method and device

Abstract

The present disclosure relates to an image segmentation method and device, a training method and device for an image segmentation model, an electronic device, and a computer-readable storage medium. The image segmentation method includes: obtaining coordinate information of an image to be segmented and a designated point in the image to be segmented; obtaining a first depth map of the image to be segmented; inputting the image to be segmented, the coordinate information of the designated point in the image to be segmented, and the first depth map into an image segmentation model, and the image segmentation model determines the area corresponding to the plane where the designated point is located in the three-dimensional space in the image to be segmented in the image to be segmented. In this way, accurate image segmentation can be performed without being restricted by image texture and semantic categories.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to an image segmentation method and device, and an image segmentation model training method and device. Background Art

[0002] Image segmentation refers to the process of segmenting an image into regions or objects of interest. Depending on whether the user's participation is required in the segmentation process, image segmentation can be divided into interactive image segmentation and unsupervised (automatic) image segmentation. Interactive image segmentation simply refers to segmenting the region or object desired by the user according to the user's needs. For interactive image segmentation, which segments the region of the plane where the position is located in the three-dimensional space of the image according to the position specified by the user in the image, the traditional method currently relies on color features for diffusion or clustering to predict the region. Among them, the diffusion method is to compare the color features of the current point and the surrounding points starting from the specified point, diffuse according to the similarity of the color features, and finally obtain a region with similar color features near the specified point. The clustering method refers to first randomly generating some cluster centers, and then continuously iterating the region corresponding to each center of the cluster center according to the color features. After convergence and stability, the region where the specified point is located is the prediction result. However, since the traditional method only relies on color features for prediction, it can only deal with the situation where the internal texture of the region where the specified point is located in the three-dimensional space of the image is simple and consistent and has obvious differences in texture from other regions. In addition, the region predicted by the traditional method cannot be guaranteed to be a plane in the three-dimensional space, but is only a region with similar color features.

[0003] In addition, with the gradual deepening of deep learning technology, methods based on deep learning to predict the area through semantic segmentation or instance segmentation have emerged. Semantic segmentation can predict the semantic category of each point in the image, and take the area with the same semantics as the specified point as the prediction result. Instance segmentation is to add a detection link before semantic segmentation, first detect the approximate area of ​​each instance, then perform semantic segmentation on the area, and finally take the area with the same semantics as the specified point as the prediction result. However, the area that can be predicted by the above method is limited by the number of categories, and can only predict limited categories such as walls and ground, and cannot be applied to all planes. In addition, semantic segmentation cannot handle situations where the semantics are the same but not on the same plane. Moreover, instance segmentation can generally only be used for foreground objects, and cannot be applied to backgrounds without obvious boundaries (such as walls). Summary of the invention

[0004] The present disclosure provides an image segmentation method and device, an image segmentation model training method and device, an electronic device, a computer-readable storage medium, and a computer program product to at least solve the above-mentioned problems existing in the prior art.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image segmentation method, which comprises: obtaining coordinate information of an image to be segmented and a designated point in the image to be segmented; obtaining a first depth map of the image to be segmented; inputting the image to be segmented, the coordinate information of the designated point in the image to be segmented, and the first depth map into an image segmentation model, and determining by the image segmentation model an area in the image to be segmented corresponding to a plane in the three-dimensional space of the image to be segmented where the designated point is located.

[0006] Optionally, the step of acquiring a first depth map of the image to be segmented includes: inputting the image to be segmented into a depth estimation network, and obtaining the first depth map by the depth estimation network.

[0007] Optionally, the depth estimation network is obtained by performing supervised training on a convolutional neural network using a depth estimation training image and a noise-free depth map of the depth estimation training image.

[0008] Optionally, the step of determining by the image segmentation model the area in the image to be segmented corresponding to the plane in the three-dimensional space of the specified point in the image to be segmented, includes: obtaining a coordinate image containing the coordinate information based on the coordinate information; inputting the image to be segmented, the coordinate image and the first depth map into the image segmentation model, and determining by the image segmentation model the area in the image to be segmented corresponding to the plane in the three-dimensional space of the image to be segmented, where the specified point is located.

[0009] Optionally, the coordinate image is an image with the same size as the image to be segmented, in which the pixel values ​​of the points corresponding to the coordinate information are 1 and the pixel values ​​of other points are 0.

[0010] Optionally, the image segmentation model includes: a semantic feature encoder, a structural feature encoder and a decoder, wherein the image to be segmented, the coordinate information of the specified point in the image to be segmented and the first depth map are input into the image segmentation model, and the step of determining by the image segmentation model the area corresponding to the plane in the three-dimensional space of the image to be segmented where the specified point is located in the image to be segmented includes: inputting the coordinate information of the image to be segmented and the specified point in the image to be segmented into the semantic feature encoder to obtain the semantic feature of the specified point; inputting the coordinate information of the specified point in the image to be segmented and the first depth map into the structural feature encoder to obtain the structural feature of the specified point; inputting the semantic feature and the structural feature into the decoder to obtain the area corresponding to the plane in the three-dimensional space of the image to be segmented where the specified point is located in the image to be segmented.

[0011] Optionally, before the image to be segmented, the coordinate information of the designated point in the image to be segmented, and the first depth map are input into the image segmentation model, the first depth map is guided filtered using the image to be segmented, and the filtered first depth map, the image to be segmented, and the coordinate information of the designated point in the image to be segmented are input into the image segmentation model, and the image segmentation model determines the area in the image to be segmented corresponding to the plane where the designated point in the three-dimensional space in the image to be segmented is located.

[0012] According to a second aspect of an embodiment of the present disclosure, a method for training an image segmentation model is provided, which includes: obtaining a training image, coordinate information of a specified point in the training image, and a real area in the training image corresponding to a plane where the specified point in the three-dimensional space of the training image is located; obtaining a first depth map of the training image; inputting the training image, the coordinate information of the specified point in the training image, and the first depth map into an image segmentation model, and determining by the image segmentation model a predicted area in the training image corresponding to a plane where the specified point in the three-dimensional space of the training image is located; and adjusting parameters of the image segmentation model based on the difference between the predicted area and the real area, and training the image segmentation model.

[0013] Optionally, the step of obtaining the real area in the training image corresponding to the plane where the specified point in the three-dimensional space in the training image is located includes: obtaining a second noise-free depth map of the training image; and based on the depth information of the second depth map, obtaining the real area in the training image corresponding to the plane where the specified point in the three-dimensional space in the training image is located.

[0014] Optionally, based on the depth information of the second depth map, the step of obtaining the real area in the training image corresponding to the plane where the specified point in the three-dimensional space in the training image is located includes: obtaining the depth values ​​of the specified point and two points adjacent to the specified point according to the depth information of the second depth map and the coordinate information of the specified point; obtaining the equation of the plane where the specified point in the three-dimensional space in the training image is located based on the coordinate information and depth value of the specified point and the two points adjacent to the specified point; determining the point in the second depth map located on the plane according to the equation of the plane, and taking the set of points in the training image corresponding to the point as the real area in the training image corresponding to the plane where the specified point in the three-dimensional space in the training image is located.

[0015] Optionally, the two points adjacent to the designated point include a point adjacent to the designated point in a transverse direction and a point adjacent to the designated point in a longitudinal direction.

[0016] Optionally, the step of determining by the image segmentation model the predicted area in the training image corresponding to the plane in the three-dimensional space of the specified point in the training image includes: obtaining a coordinate image containing the coordinate information based on the coordinate information; inputting the training image, the coordinate image and the first depth map into the image segmentation model, and determining by the image segmentation model the predicted area in the training image corresponding to the plane in the three-dimensional space of the training image where the specified point is located.

[0017] Optionally, the coordinate image is an image with the same size as the training image, in which the pixel values ​​of the points corresponding to the coordinate information are 1 and the pixel values ​​of other points are 0.

[0018] Optionally, the step of acquiring the first depth map of the training image includes: inputting the training image into a depth estimation network, and obtaining the first depth map by the depth estimation network.

[0019] Optionally, the depth estimation network is obtained by performing supervised training on a convolutional neural network using a depth estimation training image and a noise-free depth map of the depth estimation training image.

[0020] Optionally, the image segmentation model includes: a semantic feature encoder, a structural feature encoder and a decoder, wherein the training image, the coordinate information of the designated point in the training image and the first depth map are input into the image segmentation model, and the step of determining by the image segmentation model the predicted area in the training image corresponding to the plane where the designated point in the three-dimensional space of the training image is located includes: inputting the training image and the coordinate information of the designated point in the training image into the semantic feature encoder to obtain the semantic feature of the designated point; inputting the coordinate information of the designated point in the training image and the first depth map into the structural feature encoder to obtain the structural feature of the designated point; inputting the semantic feature and the structural feature into the decoder to obtain the predicted area in the training image corresponding to the plane where the designated point in the three-dimensional space of the training image is located.

[0021] Optionally, before inputting the training image, the coordinate information of the designated point in the training image, and the first depth map into the image segmentation model, the training image is used to perform guided filtering on the first depth map, and the filtered first depth map, the training image, and the coordinate information of the designated point in the training image are input into the image segmentation model, and the image segmentation model determines the predicted area in the training image corresponding to the plane where the designated point in the three-dimensional space in the training image is located.

[0022] According to a third aspect of an embodiment of the present disclosure, an image segmentation device is provided, comprising: a first acquisition unit, configured to acquire coordinate information of an image to be segmented and a designated point in the image to be segmented; a second acquisition unit, configured to acquire a first depth map of the image to be segmented; and a segmentation unit, configured to input the image to be segmented, the coordinate information of the designated point in the image to be segmented, and the first depth map into an image segmentation model, and the image segmentation model determines the area in the image to be segmented corresponding to the plane where the designated point is located in the three-dimensional space in the image to be segmented.

[0023] Optionally, the second acquisition unit is configured to: input the image to be segmented into a depth estimation network, and obtain the first depth map through the depth estimation network.

[0024] Optionally, the depth estimation network is obtained by performing supervised training on a convolutional neural network using a depth estimation training image and a noise-free depth map of the depth estimation training image.

[0025] Optionally, the segmentation unit is configured to: obtain a coordinate image containing the coordinate information based on the coordinate information; input the image to be segmented, the coordinate image and the first depth map into an image segmentation model, and use the image segmentation model to determine the area in the image to be segmented corresponding to the plane where the specified point is located in the three-dimensional space in the image to be segmented.

[0026] Optionally, the coordinate image is an image with the same size as the image to be segmented, in which the pixel values ​​of the points corresponding to the coordinate information are 1 and the pixel values ​​of other points are 0.

[0027] Optionally, the image segmentation model includes: a semantic feature encoder, a structural feature encoder and a decoder, wherein the segmentation unit is configured to: input the image to be segmented and the coordinate information of the designated point in the image to be segmented into the semantic feature encoder to obtain the semantic feature of the designated point; input the coordinate information of the designated point in the image to be segmented and the first depth map into the structural feature encoder to obtain the structural feature of the designated point; input the semantic feature and the structural feature into the decoder to obtain the area in the image to be segmented where the plane in the three-dimensional space of the designated point is located.

[0028] Optionally, the segmentation unit is configured to: before inputting the image to be segmented, the coordinate information of the designated point in the image to be segmented, and the first depth map into the image segmentation model, use the image to be segmented to perform guided filtering on the first depth map acquired by the second acquisition unit, and input the filtered first depth map and the image to be segmented and the coordinate information of the designated point in the image to be segmented into the image segmentation model, and the image segmentation model determines the area in the image to be segmented corresponding to the plane where the designated point is located in the three-dimensional space in the image to be segmented.

[0029] According to a fourth aspect of an embodiment of the present disclosure, a training device for an image segmentation model is provided, comprising: a first acquisition unit, configured to acquire a training image, coordinate information of a designated point in the training image, and a real area in the training image corresponding to a plane where the designated point in the three-dimensional space of the training image is located; a second acquisition unit, configured to acquire a first depth map of the training image; a prediction unit, configured to input the training image, the coordinate information of the designated point in the training image, and the first depth map into an image segmentation model, and the image segmentation model determines a predicted area in the training image corresponding to a plane where the designated point in the three-dimensional space of the training image is located; and a training unit, configured to adjust parameters of the image segmentation model based on the difference between the predicted area and the real area, and train the image segmentation model.

[0030] Optionally, the first acquisition unit is configured to: acquire a noise-free second depth map of the training image; and based on the depth information of the second depth map, acquire a real area in the training image corresponding to the plane where the specified point in the three-dimensional space of the training image is located.

[0031] Optionally, the first acquisition unit is further configured to: acquire the depth values ​​of the designated point and two points adjacent to the designated point according to the depth information of the second depth map and the coordinate information of the designated point; obtain the equation of the plane in which the designated point is located in the three-dimensional space in the training image based on the coordinate information and depth values ​​of the designated point and the two points adjacent to the designated point; determine the point in the second depth map located on the plane according to the equation of the plane, and use the set of points in the training image corresponding to the point as the real area in the training image corresponding to the plane in which the designated point is located in the three-dimensional space in the training image.

[0032] Optionally, the two points adjacent to the designated point include a point adjacent to the designated point in a transverse direction and a point adjacent to the designated point in a longitudinal direction.

[0033] Optionally, the prediction unit is configured to: obtain a coordinate image containing the coordinate information based on the coordinate information; input the training image, the coordinate image and the first depth map into an image segmentation model, and use the image segmentation model to determine the predicted area in the training image corresponding to the plane where the specified point in the three-dimensional space in the training image is located.

[0034] Optionally, the coordinate image is an image with the same size as the training image, in which the pixel values ​​of the points corresponding to the coordinate information are 1 and the pixel values ​​of other points are 0.

[0035] Optionally, the second acquisition unit is configured to: input the training image into a depth estimation network, and obtain the first depth map through the depth estimation network.

[0036] Optionally, the depth estimation network is obtained by performing supervised training on a convolutional neural network using a depth estimation training image and a noise-free depth map of the depth estimation training image.

[0037] Optionally, the image segmentation model includes: a semantic feature encoder, a structural feature encoder and a decoder, wherein the prediction unit is configured to: input the training image and the coordinate information of the designated point in the training image into the semantic feature encoder to obtain the semantic feature of the designated point; input the coordinate information of the designated point in the training image and the first depth map into the structural feature encoder to obtain the structural feature of the designated point; input the semantic feature and the structural feature into the decoder to obtain the predicted area in the training image corresponding to the plane where the designated point is located in the three-dimensional space in the training image.

[0038] Optionally, the prediction unit is configured to: before inputting the training image, the coordinate information of the designated point in the training image and the first depth map into the image segmentation model, use the training image to perform guided filtering on the first depth map acquired by the second acquisition unit, and input the filtered first depth map and the training image and the coordinate information of the designated point in the training image into the image segmentation model, and the image segmentation model determines the predicted area in the training image corresponding to the plane where the designated point is located in the three-dimensional space in the training image.

[0039] According to a fifth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the image segmentation method or the image segmentation model training method as described above.

[0040] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device executes the image segmentation method or the image segmentation model training method as described above.

[0041] According to a seventh aspect of an embodiment of the present disclosure, there is provided a computer program product, comprising computer instructions, wherein when the computer instructions are executed by a processor, the image segmentation method or the image segmentation model training method as described above is implemented.

[0042] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0043] According to the training method and device of the image segmentation model and the image segmentation method and device disclosed in the present invention, the semantic features and spatial structure features of the image are comprehensively considered, and accurate image segmentation can be performed without being restricted by image texture and semantic categories.

[0044] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0046] Figure 1 is a schematic diagram showing an implementation scenario of a training method for an image segmentation model and an image segmentation method according to an exemplary embodiment of the present disclosure;

[0047] Figure 2 is a flowchart illustrating a training method of an image segmentation model according to an exemplary embodiment of the present disclosure.

[0048] Figure 3 is a schematic diagram showing the 8-neighborhood of a specified point P.

[0049] Figure 4 is a schematic diagram illustrating a structure of an image segmentation model according to an exemplary embodiment of the present disclosure.

[0050] Figure 5 is a flowchart illustrating an image segmentation method according to an exemplary embodiment of the present disclosure.

[0051] Figure 6 is an outline diagram illustrating an image segmentation method according to an exemplary embodiment of the present disclosure.

[0052] Figure 7is a diagram showing an image segmentation effect according to an exemplary embodiment of the present disclosure.

[0053] Figure 8 is a block diagram illustrating a training apparatus for an image segmentation model according to an exemplary embodiment of the present disclosure.

[0054] Fig. 9 is a block diagram illustrating an image segmentation apparatus according to an exemplary embodiment of the present disclosure.

[0055] Fig.10 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0056] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0057] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0058] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0059] Currently, in virtual reality applications, there is a need to achieve special effects on a specific plane or track a specific plane. However, there are the following difficulties in segmenting the area corresponding to the plane in the three-dimensional space where the user-specified point is located using existing image segmentation methods: (1) The specified point is arbitrarily specified by the user and is not limited to segmenting common limited categories such as walls and floors, so it needs to have strong versatility; (2) The result of image segmentation is required to be the area corresponding to the plane in the three-dimensional space of the image, which is sometimes difficult to distinguish from other areas in the image in terms of semantics or texture characteristics.

[0060] The present disclosure provides a training method and device for an image segmentation model and an image segmentation method and device, which use an image segmentation model trained based on a training image, coordinate information of a specified point in the training image, a real area in the training image corresponding to a plane where the specified point is located in the three-dimensional space of the training image, and a first depth map of the training image to perform image segmentation. The method not only takes semantic features into consideration but also spatial structural features. Therefore, the method can segment the area corresponding to the plane where any point specified by the user is located, and has strong versatility.

[0061] Figure 1 1 is a schematic diagram showing an implementation scenario of an image segmentation model training method and an image segmentation method according to an exemplary embodiment of the present disclosure. Figure 1 As shown, the implementation scenario includes a server 100, a user terminal 110, a user terminal 120, and a user terminal 130. The user terminals are not limited to the number and types shown in the figure, and include but are not limited to electronic devices such as smart phones, personal computers, virtual reality devices, tablet computers, and may also include any other electronic devices that require image segmentation processing. The server 100 may be a single server, a server cluster consisting of several servers, or a cloud computing platform or a virtualization center.

[0062] The server 100 can be used to train the image segmentation model. Before training, the server 100 needs to obtain enough training sets, which include: training images, coordinate information of designated points in the training images, and the real area corresponding to the plane in the three-dimensional space of the training image where the designated point is located. The amount of various data information in the training set can be arbitrary, so as to enable the training of the image segmentation model to converge. In addition, the server 100 also needs to obtain the first depth map of the training image, so as to introduce the spatial structure features in the training image into the image segmentation model for image segmentation. Then, the server 100 inputs the acquired training image, the coordinate information of the designated point in the training image, and the first depth map into the image segmentation model, and determines the predicted area corresponding to the plane in the three-dimensional space of the training image where the designated point is located in the training image through the image segmentation model (that is, the area corresponding to the plane in the three-dimensional space where the designated point is located is predicted by the image segmentation model). Finally, the server 100 adjusts the parameters of the image segmentation model based on the difference between the predicted area and the real area, thereby training the image segmentation model. In this way, the image segmentation model obtained by training the image segmentation model using the training set can accurately segment the area corresponding to the plane in the three-dimensional space where the specified point is located according to the user's specified point in the image.

[0063] When it is necessary to segment the image to be segmented, the user can use the user terminal 110~130 to set a designated point on the image to be segmented, and then the user terminal 110~130 can determine the area corresponding to the plane where the designated point is located in the three-dimensional space of the image to be segmented according to the coordinate information of the image to be segmented and the designated point and the first depth map of the image to be segmented obtained, through the image segmentation model trained by the server 100. Since the semantic features and spatial structure features are taken into account, accurate image segmentation can be performed. Furthermore, it is possible to achieve special effects on a specific plane in, for example, virtual reality, and tracking of a specific plane in a tracking task. According to an exemplary embodiment of the present disclosure, the user terminal can send the image to be segmented and the relevant information of the designated point to the server for online processing and return the segmentation result, or download the image segmentation model from the server to the local for local processing, and the present disclosure is not limited to this.

[0064] Next, refer to Figure 2 to Figure 9 A training method and apparatus for an image segmentation model and an image segmentation method and apparatus according to exemplary embodiments of the present disclosure are described in detail.

[0065] Figure 2 is a flowchart illustrating a training method of an image segmentation model according to an exemplary embodiment of the present disclosure.

[0066] like Figure 2 As shown, in step S210, the training image, the coordinate information of the designated point in the training image, and the real area in the training image corresponding to the plane where the designated point is located in the three-dimensional space in the training image are obtained. Here, the training image can be any image that can be used for image segmentation model training, for example, it can be a three-channel (RGB) image, etc. Figure 1 The user terminals 110 to 130 in the training image can obtain the training image in various ways. The coordinate information of the specified point in the training image can be obtained by the user through Figure 1 The coordinate information of the point specified by the user terminal 110-130 in the training image. The coordinate information of the specified point may include the coordinate value of the specified point in the training image. The specified point may be a specified pixel point.

[0067] Various methods can be used to obtain the real area in the training image corresponding to the plane where the specified point used for image segmentation model training is located in the three-dimensional space in the training image.

[0068] According to an exemplary embodiment of the present disclosure, the step of obtaining the real area in the training image corresponding to the plane where the specified point is located in the three-dimensional space in the training image may include the step of obtaining a noise-free second depth map of the training image. Here, the depth map may be a map containing depth information collected by a device such as a depth sensor, wherein each pixel value (i.e., depth value) represents an image of the distance from the device such as the depth sensor to each point in the scene. That is, each pixel value in the depth map represents the distance between the corresponding point in the scene and the depth sensor. The noise-free second depth map of the training image can be obtained by various methods. For example, the depth map acquisition method includes: laser radar depth imaging method, computer stereoscopic vision imaging, coordinate measuring machine method, moiré fringe method, structured light method, etc. As an example, the noise-free second depth map can be obtained based on the depth data of the three-dimensional virtual scene. At this time, the corresponding training image is an image obtained based on the three-dimensional virtual scene. The noise-free second depth map obtained at this time is ideal data for obtaining the real area corresponding to the plane where the specified point is located in the three-dimensional space in the training image in the training image. By obtaining the noise-free second depth map, the image segmentation model can be trained more accurately. By acquiring a noise-free second depth map of the training image, the three-dimensional coordinate values ​​of all points in the training image can be obtained (the depth value and the coordinate value in the training image constitute the coordinates in a three-dimensional rectangular coordinate system).

[0069] Then, after obtaining the second depth map of the training image without noise, a step of obtaining the real area in the training image corresponding to the plane where the specified point is located in the three-dimensional space in the training image based on the depth information of the second depth map can be performed. Here, the real area can be obtained based on the depth information of the second depth map by various methods.

[0070] According to an exemplary embodiment of the present disclosure, based on the depth information of the second depth map, the step of obtaining the real area in the training image corresponding to the plane where the specified point is located in the three-dimensional space in the training image may include the step of obtaining the depth values ​​of the specified point and two points adjacent to the specified point according to the depth information of the second depth map and the coordinate information of the specified point. According to an exemplary embodiment of the present disclosure, the two points adjacent to the specified point may include a point adjacent to the specified point in the horizontal direction and a point adjacent to the specified point in the vertical direction. Thus, it is possible to simply determine two adjacent points in the same plane as the specified point. For example, when the specified point is P(x, y), the points adjacent to the specified point P(x, y) may be P2(x+1, y) and P3(x, y+1). It should be understood that the coordinates of the above adjacent points are only an example of an equation for determining a plane, and the coordinates of the pixel points in the 8-neighborhood of the specified point P(x, y) may be used to determine the equation of the plane. Figure 3The figure shows the pixel points in the 8-neighborhood of the specified point P. The 8 pixels around the specified point P are the 8-neighborhood of the specified point P.

[0071] Then, after obtaining the depth values ​​of the designated point and the two points adjacent to the designated point, a step of obtaining the equation of the plane where the designated point is located in the three-dimensional space in the training image based on the coordinate information and depth value of the designated point and the two points adjacent to the designated point can be performed. Here, the depth value is used as the vertical coordinate in the three-dimensional rectangular coordinate system, and the coordinate value in the training image is used as the horizontal coordinate and the vertical coordinate in the three-dimensional rectangular coordinate system respectively. Thus, the three-dimensional coordinate values ​​of the designated point and its two adjacent points are obtained. In this way, the three-dimensional coordinate values ​​of the three points are known to determine the equation of the plane where the three points are located. Then, after obtaining the equation of the plane where the designated point is located, it is possible to determine the points located on the plane in the second depth map according to the equation of the plane, and use the set of points in the training image corresponding to the point as the real area corresponding to the plane where the designated point is located in the three-dimensional space in the training image. By determining the equation of the plane where the designated point is located as described above, since only two points adjacent to the designated point are used, the plane where the designated point is located is more accurate, and thus the real area corresponding to the plane where the designated point is located in the training image is more accurate.

[0072] In this way, the training image, the coordinate information of the designated point in the training image, and the real area in the training image corresponding to the plane where the designated point is located in the three-dimensional space in the training image are obtained. Then, Figure 2 As shown, in step S220, a first depth map of the training image obtained in step S210 is obtained. Here, the first depth map may be a depth map containing noise or a depth map without noise.

[0073] According to an exemplary embodiment of the present disclosure, the first depth map of the training image can be obtained by inputting the training image into a trained depth estimation network. Thus, the depth map of the training image can be simply obtained. The depth estimation network can be obtained by supervised training of a convolutional neural network using a depth estimation training image and a depth map of the depth estimation training image without noise. It can also be obtained by training with other related technologies. By training the depth estimation network with a depth map without noise, the accuracy of the depth map obtained by the depth estimation network is improved.

[0074] Here, as described above, a noise-free depth map of a depth estimation training image can be obtained by various methods. As an example, the noise-free depth map can be obtained based on the depth data of a three-dimensional virtual scene. At this time, the corresponding depth estimation training image is an image obtained based on the three-dimensional virtual scene. In this way, paired training data such as [(training image M, coordinate information Mp of a specified point in the training image, first depth map D of the training image)->the plane where the specified point is located in the three-dimensional space in the training image corresponds to the real area m in the training image] is obtained.

[0075] Next, in step S230, the training image obtained in step S210 and step S220, the coordinate information of the designated point in the training image, and the first depth map are input into the image segmentation model, and the image segmentation model determines the predicted area in the training image corresponding to the plane where the designated point is located in the three-dimensional space in the training image. Here, the image segmentation model may have the following structure: Figure 4 The structure shown.

[0076] Figure 4 4 is a schematic diagram showing the structure of an image segmentation model according to an exemplary embodiment of the present disclosure, wherein the dashed box represents the image segmentation model 400, the left side of the dashed box is the input of the image segmentation model 400, and the right side of the dashed box is the output of the image segmentation model 400.

[0077] like Figure 4 As shown, the image segmentation model 400 may include: a semantic feature encoder 410, a structural feature encoder 420 and a decoder 430. Here, the semantic feature encoder 410, the structural feature encoder 420 and the decoder 430 may be various semantic feature encoders, structural feature encoders and decoders. Among them, the semantic feature encoder 410 is used to extract semantic features, the structural feature encoder 420 is used to extract structural features, and the decoder 430 is used to integrate the two features. Here, the semantic features may represent, for example, texture, color, shape category, etc., and the structural features may represent, for example, three-dimensional structural features, etc.

[0078] In this case, the training image obtained in step S210 and step S220, the coordinate information of the designated point in the training image, and the first depth map are input into the image segmentation model 400, and the step of determining by the image segmentation model 400 the predicted area corresponding to the plane where the designated point in the three-dimensional space in the training image is located in the training image and the predicted area corresponding to the plane in the training image and the first depth map may include: inputting the training image and the coordinate information of the designated point in the training image into the semantic feature encoder 410, obtaining the semantic feature of the designated point through the semantic feature encoder 410, then inputting the coordinate information of the designated point in the training image and the first depth map into the structural feature encoder 420, obtaining the structural feature of the designated point through the structural feature encoder 420; then inputting the semantic feature and the structural feature into the decoder 430, thereby obtaining the predicted area corresponding to the plane where the designated point in the three-dimensional space in the training image is located in the training image and the predicted area corresponding to the plane in the training image and the first depth map. Thus, the semantic information in the training image and the structural information in the depth map can be effectively used for image segmentation.

[0079] Since the first depth map obtained by the above-mentioned trained depth estimation network has some unavoidable errors relative to the depth map obtained by hardware such as the depth sensor, for example, the depth of the plane area will fluctuate, and the depth of the boundary area will have a transition zone, etc. Therefore, according to an exemplary embodiment of the present disclosure, before the training image, the coordinate information of the designated point in the training image, and the first depth map are input into the image segmentation model, the training image can be used to perform guided filtering on the first depth map. Then, the filtered first depth map and the training image, the coordinate information of the designated point in the training image are input into the image segmentation model, and the image segmentation model determines the predicted area in the training image corresponding to the plane where the designated point is located in the three-dimensional space in the training image. By using the training image to perform guided filtering on the first depth map, the boundary of the first depth map is more closely aligned with the training image, and obvious noise is filtered out.

[0080] According to an exemplary embodiment of the present disclosure, the step of determining by an image segmentation model the predicted area in the training image corresponding to the plane where the specified point is located in the three-dimensional space in the training image, may include: based on the coordinate information of the specified point, obtaining a coordinate image containing the coordinate information of the specified point (i.e., converting the coordinate information of the specified point into an image form). Then, the training image, the coordinate image, and the first depth map are input into the image segmentation model, and the image segmentation model determines the predicted area in the training image corresponding to the plane where the specified point is located in the three-dimensional space in the training image. Thus, the coordinate information can be input into the convolutional neural network (i.e., the image segmentation model) in the form of an image in the same manner as the training image and the first depth map, which facilitates model training.

[0081] The coordinate image may be any image containing coordinate information of a specified point. According to an exemplary embodiment of the present disclosure, the coordinate image may be an image of the same size as the training image, in which the pixel value of the point corresponding to the coordinate information of the specified point is 1 and the pixel value of other points is 0. Thus, model training can be further facilitated.

[0082] After determining the predicted region in the training image corresponding to the plane where the specified point is located in the three-dimensional space, step S240 is performed to adjust the parameters of the image segmentation model 400 based on the difference between the predicted region and the true region, and train the image segmentation model 400. Specifically, the parameters of the image segmentation model 400 are adjusted in such a way that the predicted region obtained by the image segmentation model 400 converges to the true region in the training image corresponding to the plane where the specified point is located in the three-dimensional space, and the training is performed using paired training data in the training set. For example, a loss function is used to calculate the difference between the predicted region obtained by the image segmentation model and the true region, and the parameters of the image segmentation model 400 are adjusted based on the size of the difference to minimize the difference.

[0083] The image segmentation model 400 obtained by the training method according to the exemplary embodiment of the present disclosure effectively utilizes the semantic information of the training image and the structural information in the depth map, so that it can accurately segment the area corresponding to the plane where the specified point is located in the three-dimensional space of the image according to the user's specified point in the image.

[0084] Figure 5 is a flowchart illustrating an image segmentation method according to an exemplary embodiment of the present disclosure.

[0085] like Figure 5 As shown, in step S510, the image to be segmented and the coordinate information of the designated point in the image to be segmented are obtained. Here, the image to be segmented can be an image showing a plane in a three-dimensional space, such as a three-channel (RGB) image. Figure 1 The user terminals 110 to 130 in the embodiment obtain the image to be segmented in various ways. The coordinate information of the specified point in the image to be segmented can be obtained by the user through Figure 1 The coordinate information of the point specified by the user terminal 110-130 in the image to be segmented may include the coordinate value of the specified point in the image to be segmented. The specified point may be a specified pixel point.

[0086] Next, in step S520, the first depth map of the image to be segmented obtained in step S510 is obtained. Here, the first depth map may be a depth map containing noise or a depth map without noise.

[0087] According to an exemplary embodiment of the present disclosure, the first depth map of the image to be segmented can be obtained by inputting the image to be segmented into a trained depth estimation network. Thus, the depth map of the image to be segmented can be simply obtained. The depth estimation network can be obtained by supervised training of a convolutional neural network using a depth estimation training image and a depth map of the depth estimation training image without noise. It can also be obtained by training with other related technologies. By training the depth estimation network with a depth map without noise, the accuracy of the depth map obtained by the depth estimation network is improved.

[0088] Here, as described above, a noise-free depth map of a depth estimation training image can be obtained by various methods. As an example, the noise-free depth map can be obtained based on the depth data of a three-dimensional virtual scene. In this case, the corresponding depth estimation training image is an image obtained based on the three-dimensional virtual scene.

[0089] Next, in step S530, the image to be segmented obtained in step S510 and step S520, the coordinate information of the specified point in the image to be segmented, and the first depth map are input into the image segmentation model, and the image segmentation model determines the area in the image to be segmented corresponding to the plane where the specified point is located in the three-dimensional space in the image to be segmented.

[0090] Here, the image segmentation model 400 may have Figure 4 As described above, the image segmentation model 400 may include: a semantic feature encoder 410, a structural feature encoder 420, and a decoder 430. Here, the semantic feature encoder 410, the structural feature encoder 420, and the decoder 430 may be various semantic feature encoders, structural feature encoders, and decoders. Among them, the semantic feature encoder 410 is used to extract semantic features, the structural feature encoder 420 is used to extract structural features, and the decoder 430 is used to integrate the two features. Here, the semantic features may represent, for example, texture, color, shape category, etc., and the structural features may represent, for example, three-dimensional structural features, etc.

[0091] In this case, the image to be segmented, the coordinate information of the designated point in the image to be segmented, and the first depth map obtained in step S510 and step S520 are input into the image segmentation model 400, and the step of determining by the image segmentation model 400 the region corresponding to the plane where the designated point is located in the three-dimensional space in the image to be segmented and in the image to be segmented, may include: inputting the image to be segmented and the coordinate information of the designated point in the image to be segmented into the semantic feature encoder 410, obtaining the semantic feature of the designated point through the semantic feature encoder 410, then inputting the coordinate information of the designated point in the image to be segmented and the first depth map into the structural feature encoder 420, obtaining the structural feature of the designated point through the structural feature encoder 420; then inputting the semantic feature and the structural feature into the decoder 430, thereby obtaining the region corresponding to the plane where the designated point is located in the three-dimensional space in the image to be segmented and in the image to be segmented. Thus, the semantic information in the image to be segmented and the structural information in the depth map can be effectively used for image segmentation.

[0092] Since the first depth map obtained by the above-mentioned trained depth estimation network has some unavoidable errors relative to the depth map obtained by hardware such as the depth sensor, for example, the depth of the plane area will fluctuate, and the depth of the boundary area will have a transition zone, etc. Therefore, according to an exemplary embodiment of the present disclosure, before the image to be segmented, the coordinate information of the designated point in the image to be segmented, and the first depth map are input into the image segmentation model, the image to be segmented can be used to perform guided filtering on the first depth map. Then, the filtered first depth map and the image to be segmented and the coordinate information of the designated point in the image to be segmented are input into the image segmentation model, and the image segmentation model determines the area corresponding to the plane where the designated point is located in the three-dimensional space in the image to be segmented in the image to be segmented. By using the image to be segmented to perform guided filtering on the first depth map, the boundary of the first depth map is more closely aligned with the image to be segmented, and obvious noise is filtered out at the same time.

[0093] According to an exemplary embodiment of the present disclosure, the step of determining by an image segmentation model the area corresponding to the plane where the specified point is located in the three-dimensional space in the image to be segmented, in the image to be segmented, may include: based on the coordinate information of the specified point, obtaining a coordinate image containing the coordinate information of the specified point (i.e., converting the coordinate information of the specified point into an image form). Then, the image to be segmented, the coordinate image, and the first depth map are input into the image segmentation model, and the image segmentation model determines the area corresponding to the plane where the specified point is located in the three-dimensional space in the image to be segmented, in the image to be segmented. Thus, the coordinate information can be input into the convolutional neural network (i.e., the image segmentation model) in the form of an image in the same manner as the image to be segmented and the first depth map, which facilitates model training.

[0094] The coordinate image may be any image containing coordinate information of the specified point. According to an exemplary embodiment of the present disclosure, the coordinate image may be an image having the same size as the image to be segmented, in which the pixel value of the point corresponding to the coordinate information of the specified point is 1 and the pixel value of other points is 0. Thus, image segmentation can be further facilitated.

[0095] Figure 6 is a schematic diagram showing an image segmentation method according to an exemplary embodiment of the present disclosure. Figure 6 As shown, the image to be segmented is input into the depth estimation network to obtain the corresponding first depth map, and then the first depth map output by the depth estimation network is guided filtered to output the filtered first depth map, and the coordinates (x, y) of the designated point are converted into an image (i.e., a mask image) with the same size as the image to be segmented, where only the pixel value at the coordinates (x, y) is 1 and the pixel values ​​at other points are 0. The image to be segmented, the filtered first depth map and the mask image are input into the image segmentation model together, and the image segmentation model obtains the area corresponding to the plane where the designated point is located in the three-dimensional space in the image to be segmented.

[0096] Figure 7 is a diagram showing the image segmentation effect according to an exemplary embodiment of the present disclosure. Figure 7 As shown in the figure, the left side is the image to be segmented, and the designated point in the image to be segmented is P (i.e., the wall on the right side behind the chair in the image to be segmented). It can be seen from the figure that the wall on the left side behind the chair is basically the same as the wall on the right side in terms of semantics, texture features, and color features. Therefore, the existing methods cannot distinguish the wall on the left side behind the chair from the wall on the right side. In contrast, Figure 7 As shown in the right figure in the middle, the image segmentation method disclosed in the present invention accurately segments the wall on the right.

[0097] According to the image segmentation method disclosed in the present invention, the semantic information of the image to be segmented and the structural information in the corresponding depth map are effectively utilized by using the trained image segmentation model, so that the area corresponding to the plane where the specified point in the three-dimensional space of the image to be segmented is located in the image to be segmented can be accurately segmented according to the specified point in the image by the user. Furthermore, special effects on a specific plane can be realized in virtual reality, and tracking of a specific plane can be realized in tracking tasks.

[0098] Figure 8 is a block diagram illustrating a training apparatus for an image segmentation model according to an exemplary embodiment of the present disclosure.

[0099] like Figure 8 As shown, the training device 800 of the image segmentation model includes: a first acquisition unit 810, a second acquisition unit 820, a prediction unit 830 and a training unit 840.

[0100] The first acquisition unit 810 is configured to: acquire the training image, coordinate information of a designated point in the training image, and a real area in the training image corresponding to a plane where the designated point is located in the three-dimensional space in the training image.

[0101] According to an exemplary embodiment of the present disclosure, the first acquisition unit 810 can be configured to: acquire a noise-free second depth map of the training image; and based on the depth information of the second depth map, acquire a real area in the training image corresponding to a plane where a specified point in the three-dimensional space of the training image is located.

[0102] Further, according to an exemplary embodiment of the present disclosure, the first acquisition unit 810 can also be configured to: acquire the depth value of the specified point and two points adjacent to the specified point based on the depth information of the second depth map and the coordinate information of the specified point; obtain the equation of the plane where the specified point is located in the three-dimensional space in the training image based on the coordinate information and depth value of the specified point and the two points adjacent to the specified point; determine the point in the second depth map that is located on the plane based on the equation of the plane, and use the set of points in the training image corresponding to the point as the real area in the training image corresponding to the plane where the specified point is located in the three-dimensional space in the training image.

[0103] Here, according to an exemplary embodiment of the present disclosure, the two points adjacent to the designated point may include one point adjacent to the designated point in a transverse direction and one point adjacent to the designated point in a longitudinal direction.

[0104] In addition, the second acquisition unit 820 is configured to acquire a first depth map of the training image acquired by the first acquisition unit 810 .

[0105] According to an exemplary embodiment of the present disclosure, the second acquisition unit 820 may be configured to: input the training image into a trained depth estimation network, and the depth estimation network acquires a first depth map of the training image.

[0106] According to an exemplary embodiment of the present disclosure, the depth estimation network may be obtained by performing supervised training on a convolutional neural network using a depth estimation training image and a depth map of the depth estimation training image without noise.

[0107] In addition, the prediction unit 830 is configured to: input the training image acquired by the first acquisition unit 810 and the second acquisition unit 820, the coordinate information of the specified point in the training image, and the first depth map into the image segmentation model 400, and the image segmentation model 400 determines the prediction area in the training image corresponding to the plane where the specified point is located in the three-dimensional space in the training image.

[0108] According to an exemplary embodiment of the present disclosure, the image segmentation model 400 may include: a semantic feature encoder 410, a structural feature encoder 420, and a decoder 430. In this case, the prediction unit 830 may be configured to: input the training image and the coordinate information of the designated point in the training image into the semantic feature encoder 410, and obtain the semantic feature of the designated point through the semantic feature encoder 410; input the coordinate information of the designated point in the training image and the first depth map into the structural feature encoder 420, and obtain the structural feature of the designated point through the structural feature encoder 420; input the semantic feature and the structural feature into the decoder 430, so as to obtain the predicted area corresponding to the plane where the designated point is located in the three-dimensional space in the training image in the training image.

[0109] According to an exemplary embodiment of the present disclosure, the prediction unit 830 may be configured to: before inputting the training image, the coordinate information of the designated point in the training image, and the first depth map into the image segmentation model, use the training image to perform guided filtering on the first depth map acquired by the second acquisition unit 820. Then, the filtered first depth map, the training image, and the coordinate information of the designated point in the training image are input into the image segmentation model, and the image segmentation model determines the predicted area in the training image corresponding to the plane where the designated point is located in the three-dimensional space in the training image.

[0110] In addition, the training unit 840 is configured to adjust the parameters of the image segmentation model 400 based on the difference between the predicted area and the real area, and train the image segmentation model 400.

[0111] According to an exemplary embodiment of the present disclosure, the prediction unit 830 may be configured to: obtain a coordinate image containing the coordinate information of the specified point based on the coordinate information of the specified point, and then input the training image, the coordinate image, and the first depth map into the image segmentation model, and the image segmentation model determines the prediction area in the training image corresponding to the plane where the specified point is located in the three-dimensional space in the training image.

[0112] According to an exemplary embodiment of the present disclosure, the coordinate image may be an image having the same size as the training image, in which the pixel values ​​of points corresponding to the coordinate information are 1 and the pixel values ​​of other points are 0.

[0113] The image segmentation model 400 trained by the image segmentation model training device in this way effectively utilizes the semantic information of the training image and the structural information in the depth map, so that it can accurately segment the area corresponding to the plane where the specified point is located in the three-dimensional space in the image according to the user's specified point in the image.

[0114] Fig. 9 is a block diagram showing an image segmentation apparatus according to an exemplary embodiment of the present disclosure. Fig. 9 As shown, the image segmentation device 900 includes: a first acquisition unit 910 , a second acquisition unit 920 and a segmentation unit 930 .

[0115] The first acquisition unit 910 is configured to acquire the image to be segmented and the coordinate information of the designated point in the image to be segmented.

[0116] The second acquisition unit 920 is configured to acquire the first depth map of the image to be segmented acquired by the first acquisition unit 910 .

[0117] According to an exemplary embodiment of the present disclosure, the second acquisition unit 920 may be configured to: input the image to be segmented into a trained depth estimation network, and use the depth estimation network to acquire a first depth map of the image to be segmented.

[0118] According to an exemplary embodiment of the present disclosure, the depth estimation network may be obtained by performing supervised training on a convolutional neural network using a depth estimation training image and a depth map of the depth estimation training image without noise.

[0119] In addition, the segmentation unit 930 is configured to: input the image to be segmented acquired by the first acquisition unit 910 and the second acquisition unit 920, the coordinate information of the specified point in the image to be segmented, and the first depth map into the image segmentation model 400, and the image segmentation model 400 determines the area in the image to be segmented corresponding to the plane where the specified point is located in the three-dimensional space in the image to be segmented.

[0120] According to an exemplary embodiment of the present disclosure, the image segmentation model 400 may include: a semantic feature encoder 410, a structural feature encoder 420, and a decoder 430. In this case, the segmentation unit 930 may be configured to: input the coordinate information of the image to be segmented and the designated point in the image to be segmented into the semantic feature encoder 410, and obtain the semantic feature of the designated point through the semantic feature encoder 410; input the coordinate information of the designated point in the image to be segmented and the first depth map into the structural feature encoder 420, and obtain the structural feature of the designated point through the structural feature encoder 420; input the semantic feature and the structural feature into the decoder 430, so as to obtain the area corresponding to the plane where the designated point is located in the three-dimensional space in the image to be segmented in the image to be segmented.

[0121] According to an exemplary embodiment of the present disclosure, the segmentation unit 930 may be configured to: before inputting the image to be segmented, the coordinate information of the designated point in the image to be segmented, and the first depth map into the image segmentation model, use the image to be segmented to perform guided filtering on the first depth map acquired by the second acquisition unit 920. Then, the filtered first depth map, the image to be segmented, and the coordinate information of the designated point in the image to be segmented are input into the image segmentation model, and the image segmentation model determines the area corresponding to the plane where the designated point is located in the three-dimensional space in the image to be segmented in the image to be segmented.

[0122] According to an exemplary embodiment of the present disclosure, the segmentation unit 930 may be configured to: obtain a coordinate image containing the coordinate information of the specified point based on the coordinate information of the specified point, and then input the image to be segmented, the coordinate image, and the first depth map into the image segmentation model, and the image segmentation model determines the area in the image to be segmented corresponding to the plane where the specified point is located in the three-dimensional space in the image to be segmented.

[0123] According to an exemplary embodiment of the present disclosure, the coordinate image may be an image having the same size as the image to be segmented, in which the pixel values ​​of points corresponding to the coordinate information of the designated points are 1 and the pixel values ​​of other points are 0.

[0124] According to the image segmentation device disclosed in the present invention, the semantic information of the image to be segmented and the structural information in the corresponding depth map are effectively utilized by using the trained image segmentation model, so that the area corresponding to the plane where the specified point in the three-dimensional space of the image to be segmented is located in the image to be segmented can be accurately segmented according to the specified point in the image by the user. Furthermore, special effects on a specific plane can be realized in virtual reality, and tracking of a specific plane can be realized in tracking tasks.

[0125] It should be understood that the image segmentation model training device and the image segmentation device according to the exemplary embodiments of the present disclosure may be executed with reference to the above Figure 2 and Figure 5 To avoid repetition, the described methods will not be described again here.

[0126] Fig.10 1 is a block diagram showing an electronic device according to an exemplary embodiment of the present disclosure. The electronic device 1000 includes at least one memory 1010 and at least one processor 1020, wherein the at least one memory stores a set of computer executable instructions, and when the computer executable instruction set is executed by the at least one processor, the training method of the image segmentation model and the image segmentation method according to the exemplary embodiment of the present disclosure are executed.

[0127] As an example, the electronic device 1000 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 1000 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 1000 may also be part of an integrated control system or a system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0128] In the electronic device 1000, the processor 1020 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not a limitation, the processor 1020 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0129] The processor 1020 may execute instructions or codes stored in the memory, wherein the memory 1010 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0130] The memory 1010 may be integrated with the processor 1020, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 1010 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 1010 and the processor 1020 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 1020 can read files stored in the memory 1010.

[0131] In addition, the electronic device 1000 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device may be connected to each other via a bus and / or a network.

[0132] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium may also be provided, wherein when the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the training method of the image segmentation model and the image segmentation method of the exemplary embodiment of the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0133] According to an exemplary embodiment of the present disclosure, a computer program product is provided, including computer instructions, which, when executed by a processor, implement the image segmentation model training method and image segmentation method of the exemplary embodiment of the present disclosure.

[0134] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0135] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image segmentation method, characterized in that: include: Acquire the image to be segmented and the coordinate information of the designated point in the image to be segmented, wherein the designated point is located on a plane in the three-dimensional space of the image to be segmented; Acquire a first depth map of the image to be segmented; Inputting the image to be segmented, coordinate information of a designated point in the image to be segmented, and the first depth map into an image segmentation model, and determining, by the image segmentation model, an area in the image to be segmented corresponding to a plane where the designated point in the three-dimensional space of the image to be segmented is located; The image segmentation model includes a semantic feature encoder, a structural feature encoder and a decoder. The step of inputting the image to be segmented, the coordinate information of the designated point in the image to be segmented, and the first depth map into the image segmentation model, and determining by the image segmentation model the area in the image to be segmented corresponding to the plane where the designated point is located in the three-dimensional space in the image to be segmented comprises: Inputting the image to be segmented and coordinate information of a designated point in the image to be segmented into the semantic feature encoder to obtain a semantic feature of the designated point; Inputting coordinate information of a designated point in the image to be segmented and the first depth map into the structural feature encoder to obtain a structural feature of the designated point; The semantic features and the structural features are input into the decoder to obtain the region in the image to be segmented corresponding to the plane where the designated point is located in the three-dimensional space in the image to be segmented.

2. The image segmentation method according to claim 1, characterized in that: The step of obtaining a first depth map of the image to be segmented comprises: The image to be segmented is input into a depth estimation network, and the first depth map is obtained by the depth estimation network.

3. The image segmentation method according to claim 2, characterized in that: The depth estimation network is obtained by performing supervised training on a convolutional neural network using a depth estimation training image and a depth map of the depth estimation training image without noise.

4. The image segmentation method according to claim 1, characterized in that: The step of determining, by the image segmentation model, a region in the image to be segmented corresponding to a plane where the designated point is located in the three-dimensional space in the image to be segmented, comprises: Based on the coordinate information, obtaining a coordinate image including the coordinate information; The image to be segmented, the coordinate image and the first depth map are input into an image segmentation model, and the image segmentation model determines the area in the image to be segmented corresponding to the plane where the specified point in the three-dimensional space of the image to be segmented is located.

5. The image segmentation method according to claim 4, characterized in that: The coordinate image is an image having the same size as the image to be segmented, in which the pixel values ​​of points corresponding to the coordinate information are 1 and the pixel values ​​of other points are 0.

6. The image segmentation method according to claim 3, characterized in that: Before the image to be segmented, the coordinate information of the designated point in the image to be segmented, and the first depth map are input into the image segmentation model, the first depth map is guided filtered using the image to be segmented, and the filtered first depth map, the image to be segmented, and the coordinate information of the designated point in the image to be segmented are input into the image segmentation model, and the image segmentation model determines the area in the image to be segmented corresponding to the plane where the designated point in the three-dimensional space of the image to be segmented is located.

7. A training method for an image segmentation model, characterized in that: include: Acquire a training image, coordinate information of a designated point in the training image, and a real area in the training image corresponding to a plane where the designated point is located in the three-dimensional space in the training image, wherein the designated point is located in the plane in the three-dimensional space in the training image; Acquire a first depth map of the training image; Inputting the training image, coordinate information of the designated point in the training image, and the first depth map into an image segmentation model, and determining by the image segmentation model a prediction region in the training image corresponding to a plane where the designated point in the three-dimensional space of the training image is located; Based on the difference between the predicted area and the real area, adjusting the parameters of the image segmentation model and training the image segmentation model; The image segmentation model includes a semantic feature encoder, a structural feature encoder and a decoder. The step of inputting the training image, the coordinate information of the designated point in the training image, and the first depth map into the image segmentation model, and determining by the image segmentation model the predicted area in the training image corresponding to the plane where the designated point in the three-dimensional space in the training image is located includes: Inputting the training image and coordinate information of a designated point in the training image into the semantic feature encoder to obtain a semantic feature of the designated point; Inputting coordinate information of a designated point in the training image and the first depth map into the structural feature encoder to obtain a structural feature of the designated point; The semantic features and the structural features are input into the decoder to obtain a predicted region in the training image corresponding to a plane where the designated point in the three-dimensional space of the training image is located.

8. The training method according to claim 7, characterized in that: The step of obtaining a real area in the training image corresponding to a plane where the designated point in the three-dimensional space in the training image is located comprises: Acquire a second depth map of the training image without noise; Based on the depth information of the second depth map, a real area in the training image corresponding to the plane where the designated point is located in the three-dimensional space in the training image is obtained.

9. The training method according to claim 8, characterized in that: Based on the depth information of the second depth map, the step of obtaining a real area in the training image corresponding to the plane where the specified point is located in the three-dimensional space in the training image comprises: Acquire depth values ​​of the designated point and two points adjacent to the designated point according to the depth information of the second depth map and the coordinate information of the designated point; Based on the coordinate information and depth values ​​of the designated point and two points adjacent to the designated point, an equation of a plane where the designated point is located in the three-dimensional space in the training image is obtained; According to the equation of the plane, a point in the second depth map located on the plane is determined, and a set of points in the training image corresponding to the point is used as a real area in the training image corresponding to the plane where the specified point in the three-dimensional space in the training image is located.

10. The training method according to claim 9, characterized in that: The two points adjacent to the designated point include one point adjacent to the designated point in the transverse direction and one point adjacent to the designated point in the longitudinal direction.

11. The training method according to claim 7, characterized in that: The step of determining, by the image segmentation model, a prediction region in the training image corresponding to a plane where the designated point is located in the three-dimensional space in the training image comprises: Based on the coordinate information, obtaining a coordinate image including the coordinate information; The training image, the coordinate image and the first depth map are input into an image segmentation model, and the image segmentation model determines a predicted area in the training image corresponding to a plane where the designated point in the three-dimensional space of the training image is located.

12. The training method according to claim 11, characterized in that: The coordinate image is an image having the same size as the training image, in which the pixel values ​​of points corresponding to the coordinate information are 1 and the pixel values ​​of other points are 0.

13. The training method according to claim 7, characterized in that: The step of obtaining a first depth map of the training image comprises: The training image is input into a depth estimation network, and the first depth map is obtained by the depth estimation network.

14. The training method according to claim 13, characterized in that: The depth estimation network is obtained by performing supervised training on a convolutional neural network using a depth estimation training image and a depth map of the depth estimation training image without noise.

15. The training method according to claim 14, characterized in that: Before the training image, the coordinate information of the designated point in the training image, and the first depth map are input into the image segmentation model, the first depth map is guided filtered using the training image, and the filtered first depth map, the training image, and the coordinate information of the designated point in the training image are input into the image segmentation model, and the image segmentation model determines the predicted area in the training image corresponding to the plane where the designated point in the three-dimensional space in the training image is located.

16. An image segmentation device, characterized in that: include: A first acquisition unit is configured to: acquire the image to be segmented and coordinate information of a designated point in the image to be segmented, wherein the designated point is located on a plane in a three-dimensional space in the image to be segmented; A second acquisition unit is configured to: acquire a first depth map of the image to be segmented; A segmentation unit is configured to: input the image to be segmented, the coordinate information of the designated point in the image to be segmented, and the first depth map into an image segmentation model, and determine, by the image segmentation model, an area in the image to be segmented corresponding to a plane where the designated point in the three-dimensional space of the image to be segmented is located; The image segmentation model includes a semantic feature encoder, a structural feature encoder and a decoder. Wherein, the segmentation unit is configured as follows: Inputting the image to be segmented and coordinate information of a designated point in the image to be segmented into the semantic feature encoder to obtain a semantic feature of the designated point; Inputting coordinate information of a designated point in the image to be segmented and the first depth map into the structural feature encoder to obtain a structural feature of the designated point; The semantic features and the structural features are input into the decoder to obtain the region in the image to be segmented corresponding to the plane where the designated point is located in the three-dimensional space in the image to be segmented.

17. The image segmentation device according to claim 16, characterized in that: The second acquisition unit is configured as follows: The image to be segmented is input into a depth estimation network, and the first depth map is obtained by the depth estimation network.

18. The image segmentation device according to claim 17, characterized in that: The depth estimation network is obtained by performing supervised training on a convolutional neural network using a depth estimation training image and a depth map of the depth estimation training image without noise.

19. The image segmentation device according to claim 16, characterized in that: The segmentation unit is configured as follows: Based on the coordinate information, obtaining a coordinate image including the coordinate information; The image to be segmented, the coordinate image and the first depth map are input into an image segmentation model, and the image segmentation model determines the area in the image to be segmented corresponding to the plane where the specified point in the three-dimensional space of the image to be segmented is located.

20. The image segmentation device according to claim 19, characterized in that: The coordinate image is an image having the same size as the image to be segmented, in which the pixel values ​​of points corresponding to the coordinate information are 1 and the pixel values ​​of other points are 0.

21. The image segmentation device according to claim 18, characterized in that: The segmentation unit is configured to: before inputting the image to be segmented, the coordinate information of the designated point in the image to be segmented, and the first depth map into the image segmentation model, use the image to be segmented to perform guided filtering on the first depth map acquired by the second acquisition unit, and input the filtered first depth map and the image to be segmented and the coordinate information of the designated point in the image to be segmented into the image segmentation model, and the image segmentation model determines the area in the image to be segmented corresponding to the plane where the designated point is located in the three-dimensional space in the image to be segmented.

22. A training device for an image segmentation model, characterized in that: include: A first acquisition unit is configured to: acquire a training image, coordinate information of a designated point in the training image, and a real area in the training image corresponding to a plane where the designated point is located in the three-dimensional space in the training image, wherein the designated point is located in the plane in the three-dimensional space in the training image; A second acquisition unit is configured to: acquire a first depth map of the training image; A prediction unit is configured to: input the training image, the coordinate information of the designated point in the training image, and the first depth map into an image segmentation model, and determine, by the image segmentation model, a prediction area in the training image corresponding to a plane where the designated point in the three-dimensional space of the training image is located; A training unit is configured to: adjust the parameters of the image segmentation model based on the difference between the predicted area and the real area, and train the image segmentation model; The image segmentation model includes a semantic feature encoder, a structural feature encoder and a decoder. Wherein, the prediction unit is configured as: Inputting the training image and coordinate information of a designated point in the training image into the semantic feature encoder to obtain a semantic feature of the designated point; Inputting coordinate information of a designated point in the training image and the first depth map into the structural feature encoder to obtain a structural feature of the designated point; The semantic features and the structural features are input into the decoder to obtain a predicted region in the training image corresponding to a plane where the designated point in the three-dimensional space of the training image is located.

23. The training device according to claim 22, characterized in that The first acquisition unit is configured as follows: Acquire a second depth map of the training image without noise; Based on the depth information of the second depth map, a real area in the training image corresponding to the plane where the designated point is located in the three-dimensional space in the training image is obtained.

24. The training device according to claim 23, characterized in that The first acquiring unit is further configured to: Acquire depth values ​​of the designated point and two points adjacent to the designated point according to the depth information of the second depth map and the coordinate information of the designated point; Based on the coordinate information and depth values ​​of the designated point and two points adjacent to the designated point, an equation of a plane where the designated point is located in the three-dimensional space in the training image is obtained; According to the equation of the plane, a point in the second depth map located on the plane is determined, and a set of points in the training image corresponding to the point is used as a real area in the training image corresponding to the plane where the specified point in the three-dimensional space in the training image is located.

25. The training device according to claim 24, characterized in that The two points adjacent to the designated point include one point adjacent to the designated point in the transverse direction and one point adjacent to the designated point in the longitudinal direction.

26. The training device according to claim 22, characterized in that The prediction unit is configured as follows: Based on the coordinate information, obtaining a coordinate image including the coordinate information; The training image, the coordinate image and the first depth map are input into an image segmentation model, and the image segmentation model determines a predicted area in the training image corresponding to a plane where the designated point in the three-dimensional space of the training image is located.

27. The training device according to claim 26, characterized in that The coordinate image is an image having the same size as the training image, in which the pixel values ​​of points corresponding to the coordinate information are 1 and the pixel values ​​of other points are 0.

28. The training device according to claim 22, characterized in that The second acquisition unit is configured as follows: The training image is input into a depth estimation network, and the first depth map is obtained by the depth estimation network.

29. The training device according to claim 28, characterized in that The depth estimation network is obtained by performing supervised training on a convolutional neural network using a depth estimation training image and a depth map of the depth estimation training image without noise.

30. The training device according to claim 29, characterized in that The prediction unit is configured to: before inputting the training image, the coordinate information of the designated point in the training image, and the first depth map into the image segmentation model, use the training image to perform guided filtering on the first depth map acquired by the second acquisition unit, and input the filtered first depth map, the training image, and the coordinate information of the designated point in the training image into the image segmentation model, and the image segmentation model determines the prediction area in the training image corresponding to the plane where the designated point is located in the three-dimensional space in the training image.

31. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image segmentation method as described in any one of claims 1 to 6 or the training method of the image segmentation model as described in any one of claims 7 to 15.

32. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device executes the image segmentation method as described in any one of claims 1 to 6 or the training method of the image segmentation model as described in any one of claims 7 to 15.

33. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the image segmentation method according to any one of claims 1 to 6 or the image segmentation model training method according to any one of claims 7 to 15 is implemented.

Images