Image segmentation method and device, electronic equipment and computer readable storage medium

Abstract

Embodiments of the present application provide an image segmentation method and device, electronic equipment and computer readable storage medium, the method comprising: obtaining a target image to be processed, and extracting initial entity features in the target image; obtaining a sample image corresponding to the target image, and extracting entity contact information from the sample image; processing the initial entity features according to the entity contact information to obtain updated entity feature information; determining the entity segmentation result of the target image according to the updated entity feature information. The image segmentation method provided by the embodiments of the present application processes the initial entity features by extracting the entity contact information from the sample image, so that the correlation between different entities in a specific scene is fused in the obtained updated entity feature information, that is, the explicit dependency relationship between entities can be fully utilized to infer the entity result of image segmentation, thereby achieving better semantic segmentation effect.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to an image segmentation method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Image segmentation refers to separating various entities from an image. For example, a common image semantic segmentation involves dividing an image into several meaningful parts by assigning a category label to each pixel.

[0003] However, current image segmentation methods still suffer from insufficient image segmentation accuracy. Summary of the Invention

[0004] This application provides an image segmentation method, apparatus, electronic device, and computer-readable storage medium, aiming to solve the problem of insufficient segmentation accuracy in existing image segmentation methods.

[0005] On one hand, embodiments of this application provide an image segmentation method, including:

[0006] Acquire the target image to be processed and extract the initial entity features from the target image;

[0007] Obtain a sample image corresponding to the target image, and extract the contact information between entities from the sample image;

[0008] The initial entity features are processed based on the inter-entity contact information to obtain updated entity feature information;

[0009] The entity segmentation result of the target image is determined based on the updated entity feature information.

[0010] As a feasible embodiment of this application, the step of extracting inter-entity contact information from the sample image includes:

[0011] The sample image is segmented to obtain the segmented regions corresponding to each entity in the sample image;

[0012] The contact frequency and / or contact boundary length between entities are determined based on the segmented regions corresponding to each entity in the sample image.

[0013] The contact frequency and / or contact boundary length are determined as the contact information between entities.

[0014] As a feasible embodiment of this application, the step of processing the initial entity features based on the inter-entity contact information to obtain updated entity feature information includes:

[0015] The contact frequency between entities in the entity contact information is normalized to obtain the first dependency between entities.

[0016] The contact boundary lengths between entities in the inter-entity contact information are normalized to obtain the second dependency between entities.

[0017] Generate the fusion dependency between entities based on the first dependency and the second dependency between entities;

[0018] The initial entity features are processed based on the fusion dependency between entities to obtain updated entity feature information.

[0019] As a feasible embodiment of this application, determining the entity segmentation result of the target image based on the updated entity feature information includes:

[0020] For each entity, an associated image corresponding to the entity is extracted from the sample image; the associated image is a sample image containing the entity.

[0021] The existence probability of the entity is determined based on the number of associated images corresponding to the entity.

[0022] The updated entity feature information is processed according to the existence probability of each entity to obtain fused entity feature information;

[0023] The entity segmentation result in the target image is determined based on the fused entity feature information.

[0024] As a feasible embodiment of this application, the extraction of initial entity features from the target image includes:

[0025] The target image is subjected to convolution processing to obtain the initial convolutional feature map of the target image;

[0026] The initial convolutional feature map is input into the trained image segmentation model to obtain several entity feature representation vectors;

[0027] For each entity feature representation vector, the entity feature representation vector is grouped to obtain several grouped entity feature representation vectors;

[0028] The grouped entity feature representation vectors are weighted and fused to obtain the initial entity features.

[0029] As a feasible embodiment of this application, the image segmentation model is trained through the following steps:

[0030] Obtain the training images and the entity segmentation labels corresponding to the training images;

[0031] The training image is input into a preset initial image segmentation model to obtain the predicted feature representation vector corresponding to the training image;

[0032] The initial image segmentation model is trained using the predicted feature representation vector and the label feature representation vector corresponding to the entity segmentation label to obtain the trained image segmentation model.

[0033] As a feasible embodiment of this application, after extracting the inter-entity contact information from the sample image, the method further includes:

[0034] Obtain scene information corresponding to the sample image, and associate the contact information between entities with the scene information and store it in a preset database;

[0035] The step of processing the initial entity features based on the inter-entity contact information to obtain updated entity feature information includes:

[0036] Based on the scene information of the target image, the preset database is queried to obtain the contact information between associated entities stored in association with the scene information;

[0037] The initial entity features are processed based on the contact information between the associated entities to obtain updated entity feature information.

[0038] On the other hand, embodiments of this application also provide an image segmentation apparatus, including:

[0039] The first acquisition module is used to acquire the target image to be processed and extract the initial entity features from the target image;

[0040] The second acquisition module is used to acquire a sample image corresponding to the target image and extract inter-entity contact information from the sample image.

[0041] The processing module is used to process the initial entity features based on the inter-entity contact information to obtain updated entity feature information;

[0042] The determination module is used to determine the entity segmentation result of the target image based on the updated entity feature information.

[0043] On the other hand, embodiments of this application also provide an electronic device, which includes a processor, a memory, and an image segmentation program stored in the memory and executable on the processor. The processor executes the image segmentation program to implement the steps in the image segmentation method described above.

[0044] On the other hand, embodiments of this application also provide a computer-readable storage medium storing an image segmentation program, which is executed by a processor to implement the steps in the image segmentation method described above.

[0045] The image segmentation method provided in this application, after extracting the initial entity feature information in the target image, updates the initial entity features by extracting the contact information between entities from the sample image. This allows the updated entity feature information to incorporate the association relationships between different entities in a specific scenario. In other words, the image segmentation method provided in this application can fully utilize the explicit dependencies between entities to reason about the entity results of image segmentation, thereby achieving better semantic segmentation results. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A schematic diagram illustrating an implementation scenario of an image segmentation method provided in this application embodiment;

[0048] Figure 2 This is a flowchart illustrating the steps of an image segmentation method provided in an embodiment of this application.

[0049] Figure 3 This application provides a flowchart illustrating the steps for determining inter-entity association features.

[0050] Figure 4 This application provides a schematic diagram of a process for associating inter-entity association features with scene types, as illustrated in an embodiment of the present application.

[0051] Figure 5 This application provides a schematic flowchart illustrating the steps for image entity segmentation by fusing entity existence probability information in an embodiment of the present application.

[0052] Figure 6 This is a schematic flowchart illustrating another step of image entity segmentation provided in an embodiment of this application;

[0053] Figure 7 A flowchart illustrating the steps for updating feature information based on intra-entity relation reasoning and inter-entity relation reasoning, provided in this application embodiment;

[0054] Figure 8This application provides a schematic diagram of the steps involved in training an image segmentation model according to an embodiment of the present application.

[0055] Figure 9 This is a schematic diagram of the structure of an image segmentation device provided in an embodiment of this application;

[0056] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0057] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of the present invention.

[0058] In this application, the term "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to implement and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be implemented without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in the embodiments of this application.

[0059] Currently, there are many commonly used image segmentation methods. Among them, some are based on neural network models. For example, image segmentation algorithms that utilize multi-level context aggregation features can collect rich contextual information from images at deeper layers. By constructing a feature pyramid, they aggregate multi-scale context in the image, thereby achieving better image segmentation results. There are also image segmentation algorithms implemented using graph convolutional network models. For instance, BGRnet constructs a graph by treating each object category as a node, and each category is represented as a feature vector, thus achieving image segmentation.

[0060] However, regardless of the image segmentation algorithm used, limitations in the model parameters themselves inevitably lead to image segmentation anomalies. For example, common issues include incorrect segmentation regions of entities, resulting in insufficient segmentation accuracy, or the inclusion of adjacent entities that should not or are unlikely to be encountered in a real-world scenario, further complicating the segmentation. This application aims to address these problems by proposing an image segmentation algorithm that guides model reasoning based on explicit dependencies between entities. This ensures that the final image segmentation result fully considers the relationships between entities within a specific scene, thereby improving the final segmentation accuracy. Furthermore, the image segmentation method provided in this application can be combined with existing neural network-based image segmentation methods, meaning it can be integrated into existing image segmentation models to enhance their segmentation performance.

[0061] Specifically, embodiments of this application provide an image segmentation method, apparatus, electronic device, and computer-readable storage medium, which will be described in detail below.

[0062] In this embodiment, the image segmentation method is deployed as a program on an image segmentation device, which is installed in an electronic device as a processor. The image segmentation device in the electronic device executes the following steps by running the program corresponding to the image segmentation method: acquiring a target image to be processed and extracting initial entity features from the target image; acquiring a sample image corresponding to the target image and extracting inter-entity contact information from the sample image; processing the initial entity features according to the inter-entity contact information to obtain updated entity feature information; and determining the entity segmentation result of the target image according to the updated entity feature information.

[0063] like Figure 1 As shown, Figure 1 This is a schematic diagram illustrating an implementation scenario of an image segmentation method provided in this application embodiment. The implementation scenario provided in this application embodiment includes an image acquisition device 100 and an image segmentation device 200. The image acquisition device 100 is mainly used to acquire the target image to be processed and transmit it to the image segmentation device 200. After acquiring the target image transmitted by the image acquisition device 100, the image segmentation device 200 performs image segmentation by executing a preset image segmentation program.

[0064] It should be noted that, Figure 1 The illustrated scenario of the image segmentation method is merely an example. The illustrated scenario of the image segmentation method described in the embodiments of this application is for the purpose of more clearly illustrating the technical solution of the embodiments of this application, and does not constitute a limitation on the technical solution provided in the embodiments of this application.

[0065] Based on the above-described implementation scenario diagram of the image segmentation method, a specific embodiment of the image segmentation method is proposed.

[0066] like Figure 2 As shown, Figure 2 This is a flowchart illustrating the steps of an image segmentation method provided in an embodiment of this application. The image segmentation method in this embodiment includes steps 201 to 204:

[0067] 201. Obtain the target image to be processed and extract the initial entity features from the target image.

[0068] In this embodiment, the target image to be processed typically refers to an image that needs to be segmented. The target image is usually an image of a specific scene, such as an office scene, a street scene, etc., which typically contains several entities, such as tables, chairs, vehicles, etc. These entities usually exist as entity features in image segmentation.

[0069] The initial entity feature information in the target image can be obtained by processing it using existing image segmentation algorithms. For example, it can be obtained using the aforementioned image segmentation algorithm that utilizes multi-level context aggregation features, or by processing the target image using a graph convolutional network model. Of course, it is also feasible to use other neural network-based image segmentation algorithms to process the target image to obtain the initial entity feature information. Specifically, as a feasible embodiment of this application, after extracting entity features through image convolution and image segmentation models, the internal relationships between entities can be further fused to obtain the initial entity features. Specific implementation schemes can be found in subsequent sections. Figure 6 And its explanations and descriptions.

[0070] It should be noted that, in general, the initial entity features obtained here are coarse entity information, that is, entity information that may differ from the actual segmentation results due to the influence of the model's own parameters.

[0071] The initial entity features mainly include two types of information: entity category information, which describes the possible entity categories, such as table, chair, water glass, human body, etc.; and entity location information, which indicates the spatial location of the entity in the image, which can be simply understood as the segmented region in the image corresponding to the entity. Of course, in addition to the above information, the initial entity feature information can also include other information, such as entity probability information, which describes the probability of corresponding to each entity category while describing the possible entity categories. For ease of explanation, the initial entity feature information described below will be based on an example that simultaneously includes entity category information, entity probability information, and spatial location information.

[0072] Furthermore, as another feasible implementation, considering that the number of entity types in an image is limited in certain specific scenarios, the initial entity features can be described in the form of feature vectors to represent the entity category and probability corresponding to a certain entity. The dimension of this feature vector is equal to the number of entity types, and each dimension corresponds to a different entity, such as the first dimension representing a human body, the second dimension representing a table, and the third dimension representing a chair, etc. In this case, the value on each dimension of the feature vector represents the probability of that entity corresponding to each of its respective entity types. Specifically, for ease of understanding, taking the aforementioned example of the first dimension representing a human body, the second dimension representing a table, and the third dimension representing a chair, (0.85, 0.1, 0.05) indicates that the probability of the entity being a human body is 0.8, the probability of it being a table is 0.1, and the probability of it being a chair is 0.05. Of course, for scenarios containing a larger number of entity types, the probability of each entity in the scene corresponding to each of its respective entity types can be deduced similarly.

[0073] 202. Obtain the sample image corresponding to the target image, and extract the contact information between entities from the sample image.

[0074] In this embodiment, the sample image corresponding to the target image typically refers to an image with the same scene as the target image, such as other office scene images or street images. The entity contact information in the sample image typically refers to the contact information between different entity regions in the sample image. This usually includes the contact frequency between entity regions corresponding to two different entities, or the contact boundary length between two entity regions. Specifically, to facilitate understanding of entity contact information, consider the following example: In most scenarios, a water cup is placed on a table or held in the hand, but rarely on a chair. That is, in this scenario, the water cup entity has a higher probability of contact with the table entity, or with the person entity, while the probability of contact with the chair entity is low. Alternatively, the contact information between the cup entity and the table entity is usually related to the bottom information of the cup, such as the area or perimeter of the bottom of the cup, while the contact information with the kettle entity is usually related to the information of the spout. In other words, the contact area between the water cup entity and the table entity is larger, while the contact area between the water cup entity and the kettle entity is usually smaller. This information describing the probability of entity contact or the size of the contact area is the entity contact information. Specifically, the following... Figure 3 This provides a specific and feasible solution for extracting contact information between entities.

[0075] Based on this, after extracting entity contact information for a specific scene from sample images identical to the target image scene, to simplify subsequent image segmentation, this entity contact information can be associated and stored with the scene information. This allows for direct database lookup when initial entity features need to be processed using the entity contact information, eliminating the need to repeatedly process the sample images. In other words, after extracting entity contact information from the sample images, the image segmentation device obtains the scene information corresponding to the sample images and associates the entity contact information with the scene information in a preset database. Subsequently, when the initial entity features of the image to be processed need to be processed using the entity contact information for a specific scene, the preset database can be queried directly using the scene information of the image to be processed to obtain the associated entity contact information stored with that scene information, which can then be used for subsequent processing of the initial entity features.

[0076] 203. The initial entity features are processed based on the inter-entity contact information to obtain updated entity feature information.

[0077] In this embodiment, based on the contact information between entities obtained from sample images, such as the aforementioned higher probability of contact between the water cup entity and the table entity or the person entity, the contact area between the water cup entity and the table entity is larger, and the contact area between the water cup entity and the kettle entity is usually smaller, by using this information to update the initial entity features, some erroneous feature representations that may exist in the original entity features can be eliminated, thereby obtaining optimized updated entity feature information.

[0078] In this embodiment, the initial entity feature information is processed using the obtained inter-entity association features. The resulting updated entity feature information incorporates the inter-entity association relationships, leading to greater accuracy. For example, if the entity category or probability information in the initial entity feature information indicates that an entity has a higher probability of being a chair and a lower probability of being a table, and the entity location information in the initial entity feature information indicates that the entity is in contact with a water cup in the image, then processing the feature information using the inter-entity association features will change the final updated entity feature information. Since the inter-entity association features indicate that the water cup entity is more likely to be in contact with the table entity and less likely to be in contact with the chair entity, the entity category or probability information will change accordingly, indicating that the entity has a higher probability of being a table and a lower probability of being a chair. Therefore, the final output entity segmentation result is a table.

[0079] Of course, it should be noted that processing feature information using inter-entity association features is merely to correct potential biases in the initial entity feature information using contact information between entities. The final updated entity feature information will fully consider both types of information: inter-entity association features and the initial entity feature information. Specifically, taking the aforementioned example again, if the entity category information or entity probability in the initial entity feature information indicates that a certain entity has a very high probability of being a chair, for example, over 95%, and only a very low probability of being a table, then in the final updated entity feature information, the entity category information or entity probability will still indicate that the entity has a higher probability of being a chair.

[0080] Of course, the above example only illustrates the contact information between the chair entity and the cup entity. In fact, the inter-entity association features contain rich contact information between the entities. By using these inter-entity association features to process the initial entity feature information, it is possible to effectively correct the entity category information, entity probability information, and entity position information in the initial entity feature information, thereby obtaining more accurate updated entity feature information.

[0081] 204. Determine the entity segmentation result of the target image based on the updated entity feature information.

[0082] In this embodiment, as described above, updating entity feature information is achieved by updating the entity category information, entity probability information, and entity location information in the feature information based on the inter-entity association features. Therefore, the updated entity feature information also includes entity category information, entity probability information, and entity location information. The updated entity category information and entity probability information can be used to determine segmented entities in the image, while the updated entity location information describes the segmentation region corresponding to that segmented entity. The segmented entity and its corresponding segmentation region constitute the final entity segmentation result of the target image.

[0083] Furthermore, the image segmentation method provided in this application embodiment utilizes the relationships between entities to process the initial image segmentation result, ensuring that the processed initial image segmentation result fully considers the explicit dependencies between entities in the scene, thereby improving the image segmentation effect. In fact, besides utilizing the relationships between entities to process the initial image segmentation result, the image segmentation method provided in this application embodiment can also utilize entity information from more dimensions to optimize the initial image segmentation result, thereby further improving the image segmentation effect. For example, as an optional embodiment of this application, some global information in the scene can also be used to optimize the image segmentation result. Specific implementation schemes can be found in the following sections. Figure 5 And its explanations and descriptions.

[0084] The image segmentation method provided in this application, after extracting the initial entity feature information in the target image, updates the initial entity feature information by extracting the contact information between entities from the sample image. This allows the updated entity feature information to incorporate the association relationships between different entities in a specific scenario. In other words, the image segmentation method provided in this application can fully utilize the explicit dependencies between entities to reason about the entity results of image segmentation, thereby achieving better semantic segmentation results.

[0085] like Figure 3 As shown, Figure 3 A flowchart illustrating the steps for extracting contact information between entities, as provided in this application embodiment, is described in detail below.

[0086] In this embodiment of the application, a technical solution is provided that uses the contact frequency and / or contact boundary length between entities in a sample image as contact information for subsequent correction of the initial entity feature information, specifically including steps 301 to 303:

[0087] 301. Perform image segmentation on the sample image to obtain the segmented regions corresponding to each entity in the sample image.

[0088] In this embodiment of the application, by performing image segmentation on the sample image, the segmentation region corresponding to each entity in the sample image can be determined. Of course, it is also feasible to obtain the segmentation region corresponding to each entity by manual annotation.

[0089] It should be noted that the sample images used in the embodiments of this application usually include multiple images. The specific number can be set based on actual needs. For example, the more sample images used, the more accurate the extracted contact information between entities will be, but the corresponding amount of computation will also be higher.

[0090] 302. Determine the contact frequency and / or contact boundary length between entities based on the segmented regions corresponding to each entity in the sample image.

[0091] In this embodiment of the application, after determining the segmentation region corresponding to each entity in the sample image, the image segmentation device can obtain the contact frequency and / or contact boundary length between entities by performing statistics on the segmentation region corresponding to each entity.

[0092] Specifically, the contact frequency between entities refers to the probability that two entities in a sample image will come into contact. For example, for the first entity and the second entity, if there are N sample images out of all M sample images in which the first entity and the second entity come into contact, then the contact frequency between the first entity and the second entity is N / M. The higher the contact frequency between entities, the greater the probability that the two entities will come into contact.

[0093] Based on this, by analyzing the length of the overlapping boundary between the segmented regions corresponding to two entities in the sample image, the contact boundary length between the entities can be obtained. Specifically, the contact boundary length can be obtained from the statistical value of the length of the overlapping boundary between the segmented regions corresponding to two entities in the sample image. For example, taking the first entity and the second entity mentioned above as examples, if there are N sample images in which the first entity and the second entity are in contact, the lengths of the overlapping boundary corresponding to each sample image are L1, L2, ..., L... N Then the final contact boundary length between the first entity and the second entity is (L 1+ L2+...+L N ) / N.

[0094] 303, The contact frequency and / or contact boundary length are determined as inter-entity contact information.

[0095] In this embodiment, either the contact frequency or the contact boundary length obtained above can be determined as the inter-entity contact information. Alternatively, both the contact frequency and the contact boundary length can be selected as the inter-entity contact information to improve the updating effect of the initial entity features. Based on this, when the inter-entity contact information includes both the contact frequency and the contact boundary length between entities, the specific implementation scheme for processing the initial entity features according to the inter-entity contact information to obtain the updated entity feature information can be found in subsequent sections. Figure 4 And its explanations and descriptions.

[0096] like Figure 4 As shown, Figure 4 This application provides a flowchart illustrating the steps for processing initial entity features, specifically including steps 401 to 404:

[0097] 401, normalize the contact frequency between each entity in the entity contact information to obtain the first dependency between each entity.

[0098] In this embodiment of the application, by normalizing the contact frequency between each entity, that is, by mapping the contact frequency between each entity to a value between 0 and 1 based on the maximum and minimum values ​​of the contact frequency between each entity, the first dependency between each entity can be obtained.

[0099] 402, normalize the contact boundary lengths between entities in the entity contact information to obtain the second dependency between entities.

[0100] In this embodiment, by normalizing the contact boundary length between entities, that is, by mapping the contact boundary length between entities to a value between 0 and 1 based on the maximum and minimum values ​​of the contact boundary length between entities, the second dependency between entities can be obtained.

[0101] 403. Generate the fusion dependency between entities based on the first dependency and the second dependency between entities.

[0102] In this embodiment, after the first dependency and the second dependency, the two are weighted and fused according to a certain weight coefficient. For example, the most common method is average weighting, which yields the fused dependency between the first entity and the second entity. Specifically, the fused dependency between entities can be described by an entity association feature matrix in matrix form. That is, the element value in the i-th row and j-th column of the entity association feature matrix is ​​the fused dependency between the i-th entity and the j-th entity.

[0103] 404. The initial entity features are processed according to the fusion dependency between entities to obtain updated entity feature information.

[0104] In this embodiment, the fusion dependency between entities is utilized, that is, the initial entity features are convolved with the aforementioned inter-entity association features to obtain the updated entity feature information.

[0105] Of course, the above provides an embodiment for generating inter-entity association features based on contact frequency and contact boundary length. In fact, it is also feasible to generate inter-entity association features based solely on contact frequency or solely on contact boundary length. However, the inter-entity association features generated using both contact frequency and contact boundary length contain richer contact information and can be better used for subsequent processing of feature information.

[0106] Of course, besides utilizing inter-entity association features to improve entity segmentation of target images, another optional embodiment of this application can further improve entity segmentation through global information reasoning. Global information reasoning refers to further refining the entity-specific representation in the image based on the entity probabilities corresponding to scene information, taking into account the different existence probabilities of entities in different real-world scenarios. Specifically, such as... Figure 5 As shown, Figure 5A flowchart illustrating the steps for image entity segmentation by fusing entity presence probability information, as provided in this application embodiment, includes steps 501 to 504:

[0107] 501. For each entity, extract the associated image corresponding to the entity from the sample image.

[0108] In this embodiment of the application, the associated image refers to the image in the sample image that contains the entity. That is, for each entity, the image segmentation device will extract the associated image containing the entity from the sample image.

[0109] 502. Based on the number of associated images corresponding to the entity, determine the existence probability of the entity.

[0110] In this embodiment, the existence probability of an entity can be obtained by counting the number of associated images and calculating the ratio of this number to the total number of sample images. For example, if there are P associated images containing the entity among all M sample images, then the existence probability of that entity is P / M. Specifically, typically, entity probability features existing in vector form can be used to describe the existence probability of each entity; that is, the value of the i-th dimension in the entity probability feature is the existence probability of the i-th entity.

[0111] 503. The updated entity feature information is processed according to the existence probability of each entity to obtain fused entity feature information.

[0112] In this embodiment, similar to processing the initial entity feature information using inter-entity association features, the updated entity feature information is processed using the existence probability of each entity. The resulting fused entity feature information is further integrated into the global scene information. For example, if an entity has a low existence probability in the scene but is incorrectly identified, processing the updated entity feature information using entity probability features can correct the aforementioned incorrect identification by multiplying it by a very small probability, thereby improving image segmentation accuracy.

[0113] 504. Determine the entity segmentation result in the target image based on the fused entity feature information.

[0114] In this embodiment of the application, as can be seen from the foregoing description, the fusion of entity feature information not only integrates the explicit dependencies between entities, but also integrates global scene information. Through the fusion of multi-dimensional information, the accuracy and precision of entity segmentation results are further improved.

[0115] like Figure 6 As shown, Figure 6 The flowchart illustrating the steps for extracting initial entity features provided in this application embodiment is described in detail below.

[0116] This application provides a method for extracting initial entity features based on a neural network model and simultaneously fusing intra-entity relationships, specifically including steps 601-604:

[0117] 601, Perform convolution processing on the target image to obtain the initial convolution feature map of the target image.

[0118] In this embodiment, the initial convolutional features of the image can be obtained by inputting the target image to be processed into a 3x3 convolutional layer. Figure X 1.

[0119] 602. The initial convolutional feature map is input into the trained image segmentation model to obtain several entity feature representation vectors.

[0120] In this embodiment of the application, based on the foregoing, the initial convolutional features are... Figure X 1. Inputting the initial convolutional features into a trained image segmentation model can transform the initial convolutional features into the model's input. Figure X 1. Decoupling into feature representations of k entities, that is, the entity feature representation vector X corresponding to the k entities. k ∈R Z×H×W Where k∈{1,...,K}, K is the number of entity types, Z represents the number of channels, and H and W represent the height and width. Specifically, the image segmentation model is pre-trained; the specific training process can be found in subsequent articles. Figure 7 And its explanations and descriptions.

[0121] 603. For each entity feature representation vector, the entity feature representation vector is grouped to obtain several grouped entity feature representation vectors.

[0122] In this embodiment, for each entity feature representation vector, two grouped convolutional layers can be applied to perform entity-internal relation inference for that entity. This allows each entity to update its representation, i.e., its entity feature representation vector, based on the internal structure of its corresponding entity category. Specifically, by using the entity feature representation vector X... k ∈R Z×H×W Dividing the entity into M groups along the channel dimension, we can obtain the grouped entity feature representation vector for each group.

[0123] 604. The grouped entity feature representation vectors are weighted and fused to obtain the initial entity features.

[0124] In this embodiment of the application, the grouped entity feature representation vector is obtained as described above. Then, by performing convolution on each group separately and concatenating the new feature mapping groups, we can obtain the initial entity features X containing the internal relationships of the entities. k1 Specifically, the formula for calculating the fusion of the above grouped convolutions is as follows:

[0125]

[0126] Here, Concat represents the array concatenation operation, which connects the feature mapping groups together. These represent the feature representation vectors of each grouped entity. Regarding the weights, it should be noted that... The weights can be determined through pre-training.

[0127] like Figure 7 As shown, Figure 7 This application provides a flowchart illustrating the steps involved in training an image segmentation model, specifically including steps 701 to 703:

[0128] 701, Obtain the training image and the entity segmentation label corresponding to the training image.

[0129] In this embodiment, the training images also refer to a batch of manually labeled entity segmentation images obtained in advance. The source of the training images can be similar to the sample images mentioned above. This embodiment will not elaborate on the training images here.

[0130] 702, The training image is input into a preset initial image segmentation model to obtain the predicted feature representation vector corresponding to the training image.

[0131] In this embodiment, the training image is input into an untrained initial image segmentation model for processing. The initial image segmentation model outputs a predicted entity segmentation result for the training image, which is a predicted feature representation vector. Of course, in most cases, the predicted entity segmentation result will differ from the true entity segmentation result, i.e., the entity segmentation label.

[0132] 703. The initial image segmentation model is trained based on the predicted feature representation vector and the label feature representation vector corresponding to the entity segmentation label to obtain the trained image segmentation model.

[0133] In this embodiment, after obtaining the predicted feature representation vector that differs from the actual entity segmentation result, the image segmentation model updates the model parameters in the initial image segmentation model based on this difference using the backpropagation algorithm. This makes the predicted feature representation vector obtained by the initial image segmentation model from processing training images increasingly closer to the actual entity segmentation result. After several iterations of training, a trained image segmentation model that can accurately segment the entity of the training image is obtained. Specifically, when the difference between the feature representation vector obtained by inputting the training image into the currently trained model and the actual entity segmentation label is sufficiently small, the model can be considered to have been trained successfully. At this point, the currently trained model is the final image segmentation model used to achieve image segmentation.

[0134] like Figure 8 As shown, Figure 8 An implementation scheme for associating and storing inter-entity contact information with scene information, provided in this application embodiment, specifically includes steps 801 to 803:

[0135] 801. Obtain the scene information corresponding to the sample image, and associate the contact information between entities with the scene information and store it in a preset database.

[0136] In this embodiment of the application, the scene information corresponding to the sample image can exist in the form of pre-labeled scene tags. For example, common scene tags include office, road, park, etc. Under different scene information, there are different entities and contact information between entities. Therefore, the contact information between entities extracted from the sample image under a specific scene, such as the contact frequency between entities and the contact boundary length between entities, can be associated with the scene information and stored in a preset database so that when a certain image to be processed needs to be processed later, the corresponding contact information between entities can be directly extracted from the preset database.

[0137] 802. Query the preset database based on the scene information of the target image to obtain contact information between associated entities stored in association with the scene information.

[0138] In this embodiment of the application, after extracting the contact information between entities in different scenarios through the aforementioned method and storing it in the database in association with the corresponding scenario information, when it is necessary to complete the image segmentation of the target image, the scene information of the target image can be directly used to query the preset database to obtain the contact information between entities associated with the scene information.

[0139] 803. The initial entity features are processed based on the contact information between the associated entities to obtain updated entity feature information.

[0140] In this embodiment, after extracting the contact information between related entities from a preset database, the image segmentation device processes the initial entity features based on the contact information. The specific processing procedure is the same as in step 203 described above; for details, please refer to step 203 and... Figure 4 And its explanations and descriptions.

[0141] To better implement the image segmentation method in the embodiments of this application, an image segmentation apparatus is also provided in the embodiments of this application, based on the image segmentation method. For example... Figure 9 As shown, Figure 9 This is a schematic diagram of an image segmentation apparatus provided in an embodiment of this application. Specifically, the image segmentation apparatus includes:

[0142] The first acquisition module 901 is used to acquire the target image to be processed and extract the initial entity features from the target image;

[0143] The second acquisition module 902 is used to acquire a sample image corresponding to the target image and extract contact information between entities from the sample image.

[0144] Processing module 903 is used to process the initial entity features based on the inter-entity contact information to obtain updated entity feature information;

[0145] The determination module 904 is used to determine the entity segmentation result of the target image based on the updated entity feature information.

[0146] In some embodiments of this application, the second acquisition module is used to perform image segmentation on the sample image to obtain segmented regions corresponding to each entity in the sample image; determine the contact frequency and / or contact boundary length between each entity based on the segmented regions corresponding to each entity in the sample image; and determine the contact frequency and / or contact boundary length as contact information between entities.

[0147] In some embodiments of this application, the processing module is used to normalize the contact frequency between entities in the inter-entity contact information to obtain a first dependency between entities; normalize the contact boundary length between entities in the inter-entity contact information to obtain a second dependency between entities; generate a fusion dependency between entities based on the first dependency and the second dependency; and process the initial entity features based on the fusion dependency between entities to obtain updated entity feature information.

[0148] In some embodiments of this application, the determining module is used to extract the associated image corresponding to the entity from the sample image for each entity; the associated image is a sample image containing the entity; based on the number of associated images corresponding to the entity, determine the existence probability corresponding to the entity; process the updated entity feature information according to the existence probability corresponding to each entity to obtain fused entity feature information; and determine the entity segmentation result in the target image according to the fused entity feature information.

[0149] In some embodiments of this application, the first acquisition module is used to perform convolution processing on the target image to obtain an initial convolutional feature map of the target image; input the initial convolutional feature map into a trained image segmentation model to obtain several entity feature representation vectors; for each entity feature representation vector, group the entity feature representation vectors to obtain several grouped entity feature representation vectors; and perform weighted fusion on the grouped entity feature representation vectors to obtain initial entity features.

[0150] In some embodiments of this application, the first acquisition module is used to acquire a training image and an entity segmentation label corresponding to the training image; input the training image into a preset initial image segmentation model to obtain a predicted feature representation vector corresponding to the training image; and train the initial image segmentation model according to the predicted feature representation vector and the label feature representation vector corresponding to the entity segmentation label to obtain a trained image segmentation model.

[0151] In some embodiments of this application, the second acquisition module is used to acquire scene information corresponding to the sample image and associate the entity contact information with the scene information and store it in a preset database; the processing module is used to query the preset database according to the scene information of the target image to obtain associated entity contact information stored with the scene information; and to process the initial entity features according to the associated entity contact information to obtain updated entity feature information.

[0152] This application also provides an image segmentation device, such as... Figure 10 As shown, Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0153] The electronic device includes a memory, a processor, and an image segmentation program stored in the memory and executable on the processor. When the processor executes the image segmentation program, it implements the steps in the image segmentation method provided in any embodiment of this application.

[0154] Specifically, an electronic device may include components such as a processor 1001 with one or more processing cores, a memory 1002 with one or more storage media, a power supply 1003, and an input unit 1004. Those skilled in the art will understand that... Figure 10 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0155] The processor 1001 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 1002, and by calling data stored in the memory 1002, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. Optionally, the processor 1001 may include one or more processing cores; preferably, the processor 1001 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 1001.

[0156] The memory 1002 can be used to store software programs and modules. The processor 1001 executes various functional applications and data processing by running the software programs and modules stored in the memory 1002. The memory 1002 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1002 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 1002 may also include a memory controller to provide the processor 1001 with access to the memory 1002.

[0157] The electronic device also includes a power supply 1003 that supplies power to various components. Preferably, the power supply 1003 can be logically connected to the processor 1001 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 1003 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0158] The electronic device may also include an input unit 1004, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0159] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 1001 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 1002 according to the following instructions, and the processor 1001 runs the applications stored in the memory 1002, thereby implementing the steps in the image segmentation method provided in any embodiment of this application.

[0160] Therefore, embodiments of this application provide a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc. The computer-readable storage medium stores an image segmentation program, which, when executed by a processor, implements the steps of the image segmentation method provided in any embodiment of this application.

[0161] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed descriptions of other embodiments above, which will not be repeated here.

[0162] In practice, each of the above units or structures can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units or structures, please refer to the previous method embodiments, which will not be repeated here.

[0163] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0164] The above provides a detailed description of an image segmentation method provided by the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An image segmentation method, characterized in that, include: Acquire the target image to be processed and extract the initial entity features from the target image; Obtain a sample image corresponding to the target image, and extract the contact information between entities from the sample image; The initial entity features are processed based on the inter-entity contact information to obtain updated entity feature information; The entity segmentation result of the target image is determined based on the updated entity feature information; The step of extracting inter-entity contact information from the sample image includes: performing image segmentation on the sample image to obtain segmented regions corresponding to each entity in the sample image; determining the contact frequency and / or contact boundary length between each entity based on the segmented regions corresponding to each entity in the sample image; and determining the contact frequency and / or contact boundary length as inter-entity contact information. The step of processing the initial entity features based on the inter-entity contact information to obtain updated entity feature information includes: normalizing the contact frequency between entities in the inter-entity contact information to obtain a first dependency between entities; normalizing the contact boundary length between entities in the inter-entity contact information to obtain a second dependency between entities; generating a fusion dependency between entities based on the first dependency and the second dependency; and processing the initial entity features based on the fusion dependency between entities to obtain updated entity feature information.

2. The method according to claim 1, characterized in that, Determining the entity segmentation result of the target image based on the updated entity feature information includes: For each entity, an associated image corresponding to the entity is extracted from the sample image; the associated image is a sample image containing the entity. The existence probability of the entity is determined based on the number of associated images corresponding to the entity. The updated entity feature information is processed according to the existence probability of each entity to obtain fused entity feature information; The entity segmentation result in the target image is determined based on the fused entity feature information.

3. The method according to claim 1, characterized in that, The extraction of initial entity features from the target image includes: The target image is subjected to convolution processing to obtain the initial convolutional feature map of the target image; The initial convolutional feature map is input into the trained image segmentation model to obtain several entity feature representation vectors; For each entity feature representation vector, the entity feature representation vector is grouped to obtain several grouped entity feature representation vectors; The grouped entity feature representation vectors are weighted and fused to obtain the initial entity features.

4. The method according to claim 3, characterized in that, The image segmentation model is trained through the following steps: Obtain the training images and the entity segmentation labels corresponding to the training images; The training image is input into a preset initial image segmentation model to obtain the predicted feature representation vector corresponding to the training image; The initial image segmentation model is trained based on the predicted feature representation vector and the label feature representation vector corresponding to the entity segmentation label to obtain the trained image segmentation model.

5. The method according to any one of claims 1 to 4, characterized in that, After extracting the inter-entity contact information from the sample image, the method further includes: Obtain scene information corresponding to the sample image, and associate the contact information between entities with the scene information and store it in a preset database; The step of processing the initial entity features based on the inter-entity contact information to obtain updated entity feature information includes: Based on the scene information of the target image, the preset database is queried to obtain the contact information between associated entities stored in association with the scene information; The initial entity features are processed based on the contact information between the associated entities to obtain updated entity feature information.

6. An image segmentation apparatus, characterized in that, include: The first acquisition module is used to acquire the target image to be processed and extract the initial entity features from the target image; The second acquisition module is used to acquire a sample image corresponding to the target image and extract inter-entity contact information from the sample image. The processing module is used to process the initial entity features based on the inter-entity contact information to obtain updated entity feature information; The determination module is used to determine the entity segmentation result of the target image based on the updated entity feature information; The second acquisition module extracts inter-entity contact information from the sample image, including: performing image segmentation on the sample image to obtain segmented regions corresponding to each entity in the sample image; determining the contact frequency and / or contact boundary length between each entity based on the segmented regions corresponding to each entity in the sample image; and determining the contact frequency and / or contact boundary length as inter-entity contact information. The processing module processes the initial entity features based on the inter-entity contact information to obtain updated entity feature information, including: normalizing the contact frequency between entities in the inter-entity contact information to obtain a first dependency between entities; normalizing the contact boundary length between entities in the inter-entity contact information to obtain a second dependency between entities; generating a fusion dependency between entities based on the first dependency and the second dependency; and processing the initial entity features based on the fusion dependency between entities to obtain updated entity feature information.

7. An electronic device, characterized in that, The electronic device includes a processor, a memory, and an image segmentation program stored in the memory and executable on the processor, the processor executing the image segmentation program to implement the steps of the image segmentation method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an image segmentation program, which is executed by a processor to implement the steps of the image segmentation method according to any one of claims 1 to 5.

Images