Image semantic segmentation method, device, electronic device and storage medium
By constructing a semantic correlation matrix based on initial classification prediction results and performing feature enhancement processing, combining user-corrected data and model parameter adjustment, the problems of insufficient explanatory and lack of supervision information in the GCN model are solved, and the recognition of image semantic segmentation model is improved Effect.
Patent Information
- Application Number
- CN202110019145.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-01-07
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-01-07
AI Technical Summary
When using GCN, the existing image semantic segmentation model lacks clear interpretative adjacency matrix and supervision information, resulting in the unsatisfactory recognition effect of the trained model.
By constructing an adjacency matrix based on the initial classification prediction results, the semantic correlation between pixel points in the image is described, and the matrix is used for feature enhancement processing, combining user-corrected data and model parameter adjustments, the accuracy of semantic segmentation prediction results is improved.
The clustering accuracy and recognition effect of the image semantic segmentation model are improved, and more accurate semantic segmentation prediction results are obtained.
Smart Images

Figure CN114742996B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to an image semantic segmentation method, device, electronic device and storage medium. Background Art
[0002] With the development of technology, image semantic segmentation has been widely used in related fields, and the extraction of the relationship of contextual information has become an important way to improve the effect of image semantic segmentation models. The problem of image semantic segmentation can be regarded as a clustering problem between different pixels. The inventors of the present invention have found that in the traditional image semantic segmentation models using GCN (graph convolutional neural network), most of them use GCN networks to explore the global contextual relationship between pixels. However, the adjacency matrix used by this type of traditional image semantic segmentation model using GCN adopts a method similar to the attention mechanism, which is obtained by transforming the image features. The adjacency matrix obtained in this way does not have a clear interpretability, and due to the lack of supervised information, the model image semantic segmentation model needs to learn the connection relationship between each pixel in the adjacency matrix by itself, so the recognition effect of the trained image semantic segmentation model is not ideal. Summary of the invention
[0003] The embodiments of the present disclosure provide a method, device, electronic device and computer-readable storage medium for image semantic segmentation.
[0004] In a first aspect, an embodiment of the present disclosure provides an image semantic segmentation method, which includes:
[0005] Get the image to be processed;
[0006] Determine the initial classification prediction results of the pixels in the image to be processed;
[0007] Performing feature enhancement processing on the image features of the image to be processed based on the initial classification prediction result;
[0008] Obtain a semantic segmentation prediction result of the image to be processed according to the image features after feature enhancement processing.
[0009] Furthermore, based on the initial classification prediction result, feature enhancement processing is performed on the image features of the image to be processed, including:
[0010] Constructing an adjacency matrix of the image to be processed according to the initial classification prediction result; wherein the adjacency matrix is used to describe the semantic correlation relationship between pixels in the image to be processed;
[0011] Perform feature enhancement processing on the image features based on the adjacency matrix.
[0012] Furthermore, constructing an adjacency matrix corresponding to the image to be processed according to the initial classification prediction result includes:
[0013] The adjacency matrix is constructed by multiplying a classification matrix by the transpose of the classification matrix; the classification matrix is a matrix form of the initial classification prediction result.
[0014] Furthermore, the feature enhancement process includes multiple stages, and the feature enhancement process is performed on the image features based on the adjacency matrix, including:
[0015] Performing feature enhancement processing of the current stage on the input features based on the adjacency matrix; wherein the input features are the image features or output features obtained by feature enhancement processing of the previous stage;
[0016] Obtaining a semantic segmentation prediction result of the image to be processed according to the image features after feature enhancement processing, including:
[0017] The semantic segmentation prediction result is obtained according to the output features obtained by the feature enhancement processing in the last stage.
[0018] Furthermore, determining the initial classification prediction result of the pixel points in the image to be processed includes:
[0019] Acquire image features of the image to be processed by using a basic model in a semantic segmentation model;
[0020] Using the classification model in the semantic segmentation model to perform classification prediction on the pixel points in the image to be processed to obtain the initial classification prediction result;
[0021] Performing feature enhancement processing on the image features based on the adjacency matrix includes:
[0022] The feature enhancement model in the semantic segmentation model is utilized to perform feature enhancement processing on the image features based on the adjacency matrix.
[0023] Furthermore, the method further comprises:
[0024] When the confidence level of the semantic segmentation prediction result is lower than a preset threshold, displaying the semantic segmentation prediction result to a user;
[0025] Receiving correction data of the semantic segmentation prediction result from the user;
[0026] The semantic segmentation prediction result is adjusted according to the correction data.
[0027] Furthermore, the method further comprises:
[0028] The model parameters of the semantic segmentation model are adjusted according to the adjusted semantic segmentation prediction result and the semantic segmentation prediction result before the adjustment; wherein the semantic segmentation model is a model for obtaining the semantic segmentation prediction result before the adjustment.
[0029] Furthermore, the image to be processed is a remote sensing image, and the semantic segmentation prediction result includes a combination of one or more of the following data: contour information of different objects in the remote sensing image, resource types of different objects in the remote sensing image, and road network information in the remote sensing image.
[0030] Furthermore, the image to be processed is a remote sensing image of a target area, and the method further includes:
[0031] The target object in the target area is determined according to the semantic segmentation prediction result.
[0032] Furthermore, the image to be processed is a remote sensing image of a target area, and the method further includes:
[0033] The resource type of the ground objects included in the target area is determined according to the semantic segmentation prediction result.
[0034] Furthermore, the image to be processed is a remote sensing image of a target area, and the method further includes:
[0035] Determine the location and type of the road in the target area according to the semantic segmentation prediction result;
[0036] The road network information in the target area is determined according to the location and type of the road.
[0037] Furthermore, the image to be processed includes a first remote sensing image and a second remote sensing image, and the method further includes:
[0038] Change detection information of ground objects in the first remote sensing image and the second remote sensing image is determined according to the semantic segmentation prediction result corresponding to the first remote sensing image and the semantic segmentation prediction result corresponding to the second remote sensing image.
[0039] Furthermore, the image to be processed includes an image of a road ahead of the autonomous driving vehicle, and the method further includes:
[0040] Determine object information on the road ahead according to the semantic segmentation prediction result;
[0041] determining vehicle driving information based on the object information;
[0042] The vehicle driving information is output to the autonomous driving vehicle.
[0043] Furthermore, the image to be processed is the first image in the video, and the method further includes:
[0044] Acquire a second image in the video;
[0045] updating the initial classification prediction result of the first image according to the difference between the first image and the second image to obtain the initial classification prediction result of the second image;
[0046] A semantic segmentation prediction result of the second image is obtained based on the initial classification prediction result of the second image.
[0047] Further, updating the initial classification prediction result of the first image according to the difference between the first image and the second image to obtain the initial classification prediction result of the second image includes:
[0048] Obtaining a foreground offset between the first image and the second image;
[0049] The initial classification prediction result is updated according to the foreground offset to obtain an initial classification prediction result of the second image.
[0050] Furthermore, the image to be processed is an image including a commodity, and the method includes:
[0051] Acquire a commodity image in the image to be processed according to the semantic segmentation prediction result;
[0052] Visual description information of the product is generated based on the product image.
[0053] Furthermore, the image to be processed is an image in a live video, and the method includes:
[0054] Processing the live video according to the semantic segmentation prediction result to obtain a target video;
[0055] The target video is output.
[0056] Further, the live video is processed according to the semantic segmentation prediction result to obtain a target video, including:
[0057] Determining whether the image to be processed is a target image including a product according to the semantic segmentation prediction result;
[0058] Performing preset processing on the target image.
[0059] Further, the live video is processed according to the semantic segmentation prediction result to obtain a target video, including:
[0060] Acquire a target image including a product in the live video according to the semantic segmentation prediction result;
[0061] Visual description information of the product is generated according to the target image.
[0062] In a second aspect, an embodiment of the present disclosure provides a method for training a semantic segmentation model, which includes:
[0063] Get training images;
[0064] Obtaining initial classification prediction results of pixels in the training image using a semantic segmentation model, and performing feature enhancement processing on image features of the training image based on the initial classification prediction results;
[0065] Obtaining a semantic segmentation prediction result of the training image according to the image features after feature enhancement processing;
[0066] The semantic segmentation model is trained based on the initial classification prediction result and the semantic segmentation prediction result.
[0067] Furthermore, based on the initial classification prediction result, feature enhancement processing is performed on the image features of the training image, including:
[0068] Constructing an adjacency matrix corresponding to the training image according to the initial classification prediction result, wherein the adjacency matrix is used to describe the semantic correlation relationship between pixels in the training image;
[0069] Feature enhancement processing is performed on the image features corresponding to the training image based on the adjacency matrix.
[0070] Furthermore, constructing an adjacency matrix corresponding to the training image according to the initial classification prediction result includes:
[0071] The adjacency matrix is constructed by multiplying a classification matrix by the transpose of the classification matrix.
[0072] Furthermore, the semantic segmentation model includes a basic model, a classification model and a feature enhancement model, and the semantic segmentation model is trained based on the initial classification prediction result and the semantic segmentation prediction result, including:
[0073] The model parameters of the classification model and the basic model are adjusted by the loss between the initial classification prediction result and the corresponding true classification result; the classification model is used to obtain the initial classification prediction result; the basic model is used to obtain the image feature;
[0074] The model parameters of the feature enhancement model and the basic model are adjusted by the loss between the semantic segmentation prediction result and its corresponding semantic segmentation true result; the eigenvalue enhancement model is used to perform feature enhancement processing on the image features.
[0075] Furthermore, the feature enhancement process includes multiple stages; performing feature enhancement process on the image features corresponding to the training image based on the adjacency matrix includes:
[0076] Performing feature enhancement processing of the current stage on the input features based on the adjacency matrix to obtain output features; the input features are the image features or the output features obtained after feature enhancement processing in the previous stage;
[0077] The semantic segmentation prediction result is obtained based on the output features obtained after the feature enhancement processing in the last stage.
[0078] In a third aspect, an embodiment of the present disclosure provides an image semantic segmentation method, which includes:
[0079] Get the image to be processed;
[0080] Calling a preset service interface so that the preset service interface obtains an image to be processed, determines an initial classification prediction result of pixels in the image to be processed, performs feature enhancement processing on image features of the image to be processed based on the initial classification prediction result, and obtains a semantic segmentation prediction result of the image to be processed according to the image features after the feature enhancement processing;
[0081] Output the semantic segmentation prediction result.
[0082] In a fourth aspect, an embodiment of the present disclosure provides a method for training a semantic segmentation model, which includes:
[0083] Get training images;
[0084] Calling a preset service interface so that the preset service interface uses a semantic segmentation model to obtain an initial classification prediction result of pixels in the training image, and performs feature enhancement processing on image features of the training image based on the initial classification prediction result, and obtains a semantic segmentation prediction result of the training image according to the image features after the feature enhancement processing, and trains the semantic segmentation model based on the initial classification prediction result and the semantic segmentation prediction result;
[0085] Output the semantic segmentation model.
[0086] In a fifth aspect, an embodiment of the present disclosure provides an image semantic segmentation device, which includes:
[0087] A first acquisition module is configured to acquire an image to be processed;
[0088] A first determination module is configured to determine an initial classification prediction result of a pixel point in the image to be processed;
[0089] A first processing module is configured to perform feature enhancement processing on the image features of the image to be processed based on the initial classification prediction result;
[0090] The second acquisition module is configured to obtain the semantic segmentation prediction result of the image to be processed according to the image features after feature enhancement processing.
[0091] In a sixth aspect, an embodiment of the present disclosure provides a training device for a semantic segmentation model, comprising:
[0092] a fifth acquisition module, configured to acquire a training image;
[0093] a sixth acquisition module, configured to acquire initial classification prediction results of pixels in the training image using a semantic segmentation model, and perform feature enhancement processing on image features of the training image based on the initial classification prediction results;
[0094] a seventh acquisition module, configured to acquire a semantic segmentation prediction result of the training image according to the image features after feature enhancement processing;
[0095] A training module is configured to train the semantic segmentation model based on the initial classification prediction result and the semantic segmentation prediction result.
[0096] In a seventh aspect, an embodiment of the present disclosure provides an image semantic segmentation device, comprising:
[0097] An eighth acquisition module is configured to acquire an image to be processed;
[0098] A first calling module is configured to call a preset service interface so that the preset service interface obtains an image to be processed, determines an initial classification prediction result of a pixel in the image to be processed, performs feature enhancement processing on an image feature of the image to be processed based on the initial classification prediction result, and obtains a semantic segmentation prediction result of the image to be processed according to the image feature after the feature enhancement processing;
[0099] The second output module is configured to output the semantic segmentation prediction result.
[0100] In an eighth aspect, an embodiment of the present disclosure provides a training device for a semantic segmentation model, comprising:
[0101] a ninth acquisition module, configured to acquire a training image;
[0102] A second calling module is configured to call a preset service interface so that the preset service interface uses a semantic segmentation model to obtain an initial classification prediction result of a pixel in the training image, and performs feature enhancement processing on an image feature of the training image based on the initial classification prediction result, and obtains a semantic segmentation prediction result of the training image according to the image feature after the feature enhancement processing, and trains the semantic segmentation model based on the initial classification prediction result and the semantic segmentation prediction result;
[0103] The third output module is configured to output the semantic segmentation model.
[0104] The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the functions.
[0105] In one possible design, the structure of the above-mentioned device includes a memory and a processor, the memory is used to store one or more computer instructions that support the above-mentioned device to execute the above-mentioned corresponding method, and the processor is configured to execute the computer instructions stored in the memory. The above-mentioned device may also include a communication interface for the above-mentioned device to communicate with other devices or communication networks.
[0106] In a ninth aspect, an embodiment of the present disclosure provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described in any one of the above aspects.
[0107] In a tenth aspect, an embodiment of the present disclosure provides a computer-readable storage medium for storing computer instructions used by any of the above-mentioned devices, and when the computer instructions are executed by a processor, they are used to implement the steps of the method described in any of the above-mentioned aspects.
[0108] In an eleventh aspect, an embodiment of the present disclosure provides a computer program product, which includes computer instructions, and when the computer instructions are executed by a processor, they are used to implement the steps of the method described in any of the above aspects.
[0109] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:
[0110] In the process of semantic segmentation of an image, the disclosed embodiment first obtains the initial classification prediction results of the pixels in the image, and then performs feature enhancement processing on the image features according to the initial classification prediction results, and then uses the image features after feature enhancement processing to perform semantic segmentation on the image. In this way, since clustering pixels using the image features after feature enhancement processing can make the clustering accuracy higher, the semantic segmentation prediction results finally obtained are also more accurate.
[0111] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0112] Other features, objectives and advantages of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:
[0113] Figure 1 A flowchart of an image semantic segmentation method according to an embodiment of the present disclosure is shown;
[0114] Figure 2 A schematic diagram showing a remote sensing image semantic segmentation prediction result according to an embodiment of the present disclosure;
[0115] Figure 3 A schematic diagram showing a semantic segmentation prediction result in an autonomous driving scenario according to an embodiment of the present disclosure;
[0116] FIG4(a)-(b) are schematic diagrams showing an application of an image semantic segmentation method in an e-commerce scenario according to an embodiment of the present disclosure;
[0117] Figure 5 A flowchart showing a method for training a semantic segmentation model according to an embodiment of the present disclosure is shown;
[0118] Figure 6 A schematic diagram of a semantic segmentation model implementation structure according to an embodiment of the present disclosure is shown;
[0119] Figure 7 A schematic diagram of a CGGCN model implementation structure according to an embodiment of the present disclosure is shown;
[0120] Figure 8 Another schematic diagram of implementing the semantic segmentation model according to one embodiment of the present disclosure is shown;
[0121] Fig. 9 A flowchart showing a method for training a semantic segmentation model according to another embodiment of the present disclosure is shown;
[0122] Fig.10 A flowchart showing an image semantic segmentation method according to another embodiment of the present disclosure is shown;
[0123] Fig.11 It is a structural schematic diagram of an electronic device suitable for implementing a training method for a semantic segmentation model and / or an image semantic segmentation method according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0124] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.
[0125] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate the presence of features, numbers, steps, behaviors, components, parts, or a combination thereof disclosed in the present specification, and are not intended to exclude the possibility that one or more other features, numbers, steps, behaviors, components, parts, or a combination thereof exist or are added.
[0126] It should also be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0127] The details of the embodiments of the present disclosure are described in detail below through specific examples.
[0128] Figure 1 A flowchart of an image semantic segmentation method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the image semantic segmentation method includes the following steps:
[0129] In step S101, an image to be processed is obtained;
[0130] In step S102, the initial classification prediction results of the pixels in the image to be processed are determined;
[0131] In step S103, feature enhancement processing is performed on the image features of the image to be processed based on the initial classification prediction result;
[0132] In step S104, a semantic segmentation prediction result of the image to be processed is obtained according to the image features after feature enhancement processing.
[0133] In this embodiment, the image to be processed may be any image, for example, a road image collected by an autonomous driving vehicle using a camera installed in front of the vehicle, a remote sensing image taken by a satellite, etc. Semantic segmentation can be understood as identifying different contents in an image, for example, roads, pedestrians, vehicles, etc. can be segmented from a road image, or different ground objects can be segmented from a remote sensing image.
[0134] The disclosed embodiment can use a pre-trained semantic segmentation model to perform semantic segmentation on the image to be processed. In the semantic segmentation process, the initial classification prediction results of the pixels in the image to be processed can be first identified, and the image features of the image to be processed can be enhanced based on the initial classification prediction results, and then the semantic segmentation prediction results of the image to be processed can be obtained based on the image features after the feature enhancement processing.
[0135] Semantic segmentation can be understood as dividing the pixels in the image to be processed according to semantic correlation, dividing pixels with strong correlation into one category, and dividing pixels with weak correlation into different categories. In the disclosed embodiment, semantic segmentation is not simply performed on the image to be processed according to the initial predicted classification results of the pixels, but after feature enhancement processing is performed on the image features of the image to be processed based on the initial classification prediction results, semantic segmentation is performed using the image features that have undergone feature enhancement processing, so that in the final semantic segmentation prediction results, pixels belonging to the same type of object and adjacent in distance are divided into one category, while objects of different types or pixels belonging to the same type but not adjacent are divided into different categories.
[0136] It should be noted that since the initial classification prediction result of the image to be processed is not a true value, directly using the initial classification prediction result to perform semantic segmentation on the image to be processed may cause a large error in the semantic segmentation prediction result. Therefore, in the embodiment of the present disclosure, the image features of the image to be processed are further enhanced using the initial classification prediction result, such as semantic enhancement processing, so that the image features after feature enhancement processing can more effectively reflect the semantic correlation between each pixel in the image to be processed, and the semantic segmentation prediction result obtained using the image features after feature enhancement processing is more accurate.
[0137] In the process of semantic segmentation of an image, the disclosed embodiment first obtains the initial classification prediction results of the pixels in the image, and then performs feature enhancement processing on the image features according to the initial classification prediction results, and then uses the image features after feature enhancement processing to perform semantic segmentation on the image. In this way, since clustering pixels using the image features after feature enhancement processing can make the clustering accuracy higher, the semantic segmentation prediction results finally obtained are also more accurate.
[0138] In an optional implementation of this embodiment, step S103, i.e., the step of performing feature enhancement processing on the image features of the image to be processed based on the initial classification prediction result, further includes the following steps:
[0139] Constructing an adjacency matrix of the image to be processed according to the initial classification prediction result; wherein the adjacency matrix is used to describe the semantic correlation relationship between pixels in the image to be processed;
[0140] Perform feature enhancement processing on the image features based on the adjacency matrix.
[0141] In this optional implementation, the image to be processed can be regarded as a graph structure, and the adjacency matrix of the image to be processed can be constructed using the initial classification prediction results, so that the adjacency matrix is used to represent the semantic correlation between pixels in the image to be processed. For example, the element in the i-th row and j-th column of the adjacency matrix represents the semantic correlation between the i-th pixel and the j-th pixel in the image to be processed.
[0142] The semantic correlation relationship can be obtained from the initial classification prediction results of the pixels in the image to be processed. For example, when the categories of two pixels are the same, the semantic correlation relationship between the two pixels is strong, and when the categories of the two pixels are different, the semantic correlation relationship between the two pixels is weak. Therefore, the adjacency matrix used to describe the semantic correlation relationship between pixels in the image to be processed can be constructed by the initial classification prediction results of the pixels in the image to be processed.
[0143] When constructing semantic segmentation, the pixels can be clustered according to the strength of the semantic correlation between the pixels. For example, pixels with strong correlation can be clustered into one category, while pixels with weak correlation can be clustered into different categories. The obtained semantic segmentation prediction results may include the contours and categories of objects in the image to be processed that have been divided according to different semantics. For example, the semantic segmentation prediction results can be obtained by marking different segmented objects with different colors.
[0144] In the feature enhancement process, the image features of the processed image can be enhanced based on the adjacency matrix by using graph clustering. This process can be understood as the feature enhancement of the image features in terms of semantic relevance.
[0145] In the graph clustering method, the image to be processed is regarded as a graph structure G=(V, E) including a number of nodes and edges between two nodes, where V represents a set of nodes in the graph structure and E represents an edge set in the graph structure. The pixels in the image to be processed can be used as nodes in the graph structure, and the semantic correlation between two pixels can be used as an edge between two nodes in the graph structure. When there is a semantic correlation or a strong semantic correlation between the two pixels, there is an edge between the two pixels in the graph structure. When there is no semantic correlation or a weak semantic correlation between the two pixels, there is no edge between the two pixels in the graph structure.
[0146] The graph structure can be represented in the form of an adjacency matrix, in which the two dimensions corresponding to each element in the adjacency matrix represent two nodes in the graph structure, and the element value represents the edge between the two nodes. In the embodiment of the present disclosure, in the adjacency matrix corresponding to the image to be processed, the two dimensions corresponding to each element represent two pixels in the image to be processed, and the element value represents the semantic correlation between the two pixels. In the embodiment of the present disclosure, the image to be processed can be regarded as an undirected graph structure, so the adjacency matrix can be a symmetric matrix.
[0147] In the semantic segmentation application scenario, the semantic correlation between two pixels can represent the degree of similarity between the categories of the two pixels. That is, the semantic correlation can be determined based on the category between the two pixels, and the category can be determined based on the initial classification prediction result described in the foregoing text. The closer the categories between the two pixels are, the higher the semantic correlation between the two pixels; and the greater the category gap between the two pixels, the weaker the semantic correlation between the two pixels. In some embodiments, the initial classification prediction results of the two pixels are represented in the form of vectors, and the two vectors can be directly multiplied to obtain a floating point number between (0, 1). The floating point number is used to represent the correlation between the two pixels, which can also be called the degree of correlation. The closer the floating point number is to 1, the greater the degree of correlation between the two pixels, that is, the higher the semantic correlation between the two pixels.
[0148] In some embodiments, the image features of the processed image can be enhanced based on spectral clustering. Spectral clustering is a clustering method for graph structures, which regards each pixel as a point on the graph structure, and the basis for judging whether two pixels can be clustered into one category (that is, whether they are segmented into the same object) is whether the two pixels are connected by an edge on the graph structure, which can be directly or indirectly connected. Therefore, based on the spectral clustering method of the graph, the image features of the processed image are enhanced according to the semantic correlation relationship obtained by the method proposed in the embodiment of the present disclosure, and then the semantic segmentation prediction result is obtained based on the enhanced image features.
[0149] In the prior art, when the spectral clustering method of the graph is used to perform semantic segmentation on the processed image, the semantic segmentation uses the self-attention mechanism to construct the adjacency matrix A, which can be expressed as follows:
[0150] A=f(δ(X)ψ(X) T )
[0151] Among them, δ, ψ, f can be expressed as convolution transformation, and X is the image to be processed. The adjacency matrix A obtained in this way does not have a very clear interpretation, that is, the value of each element in the adjacency matrix does not have a very clear physical meaning, and lacks supervision information. The model needs to learn the connection relationship between each pixel in the adjacency matrix by itself during the training process.
[0152] Therefore, the embodiment of the present disclosure uses the initial classification prediction results between the pixels in the image to be processed to construct the adjacency matrix corresponding to the image to be processed, and the adjacency matrix is used to describe the semantic correlation between the pixels in the image to be processed. Therefore, the adjacency matrix in the embodiment of the present disclosure has clear explanatory power, that is, the stronger the semantic correlation represented in the adjacency matrix, the closer the categories between the two pixels are. In addition, during the training process, the actual classification results corresponding to the initial classification prediction results can also be used as supervisory information, so that the learning of the adjacency matrix has strong supervision, so that the semantic segmentation model can accurately identify the initial classification prediction results of the pixels in the image to be processed, and then a more accurate adjacency matrix can be obtained based on the initial classification prediction results.
[0153] In some embodiments, a GCN (graph convolutional network) model can be used to enhance the features to be processed. The GCN model can perform semantic segmentation on the image to be processed based on spectral clustering to obtain semantic segmentation features, which can be used to obtain semantic segmentation prediction results corresponding to the image to be processed.
[0154] The GCN network model can be expressed as: Y = σ(AXΘ). Among them, Y is the semantic segmentation feature, X is the image feature corresponding to the image to be processed (dimension is H×W×C), A is the adjacency matrix (dimension is HW×HW), and Θ is the weight matrix of the GCN network model (dimension is C×C).
[0155] The disclosed embodiment utilizes the predicted pixel-level classification prediction information to construct an adjacency matrix so that the adjacency matrix can describe the contextual semantic correlation between pixels in the image, and realizes the fusion of contextual semantic information between pixels through the adjacency matrix to obtain semantically enhanced image features, and then predicts the semantic segmentation prediction results of the image based on the semantically enhanced image features, which can improve the recognition effect of the model.
[0156] In an implementation manner of the first embodiment of the present disclosure, the step of constructing the adjacency matrix corresponding to the image to be processed according to the initial classification prediction result may further include the following steps:
[0157] The adjacency matrix is constructed by multiplying a classification matrix by the transpose of the classification matrix; the classification matrix is a matrix form of the initial classification prediction result.
[0158] In this optional implementation, the adjacency matrix can be represented as a form of multiplying a classification matrix by the transpose of the classification matrix, and the classification matrix is a matrix form of the initial classification prediction result. Each element in the classification matrix represents the category of the pixel point in the image to be processed. For example, when the image to be processed includes a total of 8 categories of objects, the category of each pixel point can be one of the 8 categories.
[0159] The adjacency matrix can be represented as follows:
[0160] A=f(P T , P)
[0161] Among them, A represents the adjacency matrix, P represents the initial classification prediction result, and f represents the dot product operation of the matrix.
[0162] In some embodiments, the GCN network can be used to perform semantic segmentation on the image to be processed, thereby obtaining a semantic segmentation prediction result. In the prior art, the adjacency matrix in the GCN network is obtained by performing multiple convolution transformations on the input image feature X corresponding to the image to be processed, which can be expressed as: A = f(δ(X)ψ(X) T ), and the dimension of the adjacency matrix obtained in this way is HW×HW. Multiplying it with the image features of dimension H×W×C and the weight matrix Θ of dimension C×C, the computational complexity is O(n 2 ), and in the embodiment of the present disclosure, since the adjacency matrix can be expressed as the multiplication of the transpose of the classification matrix and the classification matrix, and the dimensions of the classification matrix and the transpose of the classification matrix are both H×W, the computational complexity of obtaining the semantic segmentation features by multiplying the transpose of the classification matrix and the classification matrix with the image feature X and the weight matrix Θ in sequence can be reduced to O(n). Therefore, in the embodiment of the present disclosure, by expressing the adjacency matrix as the multiplication of the transpose of the classification matrix and the classification matrix, the computational complexity of the network model can be reduced and the recognition efficiency of the model can be improved.
[0163] Through the method of the embodiments of the present disclosure, since the adjacency matrix can be expressed as the form of the multiplication of the classification matrix and the transpose of the classification matrix, and the dimension of the classification matrix is smaller than the dimension of the adjacency matrix, in the process of semantic segmentation of the processed image based on the adjacency matrix, the operations involving multiplication with the adjacency matrix can be split into operations involving multiplication with the classification matrix and the transpose of the classification matrix, which can reduce the computational complexity and improve the efficiency of semantic segmentation.
[0164] In an implementation manner of the first embodiment of the present disclosure, the feature enhancement process includes multiple stages, and the step of performing feature enhancement process on the image features based on the adjacency matrix may further include the following steps:
[0165] Performing feature enhancement processing of the current stage on the input features based on the adjacency matrix; wherein the input features are the image features or output features obtained by feature enhancement processing of the previous stage;
[0166] Step S104, i.e., the step of obtaining the semantic segmentation prediction result of the image to be processed according to the image features after feature enhancement processing, further comprises the following steps:
[0167] The semantic segmentation prediction result is obtained according to the output features obtained by the feature enhancement processing in the last stage.
[0168] In this optional implementation, after obtaining the adjacency matrix using the initial classification prediction results of the pixels in the image to be processed, in the process of using the adjacency matrix to enhance the image features corresponding to the image to be processed, multiple cascaded feature enhancement stages can be used to achieve feature enhancement of the image features.
[0169] In multiple feature enhancement stages, the input of the first stage may include the image features and the adjacency matrix of the image to be processed, and starting from the second stage, the input of each stage includes the output features and the adjacency matrix of the previous stage, and the output of the last stage is the final enhanced feature, that is, the feature used to obtain the semantic segmentation prediction result.
[0170] In the disclosed embodiment, multiple cascaded feature enhancement stages are used to perform feature enhancement processing on the processed image using an adjacency matrix, thereby achieving fine-tuning of the semantic segmentation prediction results of the processed image from coarse to fine, thereby improving the accuracy of the semantic segmentation prediction results.
[0171] In the implementation of the first embodiment of the present disclosure, step S102, i.e., the step of determining the initial classification prediction result of the pixel points in the image to be processed, may further include the following steps:
[0172] Acquire image features of the image to be processed by using a basic model in a semantic segmentation model;
[0173] Using the classification model in the semantic segmentation model to perform classification prediction on the pixel points in the image to be processed to obtain the initial classification prediction result;
[0174] Step S103, i.e., the step of performing feature enhancement processing on the image features based on the adjacency matrix, may further include the following steps:
[0175] The feature enhancement model in the semantic segmentation model is utilized to perform feature enhancement processing on the image features based on the adjacency matrix.
[0176] In this optional implementation, the semantic segmentation model in the disclosed embodiment may include a basic model and a classification model; the semantic segmentation model also includes a feature enhancement model. The basic model is used to extract image features from the image to be processed, the classification model is used to predict the category of each pixel in the image to be processed based on the image features, so as to obtain the initial classification prediction result corresponding to the image to be processed; and the feature enhancement model is used to perform feature enhancement processing on the image features using the initial classification prediction result to obtain feature enhanced image features, which can be used to obtain the semantic segmentation prediction result of the image to be processed.
[0177] The image feature may be a feature extracted from the image to be processed using a basic model, for example, the image feature may be a neural network feature. In some embodiments, an existing basic model may be used to extract image features in the image to be processed, and the basic model may use a neural network model, such as HRNet-W48, ResNet101, ResNet 50, VGG16, etc. After the image to be processed is input into the basic model, the basic model may obtain the corresponding image feature by processing the image to be processed.
[0178] Image features can be understood as feature maps, and each feature point in the feature map corresponds to each pixel in the image to be processed. Therefore, the classification prediction result of each feature point in the image features obtained by the classification model processing the image features is equivalent to the classification prediction result of each pixel in the image to be processed.
[0179] The classification model can use a relatively simple model to predict the rough classification results of feature points in image features. For example, the FCN (Fully Convolutional Networks) model can be used to predict the category of feature points in image features. The FCN model can output pixel-level classification prediction results for the input image features. After the classification prediction results are obtained by the classification model, the classification prediction results are used as auxiliary input information. After the image features and the classification prediction results are processed by the feature enhancement model, semantic enhancement features can be obtained. The semantic enhancement features contain the contextual semantic correlation between pixels. Therefore, the semantic segmentation prediction results of the image to be processed can be obtained through the enhanced features.
[0180] In an optional implementation of this embodiment, the method further includes the following steps:
[0181] When the confidence level of the semantic segmentation prediction result is lower than a preset threshold, displaying the semantic segmentation prediction result to a user;
[0182] Receiving correction data of the semantic segmentation prediction result from the user;
[0183] The semantic segmentation prediction result is adjusted according to the correction data.
[0184] In this optional implementation, the semantic segmentation prediction result may include but is not limited to the category information of each pixel in the image to be processed, and the category information includes the category confidence that the pixel belongs to each category. The category corresponding to the maximum category confidence can be determined as the target category of the pixel.
[0185] The confidence of the semantic segmentation prediction result can be determined according to the category confidence corresponding to the target category of each pixel in the image to be processed. The confidence of the semantic segmentation prediction result lower than the preset threshold can be understood as the average value of the category confidence corresponding to the target category of each pixel is lower than the preset threshold, or the number of pixels with low category confidence corresponding to the target category in the image to be processed is greater than a predetermined number, then the confidence of the image to be processed is considered to be lower than the preset threshold, or other judgment methods that believe that the semantic segmentation prediction result of the image to be processed is not accurate enough.
[0186] In some embodiments, the semantic segmentation prediction results may be displayed to the user by rendering the image to be processed with different colors, for example, pixels belonging to different objects are rendered with different colors. In other embodiments, the semantic segmentation prediction results may be displayed to the user by outlining the image to be processed with lines, for example, the outlines of different objects may be outlined with lines.
[0187] The user can provide correction data for the displayed semantic segmentation results. The correction data can be obtained by directly adjusting the semantic segmentation prediction results through the image editing interface provided on the display interface. For example, the correction data can be provided by adjusting the object contour lines by dragging, stretching, scaling, etc.
[0188] In this way, in the case of inaccurate semantic segmentation, the semantic segmentation prediction results can be provided to the user for manual adjustment, so that a more accurate semantic segmentation result can be obtained in the end.
[0189] In an optional implementation of this embodiment, the method further includes the following steps:
[0190] The model parameters of the semantic segmentation model are adjusted according to the adjusted semantic segmentation prediction result and the semantic segmentation prediction result before the adjustment; wherein the semantic segmentation model is a model for obtaining the semantic segmentation prediction result before the adjustment.
[0191] In this optional implementation, during the semantic segmentation process of the image to be processed, the semantic segmentation prediction result adjusted by the user can also be used to further train the semantic segmentation model, so that the semantic segmentation model can give more correct semantic segmentation results for similar situations.
[0192] In the further training process, the semantic segmentation prediction result adjusted by the user can be used as the true result, and the semantic segmentation prediction result before adjustment obtained by the semantic segmentation model can be used as the prediction result. By fitting the loss between the true result and the prediction result, the model parameters in the semantic segmentation model can be reversely adjusted according to the loss.
[0193] In an optional implementation of this embodiment, the image to be processed is a remote sensing image, and the semantic segmentation prediction result includes a combination of one or more of the following data: contour information of different objects in the remote sensing image, resource types of different objects in the remote sensing image, and road network information in the remote sensing image.
[0194] The image semantic segmentation method in the embodiment of the present disclosure can be applied in a variety of scenarios, such as remote sensing image analysis scenarios, autonomous driving scenarios, etc. In the remote sensing image analysis scenario, the remote sensing image can be input as an image to be processed into the semantic segmentation model proposed in the embodiment of the present disclosure, and the semantic segmentation model performs semantic segmentation on the remote sensing image to obtain the corresponding semantic segmentation prediction result. The semantic segmentation prediction result may include but is not limited to a combination of one or more of the following data: contour information of different objects in the remote sensing image, resource types of different objects in the remote sensing image, and road network information in the remote sensing image.
[0195] Remote sensing images can be obtained by taking pictures with satellites or airplanes, and the remote sensing images include various ground object information on the ground. The semantic segmentation model proposed in the embodiment of the present disclosure can perform semantic segmentation on different ground objects in the remote sensing images, so that the distribution of the ground objects on the ground can be seen from the semantic segmentation prediction results, that is, the contour information of various ground objects can be seen from the semantic segmentation prediction results. In other embodiments, different colors can be used to identify the resource types of different ground objects, such as roads, mountains, buildings, fields, etc. In other embodiments, the road network information of the city can also be obtained through the remote sensing images of the city. The semantic segmentation model of the embodiment of the present disclosure can be used to perform semantic segmentation on the remote sensing images of the city, and the road network information in the city can be obtained through the semantic segmentation prediction results.
[0196] In an optional implementation of this embodiment, the image to be processed is a remote sensing image of a target area, and the method further includes the following steps:
[0197] The target object in the target area is determined according to the semantic segmentation prediction result.
[0198] In this optional implementation, the image semantic segmentation model proposed in the embodiment of the present disclosure can be used to perform semantic segmentation on the remote sensing image of the target area, and then the target object in the target area can be determined based on the semantic segmentation result.
[0199] For example, when determining the greening situation of a city, a remote sensing image of the city can be obtained from above the city, and the remote sensing image can be input into the semantic segmentation model proposed in the embodiment of the present disclosure, and the image features of the remote sensing image can be obtained by the basic model in the semantic segmentation model, and then the image features can be processed by the classification model to obtain the initial classification prediction results of the pixels in the remote sensing image; the adjacency matrix corresponding to the remote sensing image can be constructed using the initial classification prediction results, and the image features can be enhanced using the adjacency matrix and the feature enhancement model, and the image features after feature enhancement can be used to obtain the semantic segmentation results of the remote sensing image. The semantic segmentation results can include the division results of pixels belonging to the category of green plants such as trees, and the greening situation of the city can be determined based on the division results.
[0200] In an optional implementation of this embodiment, the image to be processed is a remote sensing image of a target area, and the method further includes the following steps:
[0201] The resource type of the ground objects included in the target area is determined according to the semantic segmentation prediction result.
[0202] In this optional implementation, the image semantic segmentation model proposed in the embodiment of the present disclosure can be used to perform semantic segmentation on the remote sensing image of the target area, and then the resource type in the target area can be determined based on the semantic segmentation result.
[0203] For example, for an area to be developed with a large area and complex resource types, the resource types in the area to be developed can be determined first, and then it can be decided whether it is suitable to start the current land development project. First, a remote sensing image of the area to be developed can be obtained, and the remote sensing image can be input into the semantic segmentation model proposed in the embodiment of the present disclosure. The image features of the remote sensing image are obtained by the basic model in the semantic segmentation model, and then the image features are processed by the classification model to obtain the initial classification prediction results of the pixels in the remote sensing image; the adjacency matrix corresponding to the remote sensing image is constructed using the initial classification prediction results, and the image features are enhanced using the adjacency matrix and the feature enhancement model. The image features after feature enhancement can be used to obtain the semantic segmentation results of the remote sensing image. The semantic segmentation results can include the division results and category annotations of various resources. According to the division results and category annotations, it is easy to take inventory of the area to be developed and determine whether the area to be developed is suitable for the current land development project.
[0204] In an optional implementation of this embodiment, the image to be processed is a remote sensing image of a target area, and the method further includes the following steps:
[0205] Determine the location and type of the road in the target area according to the semantic segmentation prediction result;
[0206] The road network information in the target area is determined according to the location and type of the road.
[0207] In this optional implementation, the road network information of a certain city or other area can be obtained in advance through the semantic segmentation prediction results.
[0208] For example, in order to obtain the road network information of a certain city, a remote sensing image of the city can be obtained by shooting from above the city, and the remote sensing image can be input into the semantic segmentation model proposed in the embodiment of the present disclosure. The image features of the remote sensing image can be obtained by the basic model in the semantic segmentation model, and then the image features can be processed by the classification model to obtain the initial classification prediction results of the pixels in the remote sensing image; the adjacency matrix corresponding to the remote sensing image can be constructed using the initial classification prediction results, and the image features can be enhanced using the adjacency matrix and the feature enhancement model. The image features after feature enhancement can be used to obtain the semantic segmentation results of the remote sensing image. The semantic segmentation results can include the division results of different roads in the city, such as the location and type of roads, such as trunk roads, highways, provincial roads, main roads, auxiliary roads, etc. The road network information of the city can be extracted based on the above division results.
[0209] Figure 2 A schematic diagram showing a remote sensing image semantic segmentation prediction result according to an embodiment of the present disclosure. Figure 2 It can be seen that in the semantic segmentation prediction results obtained for remote sensing images, it can be clearly seen that roads, buildings, lawns, etc. are separated, and the distribution of main roads in the corresponding area can be seen from the overall perspective.
[0210] In an optional implementation of this embodiment, the image to be processed includes a first remote sensing image and a second remote sensing image, and the method further includes the following steps:
[0211] Change detection information of ground objects in the first remote sensing image and the second remote sensing image is determined according to the semantic segmentation prediction result corresponding to the first remote sensing image and the semantic segmentation prediction result corresponding to the second remote sensing image.
[0212] In this optional implementation, remote sensing images of the same area can also be obtained at different times through the same shooting angle, that is, the first remote sensing image and the second remote sensing image can be remote sensing images of the same area obtained at different times. Currently, in other embodiments, the first remote sensing image and the second remote sensing image can also be remote sensing images of different areas with similar landforms obtained at the same or different times.
[0213] By inputting the first remote sensing image and the second remote sensing image into the semantic segmentation model proposed in the embodiment of the present disclosure, the semantic segmentation prediction results corresponding to the first remote sensing image and the second remote sensing image are obtained. The change detection information between the first remote sensing image and the second remote sensing image can be determined by comparing the semantic segmentation grammar results corresponding to the first remote sensing image and the second remote sensing image. The change detection information may include but is not limited to the difference between the first remote sensing image and the second remote sensing image, for example, the first remote sensing image has one more building or one more road than the second remote sensing image.
[0214] In an optional implementation of this embodiment, the image to be processed includes a front road image collected by the autonomous driving vehicle, and the method further includes the following steps:
[0215] Determine object information on the road ahead according to the semantic segmentation prediction result;
[0216] determining vehicle driving information based on the object information;
[0217] The vehicle driving information is output to the autonomous driving vehicle.
[0218] In this optional implementation, in the autonomous driving scenario, the semantic segmentation model proposed in the embodiment of the present disclosure can be used to semantically segment the road image in front of the autonomous driving vehicle. A camera can be installed in front of the autonomous driving vehicle, and the camera is used to collect the road image in front of the road in real time, and the road image in front of the road is transmitted back to the cloud. The cloud can pre-deploy a semantic segmentation model trained by the model training method proposed in the embodiment of the present disclosure. After the cloud receives the road image in front, the pre-deployed semantic segmentation model is used to obtain the corresponding semantic segmentation prediction result, which may include, for example, the vehicle in front, pedestrians on the road, side vehicles / pedestrians, roads, buildings, trees, and other roadblocks. After obtaining the semantic segmentation prediction result, vehicle driving information can be generated according to the situation where a roadblock appears or does not appear in the semantic segmentation prediction result. The vehicle driving information may include but is not limited to turning direction, braking, speed adjustment, etc. The vehicle driving information can be returned by the cloud to the vehicle control module set on the autonomous driving vehicle, and the vehicle control module controls the driving condition of the autonomous driving vehicle.
[0219] Figure 3 A schematic diagram showing the semantic segmentation prediction results in an autonomous driving scenario according to an embodiment of the present disclosure is shown. Figure 3 As shown in FIG. 1 , after semantic segmentation of the front road image in the left figure, the semantic segmentation prediction result in the right figure can be obtained. In the semantic segmentation prediction result, the large truck in front of the left side of the vehicle, the car in front of the vehicle, and the road on the right are all segmented.
[0220] In an optional implementation of this embodiment, the image to be processed is the first image in the video, and the method further includes the following steps:
[0221] Acquire a second image in the video;
[0222] updating the initial classification prediction result of the first image according to the difference between the first image and the second image to obtain the initial classification prediction result of the second image;
[0223] A semantic segmentation prediction result of the second image is obtained based on the initial classification prediction result of the second image.
[0224] In this optional implementation, the first image may be any frame in the video, and the second image may be an adjacent frame of the first image, or an interval frame whose distance from the first image in the time dimension does not exceed a preset threshold.
[0225] When performing semantic segmentation on images in a video, the semantic segmentation prediction result of the first image in the video can be obtained by using the image semantic segmentation method proposed in the embodiment of the present disclosure. Considering that the first image and one or more frames adjacent to it, that is, the second image, are most likely similar, the semantic segmentation prediction result of the second image will not differ too much from the semantic segmentation prediction result of the first image. Therefore, the initial classification prediction result of the first image can be updated by the difference between the first image and the second image, and then the initial classification prediction result of the second image is obtained, and the semantic segmentation prediction result of the second image is obtained based on the updated initial classification prediction result. In this way, the calculation process of semantic segmentation can be saved and the efficiency of semantic segmentation can be improved.
[0226] In an optional implementation of this embodiment, the image to be processed is the first image in the video, and the method further includes the following steps:
[0227] Obtaining a foreground offset between the first image and the second image;
[0228] The initial classification prediction result is updated according to the foreground offset to obtain an initial classification prediction result of the second image.
[0229] In this optional implementation, the foreground offset between the first image and the second image can be determined by optical flow or other methods, and the foreground offset may include but is not limited to the position offset of the pixel points of the second image relative to the foreground area of the first image. The initial classification prediction result of the first image can be updated according to the foreground offset, and then the updated initial classification prediction result is determined as the initial classification prediction result of the second image. It should be noted that the above-mentioned image semantic segmentation method proposed in the embodiment of the present disclosure is not limited to application scenarios such as remote sensing imaging and autonomous driving, but can also be applied to other scenarios. For example, it can be applied to e-commerce scenarios or e-commerce live broadcast scenarios.
[0230] In an optional implementation of this embodiment, the image to be processed is an image including a commodity, and the method further includes the following steps:
[0231] Acquire a commodity image in the image to be processed according to the semantic segmentation prediction result;
[0232] Visual description information of the product is generated based on the product image.
[0233] In this optional implementation, when the image semantic segmentation model proposed in the embodiment of the present disclosure is applied in an e-commerce scenario, the image semantic segmentation method proposed in the embodiment of the present disclosure can be used to perform semantic segmentation on the image containing the product, and the product image can be obtained according to the semantic segmentation prediction result. The product image is a partial image that only includes the product in the image to be processed, and the product image can be further used to produce visual description information of the product. The visual description information may include but is not limited to promotional videos, promotional images, and virtual animations of the product. The visual description information can be generated, for example, by embedding the product image into a live video, or by embedding the product image into a poster to form a promotional poster for the product, or by embedding the product image into a virtual scene generated by a computer to form a promotional short film for the product, etc. The visual description information can be generated specifically according to actual needs, and there is no limitation here.
[0234] In an optional implementation of this embodiment, the image to be processed is an image in a live video, and the method further includes the following steps:
[0235] Processing the live video according to the semantic segmentation prediction result to obtain a target video;
[0236] The target video is output.
[0237] In this optional implementation, when the image semantic segmentation method proposed in the embodiment of the present disclosure is applied in an e-commerce live broadcast scenario, the above-mentioned image semantic segmentation method proposed in the embodiment of the present disclosure can also be used to perform semantic segmentation on each frame of the image or part of the image in the live video, and the live video can be processed according to the semantic segmentation prediction result to obtain the target video corresponding to the live video, and the target video can be output to users watching the live broadcast. The live video processing can, for example, be through processing the image of the product included in the live video, so that when the user watches the live video, it can deepen the user's impression of the product, or can play a role in effectively promoting the product. The live video processing can also be through extracting the target image containing the product from the live video, and then using the target image to generate visual description information of the product. The specific processing can be carried out according to actual needs, and there is no limitation here.
[0238] In an optional implementation of this embodiment, the live video is processed according to the semantic segmentation prediction result to obtain a target video, further comprising the following steps:
[0239] Determining whether the image to be processed is a target image including a product according to the semantic segmentation prediction result;
[0240] Performing preset processing on the target image.
[0241] In this optional implementation, it can be determined whether the current image in the live video includes a product based on the semantic segmentation prediction result. If the product is included, the image can be used as a target image for preset processing. The preset processing can, for example, be editing and processing the target image, rendering the product image in the target image, so that the product image in the target image has a better visual effect, which can further deepen the user's impression of the product and enhance the user's interest in the product. The specific preset processing can be determined according to actual needs and is not limited here.
[0242] In an optional implementation of this embodiment, the live video is processed according to the semantic segmentation prediction result to obtain a target video, further comprising the following steps:
[0243] Acquire a target image including a product in the live video according to the semantic segmentation prediction result;
[0244] Visual description information of the product is generated according to the target image.
[0245] In this optional implementation, semantic segmentation can be performed on each frame or part of the frame image in the live video, and the target image including the product can be determined based on the semantic segmentation prediction result, and the target image can be extracted. Then, target videos such as promotional videos of the approval stock can be generated based on the target image. The target video can be promoted on the advertising platform or the product page.
[0246] Figure 4 (a)-(b) shows a schematic diagram of the application of the image semantic segmentation method in an e-commerce scenario according to an embodiment of the present disclosure. The image semantic segmentation model that implements the image semantic segmentation method proposed in the embodiment of the present disclosure can be deployed in the cloud.
[0247] As shown in FIG4(a), the e-commerce platform can provide a calling interface for the semantic segmentation model on the server. Merchants can upload images that need to be semantically segmented to the cloud through the calling interface via the user interface provided by the client of the e-commerce platform. The cloud uses the semantic segmentation model to semantically segment the image, and then returns the semantic segmentation prediction result to the server of the e-commerce platform. The server of the e-commerce platform provides the product image to the merchant based on the semantic segmentation prediction result. The merchant can create visual description information of the product based on the product image. The server of the e-commerce platform can also provide various visual description information generation interfaces. Merchants can call these generation interfaces through the client to create corresponding visual description information. Merchants can further publish the created visual description information on the e-commerce platform so that users can view the visual description information when viewing the product.
[0248] As shown in FIG4(b), the live broadcast platform may also provide a calling interface of the semantic segmentation model on the server. Live broadcast related personnel may upload each frame or part of the frame image in the live broadcast video to the cloud through the calling interface through the user interface provided by the client of the live broadcast platform. The cloud uses the semantic segmentation model to semantically segment the image, and then returns the semantic segmentation prediction result to the server of the live broadcast platform. The server of the live broadcast platform may process the live broadcast video according to the semantic segmentation prediction result. Live broadcast related personnel may pre-set the processing method for the live broadcast video according to their own needs. The server processes the live broadcast video according to the settings of the live broadcast related personnel. For example, after receiving a frame of the live broadcast video, the product in the image may be processed and the processed live broadcast video image may be output to the live broadcast audience. For another example, the server may also extract the target image including the product from the live broadcast video after the live broadcast ends, and then use these target images to form a promotional short video of the product, and provide the promotional short video to the live broadcast related personnel so that the live broadcast related personnel can promote the short video as a promotional film of the product to the user.
[0249] Figure 5A flowchart of a method for training an image semantic segmentation model according to an embodiment of the present disclosure is shown. Figure 5 As shown, the training method of the image semantic segmentation model includes the following steps:
[0250] In step S501, a training image is obtained;
[0251] In step S502, an initial classification prediction result of pixels in the training image is obtained using a semantic segmentation model, and feature enhancement processing is performed on image features of the training image based on the initial classification prediction result;
[0252] In step S503, a semantic segmentation prediction result of the training image is obtained according to the image features after feature enhancement processing;
[0253] In step S504, the semantic segmentation model is trained based on the initial classification prediction result and the semantic segmentation prediction result.
[0254] In this embodiment, the training image can be any image. The semantic segmentation model is used to perform semantic segmentation on the image. Semantic segmentation can be understood as identifying different contents in the image, for example, segmenting roads, pedestrians, vehicles, etc. from a road image, or segmenting different objects from a remote sensing image.
[0255] The semantic segmentation model can classify and predict the pixels in the training image, and perform semantic segmentation on the training image according to the initial classification prediction results. Semantic segmentation can be understood as dividing the pixels in the training image according to semantic correlation, dividing the pixels with strong correlation into one category, and dividing the pixels with weak correlation into different categories. In the disclosed embodiment, the training image is not simply segmented according to the category of the pixels, but after the image features of the training image are enhanced based on the initial classification prediction results, semantic segmentation is performed using the image features that have been enhanced, so that in the final semantic segmentation prediction results, pixels belonging to the same type of object and adjacent in distance are divided into one category, while objects of different categories or pixels of the same category but not adjacent are divided into different categories.
[0256] It should be noted that since the initial classification prediction result of the training image is not a true value, directly using the initial classification prediction result to perform semantic segmentation on the training image may cause a large error in the semantic segmentation prediction result. Therefore, in the embodiment of the present disclosure, the image features of the training image are further enhanced by using the initial classification prediction result, such as semantic enhancement processing, so that the image features after feature enhancement processing can more effectively reflect the semantic correlation between each pixel in the training image, and the semantic segmentation prediction result obtained by using the image features after feature enhancement processing is more accurate. The semantic segmentation model can process the training image using a neural network or other model that can predict the category of each pixel in the training image to obtain the initial classification prediction result for each pixel in the training image. After obtaining the initial classification prediction result for each pixel in the training image, the semantic segmentation model can also use the initial classification prediction result to perform feature enhancement processing on the image features of the training image, and then use the image features after feature enhancement to obtain the semantic segmentation prediction result corresponding to the training image.
[0257] During the training process, the embodiments of the present disclosure use the initial classification prediction results and the semantic segmentation prediction results to train the semantic segmentation model. In some embodiments, the initial classification prediction results can be used to adjust some model parameters related to the classification prediction through the back propagation method; and the semantic segmentation prediction results can be used to adjust all model parameters in the semantic segmentation model. The process of adjusting some model parameters related to the classification prediction using the initial classification prediction results and the process of adjusting all model parameters in the semantic segmentation model using the semantic segmentation prediction results can be two parallel branches.
[0258] In the model training process of the disclosed embodiment, in addition to using the semantic segmentation prediction results to learn the semantic correlation between pixels in the training image, the initial classification prediction results are used to learn the pixel-level category relationship in the training image. The semantic segmentation prediction results are used to supervise the semantic segmentation model as a whole, and the initial classification prediction results are used to supervise the classification prediction part. In this way, the semantic segmentation model can quickly and accurately learn the pixel-level semantic relationship in the training image, thereby improving the recognition accuracy of the semantic segmentation model.
[0259] In an optional implementation of this embodiment, the step of performing feature enhancement processing on the image features of the training image based on the initial classification prediction result in step S502 further includes the following steps:
[0260] Constructing an adjacency matrix corresponding to the training image according to the initial classification prediction result, wherein the adjacency matrix is used to describe the semantic correlation relationship between pixels in the training image;
[0261] Feature enhancement processing is performed on the image features corresponding to the training image based on the adjacency matrix.
[0262] In this optional implementation, the disclosed embodiment can also perform feature enhancement processing on the image features of the training image through the semantic correlation between pixels in the training image, and the image features after feature enhancement can be used to obtain the semantic segmentation prediction results of the training image. During the feature enhancement process, the adjacency matrix of the training image is first constructed using the initial classification prediction results of the pixels in the training image, and each element in the adjacency matrix is used to represent the semantic correlation between each pair of pixels in the training image; based on the semantic correlation, the image features of the training image are feature enhanced, so that the image features after feature enhancement can more effectively reflect the semantic correlation between each pixel in the image to be processed, and the semantic segmentation prediction results obtained using the image features after feature enhancement processing are more accurate.
[0263] During the feature enhancement process, the image features of the training image can be enhanced by using graph clustering. The image features after feature enhancement can be understood as enhanced features of the image features in terms of semantic relevance.
[0264] In the graph clustering method, the training image is regarded as a graph structure G = (V, E) including a number of nodes and edges between two nodes, where V represents a set of nodes in the graph structure and E represents an edge set in the graph structure. The pixels in the training image can be used as nodes in the graph structure, and the semantic correlation between two pixels can be used as an edge between two nodes in the graph structure. When there is a semantic correlation or a strong semantic correlation between the two pixels, there is an edge between the two pixels in the graph structure, and when there is no semantic correlation or a weak semantic correlation between the two pixels, there is no edge between the two pixels in the graph structure.
[0265] The graph structure can be represented in the form of an adjacency matrix, in which the two dimensions corresponding to each element in the adjacency matrix represent two nodes in the graph structure, and the element value represents the edge between the two nodes. In the embodiment of the present disclosure, in the adjacency matrix corresponding to the training image, the two dimensions corresponding to each element represent two pixels in the training image, and the element value represents the semantic correlation between the two pixels. In the embodiment of the present disclosure, the training image can be regarded as an undirected graph structure, so the adjacency matrix can be a symmetric matrix.
[0266] In the semantic segmentation application scenario, the semantic correlation between two pixels can represent the degree of similarity between the categories of the two pixels. That is, the semantic correlation can be determined based on the category between the two pixels, and the category can be determined based on the initial classification prediction result described in the foregoing text. The closer the categories between the two pixels are, the higher the semantic correlation between the two pixels; and the greater the category gap between the two pixels, the weaker the semantic correlation between the two pixels. In some embodiments, the initial classification prediction results of the two pixels are represented in the form of vectors, and the two vectors can be directly multiplied to obtain a floating point number between (0, 1). The floating point number is used to represent the correlation between the two pixels, which can also be called the degree of correlation. The closer the floating point number is to 1, the greater the degree of correlation between the two pixels, that is, the higher the semantic correlation between the two pixels.
[0267] In some embodiments, the image features of the training images can be enhanced based on spectral clustering. Spectral clustering is a clustering method for graph structures, which regards each pixel as a point on a graph structure, and the basis for judging whether two pixels can be clustered into one category (that is, whether they are segmented into the same object) is whether the two pixels are connected by an edge on the graph structure, which can be directly or indirectly connected. Therefore, based on the spectral clustering method of the graph, the image features of the training images can be enhanced according to the method proposed in the embodiment of the present disclosure, and then the semantic segmentation prediction result is obtained based on the image features after feature enhancement.
[0268] In the prior art, when the spectral clustering method of the graph is used to perform semantic segmentation on the training image, the semantic segmentation uses the self-attention mechanism to construct the adjacency matrix A, which can be expressed as follows:
[0269] A=f(δ(X)ψ(X) T )
[0270] Among them, δ, ψ, f can be expressed as convolution transformation, and X is the training image. The adjacency matrix A obtained in this way does not have a very clear interpretation, that is, the value of each element in the adjacency matrix does not have a very clear physical meaning, and lacks supervision information. The model needs to learn the connection relationship between each pixel in the adjacency matrix by itself during the training process.
[0271] Therefore, the embodiment of the present disclosure uses the initial classification prediction results between the pixels in the training image to construct the adjacency matrix corresponding to the training image, and the adjacency matrix is used to describe the semantic correlation between the pixels in the training image. Therefore, the adjacency matrix in the embodiment of the present disclosure has clear explanatory power, that is, the stronger the semantic correlation represented in the adjacency matrix, the closer the categories between the two pixels are. In addition, during the training process, the actual classification results corresponding to the initial classification prediction results can also be used as supervisory information, so that the learning of the adjacency matrix has strong supervision, so that the semantic segmentation model can accurately identify the initial classification prediction results of the pixels in the training image, and then can obtain a more accurate adjacency matrix based on the initial classification prediction results.
[0272] In some embodiments, a GCN (graph convolutional network) model can be used to implement the feature enhancement process. The GCN model can perform semantic segmentation on the training image based on spectral clustering to obtain semantic segmentation features, which can be used to obtain semantic segmentation prediction results corresponding to the training image.
[0273] The GCN network model can be expressed as: Y = σ(AXΘ). Where Y is the semantic segmentation feature, X is the image feature corresponding to the training image (dimension is H×W×C), A is the adjacency matrix (dimension is HW×HW), and Θ is the weight matrix of the GCN network model (dimension is C×C).
[0274] The disclosed embodiment utilizes the predicted pixel-level classification prediction information to construct an adjacency matrix so that the adjacency matrix can describe the contextual semantic correlation between pixels in the image, and realizes the fusion of contextual semantic information between pixels through the adjacency matrix, and then predicts the semantic segmentation prediction result of the image based on the fused contextual semantic information, which can improve the recognition effect of the model.
[0275] In an optional implementation of this embodiment, constructing an adjacency matrix corresponding to the training image according to the initial classification prediction result further includes the following steps:
[0276] In this optional implementation, the adjacency matrix can be represented as a form of multiplying a classification matrix by the transpose of the classification matrix, and the classification matrix is a matrix form of the initial classification prediction result. Each element in the classification matrix represents the category of a pixel in the training image. For example, when the training image includes a total of 8 categories of objects, the category of each pixel can be one of the 8 categories.
[0277] The adjacency matrix can be represented as follows:
[0278] A=f(P T , P)
[0279] Among them, A represents the adjacency matrix, P represents the initial classification prediction result, and f represents the dot product operation of the matrix.
[0280] In some embodiments, the GCN network can be used to enhance the image features of the training image, and then semantic segmentation can be performed based on the enhanced image features to obtain semantic segmentation prediction results. In the prior art, the adjacency matrix in the GCN network is obtained by performing multiple convolution transformations on the input, that is, the image feature X corresponding to the training image, which can be expressed as: A = f(δ(X)ψ(X) T ), and the dimension of the adjacency matrix obtained in this way is HW×HW. Multiplying it with the image features of dimension H×W×C and the weight matrix Θ of dimension C×C, the computational complexity is O(n 2 ), and in the embodiment of the present disclosure, since the adjacency matrix can be expressed as the multiplication of the transpose of the classification matrix and the classification matrix, and the dimensions of the classification matrix and the transpose of the classification matrix are both H×W, the computational complexity of obtaining the semantic segmentation features by multiplying the transpose of the classification matrix and the classification matrix with the image feature X and the weight matrix Θ in sequence can be reduced to O(n). Therefore, in the embodiment of the present disclosure, by expressing the adjacency matrix as the multiplication of the transpose of the classification matrix and the classification matrix, the computational complexity of the network model can be reduced and the recognition efficiency of the model can be improved.
[0281] In an optional implementation of this embodiment, the semantic segmentation model includes a basic model, a classification model and a feature enhancement model, and step S403, i.e., the step of training the semantic segmentation model based on the initial classification prediction result and the semantic segmentation prediction result, further includes the following steps:
[0282] The model parameters of the classification model and the basic model are adjusted by the loss between the initial classification prediction result and the corresponding true classification result; the classification model is used to obtain the initial classification prediction result; the basic model is used to obtain the image feature;
[0283] The model parameters of the feature enhancement model and the basic model are adjusted by the loss between the semantic segmentation prediction result and its corresponding semantic segmentation true result; the eigenvalue enhancement model is used to perform feature enhancement processing on the image features.
[0284] In this optional implementation, the semantic segmentation model may include a basic model, a classification model, and a feature enhancement model. The basic model may extract image features from the training image, the classification model may classify each pixel in the training image based on the image features, and thereby obtain an initial classification prediction result of the training image; the feature enhancement model may perform feature enhancement processing on the image features based on the initial classification prediction result, so that the image features after feature enhancement can be updated to accurately reflect the semantic correlation between pixel points, thereby making the semantic segmentation prediction result of the training image obtained based on the image features after feature enhancement more accurate.
[0285] The image feature may be a feature extracted from the training image using a basic model, for example, the image feature may be a neural network feature. In some embodiments, an existing basic model may be used to extract image features in the training image, and the basic model may use a neural network model, such as HRNet-W48, ResNet101, ResNet50, VGG16, etc. After the training image is input into the basic model, the basic model may obtain the corresponding image feature by processing the training image.
[0286] Image features can be understood as feature maps, and each feature point in the feature map corresponds to each pixel in the training image. Therefore, the initial classification prediction result of each feature point in the image features obtained by the classification model processing the image features is equivalent to the initial classification prediction result of each pixel in the training image. After the feature enhancement model processes the image features to obtain semantic segmentation features, the semantic segmentation prediction result obtained through the semantic segmentation features is equivalent to the semantic segmentation prediction result of the training image.
[0287] The classification model can use a relatively simple model to predict the rough classification results of feature points in image features. For example, the FCN (Fully Convolutional Networks) model can be used to predict the category of feature points in image features. The FCN model can output the initial classification prediction results at the pixel level for the input image features. After the initial classification prediction results are obtained by the classification model, the initial classification prediction results are used as auxiliary input information. After the image features and the initial classification prediction results are processed by the feature enhancement model, the semantic enhancement features can be obtained. The semantic enhancement features contain the contextual semantic correlation between pixels. Therefore, the semantic segmentation prediction results of the training image can be obtained through the enhanced features.
[0288] During the model training process, after obtaining the initial classification prediction result, the loss between the initial classification prediction result and the actual classification result corresponding to each pixel in the training image can be fitted by constructing a loss function. The loss function can refer to existing related technologies and is not limited here. The actual classification result can be obtained by manually or other reliable methods by labeling the pixels in the training image.
[0289] In addition, the input to the classification model is the image features corresponding to the training images, and the image features can be extracted from the training images using a known basic model. Therefore, in the process of adjusting the model parameters of the classification model using the initial classification prediction results, the model parameters of the basic model can also be adjusted at the same time.
[0290] After obtaining the semantic segmentation prediction result, the model parameters of the feature enhancement model can also be adjusted based on the real semantic segmentation prediction result corresponding to the semantic segmentation prediction result. During the adjustment process, the loss between the semantic segmentation prediction result and the real semantic segmentation prediction result corresponding to the training image can also be fitted by constructing a loss function. The loss function can refer to the existing related technology and is not limited here. The real semantic segmentation prediction result can be obtained by dividing the training image manually or in other reliable ways.
[0291] In addition, the input to the feature enhancement model is the image features corresponding to the training image, and the image features can be extracted from the training image using the known basic model results. Therefore, in the process of adjusting the model parameters of the feature enhancement model using the semantic segmentation prediction results, the model parameters of the basic model can also be adjusted at the same time.
[0292] Through the above method, the actual classification results of the pixels in the training image can be used as supervision information to train the classification model, so that the adjacency matrix calculated according to the initial classification prediction results output by the classification model has supervision information, making the adjacency matrix obtained using the classification model more accurate.
[0293] In an optional implementation of this embodiment, the feature enhancement process includes multiple stages; step S502, i.e., the step of performing feature enhancement process on the image features corresponding to the training image based on the adjacency matrix, further includes the following steps:
[0294] Performing feature enhancement processing of the current stage on the input features based on the adjacency matrix to obtain output features; the input features are the image features or the output features obtained after feature enhancement processing in the previous stage;
[0295] The semantic segmentation prediction result is obtained based on the output features obtained after the feature enhancement processing in the last stage.
[0296] In this optional implementation, the image features can be enhanced through multiple feature enhancement processes. In each feature enhancement process, the input features are processed using an adjacency matrix that describes the contextual semantic relationship between pixels in the image to obtain enhanced features. The enhanced features can be further used as input features of the next stage for further feature enhancement until the last feature enhancement stage. The last feature enhancement stage outputs the final features, based on which the semantic segmentation prediction results can be obtained.
[0297] In the embodiment of the present disclosure, a feature enhancement process is performed in multiple stages. Each feature enhancement process uses an adjacency matrix to perform feature enhancement processing on the input features, thereby achieving fine-tuning of the semantic segmentation prediction results of the image from coarse to fine, thereby improving the recognition effect of the semantic segmentation model.
[0298] Figure 6 FIG. 2 shows a schematic diagram of a semantic segmentation model implementation structure according to an embodiment of the present disclosure. Figure 6 As shown, the semantic segmentation model includes a basic model and a CGGCN (Category guided GCN) model. The input image (such as a training image) is passed through the basic model to obtain an image feature X. The image feature X is passed through the CGGCN model to obtain a semantic segmentation feature. The semantic segmentation feature is passed through an FCN (Fully Convolutional Networks) model to obtain a semantic segmentation prediction result.
[0299] Figure 7 FIG. 2 shows a schematic diagram of a CGGCN model implementation structure according to an embodiment of the present disclosure. Figure 7 As shown, from Figure 6 The image feature X obtained by the basic model shown is used as the input of the CGGCN model and enters two branches for processing: the main branch and the auxiliary branch.
[0300] In the auxiliary branch, the classification model is used to perform classification prediction on the image feature X to obtain the initial classification prediction result corresponding to the image feature X. The initial classification prediction result can be expressed in the form of a matrix P. The dimension of P can be C×H×W, where H and W can be the spatial dimensions of P, that is, height and width, and C is the feature dimension, that is, the length of the feature vector corresponding to each feature point. Each element value in the matrix P is a feature vector of length C, which is used to represent the category of the corresponding pixel point in the training image.
[0301] During the training process, the classification loss can be obtained by fitting the initial classification prediction results output by the classification model and the actual classification results obtained by manual annotation or other reliable methods. Afterwards, the classification loss can be used to adjust the model parameters of the classification model and the basic model through the known back propagation method.
[0302] After multiplying the classification matrix P and the transpose of the classification matrix P, the adjacency matrix A can be obtained. In some embodiments, each element value in the classification matrix can be encoded in a one-hot manner. For example, when there are a total of 8 categories, each element value can be represented by 8 strings, each character in the 8 strings corresponds to one of the categories, and only one of the 8 strings is 1, while the others are 0. The category corresponding to the character that is 1 is the category of the pixel corresponding to the element. After the classification matrix P and the transpose of the initial classification matrix P are multiplied, if the categories of the pixels corresponding to the two multiplied elements are the same, the multiplication result is also 1, and if the categories of the pixels corresponding to the two multiplied elements are different, the multiplication result is 0. Therefore, in the obtained adjacency matrix, the element value of each element can be 1 or 0. If it is 1, it indicates that there is a strong correlation between the two pixels corresponding to the element, and there is a weak correlation between the two image pixels corresponding to the element with a value of 0. It can be understood that the adjacency matrix is used to represent the correlation between the pixels in the image, and the more similar the category distribution is, the stronger the correlation between the two pixels.
[0303] In the main branch, the feature enhancement model can be used to perform feature enhancement processing on the image feature X in terms of semantic relevance based on the adjacency matrix. In this embodiment, the feature enhancement model adopts the GCN model. In the GCN model, feature enhancement processing is performed by clustering the pixels in the training image. After the image feature X is transformed by dimensionality reduction, it is multiplied by the adjacency matrix A obtained by the auxiliary branch. The result of the multiplication is transformed by dimensionality recovery (that is, transformed to the same dimensional size as the image feature X), and then passed through the convolution layer to obtain the processing result. The convolution layer is used to estimate the weight matrix of the GCN model. The processing result is spliced with the image feature X to form a semantic segmentation feature. The semantic segmentation feature obtained by the main branch is input to Figure 6 The final semantic segmentation prediction result is obtained after the FCN shown in the figure. The semantic segmentation prediction result can separate the objects of different categories and the background in the input image.
[0304] During the training process, Figure 6 In the model structure shown, the semantic segmentation prediction result and the semantic segmentation true result are fitted to obtain the semantic segmentation loss. The semantic segmentation loss can be used to adjust the model parameters of the FCN, each model in the trunk branch, and the base model through the known back propagation method.
[0305] Figure 8 FIG. 2 shows another implementation structure diagram of a semantic segmentation model according to an embodiment of the present disclosure. Figure 8 As shown in Figure 1, the semantic segmentation model includes a basic model, a classification model, and two cascaded GCN models. The input image (e.g., training image) is passed through the basic model to obtain image features X, which are then input into the classification model of the auxiliary branch to obtain the initial classification prediction results, and then the adjacency matrix is obtained based on the initial classification prediction results. The processing of the auxiliary branch can be seen in Figure 7 The description in will not be repeated here.
[0306] The image feature X is also input to the trunk branch, which includes two cascaded GCN models, and the adjacency matrix used in each GCN model is obtained from the auxiliary branch, that is, after the auxiliary branch obtains the adjacency matrix using the image feature X, it outputs the adjacency matrix to each GCN model in the trunk branch, and the GCN model obtains the output feature using the input features (image features or enhanced features of the previous GCN model) and the adjacency matrix; the input of the first GCN model in the trunk branch is the image features and the adjacency matrix obtained by the auxiliary branch, while the input of the second GCN model is the enhanced features output by the previous GCN model, and the last GCN model outputs the final feature, which is passed through an FCN model to obtain the semantic segmentation prediction result.
[0307] It should be noted that Figure 8 In the model structure shown, the number of GCN models is not limited to two, and more than two GCN models can be cascaded. The specific setting can be based on actual needs and is not limited here.
[0308] In the disclosed embodiment, the input image is passed through the basic model to obtain image features, and the image features are passed through a classification model such as an FCN model to obtain initial classification prediction results. The initial classification prediction results can obtain the semantic correlation between pixels in the image by constructing an adjacency matrix, and then obtain the semantic segmentation features corresponding to the image based on the semantic correlation and the image features. The semantic segmentation features are actually used to represent the correlation between pixels in the image. The more accurate the correlation is, the more accurate the semantic segmentation results will be. However, since the above initial classification prediction results are the coarse segmentation results of the image at the pixel level, the actual situation is that there may be some pixel misclassifications in the coarse segmentation results. Therefore, if the coarse segmentation results are directly used to perform semantic segmentation on the image, the final semantic segmentation results will have errors. Figure 7In the model structure shown, the cascaded GCN is used to continuously adjust the coarse segmentation results, so that the semantic enhancement features obtained after continuous adjustment can more accurately represent the correlation between pixels in the image, and then the image is semantically segmented according to the more accurate correlation, and the obtained semantic segmentation results are also more accurate.
[0309] Fig. 9 FIG. 4 is a flowchart showing a method for training a semantic segmentation model according to another embodiment of the present disclosure. Fig. 9 As shown, the model training method includes the following steps:
[0310] In step S901, a training image is obtained;
[0311] In step S902, a preset service interface is called so that the preset service interface obtains an initial classification prediction result of pixels in the training image using a semantic segmentation model, performs feature enhancement processing on image features of the training image based on the initial classification prediction result, obtains a semantic segmentation prediction result of the training image based on the image features after the feature enhancement processing, and trains the semantic segmentation model based on the initial classification prediction result and the semantic segmentation prediction result;
[0312] In step S903, the semantic segmentation model is output.
[0313] In this embodiment, the method can be executed in the cloud. The preset service interface can be pre-deployed in the cloud, and the preset service interface can be a SaaS (Software-as-a-service) interface. The demander can obtain the right to use the preset service interface in advance, and can train the required semantic segmentation model by calling the preset service interface when needed.
[0314] In this embodiment, the demander can provide the collected multiple training images to the preset service interface, and the preset service interface will train the corresponding semantic segmentation model based on the training images, and then output the trained semantic segmentation model to the demander.
[0315] The training image can be any image. The semantic segmentation model is used to perform semantic segmentation on the image. Semantic segmentation can be understood as identifying different contents in the image, for example, segmenting roads, pedestrians, vehicles, etc. from a road image, or segmenting different ground objects from a remote sensing image.
[0316] The semantic segmentation model can classify and predict the pixels in the training image, and perform semantic segmentation on the training image according to the initial classification prediction results. Semantic segmentation can be understood as dividing the pixels in the training image according to semantic correlation, dividing the pixels with strong correlation into one category, and dividing the pixels with weak correlation into different categories. In the disclosed embodiment, the training image is not simply segmented according to the category of the pixels, but after the image features of the training image are enhanced based on the initial classification prediction results, semantic segmentation is performed using the image features that have been enhanced, so that in the final semantic segmentation prediction results, pixels belonging to the same type of object and adjacent in distance are divided into one category, while objects of different categories or pixels of the same category but not adjacent are divided into different categories.
[0317] It should be noted that since the initial classification prediction result of the training image is not a true value, directly using the initial classification prediction result to perform semantic segmentation on the training image may cause a large error in the semantic segmentation prediction result. Therefore, in the embodiment of the present disclosure, the image features of the training image are further enhanced by using the initial classification prediction result, such as semantic enhancement processing, so that the image features after feature enhancement processing can more effectively reflect the semantic correlation between each pixel in the training image, and the semantic segmentation prediction result obtained by using the image features after feature enhancement processing is more accurate. The semantic segmentation model can process the training image using a neural network or other model that can predict the category of each pixel in the training image to obtain the initial classification prediction result for each pixel in the training image. After obtaining the initial classification prediction result for each pixel in the training image, the semantic segmentation model can also use the initial classification prediction result to perform feature enhancement processing on the image features of the training image, and then use the image features after feature enhancement to obtain the semantic segmentation prediction result corresponding to the training image.
[0318] During the training process, the embodiments of the present disclosure use the initial classification prediction results and the semantic segmentation prediction results to train the semantic segmentation model. In some embodiments, the initial classification prediction results can be used to adjust some model parameters related to the classification prediction through the back propagation method; and the semantic segmentation prediction results can be used to adjust all model parameters in the semantic segmentation model. The process of adjusting some model parameters related to the classification prediction using the initial classification prediction results and the process of adjusting all model parameters in the semantic segmentation model using the semantic segmentation prediction results can be two parallel branches.
[0319] In the model training process of the disclosed embodiment, in addition to using the semantic segmentation prediction results to learn the semantic correlation between pixels in the training image, the initial classification prediction results are used to learn the pixel-level category relationship in the training image. The semantic segmentation prediction results are used to supervise the semantic segmentation model as a whole, and the initial classification prediction results are used to supervise the classification prediction part. In this way, the semantic segmentation model can quickly and accurately learn the pixel-level semantic relationship in the training image, thereby improving the recognition accuracy of the semantic segmentation model.
[0320] Fig.10 FIG. 5 is a flowchart of an image semantic segmentation method according to another embodiment of the present disclosure. Fig.10 As shown, the image semantic segmentation method includes the following steps:
[0321] In step S1001, an image to be processed is obtained;
[0322] In step S1002, a preset service interface is called so that the preset service interface obtains an image to be processed, determines an initial classification prediction result of pixels in the image to be processed, performs feature enhancement processing on image features of the image to be processed based on the initial classification prediction result, and obtains a semantic segmentation prediction result of the image to be processed according to the image features after the feature enhancement processing;
[0323] In step S1003, the semantic segmentation prediction result is output.
[0324] In this embodiment, the method can be executed in the cloud. The preset service interface can be pre-deployed in the cloud, and the preset service interface can be a SaaS (Software-as-a-service) interface. The demander can obtain the right to use the preset service interface in advance, and can perform semantic segmentation on the image to be processed by calling the preset service interface when necessary, thereby obtaining the semantic segmentation prediction result.
[0325] Take the demand side as an example. The user uploads the image to be processed through the user interface on the client device, and the user selects the scene applicable to the currently uploaded image to be processed on the user interface, and initiates a semantic segmentation request to the cloud by clicking buttons such as submit. After the cloud receives the request, it calls the corresponding preset service interface according to the scene selected by the user. The preset service interface is used to perform semantic segmentation on the image to be processed uploaded by the user using the semantic segmentation model under the scene. The cloud can return the semantic segmentation prediction result output by the semantic segmentation model to the user's client device, and the client device displays the semantic segmentation prediction result on the user interface. The user can also correct the semantic segmentation prediction result through the interface provided on the user interface, and the client device can feed back the user's corrected data to the cloud so that the cloud can further train the semantic segmentation model based on the corrected data.
[0326] In this embodiment, semantic segmentation models suitable for different scenarios can be pre-trained for different fields, and then the semantic segmentation models suitable for different scenarios can be deployed in the cloud. Different preset service interfaces can be set for semantic segmentation models of different scenarios. The demander can provide the image to be processed to the corresponding preset service interface, and the corresponding preset service interface uses the pre-deployed semantic segmentation model to perform semantic segmentation on the image to be processed, and then outputs the semantic segmentation prediction result to the demander.
[0327] For example, geographic surveyors can provide the remote sensing images to be processed to a preset service interface suitable for semantic segmentation of remote sensing images, and obtain the types of ground resources in the remote sensing images from the preset service interface; and autonomous driving vehicles can provide the collected images of the road ahead to a preset service interface suitable for autonomous driving scenarios, and obtain roadblock information from the preset service interface.
[0328] The image to be processed can be any image, for example, a road image collected by an autonomous vehicle using a camera installed in front of the vehicle, a remote sensing image taken by a satellite, etc. Semantic segmentation can be understood as identifying different contents in an image, for example, segmenting roads, pedestrians, vehicles, etc. from a road image, or segmenting different objects from a remote sensing image.
[0329] The disclosed embodiment can use a pre-trained semantic segmentation model to perform semantic segmentation on the image to be processed. In the semantic segmentation process, the initial classification prediction results of the pixels in the image to be processed can be first identified, and the image features of the image to be processed can be enhanced based on the initial classification prediction results, and then the semantic segmentation prediction results of the image to be processed can be obtained based on the image features after the feature enhancement processing.
[0330] Semantic segmentation can be understood as dividing the pixels in the image to be processed according to semantic correlation, dividing pixels with strong correlation into one category, and dividing pixels with weak correlation into different categories. In the disclosed embodiment, semantic segmentation is not simply performed on the image to be processed according to the initial predicted classification results of the pixels, but after feature enhancement processing is performed on the image features of the image to be processed based on the initial classification prediction results, semantic segmentation is performed using the image features that have undergone feature enhancement processing, so that in the final semantic segmentation prediction results, pixels belonging to the same type of object and adjacent in distance are divided into one category, while objects of different types or pixels belonging to the same type but not adjacent are divided into different categories.
[0331] It should be noted that since the initial classification prediction result of the image to be processed is not a true value, directly using the initial classification prediction result to perform semantic segmentation on the image to be processed may cause a large error in the semantic segmentation prediction result. Therefore, in the embodiment of the present disclosure, the image features of the image to be processed are further enhanced using the initial classification prediction result, such as semantic enhancement processing, so that the image features after feature enhancement processing can more effectively reflect the semantic correlation between each pixel in the image to be processed, and the semantic segmentation prediction result obtained using the image features after feature enhancement processing is more accurate.
[0332] In the process of semantic segmentation of an image, the disclosed embodiment first obtains the initial classification prediction results of the pixels in the image, and then performs feature enhancement processing on the image features according to the initial classification prediction results, and then uses the image features after feature enhancement processing to perform semantic segmentation on the image. In this way, since clustering pixels using the image features after feature enhancement processing can make the clustering accuracy higher, the semantic segmentation prediction results finally obtained are also more accurate.
[0333] The following are embodiments of the apparatus of the present disclosure, which can be used to execute embodiments of the method of the present disclosure.
[0334] According to an image semantic segmentation device according to an embodiment of the present disclosure, the device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. The image semantic segmentation device includes:
[0335] A first acquisition module is configured to acquire an image to be processed;
[0336] A first determination module is configured to determine an initial classification prediction result of a pixel point in the image to be processed;
[0337] A first processing module is configured to perform feature enhancement processing on the image features of the image to be processed based on the initial classification prediction result;
[0338] The second acquisition module is configured to obtain the semantic segmentation prediction result of the image to be processed according to the image features after feature enhancement processing.
[0339] In an optional implementation of this embodiment, the first processing module includes:
[0340] A first construction submodule is configured to construct an adjacency matrix of the image to be processed according to the initial classification prediction result; wherein the adjacency matrix is used to describe the semantic correlation relationship between pixels in the image to be processed;
[0341] The first processing submodule is configured to perform feature enhancement processing on the image features based on the adjacency matrix.
[0342] In an optional implementation of this embodiment, the first building submodule includes:
[0343] The second construction submodule is configured to construct the adjacency matrix by multiplying the classification matrix by the transpose of the classification matrix; the classification matrix is a matrix form of the initial classification prediction result.
[0344] In an optional implementation of this embodiment, the feature enhancement process includes multiple stages, and the first processing submodule includes:
[0345] A second processing submodule is configured to perform feature enhancement processing of the current stage on the input features based on the adjacency matrix; wherein the input features are the image features or output features obtained by feature enhancement processing of the previous stage;
[0346] The second acquisition module includes:
[0347] The first acquisition submodule is configured to acquire the semantic segmentation prediction result according to the output features obtained by the feature enhancement processing in the last stage.
[0348] In an optional implementation of this embodiment, the first determining module includes:
[0349] A second acquisition submodule is configured to acquire image features of the image to be processed using a basic model in the semantic segmentation model;
[0350] A classification submodule is configured to use the classification model in the semantic segmentation model to perform classification prediction on the pixels in the image to be processed to obtain the initial classification prediction result;
[0351] The first processing submodule includes:
[0352] The third processing submodule is configured to utilize the feature enhancement model in the semantic segmentation model to perform feature enhancement processing on the image features based on the adjacency matrix.
[0353] In an optional implementation of this embodiment, the device further includes:
[0354] A display module, configured to display the semantic segmentation prediction result to a user when the confidence level of the semantic segmentation prediction result is lower than a preset threshold;
[0355] A first receiving module is configured to receive correction data of the user for the semantic segmentation prediction result;
[0356] The first adjustment module is configured to adjust the semantic segmentation prediction result according to the correction data.
[0357] In an optional implementation of this embodiment, the device further includes:
[0358] The second adjustment module is configured to adjust the model parameters of the semantic segmentation model according to the adjusted semantic segmentation prediction result and the semantic segmentation prediction result before adjustment; wherein the semantic segmentation model is a model for obtaining the semantic segmentation prediction result before adjustment.
[0359] In an optional implementation of this embodiment, the image to be processed is a remote sensing image, and the semantic segmentation prediction result includes a combination of one or more of the following data: contour information of different objects in the remote sensing image, resource types of different objects in the remote sensing image, and road network information in the remote sensing image.
[0360] In an optional implementation of this embodiment, the image to be processed is a remote sensing image of a target area, and the device further includes:
[0361] The second determination module is configured to determine the target object in the target area according to the semantic segmentation prediction result.
[0362] In an optional implementation of this embodiment, the image to be processed is a remote sensing image of a target area, and the device further includes:
[0363] The third determination module is configured to determine the resource type of the features included in the target area according to the semantic segmentation prediction result.
[0364] In an optional implementation of this embodiment, the image to be processed is a remote sensing image of a target area, and the device further includes:
[0365] A fourth determination module is configured to determine the location and type of the road in the target area according to the semantic segmentation prediction result;
[0366] The fifth determination module is configured to determine the road network information in the target area according to the location and type of the road.
[0367] In an optional implementation of this embodiment, the image to be processed includes a first remote sensing image and a second remote sensing image, and the device further includes:
[0368] The sixth determination module is configured to determine the change detection information of the ground objects in the first remote sensing image and the second remote sensing image according to the semantic segmentation prediction result corresponding to the first remote sensing image and the semantic segmentation prediction result corresponding to the second remote sensing image.
[0369] In an optional implementation of this embodiment, the image to be processed includes an image of a road ahead of the autonomous driving vehicle, and the device further includes:
[0370] a seventh determination module, configured to determine object information on the road ahead according to the semantic segmentation prediction result;
[0371] an eighth determination module, configured to determine vehicle driving information based on the object information;
[0372] The first output module is configured to output the vehicle driving information to the autonomous driving vehicle.
[0373] In an optional implementation of this embodiment, the image to be processed is the first image in the video, and the device further includes:
[0374] A third acquisition module is configured to acquire a second image in the video;
[0375] an updating module, configured to update the initial classification prediction result of the first image according to a difference between the first image and the second image, so as to obtain an initial classification prediction result of the second image;
[0376] The fourth acquisition module is configured to obtain a semantic segmentation prediction result of the second image based on the initial classification prediction result of the second image.
[0377] In an optional implementation of this embodiment, the update module includes:
[0378] A third acquisition submodule is configured to acquire a foreground offset between the first image and the second image;
[0379] An updating submodule is configured to update the initial classification prediction result according to the foreground offset to obtain an initial classification prediction result of the second image.
[0380] In an optional implementation of this embodiment, the image to be processed is an image including a commodity, and the device includes:
[0381] A module for acquiring a commodity image in the image to be processed according to the semantic segmentation prediction result;
[0382] A module for generating visual description information of the product based on the product image.
[0383] In an optional implementation of this embodiment, the image to be processed is an image in a live video, and the device includes:
[0384] A target video acquisition module is configured to process the live video according to the semantic segmentation prediction result to obtain a target video;
[0385] A module for outputting the target video.
[0386] In an optional implementation of this embodiment, the target video acquisition module includes:
[0387] Determining whether the image to be processed is a module including a target image of a commodity according to the semantic segmentation prediction result;
[0388] A module for performing preset processing on the target image.
[0389] In an optional implementation of this embodiment, the target video acquisition module includes:
[0390] A module for acquiring a target image including a product in the live video according to the semantic segmentation prediction result;
[0391] A module for generating visual description information of the product according to the target image.
[0392] The image semantic segmentation device in this embodiment Figure 1 The image semantic segmentation method in the embodiment shown and the related embodiments corresponds to each other. For details, please refer to the above Figure 1 The description of the image semantic segmentation method in the illustrated embodiment and related embodiments will not be repeated here.
[0393] According to a semantic segmentation model training device of an embodiment of the present disclosure, the device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. The semantic segmentation model training device includes:
[0394] a fifth acquisition module, configured to acquire a training image;
[0395] a sixth acquisition module, configured to acquire initial classification prediction results of pixels in the training image using a semantic segmentation model, and perform feature enhancement processing on image features of the training image based on the initial classification prediction results;
[0396] a seventh acquisition module, configured to acquire a semantic segmentation prediction result of the training image according to the image features after feature enhancement processing;
[0397] A training module is configured to train the semantic segmentation model based on the initial classification prediction result and the semantic segmentation prediction result.
[0398] In an optional implementation of this embodiment, the sixth acquisition module includes:
[0399] A third construction submodule is configured to construct an adjacency matrix corresponding to the training image according to the initial classification prediction result, wherein the adjacency matrix is used to describe the semantic correlation relationship between pixels in the training image;
[0400] The fourth processing submodule is configured to perform feature enhancement processing on the image features corresponding to the training image based on the adjacency matrix.
[0401] In an optional implementation of this embodiment, the third building submodule includes:
[0402] The fourth construction submodule is configured to construct the adjacency matrix by multiplying the classification matrix by the transpose of the classification matrix.
[0403] In an optional implementation of this embodiment, the semantic segmentation model includes a basic model, a classification model and a feature enhancement model, and the training module includes:
[0404] A first adjustment submodule is configured to adjust the model parameters of the classification model and the base model through the loss between the initial classification prediction result and the corresponding true classification result; the classification model is used to obtain the initial classification prediction result; the base model is used to obtain the image feature;
[0405] The second adjustment submodule is configured to adjust the model parameters of the feature enhancement model and the basic model through the loss between the semantic segmentation prediction result and its corresponding semantic segmentation true result; the eigenvalue enhancement model is used to perform feature enhancement processing on the image features.
[0406] In an optional implementation of this embodiment, the feature enhancement process includes multiple stages; the sixth acquisition module includes:
[0407] A fourth acquisition submodule is configured to perform feature enhancement processing on the input features at the current stage based on the adjacency matrix to obtain output features; the input features are the image features or the output features obtained after feature enhancement processing at the previous stage;
[0408] The fifth acquisition submodule is configured to acquire the semantic segmentation prediction result based on the output features obtained after the feature enhancement processing in the last stage.
[0409] The training device of the semantic segmentation model in this embodiment is Figure 5 The model training methods in the illustrated embodiment and related embodiments correspond to each other. For details, please refer to the above Figure 5 The description of the training method of the semantic segmentation model in the illustrated embodiment and related embodiments will not be repeated here.
[0410] According to another embodiment of the present disclosure, a training device for a semantic segmentation model can be implemented as part or all of an electronic device through software, hardware, or a combination of both. The training device for a semantic segmentation model includes:
[0411] a ninth acquisition module, configured to acquire a training image;
[0412] A second calling module is configured to call a preset service interface so that the preset service interface uses a semantic segmentation model to obtain an initial classification prediction result of a pixel in the training image, and performs feature enhancement processing on an image feature of the training image based on the initial classification prediction result, and obtains a semantic segmentation prediction result of the training image according to the image feature after the feature enhancement processing, and trains the semantic segmentation model based on the initial classification prediction result and the semantic segmentation prediction result;
[0413] The third output module is configured to output the semantic segmentation model.
[0414] The model training device in this embodiment is Fig. 9 The model training methods in the illustrated embodiment and related embodiments correspond to each other. For details, please refer to the above Fig. 9 The description of the model training method in the illustrated embodiment and related embodiments will not be repeated here.
[0415] According to another embodiment of the present disclosure, the image semantic segmentation device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. The image semantic segmentation device includes:
[0416] An eighth acquisition module is configured to acquire an image to be processed;
[0417] A first calling module is configured to call a preset service interface so that the preset service interface obtains an image to be processed, determines an initial classification prediction result of a pixel in the image to be processed, performs feature enhancement processing on an image feature of the image to be processed based on the initial classification prediction result, and obtains a semantic segmentation prediction result of the image to be processed according to the image feature after the feature enhancement processing;
[0418] The second output module is configured to output the semantic segmentation prediction result.
[0419] The image semantic segmentation device in this embodiment Fig.10 The image semantic segmentation method in the embodiment shown and the related embodiments corresponds to each other. For details, please refer to the above Fig.10 The description of the image semantic segmentation method in the illustrated embodiment and related embodiments will not be repeated here.
[0420] Fig.11 It is a structural schematic diagram of an electronic device suitable for implementing a training method for a semantic segmentation model and / or an image semantic segmentation method according to an embodiment of the present disclosure.
[0421] like Fig.11 As shown, the electronic device 1100 includes a processing unit 1101, which can be implemented as a processing unit such as a CPU, a GPU, an FPGA, and an NPU. The processing unit 1101 can perform various processes in the embodiments of any of the above methods of the present disclosure according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage part 1108 to a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the electronic device 1100 are also stored. The processing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.
[0422] The following components are connected to the I / O interface 1105: an input section 1106 including a keyboard, a mouse, etc.; an output section 1107 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN card, a modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the I / O interface 1105 as needed. A removable medium 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1110 as needed, so that a computer program read therefrom is installed into the storage section 1108 as needed.
[0423] In particular, according to an embodiment of the present disclosure, any method in the above referenced embodiment of the present disclosure may be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly contained on a readable medium thereof, and the computer program includes a program code for executing any method in the embodiment of the present disclosure. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 1109, and / or installed from a removable medium 1111.
[0424] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the road map or block diagram can represent a part of a module, program segment or code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a different order from the order marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0425] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The units or modules described may also be set in a processor, and the names of these units or modules do not constitute limitations on the units or modules themselves in some cases.
[0426] As another aspect, the present disclosure further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the device described in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into the device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the method described in the present disclosure.
[0427] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.
Claims
1. A method for image semantic segmentation, wherein: include: Get the image to be processed; Determine the initial classification prediction results of the pixels in the image to be processed; Performing feature enhancement processing on the image features of the image to be processed based on the initial classification prediction result to improve the clustering accuracy of clustering pixels based on the image features after the feature enhancement processing; Obtain a semantic segmentation prediction result of the image to be processed according to the image features after feature enhancement processing.
2. The method according to claim 1, wherein: Performing feature enhancement processing on the image features of the image to be processed based on the initial classification prediction result includes: Constructing an adjacency matrix of the image to be processed according to the initial classification prediction result; wherein the adjacency matrix is used to describe the semantic correlation relationship between pixels in the image to be processed; Perform feature enhancement processing on the image features based on the adjacency matrix.
3. The method according to claim 2, wherein: Constructing an adjacency matrix corresponding to the image to be processed according to the initial classification prediction result, including: The adjacency matrix is constructed by multiplying a classification matrix by the transpose of the classification matrix; the classification matrix is a matrix form of the initial classification prediction result.
4. The method according to claim 2 or 3, wherein: The feature enhancement process includes multiple stages, and the feature enhancement process is performed on the image features based on the adjacency matrix, including: Performing feature enhancement processing of the current stage on the input features based on the adjacency matrix; wherein the input features are the image features or output features obtained by feature enhancement processing of the previous stage; Obtaining a semantic segmentation prediction result of the image to be processed according to the image features after feature enhancement processing, including: The semantic segmentation prediction result is obtained according to the output features obtained from the feature enhancement processing in the last stage.
5. The method according to claim 2 or 3, wherein: Determining an initial classification prediction result of a pixel point in the image to be processed includes: Acquire image features of the image to be processed by using a basic model in a semantic segmentation model; Using the classification model in the semantic segmentation model to perform classification prediction on the pixel points in the image to be processed to obtain the initial classification prediction result; Performing feature enhancement processing on the image features based on the adjacency matrix includes: The feature enhancement model in the semantic segmentation model is utilized to perform feature enhancement processing on the image features based on the adjacency matrix.
6. The method according to any one of claims 1 to 3, wherein: The method further comprises: When the confidence level of the semantic segmentation prediction result is lower than a preset threshold, displaying the semantic segmentation prediction result to a user; Receiving correction data of the semantic segmentation prediction result from the user; The semantic segmentation prediction result is adjusted according to the correction data.
7. The method according to claim 6, wherein: The method further comprises: The model parameters of the semantic segmentation model are adjusted according to the adjusted semantic segmentation prediction result and the semantic segmentation prediction result before the adjustment; wherein the semantic segmentation model is a model for obtaining the semantic segmentation prediction result before the adjustment.
8. The method according to any one of claims 1 to 3 and 7, wherein: The image to be processed is a remote sensing image, and the semantic segmentation prediction result includes a combination of one or more of the following data: contour information of different objects in the remote sensing image, resource types of different objects in the remote sensing image, and road network information in the remote sensing image.
9. The method according to any one of claims 1 to 3 and 7, wherein: The image to be processed is a remote sensing image of a target area, and the method further comprises: The target object in the target area is determined according to the semantic segmentation prediction result.
10. The method according to any one of claims 1 to 3 and 7, wherein: The image to be processed is a remote sensing image of a target area, and the method further comprises: The resource type of the ground objects included in the target area is determined according to the semantic segmentation prediction result.
11. The method according to any one of claims 1 to 3 and 7, wherein: The image to be processed is a remote sensing image of a target area, and the method further comprises: Determine the location and type of the road in the target area according to the semantic segmentation prediction result; The road network information in the target area is determined according to the location and type of the road.
12. The method according to any one of claims 1 to 3 and 7, wherein: The image to be processed includes a first remote sensing image and a second remote sensing image, and the method further includes: Change detection information of ground objects in the first remote sensing image and the second remote sensing image is determined according to the semantic segmentation prediction result corresponding to the first remote sensing image and the semantic segmentation prediction result corresponding to the second remote sensing image.
13. The method according to any one of claims 1 to 3 and 7, wherein: The image to be processed includes an image of a road ahead of the autonomous driving vehicle, and the method further includes: Determine object information on the road ahead according to the semantic segmentation prediction result; determining vehicle driving information based on the object information; The vehicle driving information is output to the autonomous driving vehicle.
14. The method according to any one of claims 1 to 3 and 7, wherein: The image to be processed is the first image in the video, and the method further includes: Acquire a second image in the video; updating the initial classification prediction result of the first image according to the difference between the first image and the second image to obtain the initial classification prediction result of the second image; A semantic segmentation prediction result of the second image is obtained based on the initial classification prediction result of the second image.
15. The method according to claim 14, wherein: Updating the initial classification prediction result of the first image according to the difference between the first image and the second image to obtain the initial classification prediction result of the second image includes: Obtaining a foreground offset between the first image and the second image; The initial classification prediction result is updated according to the foreground offset to obtain an initial classification prediction result of the second image.
16. The method according to any one of claims 1 to 3 and 7, wherein: The image to be processed is an image including a commodity, and the method comprises: Acquire a commodity image in the image to be processed according to the semantic segmentation prediction result; Visual description information of the product is generated based on the product image.
17. The method according to any one of claims 1 to 3 and 7, wherein: The image to be processed is an image in a live video, and the method comprises: Processing the live video according to the semantic segmentation prediction result to obtain a target video; The target video is output.
18. The method according to claim 17, wherein: Processing the live video according to the semantic segmentation prediction result to obtain a target video includes: Determining whether the image to be processed is a target image including a product according to the semantic segmentation prediction result; Performing preset processing on the target image.
19. The method according to claim 17, wherein: Processing the live video according to the semantic segmentation prediction result to obtain a target video includes: Acquire a target image including a product in the live video according to the semantic segmentation prediction result; Visual description information of the product is generated according to the target image.
20. A method for training a semantic segmentation model, wherein: include: Get training images; Obtaining initial classification prediction results of pixels in the training image using a semantic segmentation model, and performing feature enhancement processing on image features of the training image based on the initial classification prediction results; Obtaining a semantic segmentation prediction result of the training image according to the image features after feature enhancement processing, so as to improve the clustering accuracy of pixel points clustered based on the image features after feature enhancement processing; The semantic segmentation model is trained based on the initial classification prediction result and the semantic segmentation prediction result.
21. The method according to claim 20, wherein: Performing feature enhancement processing on the image features of the training image based on the initial classification prediction result includes: Constructing an adjacency matrix corresponding to the training image according to the initial classification prediction result, wherein the adjacency matrix is used to describe the semantic correlation relationship between pixels in the training image; Feature enhancement processing is performed on the image features corresponding to the training image based on the adjacency matrix.
22. The method according to claim 21, wherein: Constructing an adjacency matrix corresponding to the training image according to the initial classification prediction result, including: The adjacency matrix is constructed by multiplying a classification matrix by the transpose of the classification matrix.
23. The method according to any one of claims 20 to 22, wherein: The semantic segmentation model includes a basic model, a classification model and a feature enhancement model, and the semantic segmentation model is trained based on the initial classification prediction result and the semantic segmentation prediction result, including: The model parameters of the classification model and the basic model are adjusted by the loss between the initial classification prediction result and the corresponding true classification result; the classification model is used to obtain the initial classification prediction result; the basic model is used to obtain the image feature; The model parameters of the feature enhancement model and the basic model are adjusted by the loss between the semantic segmentation prediction result and its corresponding semantic segmentation true result; the feature enhancement model is used to perform feature enhancement processing on the image features.
24. The method according to claim 21 or 22, wherein: The feature enhancement process includes multiple stages; performing feature enhancement process on the image features corresponding to the training image based on the adjacency matrix includes: Performing feature enhancement processing of the current stage on the input features based on the adjacency matrix to obtain output features; the input features are the image features or the output features obtained after feature enhancement processing in the previous stage; The semantic segmentation prediction result is obtained based on the output features obtained after the feature enhancement processing in the last stage.
25. A method for image semantic segmentation, wherein: include: Get the image to be processed; Calling a preset service interface so that the preset service interface obtains an image to be processed, determines an initial classification prediction result of pixels in the image to be processed, and performs feature enhancement processing on image features of the image to be processed based on the initial classification prediction result to improve clustering accuracy of pixel points based on the image features after feature enhancement processing, and obtains a semantic segmentation prediction result of the image to be processed according to the image features after feature enhancement processing; Output the semantic segmentation prediction result.
26. A method for training a semantic segmentation model, wherein: include: Get training images; Calling a preset service interface so that the preset service interface obtains an initial classification prediction result of pixels in the training image using a semantic segmentation model, and performs feature enhancement processing on image features of the training image based on the initial classification prediction result to improve clustering accuracy of pixel points based on the image features after feature enhancement processing, and obtains a semantic segmentation prediction result of the training image according to the image features after feature enhancement processing, and trains the semantic segmentation model based on the initial classification prediction result and the semantic segmentation prediction result; Output the semantic segmentation model.
27. An image semantic segmentation device, wherein: include: A first acquisition module is configured to acquire an image to be processed; A first determination module is configured to determine an initial classification prediction result of a pixel point in the image to be processed; A first processing module is configured to perform feature enhancement processing on the image features of the image to be processed based on the initial classification prediction result, so as to improve the clustering accuracy of clustering pixels based on the image features after the feature enhancement processing; The second acquisition module is configured to obtain the semantic segmentation prediction result of the image to be processed according to the image features after feature enhancement processing.
28. A training device for a semantic segmentation model, wherein: include: a fifth acquisition module, configured to acquire a training image; a sixth acquisition module, configured to acquire an initial classification prediction result of pixels in the training image using a semantic segmentation model, and perform feature enhancement processing on image features of the training image based on the initial classification prediction result, so as to improve clustering accuracy of clustering pixels based on the image features after feature enhancement processing; a seventh acquisition module, configured to acquire a semantic segmentation prediction result of the training image according to the image features after feature enhancement processing; A training module is configured to train the semantic segmentation model based on the initial classification prediction result and the semantic segmentation prediction result.
29. An image semantic segmentation device, wherein: include: An eighth acquisition module is configured to acquire an image to be processed; A first calling module is configured to call a preset service interface so that the preset service interface obtains an image to be processed, determines an initial classification prediction result of a pixel in the image to be processed, and performs feature enhancement processing on an image feature of the image to be processed based on the initial classification prediction result to improve clustering accuracy of clustering pixels based on the image feature after the feature enhancement processing, and obtains a semantic segmentation prediction result of the image to be processed according to the image feature after the feature enhancement processing; The second output module is configured to output the semantic segmentation prediction result.
30. A training device for a semantic segmentation model, wherein: include: a ninth acquisition module, configured to acquire a training image; A second calling module is configured to call a preset service interface so that the preset service interface uses a semantic segmentation model to obtain an initial classification prediction result of pixels in the training image, and performs feature enhancement processing on image features of the training image based on the initial classification prediction result to improve the clustering accuracy of pixel points clustered based on the image features after feature enhancement processing, and obtains a semantic segmentation prediction result of the training image according to the image features after feature enhancement processing, and trains the semantic segmentation model based on the initial classification prediction result and the semantic segmentation prediction result; The third output module is configured to output the semantic segmentation model.
31. An electronic device, wherein: The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 26.
32. A computer-readable storage medium having computer instructions stored thereon, wherein: When the computer instructions are executed by a processor, the method described in any one of claims 1 to 26 is implemented.
33. A computer program product comprising computer instructions, wherein: When the computer instructions are executed by a processor, the method described in any one of claims 1 to 26 is implemented.
Citation Information
Patent Citations
Method for roughly sorting high-resolution remote sensing image scene
CN102013017A
Image semantic segmentation method and device, training method and device, electronic equipment, storage medium and program
CN108229478A
Multi-decoder full convolutional neural network and a corresponding mesoscopic structure identification method thereof
CN109919080A
Image processing system and method based on deep learning, terminal and medium
CN110009625A
Commodity searching method, device and system
CN110110117A
