Semantic segmentation method, device, electronic device and computer-readable storage medium

By obtaining the initial region image and target region context information in semantic segmentation, and using a neural network model for feature extraction and segmentation, the problem of high computational complexity is solved, and efficient semantic segmentation is achieved.

CN114648762BActive Publication Date: 2024-11-26TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210272072.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-18
Publication Date
2024-11-26
Estimated Expiration
2042-03-18

AI Technical Summary

Technical Problem

Existing semantic segmentation methods have high computational complexity, resulting in low efficiency.

Method used

By acquiring the image to be segmented and dividing it into initial region images, determining the target pixel context information and region context information, reducing computational complexity, and using a neural network model for feature extraction and segmentation.

Benefits of technology

This reduces computational complexity and improves the accuracy and efficiency of semantic segmentation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114648762B_ABST
    Figure CN114648762B_ABST
Patent Text Reader

Abstract

The embodiments of the present application disclose a semantic segmentation method, device, electronic device and computer-readable storage medium; in the embodiments of the present application, an image to be segmented is obtained, and the image to be segmented is divided to obtain an initial region image corresponding to the image to be segmented; target pixel context information within the initial region image is determined, and a target region image corresponding to the initial region image is determined based on the target pixel context information and the initial region image; target region context information between the target region images is determined, and target pixel features of the image to be segmented are determined based on the target region image and the target region context information; the image to be segmented is segmented based on the target pixel features to obtain a segmentation result. The embodiments of the present application can reduce computational complexity. The embodiments of the present application can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and autonomous driving.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a semantic segmentation method, device, electronic device and computer-readable storage medium. Background Art

[0002] Semantic segmentation is to classify each pixel in the image. Pixels belonging to the same class are divided into the same area, thereby achieving image segmentation.

[0003] In order to improve the image segmentation accuracy, an attention mechanism is added to the semantic segmentation to calculate the image context information. This method has a high computational complexity. Summary of the invention

[0004] The embodiments of the present application provide a semantic segmentation method, device, electronic device and computer-readable storage medium, which can solve the technical problem of high computational complexity in semantic segmentation.

[0005] A semantic segmentation method, comprising:

[0006] Acquire an image to be segmented, and divide the image to be segmented to obtain an initial region image corresponding to the image to be segmented;

[0007] Determine target pixel context information within the initial region image, and determine a target region image corresponding to the initial region image based on the target pixel context information and the initial region image;

[0008] Determine the target region context information between the target region images, and determine the target pixel features of the image to be segmented based on the target region images and the target region context information;

[0009] According to the target pixel features, the image to be segmented is segmented to obtain a segmentation result.

[0010] Accordingly, an embodiment of the present application provides a semantic segmentation device, comprising:

[0011] An image acquisition module is used to acquire an image to be segmented, and divide the image to be segmented to obtain an initial region image corresponding to the image to be segmented;

[0012] A first determination module is used to determine target pixel context information in the initial region image, and determine a target region image corresponding to the initial region image according to the target pixel context information and the initial region image;

[0013] A second determination module is used to determine the target area context information between the target area images, and determine the target pixel features of the image to be segmented according to the target area images and the target area context information;

[0014] The image segmentation module is used to segment the image to be segmented according to the target pixel features to obtain a segmentation result.

[0015] Optionally, the image acquisition module is specifically used to execute:

[0016] Performing feature extraction on the above-mentioned image to be segmented to obtain a feature image;

[0017] Obtaining a preset grid, and performing displacement prediction on the vertices of the preset grid according to the feature image and the preset grid to obtain a target deformed grid;

[0018] The feature image is divided according to the target deformed grid to obtain an initial region image corresponding to the image to be segmented.

[0019] Optionally, the image acquisition module is specifically used to execute:

[0020] Determine the coordinates of the vertices in the above preset grid;

[0021] According to the vertex coordinates, a vertex feature search is performed in the feature image to obtain an initial feature corresponding to the vertex coordinates;

[0022] Determine the target feature corresponding to the vertex coordinates according to the initial feature and the context information of the vertex coordinates;

[0023] The target displacement of the vertex coordinates is predicted according to the target features, and the vertex coordinates are moved according to the target displacement to obtain a target deformable mesh.

[0024] Optionally, the second determining module is specifically configured to execute:

[0025] Determining an initial region feature corresponding to the target region image according to a mean value of pixels of the target region image;

[0026] Determining target region context information between the target region images according to the initial region features;

[0027] Determine the target area features according to the initial area features and the target area context information;

[0028] The target region features are mapped onto the target deformable grid to obtain the target pixel features of the image to be segmented.

[0029] Optionally, the semantic segmentation device further includes:

[0030] Training modules for:

[0031] Obtaining a training sample set, and determining a target weight corresponding to a category of the training pixel according to the number of training pixels in a training image of the training sample set;

[0032] Dividing the above training image to obtain a training area image;

[0033] Determine initial pixel context information in the training area image through a pixel context layer in the neural network model to be trained, and determine a first area image corresponding to the training image based on the initial pixel context information and the training area image;

[0034] Determining initial regional context information between the first regional images through the regional context layer in the neural network model to be trained, and determining initial pixel features of the training image based on the first regional images and the initial regional context information;

[0035] Determining the target category of the training pixel corresponding to the initial pixel feature through the segmentation layer in the neural network model to be trained;

[0036] Determine a target loss value according to the target category, the label of the training pixel and the target weight;

[0037] Obtain the training times of the neural network model to be trained;

[0038] Based on the above target loss value and the above training times, the above neural network model to be trained is trained to obtain the above trained neural network model.

[0039] Optionally, the training module is specifically used to perform:

[0040] By using the feature extraction layer in the neural network model to be trained, extracting features from the training image to obtain a training feature image;

[0041] By using the deformable network layer in the neural network model to be trained, displacement prediction is performed on the vertices of the preset grid according to the training feature image and the preset grid to obtain an initial deformable grid;

[0042] By using the pixel context layer in the neural network model to be trained, the training feature image is divided according to the initial deformation grid to obtain a training area image;

[0043] Determine a first target loss value according to the target category, the label of the training pixel and the target weight;

[0044] Determine a second target loss value according to the initial pixel feature and the average of the initial pixel feature;

[0045] Determine a target loss value according to the first target loss value and the second target loss value;

[0046] If the target loss value does not meet the preset conditions and / or the training times of the neural network model to be trained is less than the preset threshold, the training times are increased by 1, and the network parameters of the feature extraction layer, the pixel context layer, the region context layer and the segmentation layer in the neural network model to be trained are updated according to the first target loss value, and the network parameters of the deformation network layer in the neural network model to be trained are updated according to the second target loss value, and the execution is returned to perform feature extraction on the training image through the feature extraction layer in the neural network model to be trained to obtain a training feature image;

[0047] If the target loss value satisfies a preset condition and / or the number of trainings of the neural network model to be trained is equal to a preset threshold, the training is stopped to obtain the trained neural network model.

[0048] Optionally, the training module is specifically used to perform:

[0049] Determine the sub-area of ​​each grid in the initial deformed grid and the total area of ​​the image to be segmented;

[0050] determining a third target loss value according to the sub-area and the total area;

[0051] Determining a target loss value according to the first target loss value, the second target loss value and the third target loss value;

[0052] If the target loss value does not meet the preset conditions and / or the training times of the neural network model to be trained is less than the preset threshold, the training times are increased by 1, and the network parameters of the feature extraction layer, the pixel context layer, the region context layer and the segmentation layer in the neural network model to be trained are updated according to the first target loss value, and the network parameters of the deformation network layer in the neural network model to be trained are updated according to the second target loss value and the third target loss value, and the execution is returned to perform feature extraction on the training image through the feature extraction layer in the neural network model to be trained to obtain a training feature image.

[0053] In addition, an embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the semantic segmentation method provided in the embodiment of the present application.

[0054] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for loading by a processor to execute any semantic segmentation method provided in the embodiment of the present application.

[0055] In addition, an embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements any semantic segmentation method provided in the embodiment of the present application.

[0056] In an embodiment of the present application, an image to be segmented is first obtained, and the image to be segmented is divided to obtain an initial region image corresponding to the image to be segmented. Then, the target pixel context information within the initial region image is determined, and the target region image corresponding to the initial region image is determined based on the target pixel context information and the initial region image. Then, the target region context information between the target region images is determined, and the target pixel features of the image to be segmented are determined based on the target region image and the target region context information. Finally, the image to be segmented is identified and segmented based on the target pixel features.

[0057] That is, in an embodiment of the present application, the target pixel context information within the initial area image is first determined, and the target area image corresponding to the initial area image is determined based on the target pixel context information and the initial area image, and then the target area context information between the target area images is determined, and based on the target area image and the target area context information, the target pixel features of the image to be segmented are determined, that is, by determining the target area context information between two target area images, there is no need to calculate the context information between each pixel in the target area image and all pixels in other target area images, that is, there is no need to calculate the context information between each pixel in the image to be segmented and all other pixels in the image to be segmented, thereby reducing the computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 is a scene schematic diagram of the semantic segmentation process provided in an embodiment of the present application;

[0060] Figure 2 It is a flowchart of the semantic segmentation method provided in the embodiment of the present application;

[0061] Figure 3 is a schematic diagram of an initial region image provided in an embodiment of the present application;

[0062] Figure 4 is a schematic diagram of a preset grid and a target deformable grid provided in an embodiment of the present application;

[0063] Figure 5 It is a flowchart of a training method for a neural network model to be trained provided in an embodiment of the present application;

[0064] Figure 6 is a schematic diagram of the structure of a neural network model to be trained provided in an embodiment of the present application;

[0065] Figure 7 It is a flowchart of an application method of a trained neural network model provided in an embodiment of the present application;

[0066] Figure 8 is a schematic diagram of the segmentation result of the trained neural network model provided in an embodiment of the present application;

[0067] Fig. 9 is a structural diagram of a semantic segmentation device provided in an embodiment of the present application;

[0068] Fig.10 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0070] The embodiments of the present application provide a semantic segmentation method, device, electronic device and computer-readable storage medium. The semantic segmentation device can be integrated in an electronic device, which can be a server or a terminal.

[0071] Among them, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms.

[0072] Furthermore, multiple servers may form a blockchain, and the servers are nodes on the blockchain.

[0073] The terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0074] For example, Figure 1 As shown, the terminal can obtain the image to be segmented, and divide the image to be segmented to obtain the initial area image corresponding to the image to be segmented; determine the target pixel context information within the initial area image, and determine the target area image corresponding to the initial area image based on the target pixel context information and the initial area image; determine the target area context information between the target area images, and determine the target pixel features of the image to be segmented based on the target area image and the target area context information; segment the image to be segmented based on the target pixel features to obtain a segmentation result.

[0075] In addition, the term "plurality" in the embodiments of the present application refers to two or more than two. The terms "first" and "second" in the embodiments of the present application are used to distinguish descriptions and should not be understood as implying relative importance.

[0076] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0077] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0078] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing so that computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0079] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0080] The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, autonomous driving, and remote sensing, etc. Autonomous driving generally includes technologies such as high-precision maps, environmental perception, behavioral decision-making, path planning, and motion control.

[0081] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0082] See also Figure 2 , Figure 2 : is a flowchart of a semantic segmentation method provided by an embodiment of the present application. The semantic segmentation method may include:

[0083] S201 , obtaining an image to be segmented, and dividing the image to be segmented to obtain an initial region image corresponding to the image to be segmented.

[0084] The image to be segmented refers to an image that contains the area of ​​interest to the user. For example, image A includes a vehicle, trees, and the sky, and the user wants to obtain the vehicle image in image A, then image A is the image to be segmented.

[0085] The image to be segmented may be an image of an autonomous driving scene or a remote sensing image. This embodiment does not limit the type of the image to be segmented.

[0086] The terminal may shoot with its own camera when receiving the acquisition instruction, thereby acquiring the image to be segmented. Alternatively, the terminal may acquire the image to be segmented locally when receiving the acquisition instruction.

[0087] Alternatively, when receiving the acquisition instruction, the terminal may forward the acquisition instruction to other terminals, and the other terminals may acquire the image to be segmented from the local based on the acquisition instruction, or shoot based on the acquisition instruction, thereby acquiring the image to be segmented. Then, the other terminals may send the image to be segmented to the terminal, and the terminal may acquire the image to be segmented.

[0088] Regarding the method for the terminal to obtain the image to be segmented, the user can select it according to the actual situation, and this embodiment does not limit it here.

[0089] After acquiring the image to be segmented, the terminal may divide the image to be segmented based on a preset grid to obtain an initial region image corresponding to the image to be segmented.

[0090] Segmentation refers to dividing an image into individual region images, but the region images still have connections. For example, the image to be segmented is Figure 3 As shown, the initial region image can be Figure 3 As shown, the target refers to the object in the image to be segmented, for example, the target can be a vehicle, a person, or a tree.

[0091] Since the position of the target in each image to be segmented is different, if the image to be segmented is directly divided based on the preset grid, the initial region image obtained is not accurate, that is, at this time, the edge of the preset grid does not coincide with the boundary of the target in the image to be segmented. Therefore, in order to improve the accuracy of the initial region image, in some embodiments, the image to be segmented is divided to obtain the initial region image corresponding to the image to be segmented, including:

[0092] Perform feature extraction on the image to be segmented to obtain a feature image;

[0093] Obtain a preset mesh, and perform displacement prediction on vertices of the preset mesh according to the feature image and the preset mesh to obtain a target deformable mesh;

[0094] The feature image is divided according to the target deformed grid to obtain the initial region image corresponding to the image to be segmented.

[0095] In this embodiment, the grid may be initialized to obtain a preset grid. The grid may be initialized according to the following three aspects:

[0096] First, the mesh topology remains unchanged before and after deformation, that is, the topology of the preset mesh is the same as the topology of the target deformed mesh.

[0097] Secondly, the edges of the preset grid are flexible and diverse, so that the target deformed grid can be obtained after the preset grid has the smallest possible displacement deviation.

[0098] Thirdly, the number of grids in the preset grid is constant, which facilitates the subsequent structured batch processing of images.

[0099] The grid is initialized according to the above three aspects, and the preset grid can be obtained as follows Figure 4 as shown in .

[0100] After obtaining the preset grid, the terminal can displace the vertices of the preset grid according to the feature image to obtain a target deformed grid, so that the edges of the target deformed grid coincide with the boundaries of the targets in the feature image, that is, the same target in the feature image is in the same grid in the target deformed grid, that is, the targets in the same grid in the target deformed grid are color homogenized (same color) or semantically homogenized (belonging to the same category).

[0101] For example, yes Figure 4 After displacing the vertices of the preset mesh in , the target deformed mesh can be obtained as Figure 4 shown.

[0102] After the terminal obtains the target deformed mesh, it divides the feature image according to the target deformed mesh to obtain the initial region image corresponding to the image to be segmented.

[0103] In this embodiment, after extracting features from the image to be segmented to obtain a feature image, displacement prediction is performed on the vertices of the preset grid based on the feature image and the preset grid to obtain a target deformed grid, and then the feature image is divided according to the target deformed grid to obtain an initial area image corresponding to the image to be segmented, so that the grid edge of the target deformed grid coincides with the boundary of the target in the image.

[0104] In some embodiments, performing displacement prediction on vertices of a preset mesh according to the feature image and the preset mesh to obtain a target deformable mesh includes:

[0105] Determine the coordinates of vertices in the preset grid;

[0106] According to the vertex coordinates, vertex features are searched in the feature image to obtain the initial features corresponding to the vertex coordinates;

[0107] Determine the target feature corresponding to the vertex coordinates according to the initial features and the context information of the vertex coordinates;

[0108] The target displacement of the vertex coordinates is predicted according to the target features, and the vertex coordinates are moved according to the target displacement to obtain the target deformable mesh.

[0109] In the object detection of an image, the offset of the position of the center point, the upper left vertex or the lower right vertex of the detection box is predicted by a fully connected layer or a convolutional layer. However, the vertices or center points of different detection boxes in the object detection do not interfere with each other on the same image. Semantic segmentation is a dense prediction task, that is, the vertices of each grid are not isolated from each other, but are connected to each other through the edges of the grid, and are a group of points with contextual dependencies. Therefore, in this embodiment, after obtaining the initial features corresponding to the vertices, the target features corresponding to the vertex coordinates are determined based on the initial features and the contextual information of the vertices, so that the target displacement can be obtained more accurately, and then the target deformation grid can be obtained more accurately.

[0110] Wherein, determining the target feature corresponding to the vertex coordinates according to the initial feature and the context information of the vertex includes:

[0111] The initial features are transformed point by point to obtain intermediate features, and then the target features are determined based on the intermediate features and the context information of the vertices.

[0112] It should be noted that the context information of the vertex can be obtained through the self-attention mechanism layer of the deformable network layer in the trained neural network, and the point-by-point feature transformation of the initial features can be achieved through the point-by-point convolution layer in the deformable network layer in the trained neural network. Then, the prediction convolution layer of the deformable network layer in the trained neural network can be used to determine the target features corresponding to the vertex coordinates based on the initial features and the context information of the vertex.

[0113] In addition, the deformation network layer may include multiple layers of point-by-point convolution layers and self-attention mechanism layers, so as to realize multiple feature transformations of the initial features and multiple acquisitions of context information of the vertices. For example, the deformation network layer may include 6 layers of point-by-point convolution layers and self-attention mechanism layers.

[0114] In order to obtain the intermediate features more accurately, after obtaining the initial features, the vertex coordinates can be fused with the initial features (for example, the vertex coordinates are connected in series with the initial features) to obtain candidate features, and then the candidate features are transformed point by point to obtain the intermediate features.

[0115] Among them, the vertex coordinates can be fused with the initial features through the CoordConv layer of the deformation network layer in the trained neural network.

[0116] In other embodiments, extracting features from the image to be segmented to obtain a feature image includes: extracting features from the image to be segmented through a feature extraction layer in a trained neural network model to obtain a feature image.

[0117] The feature extraction layer in the trained neural network model can be a residual neural network (ResNet) or a convolutional network. Optionally, when the feature extraction layer is a residual neural network, in order to reduce the loss of resolution and expand the receptive field, some pooling layers in the residual neural network can be replaced with extended convolutional layers, that is, at this time, the feature extraction layer is a residual neural network including an extended convolutional layer.

[0118] S202: Determine target pixel context information in the initial region image, and determine a target region image corresponding to the initial region image according to the target pixel context information and the initial region image.

[0119] After obtaining the initial region image, the terminal obtains target pixel context information between pixels in the initial region image, and then determines the target region image corresponding to the initial region image according to the target pixel context information and the initial region image.

[0120] For example, if there are pixel 1, pixel 2 and pixel 3 in the initial area image a, the target pixel context information between pixel 1 and pixel 2 is calculated, the target pixel context information between pixel 1 and pixel 3 is calculated, and the target pixel context information between pixel 2 and pixel 3 is calculated.

[0121] Among them, the target pixel context information in the initial area image can be determined through the pixel context layer in the trained neural network model, and the target area image corresponding to the initial area image can be determined according to the target pixel context information and the initial area image. The pixel context layer can be a self-attention mechanism layer.

[0122] Alternatively, the target pixel context information in the initial region image may be determined by a Euclidean distance algorithm or a cosine similarity algorithm, and the target region image corresponding to the initial region image may be determined based on the target pixel context information and the initial region image.

[0123] Context information refers to the interaction information between different objects. Pixel context information refers to the interaction information between different pixels in the same image.

[0124] S203 , determining target region context information between target region images, and determining target pixel features of the image to be segmented according to the target region images and the target region context information.

[0125] In the related art, through the self-attention mechanism layer, not only the context information of the target pixels between the pixels in the initial area image needs to be calculated, but also the context information between the pixels of different initial area images needs to be calculated, resulting in high computational complexity.

[0126] For example, there are pixels 1 and 2 in the initial area image a, and there are pixels 4 and 5 in the initial area image b. Then, through the self-attention mechanism layer, not only the target pixel context information between pixel 1 and pixel 2, and the target pixel context information between pixel 4 and pixel 5 need to be calculated, but also the target pixel context information between pixel 1 and pixel 4, the target pixel context information between pixel 1 and pixel 5, the target pixel context information between pixel 2 and pixel 4, and the target pixel context information between pixel 2 and pixel 5 need to be calculated.

[0127] In the present embodiment, after obtaining the target pixel context between pixels in the initial area image, the target area context information between the target area images is calculated, and then the target pixel features of the image to be segmented are determined based on the target area image and the target area context information. There is no need to calculate the pixel context information between pixels of different target area images, thereby reducing the computational complexity, thereby achieving higher semantic segmentation accuracy based on lower computational complexity and computational redundancy.

[0128] For example, there are pixels 1 and 2 in the initial area image a, there are pixels 4 and 5 in the initial area image b, there are pixels 6 and 7 in the target area image a1, and there are pixels 8 and 9 in the target area image b1. In this embodiment, the area context information between the target area image a1 and the target area image b1 is calculated, so that there is no need to calculate the target pixel context information between pixel 1 and pixel 4, there is no need to calculate the target pixel context information between pixel 1 and pixel 5, there is no need to calculate the target pixel context information between pixel 2 and pixel 4, and there is no need to calculate the target pixel context information between pixel 2 and pixel 5.

[0129] Among them, the target region context information between the target region images can be determined through the regional context layer in the trained neural network model, and the target pixel features of the image to be segmented can be determined based on the target region image and the target region context information.

[0130] Alternatively, the target region context information between the target region images may be determined by a Euclidean distance algorithm or a cosine similarity algorithm, and the target pixel features of the image to be segmented may be determined based on the target region image and the target region context information.

[0131] In some embodiments, determining the target region context information between the target region images, and determining the target pixel features of the image to be segmented according to the target region images and the target region context information, includes:

[0132] Determine the initial region feature corresponding to the target region image according to the mean value of the pixels of the target region image;

[0133] Determine the target region context information between the target region images according to the initial region features;

[0134] Determine the target region features according to the initial region features and the target region context information;

[0135] The target area features are mapped onto the target deformable grid to obtain the target pixel features of the image to be segmented.

[0136] In this embodiment, the mean value of the pixels of the target area image is used as the initial area feature corresponding to the target area image, and then the target area context information between the target area images is calculated based on the initial area feature corresponding to the target area image, so that the target area context information between the two target area images only needs to be calculated once instead of multiple times, thereby greatly reducing the calculation complexity.

[0137] When the target region context information between the target region images is determined by the regional context layer in the trained neural network model, and the target pixel features of the image to be segmented are determined based on the target region image and the target region context information, the initial region features corresponding to the target region image can be determined by the regional pooling layer in the regional context layer based on the mean of the pixels of the target region image. The target region context information between the target region images can be determined based on the initial region features by the regional level self-attention mechanism layer in the regional context layer, and the target region features can be determined based on the initial region features and the target region context information. The target region features can be mapped to the target deformable mesh by the regional anti-pooling layer in the regional context layer to obtain the target pixel features of the image to be segmented.

[0138] S204: Segment the image to be segmented according to the target pixel features to obtain a segmentation result.

[0139] After obtaining the target pixel features, the terminal segments the image to be segmented according to the target pixel features to obtain the segmentation result.

[0140] In some embodiments, the image to be segmented is segmented according to the target pixel features to obtain the segmentation result, including: using the segmentation layer in the trained neural network model to segment the image to be segmented according to the target pixel features to obtain the segmentation result.

[0141] Optionally, since the target area image contains detail information and the feature image includes all information, in order to make the segmentation more accurate, the image to be segmented can be segmented according to the feature image, the target area image and the target pixel features through the segmentation layer in the trained neural network model to obtain the segmentation result.

[0142] In some other embodiments, when the target pixel context information and the target region image are determined by the pixel context layer in the trained neural network model, the target region context information and the target pixel features are determined by the region context layer in the trained neural network model, and the segmentation result is determined by the segmentation layer in the trained neural network model, the method further includes:

[0143] Obtain a training sample set, and determine a target weight corresponding to a category of the training pixel according to the number of training pixels in a training image of the training sample set;

[0144] Divide the training image to obtain a training area image;

[0145] Determine initial pixel context information in the training area image through the pixel context layer in the neural network model to be trained, and determine a first area image corresponding to the training image according to the initial pixel context information and the training area image;

[0146] Determining initial regional context information between the first regional images through the regional context layer in the neural network model to be trained, and determining initial pixel features of the training image based on the first regional images and the initial regional context information;

[0147] Determine the target category of the training pixel corresponding to the initial pixel feature through the segmentation layer in the neural network model to be trained;

[0148] Determine the target loss value based on the target category, the label of the training pixel, and the target weight;

[0149] Get the number of training times of the neural network model to be trained;

[0150] Based on the target loss value and the number of training times, the neural network model to be trained is trained to obtain a trained neural network model.

[0151] Among them, the number of training pixels in the training image of the training sample set can be substituted into the following formula to obtain the target weight corresponding to the category of the training pixel:

[0152]

[0153]

[0154] Among them, i represents the i-th category of the training pixel, n i represents the number of pixels in the i-th category, z represents the number of categories, and q i represents the frequency of pixels in the i-th category, q m represents the median frequency, w i Represents the target weight corresponding to the pixel of the i-th category.

[0155] Substitute the target category, training pixel label, and target weight into the following formula to get the target loss value:

[0156]

[0157] Among them, y i represents the label of the training pixel, p i represents the target category, L c Represents the target loss value.

[0158] Since the number of pixels in each category in the training sample set varies greatly, the weighted cross entropy loss function is used to calculate the target loss value, and the target weight is determined according to the number of training pixels to avoid overfitting, thereby making the obtained trained neural network model more accurate in semantic segmentation.

[0159] Optionally, determining initial regional context information between the first regional images through a regional context layer in a neural network model to be trained, and determining initial pixel features of the training image according to the first regional images and the initial regional context information, including:

[0160] Determining a first region feature corresponding to the first region image according to a mean value of pixels of the first region image;

[0161] Determine initial region context information between first region images according to the first region feature, and obtain second region features according to the initial region context information and the first region feature;

[0162] The second region feature is mapped onto the initial deformed mesh to obtain the initial pixel feature of the training image.

[0163] Optionally, based on the target loss value and the number of training times, the neural network model to be trained is trained to obtain a trained neural network model, including:

[0164] If the number of training times is less than the preset threshold, the number of training times is increased by 1, and the network parameters of the neural network model to be trained are updated according to the target loss value, and then the training image is divided to obtain the training area image. If the number of training times is equal to the preset threshold, the neural network model to be trained is used as the trained neural network model.

[0165] Alternatively, based on the target loss value and the number of training times, the neural network model to be trained is trained to obtain a trained neural network model, which also includes:

[0166] If the target loss value does not meet the preset conditions, the network parameters of the neural network model to be trained are updated according to the target loss value, and then the training image is divided back to obtain the training area image. If the target loss value meets the preset conditions, the neural network model to be trained is used as the trained neural network model.

[0167] Alternatively, based on the target loss value and the number of training times, the neural network model to be trained is trained to obtain a trained neural network model, which also includes:

[0168] If the target loss value does not meet the preset conditions and the number of training times is less than the preset threshold, the number of training times is increased by 1, and the network parameters of the neural network model to be trained are updated according to the target loss value, and then the execution is returned to divide the training image to obtain the training area image;

[0169] If the target loss value meets the preset conditions and the number of training times is equal to the preset threshold, the neural network model to be trained is regarded as the trained neural network model.

[0170] When the target deformable mesh is obtained through the deformable network layer in the trained neural network model, the training image is divided to obtain the training area image, including:

[0171] Extracting features from the training image through a feature extraction layer in the neural network model to be trained to obtain a training feature image;

[0172] By using the deformation network layer in the neural network model to be trained, displacement prediction is performed on the vertices of the preset grid according to the training feature image and the preset grid to obtain an initial deformation grid;

[0173] The training feature image is divided according to the initial deformation grid through the pixel context layer in the neural network model to be trained to obtain the training area image;

[0174] According to the target category, the label of the training pixel and the target weight, the target loss value is determined, including:

[0175] Determine a first target loss value based on the target category, the label of the training pixel, and the target weight;

[0176] Determine a second target loss value according to the initial pixel feature and the average of the initial pixel feature;

[0177] Determine a target loss value according to the first target loss value and the second target loss value;

[0178] Based on the target loss value and the number of training times, the neural network model to be trained is trained to obtain a trained neural network model, including:

[0179] If the target loss value does not meet the preset conditions and / or the number of training times of the neural network model to be trained is less than the preset threshold, the number of training times is increased by 1, and the network parameters of the feature extraction layer, the network parameters of the pixel context layer, the network parameters of the region context layer, and the network parameters of the segmentation layer in the neural network model to be trained are updated according to the first target loss value, and the network parameters of the deformation network layer in the neural network model to be trained are updated according to the second target loss value, and the execution is returned to extract features from the training image through the feature extraction layer in the neural network model to be trained to obtain a training feature image;

[0180] If the target loss value meets the preset conditions and / or the number of training times of the neural network model to be trained is equal to the preset threshold, the training is stopped to obtain the trained neural network model.

[0181] Because the positions of many boundary points or mutation points in the image may be the positions of the vertices in the preset grid after displacement, that is, many boundary points or mutation points in the image may coincide with the vertices in the preset grid after displacement, so the accuracy of the target displacement of the vertices in the preset grid is low. In addition, optimizing the positions of the vertices in the preset grid alone will ignore the topological relationship between the vertices in the preset grid and the overall distribution of the preset grid.

[0182] Therefore, in this embodiment, a second target loss value is set, and then when the target loss value does not meet the preset conditions and / or the number of training times of the neural network model to be trained is less than a preset threshold, the deformable network layer in the neural network model to be trained is updated according to the second target loss value, so that when the target displacement of the vertices in the preset grid is predicted by the deformable network layer, the predicted target displacement is more accurate.

[0183] Among them, the initial pixel features and the mean of the initial pixel features can be substituted into the following formula for calculation to obtain the second target loss value:

[0184]

[0185] L var represents the second target loss value, N represents the number of first region images, j represents the jth first region image, p mrepresents the coordinates of the mth initial pixel feature in the jth first region image, f m represents the mth initial pixel feature in the jth first region image, f j represents the mean of the initial pixel features of the jth first region image. ||||2 represents the 2-norm operation.

[0186] Through the deformation network layer in the neural network model to be trained, the displacement of the vertices of the preset grid is predicted according to the training feature image and the preset grid to obtain the initial deformation grid, including:

[0187] Determine the coordinates of vertices in the preset grid;

[0188] According to the vertex coordinates, a vertex feature search is performed in the training feature image to obtain a first feature corresponding to the vertex coordinates;

[0189] Determine a second feature corresponding to the vertex coordinates according to the first feature and context information of the vertex coordinates;

[0190] The initial displacement of the vertex coordinates is predicted according to the second feature, and the vertex coordinates are moved according to the initial displacement to obtain an initial deformable mesh.

[0191] In some other embodiments, determining the target loss value according to the first target loss value and the second target loss value includes:

[0192] Determine the sub-area of ​​each grid in the initial deformed grid and the total area of ​​the image to be segmented;

[0193] determining a third target loss value based on the sub-area and the total area;

[0194] Determining a target loss value according to the first target loss value, the second target loss value, and the third target loss value;

[0195] If the target loss value does not meet the preset conditions and / or the number of training times of the neural network model to be trained is less than the preset threshold, the number of training times is increased by 1, and the network parameters of the feature extraction layer, the network parameters of the pixel context layer, the network parameters of the region context layer, and the network parameters of the segmentation layer in the neural network model to be trained are updated according to the first target loss value, and the network parameters of the deformation network layer in the neural network model to be trained are updated according to the second target loss value, and the execution is returned to extract features from the training image through the feature extraction layer in the neural network model to be trained to obtain a training feature image, including:

[0196] If the target loss value does not meet the preset conditions and / or the training times of the neural network model to be trained is less than the preset threshold, the training times are increased by 1, and the network parameters of the feature extraction layer, the network parameters of the pixel context layer, the network parameters of the region context layer, and the network parameters of the segmentation layer in the neural network model to be trained are updated according to the first target loss value, and the network parameters of the deformation network layer in the neural network model to be trained are updated according to the second target loss value and the third target loss value, and the execution is returned to perform feature extraction on the training image through the feature extraction layer in the neural network model to be trained to obtain a training feature image.

[0197] In order to avoid the problem of intersection of adjacent grids in the initial deformed mesh, that is, the grids in the initial deformed mesh overlap, resulting in the sum of the sub-areas of all grids in the initial deformed mesh exceeding the total area of ​​the image to be segmented, so this embodiment also updates the deformable network layer according to the third target loss value, thereby avoiding the phenomenon of grid intersection in the initial deformed mesh, and then when the target displacement of the vertices in the preset grid is predicted by the deformable network layer, the predicted target displacement is more accurate, that is, the target deformable mesh predicted by the deformable network layer will not cross.

[0198] Among them, the sub-area and the total area can be substituted into the following formula to obtain the third objective loss value:

[0199]

[0200] L area Represents the third target loss value, area j represents the sub-area of ​​the jth first region image, that is, the sub-area of ​​the jth grid, area img Represents the total area of ​​the image to be segmented.

[0201] Substitute the first target loss value, the second target loss value, and the third target loss value into the following formula to obtain the target loss value L:

[0202] L=L c +αL var +βL area

[0203] α represents the weight of the first target loss value, and β represents the weight of the second target loss value.

[0204] It should be noted that when the number of training times is used as a condition for determining whether to terminate the training of the neural network model to be trained, determining the target loss value based on the first target loss value and the second target loss value may also mean taking the first target loss value and the second target loss value as the target loss values, that is, the target loss value includes the first target loss value and the second target loss value.

[0205] Similarly, when the number of training times is used as a condition for determining whether to terminate the training of the neural network model to be trained, determining the target loss value based on the first target loss value, the second target loss value, and the third target loss value can also mean taking the first target loss value, the second target loss value, and the third target loss value as the target loss value, that is, the target loss value includes the first target loss value, the second target loss value, and the third target loss value.

[0206] As can be seen from the above, in the embodiment of the present application, the image to be segmented is first obtained, and the image to be segmented is divided to obtain the initial region image corresponding to the image to be segmented. Then the target pixel context information in the initial region image is determined, and the target region image corresponding to the initial region image is determined based on the target pixel context information and the initial region image. Then the target region context information between the target region images is determined, and the target pixel features of the image to be segmented are determined based on the target region image and the target region context information. Finally, the image to be segmented is identified and segmented based on the target pixel features.

[0207] That is, in an embodiment of the present application, the target pixel context information within the initial area image is first determined, and the target area image corresponding to the initial area image is determined based on the target pixel context information and the initial area image, and then the target area context information between the target area images is determined, and based on the target area image and the target area context information, the target pixel features of the image to be segmented are determined, that is, by determining the target area context information between two target area images, there is no need to calculate the context information between each pixel in the target area image and all pixels in other target area images, that is, there is no need to calculate the context information between each pixel in the image to be segmented and all other pixels in the image to be segmented, thereby reducing the computational complexity.

[0208] The method described in the above embodiment is further described in detail below with examples.

[0209] This embodiment takes the semantic segmentation device integrated in the terminal as an example. The semantic segmentation method may include an application method of a trained neural network model and a training method of a trained neural network model. Specifically, Figure 5 is a flowchart of the training method of the trained neural network model. Figure 6 Schematic diagram of the flow of the application method of the trained neural network model.

[0210] See also Figure 5 , the training methods of the trained neural network model include:

[0211] S501: The terminal obtains a training sample set, and determines a target weight corresponding to a category of a training pixel according to the number of training pixels in a training image of the training sample set.

[0212] S502: The terminal extracts features from the training image through a feature extraction layer in the neural network model to be trained to obtain a training feature image.

[0213] In this embodiment, the structure of the neural network model to be trained can be as follows: Figure 6 As shown, the neural network model to be trained includes a feature extraction layer, a pixel context layer, a region context layer, a deformation network layer and a segmentation layer, wherein the feature extraction layer can be a residual neural network, and then the training feature image F is obtained through the feature extraction layer.

[0214] It should be understood that after obtaining the training feature image, the dimension of the training feature image can be reduced to obtain the reduced-dimensional training feature image X, and then the reduced-dimensional training feature image is input into the deformation network layer and the pixel context layer.

[0215] S503: The terminal determines the vertex coordinates in the preset grid through the deformable network layer in the neural network model to be trained, and performs vertex feature search in the training feature image according to the vertex coordinates to obtain a first feature corresponding to the vertex coordinates.

[0216] For example, Figure 6 As shown in the figure, the deformation network layer includes an extraction layer, a CoordConv layer, 6 point-by-point convolution layers, a self-attention layer, and a prediction convolution layer. The terminal determines the vertex coordinates (t r ,s r ), and according to the vertex coordinates, perform vertex feature search in the training feature image to obtain the first feature corresponding to the vertex coordinates.

[0217] S504. The terminal determines the second feature corresponding to the vertex coordinates according to the first feature and the context information of the vertex coordinates through the deformation network layer in the neural network model to be trained, predicts the initial displacement of the vertex coordinates according to the second feature, moves the vertex coordinates according to the initial displacement, and obtains an initial deformable mesh.

[0218] The terminal can first fuse the first feature and the vertex coordinates through the CoordConv layer in the deformation network layer to obtain the initial candidate feature. Then the terminal performs point-by-point convolution on the initial candidate feature through the point-by-point convolution layer in the deformation network layer to obtain the initial intermediate feature. Then, the self-attention layer in the deformation network layer obtains the context information of the vertex coordinates, and determines the second feature corresponding to the vertex coordinates based on the initial intermediate feature and the context information of the vertex coordinates. The point-by-point convolution process and the self-attention layer process can be performed 6 times. Finally, the terminal predicts the initial displacement (Δt r , Δs r), and move the vertex coordinates according to the initial displacement to obtain the initial deformed mesh.

[0219] S505. The terminal divides the training feature image according to the initial deformation grid through the pixel context layer in the neural network model to be trained to obtain a training area image.

[0220] S506. The terminal determines initial pixel context information in the training area image through the pixel context layer in the neural network model to be trained, and determines a first area image corresponding to the training image according to the initial pixel context information and the training area image.

[0221] For the training area image Then the first region image can be Among them, K j represents the pixel in the jth training region image, and Then each first region image can form a new feature Figure X '∈R C×H×W , where R represents the dimension, C represents the number of channels, H represents the height of the training image, and W represents the width of the training image.

[0222] S507. The terminal determines a first region feature corresponding to the first region image according to a mean value of pixels of the first region image through a region pooling layer of a region context layer in a neural network model to be trained.

[0223] S508. The terminal determines the initial regional context information between the first regional images according to the first regional features through the regional-level self-attention mechanism layer of the regional context layer in the neural network model to be trained, and obtains the second regional features according to the initial regional context information and the first regional features.

[0224] S509. The terminal maps the second region features to the initial deformable grid through the regional depooling layer of the regional context layer in the neural network model to be trained, so as to obtain the initial pixel features of the training image.

[0225] S5010. The terminal determines the target category of the training pixel corresponding to the initial pixel feature according to the initial pixel feature, the first region image and the training feature image through the segmentation layer in the neural network model to be trained.

[0226] S5011. The terminal determines a first target loss value according to the target category, the label of the training pixel and the target weight.

[0227] S5012. The terminal determines a second target loss value according to the initial pixel feature and the average of the initial pixel feature.

[0228] S5013. The terminal determines the sub-area of ​​each grid in the initial deformed grid and the total area of ​​the image to be segmented, and determines a third target loss value according to the sub-area and the total area.

[0229] S5014. The terminal obtains the number of training times of the neural network model to be trained.

[0230] S5015. If the number of training times is less than the preset threshold, the terminal increases the number of training times by 1, updates the network parameters of the feature extraction layer, the network parameters of the pixel context layer, the network parameters of the region context layer, and the network parameters of the segmentation layer in the neural network model to be trained according to the first target loss value, updates the network parameters of the deformation network layer in the neural network model to be trained according to the second target loss value and the third target loss value, and returns to execute S502.

[0231] S5016: If the number of training times is equal to the preset threshold, the terminal uses the neural network model to be trained as the trained neural network model.

[0232] Reference Figure 7 , the application methods of the trained neural network model include:

[0233] S701. The terminal obtains an image to be segmented, and performs feature extraction on the image to be segmented through a feature extraction layer in a trained neural network model to obtain a feature image.

[0234] S702, the terminal obtains a preset grid, and determines the vertex coordinates in the preset grid through the deformation network layer in the trained neural network model, and searches for vertex features in the feature image based on the vertex coordinates to obtain initial features corresponding to the vertex coordinates.

[0235] S703. The terminal determines the target features corresponding to the vertex coordinates according to the initial features and the context information of the vertex coordinates through the deformation network layer in the trained neural network model, predicts the target displacement of the vertex coordinates according to the target features, moves the vertex coordinates according to the target displacement, and obtains the target deformed mesh.

[0236] S704: The terminal divides the feature image according to the target deformation grid through the pixel context layer in the trained neural network model to obtain an initial region image corresponding to the image to be segmented.

[0237] S705. The terminal determines the target pixel context information in the initial area image through the pixel context layer in the trained neural network model, and determines the target area image corresponding to the initial area image according to the target pixel context information and the initial area image.

[0238] S706. The terminal determines the initial regional features corresponding to the target region image according to the mean value of the pixels of the target region image through the regional pooling layer of the regional context layer in the trained neural network model.

[0239] S707. The terminal determines the target region context information between the target region images through the region-level self-attention layer of the region context layer in the trained neural network model, and determines the target region features according to the initial region features and the target region context information.

[0240] S708. The terminal maps the target region features onto the target deformable grid through the regional depooling layer of the regional context layer in the trained neural network model to obtain the target pixel features of the image to be segmented.

[0241] For example, the target pixel feature can be obtained using X”∈R C×H×W .

[0242] S709. The terminal segments the image to be segmented through the segmentation layer in the trained neural network model according to the target pixel features, the target area image and the feature image to obtain a segmentation result.

[0243] The segmentation layer may be a softmax layer. For example, the segmentation result obtained by the trained neural network model may be as follows: Figure 8 As shown, Figure 8 The dashed box indicates the region of interest.

[0244] The beneficial effects and other achievable methods of this embodiment may be specifically referred to in the above-mentioned semantic segmentation method embodiment, and this embodiment will not be described in detail here.

[0245] In order to better implement the semantic segmentation method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the above semantic segmentation method. The meanings of the nouns are the same as those in the above semantic segmentation method, and the specific implementation details can refer to the description in the method embodiment.

[0246] For example, Fig. 9 As shown, the semantic segmentation device may include:

[0247] The image acquisition module 901 is used to acquire the image to be segmented, and divide the image to be segmented to obtain an initial region image corresponding to the image to be segmented.

[0248] The first determination module 902 is used to determine target pixel context information in the initial region image, and determine the target region image corresponding to the initial region image according to the target pixel context information and the initial region image.

[0249] The second determination module 903 is used to determine the target region context information between the target region images, and determine the target pixel features of the image to be segmented according to the target region images and the target region context information.

[0250] The image segmentation module 904 is used to segment the image to be segmented according to the target pixel features to obtain a segmentation result.

[0251] Optionally, the image acquisition module 901 is specifically configured to execute:

[0252] Perform feature extraction on the image to be segmented to obtain a feature image;

[0253] Obtain a preset mesh, and perform displacement prediction on vertices of the preset mesh according to the feature image and the preset mesh to obtain a target deformable mesh;

[0254] The feature image is divided according to the target deformed grid to obtain the initial region image corresponding to the image to be segmented.

[0255] Optionally, the image acquisition module 901 is specifically configured to execute:

[0256] Determine the coordinates of vertices in the preset grid;

[0257] According to the vertex coordinates, vertex features are searched in the feature image to obtain the initial features corresponding to the vertex coordinates;

[0258] Determine the target feature corresponding to the vertex coordinates according to the initial features and the context information of the vertex coordinates;

[0259] The target displacement of the vertex coordinates is predicted according to the target features, and the vertex coordinates are moved according to the target displacement to obtain the target deformable mesh.

[0260] Optionally, the second determining module 903 is specifically configured to execute:

[0261] Determine the initial region feature corresponding to the target region image according to the mean value of the pixels of the target region image;

[0262] Determine the target region context information between the target region images according to the initial region features;

[0263] Determine the target region features according to the initial region features and the target region context information;

[0264] The target area features are mapped onto the target deformable grid to obtain the target pixel features of the image to be segmented.

[0265] Optionally, the semantic segmentation device further includes:

[0266] Training module, which performs:

[0267] Obtain a training sample set, and determine a target weight corresponding to a category of the training pixel according to the number of training pixels in a training image of the training sample set;

[0268] Divide the training image to obtain a training area image;

[0269] Determine initial pixel context information in the training area image through the pixel context layer in the neural network model to be trained, and determine a first area image corresponding to the training image according to the initial pixel context information and the training area image;

[0270] Determining initial regional context information between the first regional images through the regional context layer in the neural network model to be trained, and determining initial pixel features of the training image based on the first regional images and the initial regional context information;

[0271] Determine the target category of the training pixel corresponding to the initial pixel feature through the segmentation layer in the neural network model to be trained;

[0272] Determine the target loss value based on the target category, the label of the training pixel, and the target weight;

[0273] Get the number of training times of the neural network model to be trained;

[0274] Based on the target loss value and the number of training times, the neural network model to be trained is trained to obtain a trained neural network model.

[0275] Optionally, the training module is specifically used to perform:

[0276] Extracting features from the training image through a feature extraction layer in the neural network model to be trained to obtain a training feature image;

[0277] By using the deformation network layer in the neural network model to be trained, displacement prediction is performed on the vertices of the preset grid according to the training feature image and the preset grid to obtain an initial deformation grid;

[0278] The training feature image is divided according to the initial deformation grid through the pixel context layer in the neural network model to be trained to obtain the training area image;

[0279] Determine a first target loss value based on the target category, the label of the training pixel, and the target weight;

[0280] Determine a second target loss value according to the initial pixel feature and the average of the initial pixel feature;

[0281] Determine a target loss value according to the first target loss value and the second target loss value;

[0282] If the target loss value does not meet the preset conditions and / or the number of training times of the neural network model to be trained is less than the preset threshold, the number of training times is increased by 1, and the network parameters of the feature extraction layer, the network parameters of the pixel context layer, the network parameters of the region context layer, and the network parameters of the segmentation layer in the neural network model to be trained are updated according to the first target loss value, and the network parameters of the deformation network layer in the neural network model to be trained are updated according to the second target loss value, and the execution is returned to extract features from the training image through the feature extraction layer in the neural network model to be trained to obtain a training feature image;

[0283] If the target loss value meets the preset conditions and / or the number of training times of the neural network model to be trained is equal to the preset threshold, the training is stopped to obtain the trained neural network model.

[0284] Optionally, the training module is specifically used to perform:

[0285] Determine the sub-area of ​​each grid in the initial deformed grid and the total area of ​​the image to be segmented;

[0286] determining a third target loss value based on the sub-area and the total area;

[0287] Determining a target loss value according to the first target loss value, the second target loss value, and the third target loss value;

[0288] If the target loss value does not meet the preset conditions and / or the training times of the neural network model to be trained is less than the preset threshold, the training times are increased by 1, and the network parameters of the feature extraction layer, the network parameters of the pixel context layer, the network parameters of the region context layer, and the network parameters of the segmentation layer in the neural network model to be trained are updated according to the first target loss value, and the network parameters of the deformation network layer in the neural network model to be trained are updated according to the second target loss value and the third target loss value, and the execution is returned to perform feature extraction on the training image through the feature extraction layer in the neural network model to be trained to obtain a training feature image.

[0289] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation methods and corresponding beneficial effects of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0290] The present application also provides an electronic device, which may be a server or a terminal, etc. Fig.10 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:

[0291] The electronic device may include components such as a processor 1001 with one or more processing cores, a memory 1002 with one or more computer-readable storage media, a power supply 1003, and an input unit 1004. Those skilled in the art will appreciate that Fig.10 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0292] The processor 1001 is the control center of the electronic device, which uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing computer programs and / or modules stored in the memory 1002, and calling data stored in the memory 1002. Optionally, the processor 1001 may include one or more processing cores; preferably, the processor 1001 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1001.

[0293] The memory 1002 can be used to store computer programs and modules. The processor 1001 executes various functional applications and data processing by running the computer programs and modules stored in the memory 1002. The memory 1002 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, a computer program required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1002 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 1002 may also include a memory controller to provide the processor 1001 with access to the memory 1002.

[0294] The electronic device also includes a power supply 1003 for supplying power to each component. Preferably, the power supply 1003 can be logically connected to the processor 1001 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 1003 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0295] The electronic device may further include an input unit 1004, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0296] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 1001 in the electronic device will load the executable files corresponding to the processes of one or more computer programs into the memory 1002 according to the following instructions, and the processor 1001 will run the computer programs stored in the memory 1002, thereby realizing various functions, such as:

[0297] Acquire the image to be segmented, and divide the image to be segmented to obtain an initial region image corresponding to the image to be segmented;

[0298] Determine target pixel context information in the initial region image, and determine the target region image corresponding to the initial region image according to the target pixel context information and the initial region image;

[0299] Determine target region context information between target region images, and determine target pixel features of the image to be segmented based on the target region images and the target region context information;

[0300] According to the target pixel features, the image to be segmented is segmented to obtain the segmentation result.

[0301] The specific implementation methods and corresponding beneficial effects of the above operations can be found in the detailed description of the semantic segmentation method above, which will not be elaborated here.

[0302] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by a computer program, or by controlling related hardware through a computer program. The computer program may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0303] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored, and the computer program can be loaded by a processor to execute the steps in any semantic segmentation method provided in the embodiment of the present application. For example, the computer program can execute the following steps:

[0304] Acquire the image to be segmented, and divide the image to be segmented to obtain an initial region image corresponding to the image to be segmented;

[0305] Determine target pixel context information in the initial region image, and determine the target region image corresponding to the initial region image according to the target pixel context information and the initial region image;

[0306] Determine target region context information between target region images, and determine target pixel features of the image to be segmented based on the target region images and the target region context information;

[0307] According to the target pixel features, the image to be segmented is segmented to obtain the segmentation result.

[0308] The specific implementation methods and corresponding beneficial effects of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0309] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0310] Since the computer program stored in the computer-readable storage medium can execute the steps in any one of the semantic segmentation methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the semantic segmentation methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0311] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned semantic segmentation method.

[0312] The above is a detailed introduction to a semantic segmentation method, device, electronic device and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A semantic segmentation method, characterized in that: include: Acquire an image to be segmented, and perform feature extraction on the image to be segmented to obtain a feature image; Acquire a preset grid, and perform displacement prediction on vertices of the preset grid according to the feature image and the preset grid to obtain a target deformed grid; Dividing the feature image according to the target deformed grid to obtain an initial region image corresponding to the image to be segmented; Determine target pixel context information within the initial region image, and determine a target region image corresponding to the initial region image according to the target pixel context information and the initial region image; Determine the target region context information between the target region images, and determine the target pixel features of the image to be segmented according to the target region images and the target region context information; The image to be segmented is segmented according to the target pixel features to obtain a segmentation result.

2. The semantic segmentation method according to claim 1, characterized in that: The determining of the target region context information between the target region images, and determining the target pixel features of the image to be segmented according to the target region images and the target region context information, comprises: Determining an initial region feature corresponding to the target region image according to a mean value of pixels of the target region image; Determining target region context information between the target region images according to the initial region features; Determine a target area feature according to the initial area feature and the target area context information; The target region features are mapped onto the target deformable grid to obtain target pixel features of the image to be segmented.

3. The semantic segmentation method according to any one of claims 1 to 2, characterized in that: The target pixel context information and the target region image are determined by a pixel context layer in a trained neural network model, the target region context information and the target pixel feature are determined by a region context layer in the trained neural network model, and the segmentation result is determined by a segmentation layer in the trained neural network model, and the method further comprises: Acquire a training sample set, and determine a target weight corresponding to a category of the training pixel according to the number of training pixels in a training image of the training sample set; Dividing the training image to obtain a training area image; Determining initial pixel context information in the training area image through a pixel context layer in the neural network model to be trained, and determining a first area image corresponding to the training image according to the initial pixel context information and the training area image; Determining initial regional context information between the first regional images through the regional context layer in the neural network model to be trained, and determining initial pixel features of the training image based on the first regional image and the initial regional context information; Determining the target category of the training pixel corresponding to the initial pixel feature through the segmentation layer in the neural network model to be trained; Determining a target loss value according to the target category, the label of the training pixel and the target weight; Obtaining the number of training times of the neural network model to be trained; Based on the target loss value and the number of training times, the neural network model to be trained is trained to obtain the trained neural network model.

4. The semantic segmentation method according to claim 3, characterized in that: The target deformable mesh is obtained through the deformable network layer in the trained neural network model; The step of dividing the training image to obtain a training area image includes: Extracting features from the training image through a feature extraction layer in the neural network model to be trained to obtain a training feature image; By using the deformable network layer in the neural network model to be trained, displacement prediction is performed on the vertices of the preset grid according to the training feature image and the preset grid to obtain an initial deformable grid; By using the pixel context layer in the neural network model to be trained, the training feature image is divided according to the initial deformation grid to obtain a training area image; The determining of the target loss value according to the target category, the label of the training pixel and the target weight includes: Determining a first target loss value according to the target category, the label of the training pixel, and the target weight; Determining a second target loss value according to the initial pixel feature and the average of the initial pixel feature; Determine a target loss value according to the first target loss value and the second target loss value; The step of training the neural network model to be trained based on the target loss value and the number of training times to obtain the trained neural network model includes: If the target loss value does not meet the preset conditions and / or the number of training times of the neural network model to be trained is less than a preset threshold, the number of training times is increased by 1, and the network parameters of the feature extraction layer, the network parameters of the pixel context layer, the network parameters of the region context layer, and the network parameters of the segmentation layer in the neural network model to be trained are updated according to the first target loss value, and the network parameters of the deformation network layer in the neural network model to be trained are updated according to the second target loss value, and the execution is returned to perform feature extraction on the training image through the feature extraction layer in the neural network model to be trained to obtain a training feature image; If the target loss value meets the preset conditions and / or the number of training times of the neural network model to be trained is equal to a preset threshold, the training is stopped to obtain the trained neural network model.

5. The semantic segmentation method according to claim 4, characterized in that: Determining a target loss value according to the first target loss value and the second target loss value includes: Determining the sub-area of ​​each grid in the initial deformed grid and the total area of ​​the image to be segmented; determining a third target loss value according to the sub-area and the total area; determining a target loss value according to the first target loss value, the second target loss value, and the third target loss value; If the target loss value does not meet the preset conditions and / or the number of training times of the neural network model to be trained is less than a preset threshold, the number of training times is increased by 1, and the network parameters of the feature extraction layer, the network parameters of the pixel context layer, the network parameters of the region context layer, and the network parameters of the segmentation layer in the neural network model to be trained are updated according to the first target loss value, and the network parameters of the deformation network layer in the neural network model to be trained are updated according to the second target loss value, and the execution is returned to the feature extraction layer in the neural network model to be trained to extract features from the training image to obtain a training feature image, including: If the target loss value does not meet the preset conditions and / or the training times of the neural network model to be trained is less than a preset threshold, the training times are increased by 1, and the network parameters of the feature extraction layer, the network parameters of the pixel context layer, the network parameters of the region context layer and the network parameters of the segmentation layer in the neural network model to be trained are updated according to the first target loss value, and the network parameters of the deformation network layer in the neural network model to be trained are updated according to the second target loss value and the third target loss value, and the execution is returned to perform feature extraction on the training image through the feature extraction layer in the neural network model to be trained to obtain a training feature image.

6. A semantic segmentation device, characterized in that: include: An image acquisition module is used to acquire an image to be segmented and perform feature extraction on the image to be segmented to obtain a feature image; Acquire a preset grid, and perform displacement prediction on vertices of the preset grid according to the feature image and the preset grid to obtain a target deformed grid; Dividing the feature image according to the target deformed grid to obtain an initial region image corresponding to the image to be segmented; A first determination module, configured to determine target pixel context information within the initial region image, and determine a target region image corresponding to the initial region image according to the target pixel context information and the initial region image; A second determination module is used to determine the target area context information between the target area images, and determine the target pixel features of the image to be segmented according to the target area images and the target area context information; The image segmentation module is used to segment the image to be segmented according to the target pixel features to obtain a segmentation result.

7. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the semantic segmentation method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the semantic segmentation method according to any one of claims 1 to 5.

9. A computer program product, characterized in that The computer program product stores a computer program, and the computer program is suitable for being loaded by a processor to execute the semantic segmentation method according to any one of claims 1 to 5.