Semantic segmentation model training method, semantic segmentation method and device

Abstract

The application discloses a semantic segmentation model training method, a semantic segmentation method and a device. The semantic segmentation model training method comprises the following steps: inputting a first training sample image included in a first training sub-data set into an initial semantic segmentation model; the initial semantic segmentation model comprises a first initial semantic segmentation module and a second initial semantic segmentation module; the second initial semantic segmentation module comprises a first initial task independent module; the first initial task independent module has a corresponding semantic segmentation task; the first initial semantic segmentation module performs first feature processing on the first training sample image to obtain first image features; the first initial task independent module in the second initial semantic segmentation module obtains a first semantic segmentation result according to the first image features; and the initial semantic segmentation model is trained according to the first semantic segmentation result to obtain a target semantic segmentation model. In this way, the model training cost can be effectively reduced, and the accuracy of the semantic segmentation task can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a semantic segmentation model training method, semantic segmentation method and apparatus. Background Technology

[0002] Semantic segmentation is a fundamental research area in computer vision (CV). It can assign a specific category to each pixel in an image. For example, it can analyze objects in an image or video stream and label their category pixel by pixel. Semantic segmentation has been widely used in many fields such as autonomous driving, smart cities, and medical image processing.

[0003] In recent years, with the development of deep learning technology, semantic segmentation methods based on data-driven approaches and deep neural networks have also made significant progress. The training of semantic segmentation models (such as deep neural networks) requires a training dataset consisting of large-scale training sample images (also known as image data) and corresponding fine-grained category labels, depending on the semantic segmentation task to be performed.

[0004] In practical applications, the semantic segmentation task to be achieved may vary, and the required semantic segmentation model must also change accordingly. Therefore, it is necessary to train a semantic segmentation model that can achieve the corresponding semantic segmentation task based on different semantic segmentation tasks. Since the category labels of the training sample images in the training dataset are usually manually labeled, which requires a lot of manpower, financial resources, and time, the training cost of semantic segmentation models is high.

[0005] Furthermore, in practical applications, the number of training sample images labeled with category labels (i.e., labeled data) in the training dataset is limited. Limited training sample images can lead to overfitting of the semantic segmentation model and make it unable to effectively distinguish between images of different categories, ultimately resulting in incorrect predictions by the semantic segmentation model and affecting the accuracy of the semantic segmentation task.

[0006] Therefore, current semantic segmentation models suffer from problems such as high training costs and poor accuracy. Summary of the Invention

[0007] This application provides a semantic segmentation model training method, a semantic segmentation method and apparatus, as well as a computing device cluster, a computer program product containing instructions, and a computer-readable storage medium, which can solve the problems of high training cost and poor accuracy of semantic segmentation models in the prior art. That is, it can effectively reduce the training cost of semantic segmentation models and improve the accuracy of semantic segmentation tasks.

[0008] To address the aforementioned technical problems, in a first aspect, embodiments of this application provide a semantic segmentation model training method applied to an electronic device. The method includes: inputting a first training sample image, comprising a first training subset, into an initial semantic segmentation model. The first training subset has a corresponding semantic segmentation task and includes at least one first training sample image, which includes at least one first category label. The initial semantic segmentation model includes a first initial semantic segmentation module and a second initial semantic segmentation module. The second initial semantic segmentation module includes a first initial task-independent module, which has a corresponding semantic segmentation task. The first initial semantic segmentation module performs a first feature processing on the first training sample image to obtain first image features. The first initial task-independent module in the second initial semantic segmentation module obtains a first semantic segmentation result based on the first image features. The initial semantic segmentation model is trained based on the first semantic segmentation result to obtain a target semantic segmentation model.

[0009] In this implementation, the initial semantic segmentation model comprises two parts: a first initial semantic segmentation module and a second initial semantic segmentation module. The second initial semantic segmentation module includes a first initial task-independent module, which has a corresponding semantic segmentation task. Specifically, the first initial semantic segmentation module performs a first feature processing on the first training sample image to obtain first image features, and the first initial task-independent module in the second initial semantic segmentation module obtains the first semantic segmentation result based on the first image features. This semantic segmentation model has a simple structure and is easy to train, effectively reducing model training costs. Furthermore, it has relatively small network parameters and computational load, making it suitable for device-side deployment. In addition, it can better perform semantic segmentation tasks, improving the accuracy of semantic segmentation.

[0010] In one possible implementation of the first aspect above, the first training sample image includes at least one first category label, which corresponds to the corresponding semantic segmentation task, and the first category label does not include all category labels corresponding to the semantic segmentation task, that is, the first training sample image is partially labeled data corresponding to the semantic segmentation task.

[0011] For example, a semantic segmentation task may be a 10-class semantic segmentation task (e.g., vehicle, pedestrian, fence, etc.). The first category label may include fewer than 10 labels, such as only pedestrians, or only vehicles and fences.

[0012] Therefore, in the implementation of this application, a semantic segmentation model can be trained using partially labeled data, eliminating the need to label all categories of the partially labeled data, thus significantly reducing labeling costs and consequently lowering model training costs and time. Furthermore, it can efficiently utilize partially labeled data, improving its utilization rate. Additionally, it can train a semantic segmentation model covering all categories required for a semantic segmentation task under the constraints of electronic device computing power and the scenario of partially labeled data, achieving an accuracy that reaches or exceeds that of a semantic segmentation model for a single semantic segmentation task based on partially labeled data.

[0013] In one possible implementation of the first aspect above, the training dataset to which the first training subset belongs includes multiple training subsets, the first initial semantic segmentation module is a shared module including a backbone network, each training subset corresponds to a different semantic segmentation task, and the second initial semantic segmentation module includes multiple initial task-independent modules, each initial task-independent module corresponds to a different semantic segmentation task.

[0014] Therefore, in the implementation of this application, the training subsets corresponding to different semantic segmentation tasks first pass through a first initial semantic segmentation module (which serves as a shared module) to extract features, and then pass through independent initial task modules corresponding to different tasks to obtain semantic segmentation results. Based on the design of this initial semantic segmentation model, during the semantic segmentation model training process, the corresponding target semantic segmentation model can be obtained by training the initial semantic segmentation model using the training subsets corresponding to different semantic segmentation tasks. Furthermore, the semantic segmentation model has a simple network structure and is easy to train, which not only effectively reduces the model training cost, but also has relatively small network parameters and computational load, making it suitable for device-side deployment.

[0015] In one possible implementation of the first aspect above, the semantic segmentation task corresponding to the first training subset dataset is the same as the semantic segmentation task corresponding to the first initial task independent module.

[0016] In one possible implementation of the first aspect above, training an initial semantic segmentation model based on the first semantic segmentation result includes: training a first initial semantic segmentation module based on the first semantic segmentation result to obtain a first target semantic segmentation module; and training a first initial task-independent module based on the first semantic segmentation result to obtain a first target task-independent module, so as to obtain a second target semantic segmentation module including the first target task-independent module.

[0017] In one possible implementation of the first aspect described above, the semantic segmentation task corresponding to the first training subset dataset is different from the semantic segmentation task corresponding to the first initial task independent module.

[0018] In one possible implementation of the first aspect above, training the initial semantic segmentation model based on the first semantic segmentation result includes: training the first initial semantic segmentation module based on the first semantic segmentation result to obtain the first target semantic segmentation module.

[0019] In this implementation, the semantic segmentation result for each semantic segmentation task is only used to update (i.e., train) the corresponding task-independent module. Simultaneously, the semantic segmentation result for each semantic segmentation task is also used to update (i.e., train) the first initial semantic segmentation module, which serves as a shared module. This allows the first initial semantic segmentation module, as a shared module, to learn knowledge from different semantic segmentation tasks and acquire more general features. The task-independent module for each semantic segmentation task is only updated (i.e., trained), ensuring that the learned features are specific to that task. This effectively improves the accuracy of the semantic segmentation model.

[0020] In one possible implementation of the first aspect above, the first initial task independent module includes a first initial processing submodule and a second initial processing submodule. The first initial task independent module obtains the first semantic segmentation result based on the first image features, including: the first initial processing submodule performs second feature processing on the first image features to obtain second image features; and the second initial processing submodule obtains the first semantic segmentation result based on the second image features.

[0021] In the implementation of this application, the initial task independent module includes a first initial processing submodule and a second initial processing submodule that perform different image feature processing, which can accurately obtain the corresponding semantic segmentation results.

[0022] In one possible implementation of the first aspect above, the second initial processing submodule obtains the first semantic segmentation result based on the second image features, including: the second initial processing submodule performs third feature processing on the second image features to obtain third image features; the second initial processing submodule obtains probability values ​​corresponding to different semantic segmentation results based on the third image features, and takes the semantic segmentation result with the highest probability value as the first semantic segmentation result.

[0023] In the implementation of this application, the second initial processing submodule can accurately obtain the corresponding semantic segmentation result through further image feature processing and probability values ​​corresponding to different semantic segmentation results.

[0024] In one possible implementation of the first aspect above, the first initial processing submodule is a multi-scale attention module based on an attention mechanism, and the second initial processing submodule is a segmentation head module.

[0025] In this implementation, the first initial processing submodule is a multi-scale attention module based on an attention mechanism, which can adaptively extract different scale features required by each task, merging multiple semantic segmentation tasks into one network, significantly reducing latency and facilitating edge deployment. The second initial processing submodule is a segmentation head module, which can accurately obtain the semantic segmentation results corresponding to each semantic task.

[0026] In one possible implementation of the first aspect above, the first feature processing is a general feature extraction process, the first image feature is a multi-scale general feature including multiple general features of different scales, the second feature processing is a feature fusion process, the second image feature is a single-scale feature, the third feature processing is a scale adjustment process, and the third image feature is a single-scale feature with a scale different from that of the second image feature.

[0027] In this implementation, the first initial semantic segmentation module, serving as a shared module, extracts general visual features from the training sample image input to obtain multi-scale general features. A multi-scale attention module based on an attention mechanism performs feature fusion processing based on the multi-scale general features to obtain single-scale features. The segmentation head module further performs scale adjustment and other processing based on these single-scale features to obtain new single-scale features and the corresponding semantic segmentation result. This approach yields more accurate semantic segmentation results, thereby improving the accuracy of the model's semantic segmentation.

[0028] In one possible implementation of the first aspect described above, the scale of the third image feature is consistent with the scale of its corresponding training sample image to improve the accuracy of the semantic segmentation result. Of course, the size of the third image feature can also be selected and set to other sizes as needed.

[0029] In one possible implementation of the first aspect above, the training dataset to which the first training subset belongs includes multiple training subsets, each corresponding to a different semantic segmentation task, and the second initial semantic segmentation module includes multiple initial task-independent modules, each corresponding to a different semantic segmentation task. The method further includes: inputting the training sample images included in each training subset into the initial semantic segmentation model to obtain the corresponding semantic segmentation result; training the first initial semantic segmentation module based on the semantic segmentation result corresponding to each semantic segmentation task to obtain a first target semantic segmentation module; and training the initial task-independent modules corresponding to the semantic segmentation tasks based on the semantic segmentation result corresponding to each semantic segmentation task to obtain target task-independent modules, thereby obtaining a second target semantic segmentation module including the target task-independent modules.

[0030] In this implementation, based on the design of the initial semantic segmentation model, the target semantic segmentation model can be obtained by training the initial semantic segmentation model using training subsets corresponding to different semantic segmentation tasks during the semantic segmentation model training process. Specifically, the first initial semantic segmentation module extracts multi-scale general features from each training sample image and inputs them into the second initial semantic segmentation module. The initial task-independent module in the second initial semantic segmentation module can obtain the semantic segmentation result for the corresponding semantic segmentation task based on these multi-scale general features.

[0031] Thus, a semantic segmentation model can be trained using only a partially labeled training dataset to obtain the target semantic segmentation model. This eliminates the need to address the partial labeling issues in the training dataset, meaning that training the semantic segmentation model can proceed without needing to supplement the partially labeled data. This effectively reduces model training costs and allows for efficient use of partially labeled data, improving its utilization rate. Furthermore, even under the constraints of electronic device computing power and scenarios with only partially labeled data, a semantic segmentation model covering all categories required for semantic segmentation tasks can be trained, achieving an accuracy that meets or exceeds that of a semantic segmentation model for a single semantic segmentation task based on partially labeled data.

[0032] Furthermore, the network structures of the initial semantic segmentation model and the obtained target semantic segmentation model are simple and easy to train, which not only effectively reduces the model training cost, but also has small network parameters and computational load, making it suitable for device-side deployment.

[0033] Furthermore, during the training process of the semantic segmentation model, training the model based on multi-scale general features can utilize more features and learn more robust features, thus being more friendly to the problem of class imbalance and effectively improving the accuracy of the obtained semantic segmentation model.

[0034] Furthermore, different semantic segmentation tasks share the first initial semantic segmentation module and also have independent task-independent modules. In practical applications, if the requirements of a semantic segmentation task change, only the corresponding task-independent modules need to be modified (e.g., adding or deleting task-independent modules), without modifying the first initial semantic segmentation module. Therefore, only the modified task-independent modules and the shared module need to be trained, requiring only a small number of model parameters to be modified in the task-independent modules. In this way, by adding or deleting a small number of parameters, the semantic segmentation model can be adjusted to achieve the corresponding new semantic segmentation task, effectively reducing the maintenance and training costs of the semantic segmentation model.

[0035] Furthermore, during the training of the semantic segmentation model, the initial semantic segmentation model is trained based on the semantic segmentation results. This can be achieved by updating (i.e., training) the corresponding task-independent modules based on the semantic segmentation results for each semantic segmentation task, and by updating (i.e., training) the shared modules based on the semantic segmentation results for each semantic segmentation task. In this way, the semantic segmentation results for each semantic segmentation task are only used to update (i.e., train) the corresponding task-independent modules, while simultaneously updating (i.e., training) the shared modules. This allows the shared modules to learn knowledge from different semantic segmentation tasks, acquiring more general features, while the task-independent modules for each semantic segmentation task are only updated (i.e., trained), ensuring that the learned features are specific to that task. This effectively improves the accuracy of the semantic segmentation model.

[0036] In one possible implementation of the first aspect above, the training dataset includes a first training subset and a second training subset, the first training subset corresponding to a first semantic segmentation task and the second training subset corresponding to a second semantic segmentation task. The second initial semantic segmentation module includes a first initial task-independent module and a second initial task-independent module, the first initial task-independent module corresponding to the first semantic segmentation task and the second initial task-independent module corresponding to the second semantic segmentation task. Then, based on the semantic segmentation results corresponding to each semantic segmentation task, the initial task-independent module corresponding to the semantic segmentation task is trained to obtain the target task-independent module, including: training the first initial task-independent module based on the semantic segmentation results corresponding to the first semantic segmentation task to obtain a first target task-independent module; and training the second initial task-independent module based on the semantic segmentation results corresponding to the second semantic segmentation task to obtain a second target task-independent module.

[0037] Secondly, embodiments of this application provide a semantic segmentation method applied to an electronic device. The method includes: inputting images to be classified from a dataset to be classified into a target semantic segmentation model; the dataset includes at least one image to be classified; the target semantic segmentation model includes a first target semantic segmentation module and a second target semantic segmentation module; the second target semantic segmentation module includes a first target task-independent module, which has a corresponding semantic segmentation task; the target semantic segmentation model is obtained according to the aforementioned semantic segmentation model training method; the first target semantic segmentation module performs fourth feature processing on the images to be classified to obtain fourth image features; and the first target task-independent module in the second target semantic segmentation module obtains a second semantic segmentation result based on the fourth image features.

[0038] Thus, the target semantic segmentation model, through the cooperation of the first target semantic segmentation module and the first target task independent module in the second target semantic segmentation module, can conveniently, accurately and quickly obtain the semantic segmentation results corresponding to different semantic segmentation tasks based on a simple model structure.

[0039] In one possible implementation of the second aspect described above, the fourth image feature is a multi-scale general feature that includes multiple general features at different scales. This allows for more accurate semantic segmentation results.

[0040] Thirdly, embodiments of this application provide a semantic segmentation model training apparatus, comprising: a first input module, configured to input a first training sample image included in a first training subset to an initial semantic segmentation model, wherein the first training subset has a corresponding semantic segmentation task and includes at least one first training sample image, the first training sample image including at least one first category label; the initial semantic segmentation model includes a first initial semantic segmentation module and a second initial semantic segmentation module, the second initial semantic segmentation module including a first initial task-independent module, the first initial task-independent module having a corresponding semantic segmentation task; an initial semantic segmentation model module, configured to perform a first feature processing on the first training sample image through the first initial semantic segmentation module included in the initial semantic segmentation model to obtain a first image feature, and the first initial task-independent module in the second initial semantic segmentation module obtains a first semantic segmentation result based on the first image feature; and a training module, configured to train the initial semantic segmentation model based on the first semantic segmentation result to obtain a target semantic segmentation model.

[0041] Fourthly, embodiments of this application provide a semantic segmentation apparatus, comprising: a second input module, configured to input images to be classified from a dataset to be classified into a target semantic segmentation model, the dataset to be classified including at least one image to be classified, the target semantic segmentation model including a first target semantic segmentation module and a second target semantic segmentation module, the second target semantic segmentation module including a first target task independent module, the first target task independent module having a corresponding semantic segmentation task, and the target semantic segmentation model being obtained according to the aforementioned semantic segmentation model training method; and a target semantic segmentation model module, configured to perform fourth feature processing on the images to be classified through the first target semantic segmentation module in the target semantic segmentation model included in the target semantic segmentation model module to obtain fourth image features, and the first target task independent module in the second target semantic segmentation module obtaining a second semantic segmentation result based on the fourth image features.

[0042] Fifthly, embodiments of this application provide a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so as to cause the computing device cluster to perform the aforementioned semantic segmentation model training method, or to cause the computing device cluster to perform the aforementioned semantic segmentation method.

[0043] In a sixth aspect, embodiments of this application provide a computer program product containing instructions that, when executed by a computing device cluster, cause the computing device cluster to perform the aforementioned semantic segmentation model training method, or cause the computing device cluster to perform the aforementioned semantic segmentation method.

[0044] In a seventh aspect, embodiments of this application provide a computer-readable storage medium including computer program instructions, wherein when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the aforementioned semantic segmentation model training method, or the computing device cluster executes the aforementioned semantic segmentation method.

[0045] The relevant beneficial effects of the third to seventh aspects mentioned above can be found in the relevant descriptions in the first or second aspects mentioned above, and will not be repeated here. Attached Figure Description

[0046] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0047] Figure 1 This diagram illustrates the implementation principle of a deep neural network in the prior art.

[0048] Figure 2 This diagram illustrates the implementation principle of another deep neural network in the prior art.

[0049] Figure 3 This diagram illustrates the implementation principle of another deep neural network in the prior art.

[0050] Figure 4 This is a flowchart illustrating a semantic segmentation model training method according to some embodiments of this application;

[0051] Figure 5 The present invention provides a schematic flowchart of a semantic segmentation method according to some embodiments thereof.

[0052] Figure 6 According to some embodiments of this application, the structure of an initial semantic segmentation model and the corresponding semantic segmentation principle diagram are shown.

[0053] Figure 7A schematic diagram of the structure of a backbone network 110 is shown according to some embodiments of this application;

[0054] Figure 8 This is a schematic diagram illustrating the feature processing process of a multi-scale attention module according to some embodiments of this application;

[0055] Figure 9 This is a schematic diagram illustrating the process of feature processing performed by an attention unit according to some embodiments of this application;

[0056] Figure 10 According to some embodiments of this application, the structure of an initial semantic segmentation model and a corresponding schematic diagram of another semantic segmentation principle are shown;

[0057] Figure 11 According to some embodiments of this application, the structure of an initial semantic segmentation model and a corresponding schematic diagram of another semantic segmentation principle are shown;

[0058] Figure 12A This is a schematic diagram of the structure of a target semantic segmentation model and a corresponding semantic segmentation principle according to some embodiments of this application;

[0059] Figure 12B According to some embodiments of this application, another flowchart of the semantic segmentation method is shown;

[0060] Figure 13 This is a schematic diagram illustrating an application scenario of a semantic segmentation model and a semantic segmentation method according to some embodiments of this application;

[0061] Figure 14 This is a schematic diagram of a semantic segmentation model training device according to some embodiments of this application;

[0062] Figure 15 This is a schematic diagram of a semantic segmentation apparatus according to some embodiments of the present application;

[0063] Figure 16A and 16B Some structural schematic diagrams of a computing device are shown according to some embodiments of this application;

[0064] Figure 17A and 17B The present application provides some embodiments of the schematic diagrams illustrating the structure of a computing device cluster. Detailed Implementation

[0065] The technical solution of this application will be described in further detail below with reference to the accompanying drawings.

[0066] As mentioned earlier, training semantic segmentation models (also known as semantic segmentation networks, semantic segmentation network models, deep neural networks, deep learning networks, deep neural network models, image classification models, etc.) requires training datasets containing large-scale training sample images and detailed category labels (category labels can also be called type labels or attribute labels, etc.). Manually labeling category images is extremely costly in terms of manpower, financial resources, and time, resulting in high training costs for semantic segmentation models. This problem is particularly pronounced for pixel-level tasks like semantic segmentation. The expensive labeling costs, significant investment of manpower, financial resources, and time, and the long algorithm development cycle are detrimental to the commercial application of semantic segmentation models.

[0067] Furthermore, as mentioned earlier, in practical applications, the number of training sample images labeled with category labels (i.e., labeled data) in the training dataset is limited. Limited training sample images can lead to overfitting of the semantic segmentation model and make it unable to effectively distinguish between images of different categories. Ultimately, this results in incorrect predictions by the semantic segmentation model, affecting the accuracy of semantic segmentation classification tasks, especially failing to achieve good multi-task classification.

[0068] Therefore, current semantic segmentation models suffer from problems such as high training costs, poor accuracy, and inability to perform multi-task classification well.

[0069] With the development of deep learning technology, there are now some publicly available datasets (i.e., training datasets) in the field of semantic segmentation. These datasets typically consist of training sample images with partial class annotations (also called image sample data or semantic segmentation data), and can therefore be considered partially labeled data. In practical applications, the semantic segmentation task to be achieved may change. As the semantic segmentation task changes, categories may be added, removed, or modified. Existing semantic segmentation methods may struggle to adapt to these changes and perform corresponding semantic segmentation processing. For such partially labeled data, developers typically address the issue of partial annotation in training sample images through various methods.

[0070] The following is a brief description of several existing methods for solving partial annotation of training sample images, that is, a brief description of several existing training methods for deep neural networks (i.e., semantic segmentation models) corresponding to semantic segmentation tasks.

[0071] The first deep neural network training method is a method that uses fully labeled data to train a single network.

[0072] like Figure 1As shown, in this method, the original training dataset includes training sample images, for example, images only labeled with the category of "vehicle," which is partially labeled data. If, according to the requirements of the semantic segmentation task, it is necessary to classify categories such as vehicles, roads, and fences, then the partially labeled data needs to be supplemented with the required road surface, fence, and other categories to obtain fully labeled data. Then, the fully labeled data is used to train the network model, thus training a single deep neural network (i.e., the full model) to achieve the classification of categories such as vehicles, roads, and fences. In addition, during the inference phase of the deep neural network model (i.e., the actual use phase of the model), only a single network needs to be inferred.

[0073] In this method, the deep neural network is simple to design and easy to train, without needing to focus on the training problem of data labeling. However, this method has the following drawbacks: (1) The classification categories of the deep neural network obtained by this deep neural network training method have extremely poor scalability, which is not friendly to scenarios where the requirements of semantic segmentation tasks change, such as scenarios where new classification categories need to be added due to changes in business requirements. For example, in scenarios where new classification categories need to be added due to changes in business requirements, all old data needs to be re-labeled to obtain fully labeled data with the required classification categories. Therefore, this deep neural network has poor business continuity and low efficiency. (2) This deep neural network training method is difficult to handle the problem of class imbalance. For example, in the training sample images, the distribution of different categories such as people and cars is unbalanced, with more people and fewer cars. Then, the model obtained based on this method cannot quickly improve the classification performance of the model for categories with less data (such as cars) by labeling only that category.

[0074] The second method for training deep neural networks is to train multiple networks based on the data labeling method, that is, to divide the data into multiple networks for training according to the data labeling or business requirements.

[0075] like Figure 2As shown, assuming the business requires the classification of 10 semantic segmentation tasks (e.g., vehicle, pedestrian, fence, etc.), 7 road feature attribute tasks (e.g., lane lines, ground markings, etc.), and 2 freespace (drivable area) tasks (e.g., drivable area, non-drivable area), then based on the labeled data (i.e., training datasets) corresponding to the 10 semantic segmentation tasks, 7 road feature attribute tasks, and 2 freespace tasks, a separate network is trained for each semantic segmentation task, with independent training and inference, and then multiple networks are deployed. Specifically, deep neural networks are trained and deployed based on the training dataset labeled with the 10 semantic segmentation categories to implement the 10 semantic segmentation tasks; deep neural networks are trained and deployed based on the training dataset labeled with the 7 road feature attribute categories to implement the 7 road feature attribute tasks; and deep neural networks are trained and deployed based on the training dataset labeled with the 2 freespace categories to implement the 2 freespace tasks. Then, the corresponding semantic segmentation tasks are implemented based on these three deep neural networks.

[0076] In this method, the training and inference of deep neural networks are simple, and there is no need to pay attention to the problem of partial data labeling, which is more friendly to the aforementioned class imbalance problem. However, this method has the following disadvantages: (1) Due to the large number of networks, the network parameters, computational load and inference latency will increase exponentially with the increase of the number of tasks, which is not conducive to deployment on the device side. (2) For a single task, it is not possible to make good use of the labeled data of other tasks to learn a more robust feature.

[0077] The third method for training deep neural networks is to use incremental learning to continuously adjust the training categories.

[0078] like Figure 3 As shown, for different images, Figure 3 The training method shown in the example is to first add the category "person" to train the "person" category (i.e., Step t-1), then add the category "car" (i.e., Step t) to train the "car" category, and finally add the category "bike" to train the "bike" category (i.e., Step t+1). The expectation is that the final trained model can simultaneously and well segment the three objects "person", "car" and "bike".

[0079] This method can effectively utilize partially labeled data to train a deep neural network model covering all categories. However, this method has the following drawbacks: (1) There is ambiguity in the process at different training stages. For example, when training humans, the desired ground truth is humans, but actual objects (actual objects) may also include cars. In this case, cars are suppressed by the network, leading to ambiguity in the model and making it prone to training failure. (2) The forgetting problem is serious, that is, the categories trained earlier are easily forgotten in the later stages of training.

[0080] In summary, existing training methods for deep neural network models suffer from the aforementioned problems.

[0081] Based on this, the implementation of this application provides a semantic segmentation model training method, applicable to electronic devices, such as... Figure 4 As shown, the semantic segmentation model training method includes the following steps:

[0082] S110, input the training sample images included in the training subset into the initial semantic segmentation model.

[0083] In practical applications, different training datasets corresponding to different semantic segmentation tasks can be selected according to the semantic segmentation tasks to be implemented. Therefore, in this implementation, the training dataset includes at least one training subset (e.g., n subsets, where n is greater than or equal to 1), each training subset has a corresponding semantic segmentation task (e.g., there are n semantic segmentation tasks, where n is greater than or equal to 1), and each training subset includes at least one training sample image, and each sample image includes at least one category label.

[0084] For example, suppose the business requires classifying the aforementioned 10 semantic segmentation tasks (as an example of the first semantic segmentation task), 7 road feature attribute tasks (as an example of the second semantic segmentation task), and 2 freespace tasks (as an example of the third semantic segmentation task). Then, the dataset capable of performing the 10 semantic segmentation tasks (i.e., the dataset labeled with 10 semantic segmentation categories (as an example of the first category label), serving as an example of the first training subset), the dataset capable of performing the 7 road feature attribute tasks (i.e., the dataset labeled with 7 road feature categories (as an example of the second category label), serving as an example of the second training subset), and the dataset capable of performing the 2 freespace tasks (i.e., the dataset labeled with 2 freespace categories (as an example of the third category label), serving as an example of the third training subset), will each be used as a training subset, and together they will be used as the training dataset. That is, each training subset corresponds to a different semantic segmentation task.

[0085] The initial semantic segmentation model includes an initial shared module (as an example of a first initial semantic segmentation module) and an initial task module (as an example of a second initial semantic segmentation module). The initial task module includes at least one initial task-independent module, and different initial task-independent modules correspond to different semantic segmentation tasks.

[0086] For example, the initial task module includes a first independent initial task module, which corresponds to the aforementioned 10 types of semantic segmentation tasks. Additionally, the initial task module may also include a second independent initial task module and a third independent initial task module, where the second independent initial task module corresponds to the aforementioned 7 types of road feature attribute tasks, and the third independent initial task module corresponds to the aforementioned 2 types of freespace tasks.

[0087] S120, the initial semantic segmentation model performs semantic recognition processing on the training sample images to obtain the semantic segmentation results (i.e., the first semantic segmentation results) corresponding to the training sample images.

[0088] In the initial semantic segmentation model, the initial shared module performs general feature extraction on the training sample images (as an example of the first feature processing) to obtain multi-scale general features (as an example of the first image features), and inputs the multi-scale general features into the initial task module. These general features can be, for example, features of different scales that are robust to various semantic segmentation tasks, and can be selected and set according to the semantic segmentation task to be implemented.

[0089] The initial task independent module in the initial task module obtains semantic segmentation results based on multi-scale general features.

[0090] S130, Train the initial semantic segmentation model based on the semantic segmentation results to obtain the target semantic segmentation model.

[0091] For example, based on the semantic segmentation results corresponding to each semantic segmentation task, an initial task-independent module corresponding to each semantic segmentation task is trained to obtain a corresponding target task-independent module, thereby obtaining a target task module (as an example of a second target semantic segmentation module). Based on the semantic segmentation results corresponding to each semantic segmentation task, an initial shared module is trained to obtain a target shared module (as an example of a first target semantic segmentation module), thus obtaining a target semantic segmentation model.

[0092] The semantic segmentation model training method provided in this implementation uses an initial semantic segmentation model as a network structure comprising an initial shared module and an initial task module. The initial task module includes one initial task-independent module corresponding to a specific semantic segmentation task, or multiple initial task-independent modules corresponding to different semantic segmentation tasks. The initial shared module is used by the initial task-independent modules to share and extract multi-scale general features. Based on this initial semantic segmentation model design, during the semantic segmentation model training process, the initial semantic segmentation model can be trained using training subsets corresponding to different semantic segmentation tasks (i.e., partially labeled data corresponding to different semantic segmentation tasks) to obtain the corresponding target semantic segmentation model. Specifically, the initial shared module extracts multi-scale general features from each training sample image and inputs them into the initial task module. The initial task-independent modules within the initial task module can obtain the semantic segmentation results for the corresponding semantic segmentation task based on these multi-scale general features.

[0093] Thus, the semantic segmentation model training method provided in this implementation can train the target semantic segmentation model using only a partially labeled training dataset, without needing to address the partial labeling issue in the training dataset. This means that training the semantic segmentation model can proceed without needing to supplement the partially labeled data, effectively reducing model training costs and efficiently utilizing the partially labeled data, thus improving its utilization rate. Furthermore, it can train a full-category semantic segmentation model required for semantic segmentation tasks even under the constraints of electronic device computing power and scenarios with only partially labeled data, achieving an accuracy that reaches or exceeds that of a semantic segmentation model for a single semantic segmentation task based on partially labeled data.

[0094] Furthermore, the semantic segmentation model training method provided in this implementation has a simple network structure for both the initial semantic segmentation model and the obtained target semantic segmentation model, making training convenient. This not only effectively reduces the model training cost but also minimizes network parameters and computational load, making it suitable for device-side deployment. Moreover, it avoids problems such as ambiguity or severe forgetting issues during training.

[0095] Furthermore, in this implementation, during the training process of the semantic segmentation model, training the model based on multi-scale general features can utilize more features and learn more robust features, thus being more friendly to the aforementioned class imbalance problem and effectively improving the accuracy of the obtained semantic segmentation model.

[0096] Furthermore, in this implementation, different semantic segmentation tasks share the aforementioned shared modules and also have independent task-independent modules. In practical applications, if the requirements of a semantic segmentation task change, only the corresponding task-independent modules need to be modified (e.g., adding or deleting task-independent modules), without needing to modify the shared modules or other modules. Therefore, only the modified task-independent and shared modules need to be trained, requiring only a small number of model parameters to be modified in the task-independent modules. In this way, by adding or deleting a small number of parameters, the semantic segmentation model can be adjusted to implement the corresponding new semantic segmentation task, effectively reducing the maintenance and training costs of the semantic segmentation model.

[0097] Furthermore, in this implementation, during the training of the semantic segmentation model, the initial semantic segmentation model is trained based on the semantic segmentation results. This can be achieved by updating (i.e., training) the corresponding task-independent modules based on the semantic segmentation results for each semantic segmentation task, and by updating (i.e., training) the shared modules based on the semantic segmentation results for each semantic segmentation task. In this way, the semantic segmentation results for each semantic segmentation task are only used to update (i.e. train) the corresponding task-independent modules, and simultaneously used to update (i.e. train) the shared modules. This allows the shared modules to learn knowledge from different semantic segmentation tasks, acquiring more general features, while the task-independent modules for each semantic segmentation task are only updated (i.e. trained), ensuring that the learned features are specific to that task. This effectively improves the accuracy of the semantic segmentation model.

[0098] This application also provides a semantic segmentation method for use in electronic devices, such as... Figure 5 As shown, this semantic segmentation method includes the following steps:

[0099] S210: Input the images to be classified from the dataset to the target semantic segmentation model.

[0100] The dataset to be classified includes at least one image to be classified. The target semantic segmentation model is obtained according to the above semantic segmentation model training method. The target semantic segmentation model includes a target sharing module and a target task module. The target task module includes at least one target task independent module (e.g., a first target task independent module). Each target task independent module corresponds to a different semantic segmentation task.

[0101] In addition, the dataset to be classified may also include at least one sub-dataset to be classified, and each sub-dataset to be classified includes at least one image to be classified.

[0102] S220, the target semantic segmentation model performs semantic recognition processing on the image to be classified, and obtains the semantic segmentation result (i.e. the second semantic segmentation result) corresponding to the dataset to be classified.

[0103] In the target semantic segmentation model, the target sharing module performs general feature extraction processing on the images to be classified in the dataset to be classified (as an example of the fourth feature processing) to obtain multi-scale general features (as an example of the fourth image features), and inputs the multi-scale general features into the target task module.

[0104] The target task independent module (e.g., the first target task independent module) in the target task module obtains the semantic segmentation result based on multi-scale general features, which is used as the semantic segmentation result of the image to be classified.

[0105] In this way, the target semantic segmentation model can conveniently, accurately, and quickly obtain semantic segmentation results corresponding to different semantic segmentation tasks by combining the target sharing module and the target task independent module in the target task module.

[0106] The semantic segmentation model training method provided in this application will be further explained below.

[0107] like Figure 6 As shown, the initial semantic segmentation model provided by the implementation of this application includes a shared module 100 (i.e., the initial shared module) and a task module 200 (i.e., the initial task module, which can also be called the task-independent module).

[0108] The shared module 100 includes a backbone network 110, also known as a backbone module. The shared module 100 is a shared module throughout the entire semantic segmentation model, used to extract general feature representations for different semantic segmentation tasks, such as multi-scale general features for extracting general features at multiple different scales. That is, the backbone network 110 processes the input image from the real-world scene, obtaining general visual features that are then passed to the subsequent multi-scale attention module.

[0109] Task module 200 includes multiple task-independent modules (i.e., the initial task-independent module), such as task-independent module 210, task-independent module 220, and task-independent module 2n0. Among them, task-independent module 210 corresponds to task 1, task-independent module 220 corresponds to task 2, and task-independent module 2n0 corresponds to task n.

[0110] Each task-independent module includes a Multi Scale Attention Module (MSAM) (as an example of a first initial processing submodule) and a segmentation head module (as an example of a second initial processing submodule). For example, task-independent module 210 includes a multi-scale attention module 211 and a segmentation head module 212, task-independent module 220 includes a multi-scale attention module 221 and a segmentation head module 222, and task-independent module 2n0 includes a multi-scale attention module 2n1 and a segmentation head module 2n2.

[0111] Each task-independent module is dedicated to a single semantic segmentation task. This means different semantic segmentation tasks have independent multi-scale attention modules and segmentation head modules to extract relevant feature representations and generate the task's output as the prediction result. The number of task-independent modules is related to the number of tasks; each additional task requires one additional task-independent module, which in turn requires an additional set of multi-scale attention modules and segmentation head modules.

[0112] Furthermore, each multi-scale attention module is also dedicated to a separate semantic segmentation task. Based on an attention mechanism, the multi-scale attention module is used to adaptively activate the features required by each semantic segmentation task at different scales. For example, the multi-scale attention module is used to perform feature fusion processing such as weighted fusion on the multi-scale general features extracted by the backbone network 110 (as an example of second feature processing) to obtain unified single-scale features related to the semantic segmentation task (i.e., single-scale features, as an example of second image features) which are then input into the segmentation head module.

[0113] Furthermore, each multi-scale attention module includes at least one attention unit (e.g., Figure 6 The attention units shown are 2111, 2211, 2n11, etc., which are used to perform feature enhancement processing on features.

[0114] In addition, each multi-scale attention module also includes a feature fusion unit (e.g. Figure 6 The feature fusion units 2112, 2212, and 2n12 shown are used to perform feature fusion processing such as upsampling and addition on the multi-scale general features after the attention unit has performed feature enhancement processing.

[0115] The segmentation head module is used to process the task-related single-scale features extracted by the multi-scale attention module using features such as convolution and upsampling or scale adjustment (as an example of third feature processing) to obtain an image with the same resolution as the input image (i.e., consistent scale) but different scale from the aforementioned single-scale features (i.e., single-scale features, as an example of third image features). This allows the output prediction results for each task to be obtained as semantic segmentation results.

[0116] Based on this initial semantic segmentation model, the process for extracting multi-scale general features during the semantic segmentation model training can be as follows:

[0117] During the training of semantic segmentation models, such as Figure 6 As shown, the task training images (i.e. training sample images) from the multiple training subsets included in the training dataset are first input into the shared module 100.

[0118] For example, the training images used for training different tasks, such as the training images for task 1 (as an example of the first semantic segmentation task) in training subset 1 (as an example of the first training subset), the training images for task 2 (as an example of the second semantic segmentation task) in training subset 2 (as an example of the second training subset), and the training images for task n (as an example of the nth semantic segmentation task) in training subset n (as an example of the nth training subset), are first input into the shared module 100.

[0119] The training dataset includes multiple training subsets corresponding to different semantic segmentation tasks, and each training subset can be a partially labeled dataset. Of course, open-source datasets can also be selected as training subsets as needed.

[0120] The backbone network 110 in the shared module 100 is mainly used to extract common visual features from the input images of multiple tasks, that is, to extract common visual features from the training images of each task in order to obtain multi-scale common features.

[0121] In this implementation, since each task-independent module shares the backbone network 110, and the operation of the backbone network 110 to extract multi-scale general features is independent of the task type, multi-scale general features independent of the task can be extracted.

[0122] For example, the backbone network 110 performs convolution processing (i.e., general feature extraction processing) on ​​the input image to obtain multi-scale general features of the input image at different scales. For example, such as Figure 7 As shown, the backbone network 110 performs convolution processing on the input image to obtain features Feat_8x (Stage 2 8X) downsampled by 8x, features Feat_16x (Stage 3 16X) downsampled by 16x, and features Feat_32x (Stage 4 32X) downsampled by 32x. These three features at different scales are used as multi-scale general features. Of course, the backbone network 110 can also obtain other types of multi-scale general features.

[0123] In this implementation, the backbone network 110 can adopt various architectures, such as Residual Neural Network (ResNet), MobileNet, EfficientNet, etc. The backbone network 110 can also be a Convolutional Neural Network (CNN).

[0124] In addition, the weights of the backbone network 110 can be obtained by pre-training on a semantic segmentation dataset.

[0125] In one implementation of this application, after the backbone network 110 in the sharing module 100 obtains the multi-scale general features corresponding to the training images of each task, it can input the obtained multi-scale general features into each task-independent module in the task module 200, and each task-independent module in the task module 200 obtains the corresponding semantic segmentation result based on the multi-scale general features.

[0126] For example, the backbone network 110 in the shared module 100 inputs the multi-scale general features corresponding to the training images of Task 1, the multi-scale general features corresponding to the training images of Task 2, and the multi-scale general features corresponding to the training images of Task n into the multi-scale attention module 211 in the task-independent module 210, the multi-scale attention module 221 in the task-independent module 220, and the multi-scale attention module 2n1 in the task-independent module 2n0, respectively.

[0127] Each task-independent module in task module 200 obtains semantic segmentation results based on multi-scale general features. This can be achieved by each task-independent module performing feature fusion processing and result prediction processing on the multi-scale general features corresponding to different semantic segmentation tasks to obtain prediction results corresponding to different semantic segmentation tasks as semantic segmentation results.

[0128] For example, the task prediction results obtained by the task-independent module 210 include the prediction results of task 1, task 2, and task n; the task prediction results obtained by the task-independent module 220 include the prediction results of task 1, task 2, and task n; and the task prediction results obtained by the task-independent module 2n0 include the prediction results of task 1, task 2, and task n, etc.

[0129] For example, the process by which a task-independent module obtains semantic segmentation results can be described as follows:

[0130] The multi-scale attention module in each task-independent module first obtains task-related single-scale features based on multi-scale general features.

[0131] For example, multi-scale attention module 211 obtains single-scale features corresponding to task 1, task 2, and task n based on the multi-scale general features corresponding to the training images of task 1, task 2, and task n; multi-scale attention module 221 obtains single-scale features corresponding to task 1, task 2, and task n based on the multi-scale general features corresponding to the training images of task 1, task 2, and task n; multi-scale attention module 2n1 obtains single-scale features corresponding to task 1, task 2, and task n based on the multi-scale general features corresponding to the training images of task 1, task 2, and task n.

[0132] The process by which each multi-scale attention module obtains single-scale features based on multi-scale general features can be described as follows: For example, as... Figure 8 As shown, the multi-scale attention module first takes the 32x downsampled features Feat_32x corresponding to a certain task sent by the backbone network 110, performs feature enhancement processing through an attention unit, and then inputs them into the feature fusion unit for upsampling (i.e., ... Figure 8As shown in A), a 16x downsampled feature Feat_16x' is obtained. This 16x downsampled feature Feat_16x' is then added to the 16x downsampled feature Feat_16x sent from the backbone network 110, resulting in a 16x downsampled feature Feat_16x". Next, the 16x downsampled feature Feat_16x" is upsampled to obtain an 8x downsampled feature Feat_8x'. Finally, the 8x downsampled feature Feat_8x' is added to the 8x downsampled feature Feat_8x sent from the backbone network 110, resulting in an 8x downsampled feature Feat_8x", which serves as the final multi-scale fused feature, i.e., a single-scale feature (upsampling, addition, etc., are examples of feature fusion processing).

[0133] In other words, the process of fusing multi-scale general features to obtain single-scale features (i.e., fused features) is as follows:

[0134] Fusion_feat(fusion feature) = (up(A2*(up(A1*Feat_32x)+Feat_16x))+Feat_8x)

[0135] Of course, in other implementations of this application, the multi-scale attention module can also obtain the corresponding single-scale features in other ways.

[0136] In this implementation, the multi-scale attention module can adopt various different structures, such as structures based on attention mechanisms like SE (Squeeze-and-Excitation), CBAM (Convolutional Block Attention Module), and Non-local.

[0137] Furthermore, in this implementation, the attention unit consists of attention function modules in one channel dimension. For example, such as... Figure 9 As shown, F tr It is a traditional convolutional structure, where X and U are F tr The input (H'*W'*C') and output (H*W*C) are given. For features at a certain H*W*C scale, global average pooling (i.e., Fi) is applied to the H*W dimension. sq (Processing) Reduce the dimension to 1, retain the C dimension, and obtain 1×1×C features. Then, pass a fully connected (FC) layer in between (perform F) ex The process involves processing the rectified linear unit (ReLU) layer and the fully connected (FC) layer, followed by activation using a function such as the sigmoid function. This result is then used as the scale multiplier on the C channel of U (i.e., the F channel).scale (Processing), as input data for the next level. This yields the image features after feature enhancement processing.

[0138] Then, the multi-scale attention module in each task-independent module inputs the obtained single-scale features into the corresponding segmentation head module, and the segmentation head module obtains the corresponding task prediction result based on the single-scale features, that is, the corresponding semantic segmentation result.

[0139] The process by which the segmentation head module obtains semantic segmentation results based on single-scale features can be described as follows:

[0140] For example, the segmentation head module can consist of two convolutional processing modules (i.e., two convolutional operations). The first convolutional processing module is used to further extract task-related features based on single-scale features (i.e., to obtain a new single-scale feature as an example of third feature processing). The second convolutional processing module is used to generate logits corresponding to each category based on the final single-scale feature (as an example of third image features). (Logits refer to the feature vector corresponding to the category probability calculation, or the output of the last layer of the network (i.e., the output of the last fully connected layer of the network), etc.) After processing the logits by the softmax function, the probability values ​​corresponding to different categories can be obtained. The index of the largest probability value (argmax) (i.e., the category) can be directly selected to obtain the corresponding category (i.e., the semantic segmentation result).

[0141] The process by which the segmentation head module calculates and determines the semantic segmentation result is as follows:

[0142] C(semantic segmentation result) = argmax(softmax(logits))

[0143] In this way, each segmentation head module can obtain the corresponding semantic segmentation results. For example, segmentation head module 212 obtains the semantic segmentation results of task 1 (i.e., the prediction results of task 1), the semantic segmentation results of task 2 (i.e., the prediction results of task 2), and the semantic segmentation results of task n (i.e., the prediction results of task n). Segmentation head module 222 obtains the semantic segmentation results of task 1, task 2, and task n. Segmentation head module 2n2 obtains the semantic segmentation results of task 1, task 2, and task n, and so on.

[0144] In this implementation, after obtaining the prediction results of each semantic segmentation task, the shared module 100 is trained using the prediction results of different semantic segmentation tasks (for example, the shared module 100 is trained using the semantic segmentation results of task 1, task 2, and task n respectively), and the corresponding task-independent modules in the task module 200 are trained using the prediction results of different semantic segmentation tasks (for example, the multi-scale attention module 211 and the segmentation head module 212 in the task-independent module 210 are trained using the semantic segmentation results of task 1 to obtain the corresponding target task-independent module 210' (as the first...). (An example of a target task independent module); using the semantic segmentation results of task 2, the multi-scale attention module 221 and the segmentation head module 222 in task independent module 220 are trained to obtain the corresponding target task independent module 220' (as an example of a second target task independent module); using the semantic segmentation results of task n, the multi-scale attention module 2n1 and the segmentation head module 2n2 in task independent module 2n0 are trained to obtain the corresponding target task independent module 2n0' (as an example of an nth target task independent module), etc., until the training termination condition is reached to obtain the trained target semantic segmentation model. This training termination condition may be, for example, the convergence of the target loss function of the task independent modules in shared module 100 and task module 200, or reaching a preset number of iterations. The target loss function and its convergence condition, or the number of iterations, can be specifically set as needed; this application does not impose specific limitations on this.

[0145] That is, in this implementation, when training the corresponding task-independent modules in task module 200 based on the prediction results of different semantic segmentation tasks, for example, the prediction result of task 1 corresponding to the semantic segmentation task of task-independent module 210 can be selected from the prediction results obtained from task-independent module 210 to train the candidate of task-independent module 210. For other prediction results obtained from task-independent module 210, they only need to be used for training other corresponding task-independent modules.

[0146] Of course, in this implementation, when training the corresponding task-independent modules in task module 200 based on the prediction results of different semantic segmentation tasks, for example, only the loss of the output results of the training images of each semantic segmentation task can be calculated, and the losses of the other tasks can be set to 0. In this way, the purpose of training the corresponding task-independent modules in task module 200 using the prediction results of different semantic segmentation tasks can be achieved.

[0147] In this way, the semantic segmentation results for each semantic segmentation task are only used to train the corresponding task-independent module, while the semantic segmentation results for each semantic segmentation task are also used to train the shared module. This allows the shared module to learn knowledge from different semantic segmentation tasks and acquire more general features. The task-independent module for each semantic segmentation task is only trained on itself, ensuring that the features learned in this part are specific to that task. This effectively improves the accuracy of the semantic segmentation model.

[0148] The semantic segmentation model training method provided in this application proposes a multi-task semantic segmentation network framework. For different semantic segmentation tasks, it extracts common features through a shared backbone network, and then uses these common features to extract semantic segmentation task-related features using independent multi-scale attention modules and segmentation heads, thereby realizing the fusion of multiple semantic segmentation tasks into one network.

[0149] In this training dataset, the training sample images are only partially labeled, meaning each image only performs labeling for a single task. To train multiple semantic segmentation tasks simultaneously, the network input consists of images from multiple semantic segmentation tasks. After general feature extraction via the backbone network 110 in the shared module 100, different semantic segmentation tasks extract relevant features through different multi-scale attention modules and segmentation heads, outputting prediction results (i.e., semantic segmentation results). Subsequently, during model training, for these network predictions, each semantic segmentation task only calculates the loss of its own task's training image output, setting the losses of other tasks to 0. This ensures that each task only updates (trains) its own multi-scale attention module and segmentation head, and simultaneously updates (trains) the backbone network 110. This allows the backbone network 110 to learn knowledge from different tasks and acquire more general features. Each task's independent module is updated only by itself, ensuring that the learned features are specific to that task. This effectively improves the accuracy of semantic segmentation.

[0150] Furthermore, the semantic segmentation model training method provided in this implementation uses a unified shared backbone network for different semantic segmentation tasks to extract general features independent of the semantic segmentation tasks. Each semantic segmentation task has its own independent multi-scale attention module and segmentation head to extract features relevant to the semantic segmentation task. This allows multiple semantic segmentation tasks to be integrated into a single network, significantly reducing latency. Simultaneously, sharing general features across multiple tasks improves model accuracy. In addition, data from individual semantic segmentation tasks can be fully utilized without additional annotation, greatly reducing annotation costs.

[0151] In other words, the semantic segmentation model training method provided in this implementation effectively solves the problem of exponentially increased latency caused by multiple models in Method 2, as well as the data annotation problem of a single model in Method 1, by sharing a common backbone network and independently setting multi-scale attention modules and segmentation heads for different semantic segmentation tasks. In addition, sharing common features across multiple semantic segmentation tasks can further improve model accuracy.

[0152] Furthermore, based on the semantic segmentation model training method provided in this implementation, if the requirements of the semantic segmentation task change, only the corresponding task-independent modules need to be modified as needed. For example, for each added semantic segmentation task, a new task-independent module is added, i.e., a new set of multi-scale attention modules and segmentation heads are added; for each removed semantic segmentation task, a task-independent module is deleted, i.e., a set of multi-scale attention modules and segmentation heads are deleted. No modification to the backbone network is required. Therefore, only a small number of model parameters in the task-independent module part need to be modified accordingly, effectively reducing the training cost and cycle of the semantic segmentation model. Thus, this semantic segmentation model training method is better suited to scenarios where the requirements of semantic segmentation tasks change.

[0153] In addition, for the backbone network, the semantic segmentation results of each semantic segmentation task are optimized simultaneously. For the multi-scale attention modules and segmentation heads that are independent of each task, the semantic segmentation results of the corresponding task are optimized independently, without coupling with each other, which effectively improves the model training efficiency and the model accuracy.

[0154] Furthermore, the multi-scale attention module proposed in this application, which is independent for each semantic segmentation task, can adaptively extract features of different scales required for each task.

[0155] Thus, under the constraints of device computing power and the scenario of partially labeled data, a semantic segmentation model for all categories can be trained, and the classification accuracy of this semantic segmentation model can reach or exceed that of a single semantic segmentation model obtained through partial labeling.

[0156] Furthermore, the semantic segmentation model training method provided in this implementation considers the shortcomings of existing methods and the limitations of practical application scenarios. It proposes a network structure with a shared, unified backbone network and independent multi-scale attention modules and segmentation heads for different semantic segmentation tasks. This allows for the completion of multiple segmentation tasks with only a small increase in parameters, while efficiently utilizing partially labeled data. Each semantic segmentation task only updates the corresponding segmentation head and the shared backbone network, thus solving the problem of partial labeling and significantly reducing the number of parameters and computational cost compared to multiple networks.

[0157] Furthermore, the semantic segmentation model training method provided in this implementation addresses the characteristic that semantic segmentation tasks require the fusion of high-level semantic features and low-level detailed information. It designs an independent multi-scale attention module for each segmentation head module, that is, an independent multi-scale attention module for each semantic segmentation task, which adaptively activates the features required by each task at different scales, effectively improving the segmentation effect of each task.

[0158] In another implementation of this application, after the multi-scale attention modules of each task-independent module in the aforementioned task module 200 receive the multi-scale general features sent by the backbone network 110 in the shared module 100, each multi-scale attention module can first filter out the target multi-scale general features corresponding to the semantic segmentation task of the task-independent module from the multi-scale general features.

[0159] For example, multi-scale attention module 211 selects the target multi-scale general features corresponding to task 1 from the multi-scale general features corresponding to the training images of task 1, task 2, and task n, based on their corresponding semantic segmentation tasks (i.e., task 1); multi-scale attention module 221 selects the target multi-scale general features corresponding to task 2 from the multi-scale general features corresponding to the training images of task 1, task 2, and task n, based on their corresponding semantic segmentation tasks (i.e., task 2); multi-scale attention module 2n1 selects the target multi-scale general features corresponding to task n from the multi-scale general features corresponding to the training images of task 1, task 2, and task n, based on their corresponding semantic segmentation tasks (i.e., task n).

[0160] The multi-scale attention module in each task-independent module selects the target multi-scale general features corresponding to the semantic segmentation task based on the semantic segmentation task corresponding to the task-independent module. This can be achieved based on the task supervision capability of the network.

[0161] Then, each multi-scale attention module performs feature fusion processing on the selected target multi-scale general features to obtain the corresponding single-scale features, and inputs the single-scale features into the corresponding segmentation head module, which then obtains the corresponding semantic segmentation result.

[0162] In this implementation, since each multi-scale attention module performs feature fusion processing on the multi-scale features corresponding to its respective semantic segmentation task to obtain single-scale features, each segmentation head module obtains the semantic segmentation result corresponding to its semantic segmentation task. For example, Figure 10As shown, segmentation head module 212 obtains the semantic segmentation result of task 1 (i.e., the prediction result of task 1), segmentation head module 222 obtains the semantic segmentation result of task 2 (i.e., the prediction result of task 2), segmentation head module 2n2 obtains the semantic segmentation result of task n (i.e., the prediction result of task n), and so on.

[0163] That is, in this implementation, the task-independent modules corresponding to each semantic segmentation task perform further feature extraction and semantic segmentation result prediction processing on the training images of the training subsets with the same semantic segmentation task to obtain the corresponding semantic segmentation results.

[0164] In this implementation, after obtaining the prediction results of each semantic segmentation task, the shared module 100 is trained using the prediction results of different semantic segmentation tasks (for example, the shared module 100 is trained using the semantic segmentation results of task 1, task 2, and task n respectively), and the corresponding task-independent modules in task module 200 are trained using the prediction results of different semantic segmentation tasks (for example, the multi-scale attention module 211 and segmentation head module 212 in task-independent module 210 are trained using the semantic segmentation results of task 1, the multi-scale attention module 221 and segmentation head module 222 in task-independent module 220 are trained using the semantic segmentation results of task 2, and the multi-scale attention module 2n1 and segmentation head module 2n2 in task-independent module 2n0 are trained using the semantic segmentation results of task n, etc.), until the training termination condition is reached to obtain the trained target semantic segmentation model.

[0165] In this way, the semantic segmentation results for each semantic segmentation task are only used to train the corresponding task-independent module, while the semantic segmentation results for each semantic segmentation task are also used to train the shared module. This allows the shared module to learn knowledge from different semantic segmentation tasks and acquire more general features. The task-independent module for each semantic segmentation task is only trained on itself, ensuring that the features learned in this part are specific to that task. This effectively improves the accuracy of the semantic segmentation model.

[0166] In another implementation of this application, after the backbone network 110 in the shared module 100 obtains the multi-scale general features corresponding to the training images of each task, it can also input the multi-scale general features corresponding to each semantic segmentation task into the corresponding task-independent module in the task module 200 according to the different semantic segmentation tasks.

[0167] For example, such as Figure 11As shown, the backbone network 110 in the shared module 100 inputs the multi-scale general features corresponding to the training images of Task 1 into the multi-scale attention module 211 in the task-independent module 210, inputs the multi-scale general features corresponding to the training images of Task 2 into the multi-scale attention module 221 in the task-independent module 220, and inputs the multi-scale general features corresponding to the training images of Task 3 into the multi-scale attention module 2n1 in the task-independent module 2n0. Then, the multi-scale attention modules in each task-independent module take the received multi-scale general features as the target multi-scale general features by default, perform feature fusion processing to obtain the corresponding single-scale features, and input the single-scale features into the corresponding segmentation head module, which then obtains the corresponding semantic segmentation result. For example, segmentation head module 212 obtains the semantic segmentation result of Task 1 (i.e., the prediction result of Task 1), segmentation head module 222 obtains the semantic segmentation result of Task 2 (i.e., the prediction result of Task 2), segmentation head module 2n2 obtains the semantic segmentation result of Task n (i.e., the prediction result of Task n), and so on.

[0168] In this implementation, after obtaining the prediction results of each semantic segmentation task, the shared module 100 is trained using the prediction results of different semantic segmentation tasks (for example, the shared module 100 is trained using the semantic segmentation results of task 1, task 2, and task n respectively), and the corresponding task-independent modules in task module 200 are trained using the prediction results of different semantic segmentation tasks (for example, the multi-scale attention module 211 and segmentation head module 212 in task-independent module 210 are trained using the semantic segmentation results of task 1, the multi-scale attention module 221 and segmentation head module 222 in task-independent module 220 are trained using the semantic segmentation results of task 2, and the multi-scale attention module 2n1 and segmentation head module 2n2 in task-independent module 2n0 are trained using the semantic segmentation results of task n, etc.), until the training termination condition is reached to obtain the trained target semantic segmentation model.

[0169] In this way, the semantic segmentation results for each semantic segmentation task are only used to train the corresponding task-independent module, while the semantic segmentation results for each semantic segmentation task are also used to train the shared module. This allows the shared module to learn knowledge from different semantic segmentation tasks and acquire more general features. The task-independent module for each semantic segmentation task is only trained on itself, ensuring that the features learned in this part are specific to that task. This effectively improves the accuracy of the semantic segmentation model.

[0170] The target semantic segmentation model and semantic segmentation method provided in this application will be further explained below.

[0171] like Figure 12AAs shown, the target semantic segmentation model (i.e., the overall system reasoning architecture) includes a shared module 100' (i.e., the target shared module) and a task module 200' (i.e., the target task module). The shared module 100' includes a backbone network 110', and the task module 200' includes multiple task-independent modules (i.e., target task-independent modules), such as task-independent module 210' (as an example of a first target task-independent module), task-independent module 220' (as an example of a second target task-independent module), and task-independent module 2n0' (as an example of an nth target task-independent module). Task-independent module 210' includes a multi-scale attention module 211' and a segmentation head module 212', task-independent module 220' includes a multi-scale attention module 221' and a segmentation head module 222', and task-independent module 2n0' includes a multi-scale attention module 2n1' and a segmentation head module 2n2'. The multi-scale attention module 211' includes attention unit 2111' and feature fusion unit 2112', the multi-scale attention module 221' includes attention unit 2211' and feature fusion unit 2212', the multi-scale attention module 2n1' includes attention unit 2n11' and feature fusion unit 2n12', etc.

[0172] The semantic segmentation method provided in this application, such as Figure 12A and 12B As shown, the process includes the following steps: The multi-task images to be classified (i.e., the images to be classified included in the dataset, such as image 1, image 2, image n, etc.) are input into the target semantic segmentation model obtained above. The backbone network 110' in the shared module 100' of the target semantic segmentation model extracts general features from the multi-task images to obtain multi-scale general features. Then, the shared module 100' inputs the obtained multi-scale general features into the task module 200', and the corresponding task-independent modules in the task module 200' obtain the corresponding semantic segmentation results based on the multi-scale general features. For example, the multi-scale general features corresponding to Task 1 are input into the task-independent module 210' corresponding to Task 1 in the task module, the multi-scale general features corresponding to Task 2 are input into the task-independent module 220' corresponding to Task 2 in the task module, and the multi-scale general features corresponding to Task n are input into the task-independent module 2n0' corresponding to Task n in the task module. The multi-scale attention units in each task-independent module perform task-related feature extraction 1, task-related feature extraction 2, task-related feature extraction N, etc., and are classified by the corresponding segmentation head module. Then, the outputs of different semantic segmentation tasks, such as Task 1 output (i.e., Task 1 prediction result), Task 2 output (i.e., Task 2 prediction result), and Task N output (i.e., Task n prediction result), are obtained as the semantic segmentation results corresponding to the multi-task image.

[0173] In the semantic segmentation method provided by this implementation, the input real-world scene images (i.e., multi-task images) are processed through a unified backbone network, and the output features are processed through independent modules of different tasks to produce output results for different segmentation tasks, which effectively improves the efficiency and accuracy of semantic segmentation.

[0174] Furthermore, the semantic segmentation method provided in this implementation is based on a deep neural network and can achieve multi-task semantic segmentation. Therefore, it can be understood as a new method for multi-task semantic segmentation.

[0175] The semantic segmentation method provided in this application can be applied to autonomous driving scenarios (i.e., application scenario 1). Autonomous driving scenarios are highly complex, requiring numerous perception tasks to assist in policy control and decision-making. Freespace tasks help distinguish drivable areas, while attributes such as lane lines and streetlights aid in localization. The numerous semantic segmentation tasks significantly increase computational power and present challenges due to the annotation issues of different task components. Therefore, in autonomous driving applications, it is necessary to detect drivable areas to ensure the vehicle stays within them. Furthermore, lane lines and streetlights on the road are crucial for mapping and localization; both of these pieces of information require the output of a semantic segmentation model.

[0176] For example, such as Figure 13 As shown, this semantic segmentation method can be applied to autonomous driving scenarios to achieve classification tasks such as two types of drivable areas (freespace) and seven types of road feature attributes.

[0177] Due to the numerous segmentation tasks and high real-time requirements in autonomous driving scenarios, the semantic segmentation model (i.e., multi-task semantic segmentation network) and method proposed in this implementation can address this issue. Thus, with only a small increase in parameters and latency, segmentation for two tasks can be achieved, simplifying model training and deployment. Furthermore, a single input image can output segmentation results for both tasks. In other words, the semantic segmentation method provided in this implementation, when applied to autonomous driving applications, can resolve the significant increase in computational power caused by multiple semantic segmentation tasks and the challenges of different task component annotations.

[0178] The multi-task semantic segmentation network proposed in this application uses a unified backbone network for different tasks to extract task-independent general features, while each task's independent multi-scale attention module and segmentation head module extract task-related features. This allows multiple tasks to be integrated into a single network, significantly reducing latency, while sharing general features across multiple tasks improves model accuracy. Furthermore, it fully utilizes data from previously individual tasks without requiring additional annotation, greatly reducing annotation costs.

[0179] The semantic segmentation method provided in this application can also be applied to terminal vision scenarios (i.e., application scenario 2). For example, in terminal vision scenarios, features such as background blurring and image matting in current mobile phone photography rely heavily on semantic segmentation. However, the data for each task is labeled specifically for that task, making it impossible to obtain fully labeled data, and re-labeling is extremely costly. If multiple tasks need to be run simultaneously on the device (e.g., performing different background blurring and image matting on images), using multiple models results in unacceptable latency for the device. Therefore, it is necessary and urgent to have a unified model that can process partially labeled data while significantly reducing latency. The semantic segmentation method provided in this application can effectively reduce inference latency significantly without additional labeled data.

[0180] Of course, the semantic segmentation method provided in this implementation can also be applied to other computer vision or semantic segmentation scenarios such as medical image processing, video surveillance processing, and target tracking.

[0181] This application also provides a semantic segmentation model training device, such as Figure 14 As shown, it includes: a first input module, an initial semantic segmentation model module, and a training module.

[0182] The first input module is used to input the first training sample images included in the first training subset into the initial semantic segmentation model. The first training subset has a corresponding semantic segmentation task and includes at least one first training sample image, which includes at least one first category label. The initial semantic segmentation model includes a first initial semantic segmentation module and a second initial semantic segmentation module. The second initial semantic segmentation module includes a first initial task-independent module, which has a corresponding semantic segmentation task.

[0183] The initial semantic segmentation model module is used to perform semantic recognition processing on the training sample images to obtain the semantic segmentation results corresponding to the training sample images. Specifically, the first initial semantic segmentation module in the initial semantic segmentation model included in the initial semantic segmentation module performs first feature processing on the first training sample image to obtain first image features, and the first initial task independent module in the second initial semantic segmentation module obtains the first semantic segmentation result based on the first image features.

[0184] The training module is used to train the initial semantic segmentation model based on the first semantic segmentation result to obtain the target semantic segmentation model.

[0185] The first input module, the initial semantic segmentation model module, and the training module can all be implemented in software or hardware. For example, the implementation of the first input module will be described below. Similarly, the implementation of the initial semantic segmentation model module and the training module can refer to the implementation of the first input module.

[0186] As an example of a software functional unit, the first input module may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the aforementioned computing instance may be one or more. For example, the first input module may include code running on multiple hosts / virtual machines / containers. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0187] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0188] As an example of a hardware functional unit, the first input module may include at least one computing device, such as a server. Alternatively, the first input module may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0189] The multiple computing devices included in the first input module can be distributed within the same region or in different regions. Similarly, the multiple computing devices included in the first input module can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the first input module can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0190] In other embodiments, the first input module can be used to execute any step in the semantic segmentation model training method, the initial semantic segmentation model module can be used to execute any step in the semantic segmentation model training method, and the training module can be used to execute any step in the semantic segmentation model training method. The steps implemented by the first input module, the initial semantic segmentation model module, and the training module can be specified as needed. By implementing different steps in the semantic segmentation model training method through the first input module, the initial semantic segmentation model module, and the training module, all functions of the semantic segmentation model training device can be realized.

[0191] The process of each module implementing its corresponding function in this implementation method can be found in the aforementioned content on semantic segmentation model training methods, and will not be repeated here.

[0192] In this implementation, the semantic segmentation model training device can also be applied to computing devices such as computers and servers, or to a cluster of computing devices including at least one computing device, in order to realize the semantic segmentation model training function.

[0193] This application also provides a semantic segmentation apparatus, such as Figure 15 As shown, it includes: a second input module and a target semantic segmentation model module.

[0194] The second input module is used to input the images to be classified from the dataset to the target semantic segmentation model. The dataset includes at least one image to be classified. The target semantic segmentation model includes a first target semantic segmentation module and a second target semantic segmentation module. The second target semantic segmentation module includes a first target task-independent module, which has a corresponding semantic segmentation task. The target semantic segmentation model is obtained according to the aforementioned semantic segmentation model training method.

[0195] The target semantic segmentation model module is used to perform semantic recognition processing on the image to be classified using the target semantic segmentation model included in the target semantic segmentation model module, and obtain the semantic segmentation result corresponding to the image to be classified. That is, it is used to perform, for example, general feature processing on the image to be classified using the target sharing module in the target semantic segmentation model included in the target semantic segmentation model module to obtain multi-scale general features, and the first target task independent module in the target task module obtains the second semantic segmentation result based on the multi-scale general features.

[0196] Both the second input module and the target semantic segmentation model module can be implemented in software or hardware. For example, the implementation of the second input module will be described below. Similarly, the implementation of the target semantic segmentation model module can refer to the implementation of the second input module.

[0197] As an example of a software functional unit, the second input module may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the second input module may include code running on multiple hosts / virtual machines / containers. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0198] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0199] As an example of a hardware functional unit, the second input module may include at least one computing device, such as a server. Alternatively, the second input module may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0200] The multiple computing devices included in the second input module can be distributed within the same region or in different regions. Similarly, the multiple computing devices included in the second input module can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the second input module can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0201] In other embodiments, the second input module can be used to execute any step in the semantic segmentation method, and the target semantic segmentation model module can be used to execute any step in the semantic segmentation method. The steps implemented by the second input module and the target semantic segmentation model module can be specified as needed. By implementing different steps in the semantic segmentation method through the second input module and the target semantic segmentation model module respectively, the full functions of the semantic segmentation device can be realized.

[0202] The process by which each module implements its corresponding function in this implementation method can be found in the aforementioned content related to semantic segmentation methods, and will not be repeated here.

[0203] In this implementation, the semantic segmentation device can also be applied to computing devices such as computers and servers, or to a cluster of computing devices including at least one computing device, to achieve semantic segmentation functionality.

[0204] This application also provides a computing device 10. For example... Figure 16A and 16BAs shown, the computing device 10 includes a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other via the bus 102. The computing device 10 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 10.

[0205] Bus 102 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 16A and 16B The bus 102 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 102 may include a path for transmitting information between various components of the computing device 10 (e.g., memory 106, processor 104, communication interface 108).

[0206] The processor 104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0207] Memory 106 may include volatile memory, such as random access memory (RAM). Processor 104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0208] like Figure 16A As shown, the memory 106 stores executable program code, and the processor 104 executes this executable program code to implement the functions of the aforementioned first input module, initial semantic segmentation model module, and training module, respectively, that is, to implement the functions of the aforementioned semantic segmentation model training device, thereby realizing the semantic segmentation model training method. In other words, the memory 106 stores instructions for executing the semantic segmentation model training method.

[0209] Or, such as Figure 16B As shown, the memory 106 stores executable code, and the processor 104 executes this executable code to implement the functions of the aforementioned second input module and the target semantic segmentation model module, respectively, that is, to implement the functions of the aforementioned semantic segmentation device, thereby realizing the semantic segmentation method. In other words, the memory 106 stores instructions for executing the semantic segmentation method.

[0210] The communication interface 108 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 10 and other devices or communication networks.

[0211] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0212] like Figure 17A As shown, the computing device cluster includes at least one computing device 10. The memory 106 of one or more computing devices 10 in the computing device cluster may store the same instructions for executing semantic segmentation model training methods.

[0213] In some possible implementations, the memory 106 of one or more computing devices 10 in the computing device cluster may also store partial instructions for executing the semantic segmentation model training method. In other words, a combination of one or more computing devices 10 can jointly execute the instructions for executing the semantic segmentation model training method.

[0214] The memories 106 in different computing devices 10 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the semantic segmentation model training device. That is, the instructions stored in the memories 106 of different computing devices 10 can implement the functions of one or more modules among the first input module, the initial semantic segmentation model module, and the training module.

[0215] like Figure 17B As shown, the computing device cluster includes at least one computing device 10. The memory 106 of one or more computing devices 10 in the computing device cluster may store the same instructions for performing semantic segmentation methods.

[0216] In some possible implementations, the memory 106 of one or more computing devices 10 in the computing device cluster may also store partial instructions for executing the semantic segmentation method. In other words, a combination of one or more computing devices 10 can jointly execute the instructions for executing the semantic segmentation method.

[0217] The memories 106 in different computing devices 10 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the semantic segmentation device. That is, the instructions stored in the memories 106 of different computing devices 10 can implement the functions of one or more modules in the second input module and the target semantic segmentation model module.

[0218] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN), a local area network (LAN), or similar.

[0219] This application also provides a computer program product containing instructions. The computer program product can be software or program products containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product runs on at least one computing device, it causes the at least one computing device to execute a semantic segmentation model training method, or a semantic segmentation method.

[0220] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute a semantic segmentation model training method, or instruct the computing device to execute a semantic segmentation method.

[0221] In this implementation, the aforementioned semantic segmentation model training method and / or semantic segmentation method can be applied to electronic devices. These electronic devices can be determined based on the scenario, such as vehicles, intelligent driving servers, or autonomous driving perception systems in the aforementioned intelligent driving scenario (i.e., autonomous driving scenario), or terminal devices or electronic devices such as mobile phones, tablets, and wearable devices in a terminal vision scenario. This semantic segmentation model training method and / or semantic segmentation method can also be applied to scenarios requiring semantic segmentation, such as medical imaging, smart cities, medical image processing, and scene understanding. Deploying this semantic segmentation model training method and / or semantic segmentation method on the computing nodes of relevant devices, through software modifications, it can handle more semantic segmentation tasks with only a small increase in the number of parameters and latency.

[0222] Furthermore, the aforementioned semantic segmentation model training method and / or semantic segmentation method can be applied to a computing device or a cluster of computing devices including at least one computing device. This computing device or cluster of computing devices is used to deploy the algorithms related to the semantic segmentation model training method and / or semantic segmentation method involved in the implementation of this application, to achieve the corresponding functions. The computing device can be an electronic device such as a vehicle, intelligent driving server, mobile phone, tablet computer, wearable device, or other types of devices, which can be selected and configured as needed.

[0223] In other implementations of this application, the aforementioned first initial semantic segmentation module, second initial semantic segmentation module, initial task independent module, first initial processing submodule, and second initial processing submodule can also be configured with other modules as needed. The aforementioned first feature processing, second feature processing, and third feature processing can also be configured with other image feature-related processing as needed. Furthermore, the first image feature, second image feature, and third image feature can also be other image features, which can be set as required.

[0224] In the implementation of this application, the terms "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0225] In this application, some structural or methodological features may be shown in the accompanying drawings in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, including structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0226] Although this application has been illustrated and described with reference to certain embodiments thereof, those skilled in the art should understand that the above description is a further detailed explanation of this application in conjunction with specific embodiments, and should not be construed as limiting the specific implementation of this application to these descriptions. Those skilled in the art can make various changes in form and detail, including some simple deductions or substitutions, without departing from the spirit and scope of this application.

Claims

1. A semantic segmentation model training method, applied to electronic devices, characterized in that, The method includes: The first training sample images included in the first training subset are input into the initial semantic segmentation model. The first training subset has a corresponding semantic segmentation task, and the first training subset includes at least one first training sample image, which includes at least one first category label. The initial semantic segmentation model includes a first initial semantic segmentation module and a second initial semantic segmentation module. The second initial semantic segmentation module includes multiple initial task-independent modules, each of which corresponds to a different semantic segmentation task. The multiple initial task-independent modules include the first initial task-independent module, which has a corresponding semantic segmentation task. The semantic segmentation task corresponding to the first training subset is the same as the semantic segmentation task corresponding to the first initial task-independent module. The first initial semantic segmentation module performs a first feature processing on the first training sample image to obtain the first image features; The first initial task-independent module in the second initial semantic segmentation module obtains the first semantic segmentation result based on the first image features; The initial semantic segmentation model is trained based on the first semantic segmentation result to obtain the target semantic segmentation model.

2. The semantic segmentation model training method according to claim 1, characterized in that, Training the initial semantic segmentation model based on the first semantic segmentation result includes: Based on the first semantic segmentation result, the first initial semantic segmentation module is trained to obtain the first target semantic segmentation module; and Based on the first semantic segmentation result, the first initial task-independent module is trained to obtain a first target task-independent module, thereby obtaining a second target semantic segmentation module including the first target task-independent module.

3. The semantic segmentation model training method according to claim 1, characterized in that, The semantic segmentation task corresponding to the first training subset dataset is different from the semantic segmentation task corresponding to the independent module of the first initial task.

4. The semantic segmentation model training method according to claim 3, characterized in that, Training the initial semantic segmentation model based on the first semantic segmentation result includes: Based on the first semantic segmentation result, the first initial semantic segmentation module is trained to obtain the first target semantic segmentation module.

5. The semantic segmentation model training method according to any one of claims 1-4, characterized in that, The first initial task-independent module includes a first initial processing submodule and a second initial processing submodule. The first initial task-independent module obtains a first semantic segmentation result based on the first image features, including: The first initial processing submodule performs second feature processing on the first image features to obtain second image features; The second initial processing submodule obtains the first semantic segmentation result based on the second image features.

6. The semantic segmentation model training method according to claim 5, characterized in that, The second initial processing submodule obtains the first semantic segmentation result based on the second image features, including: The second initial processing submodule performs third feature processing on the second image features to obtain third image features; The second initial processing submodule obtains probability values ​​corresponding to different semantic segmentation results based on the third image features, and takes the semantic segmentation result with the highest probability value as the first semantic segmentation result.

7. The semantic segmentation model training method according to claim 6, characterized in that, The first feature processing is a general feature extraction process, and the first image feature is a multi-scale general feature including multiple general features of different scales. The second feature processing is a feature fusion process, and the second image feature is a single-scale feature. The third feature processing is a scale adjustment process, and the third image feature is a single-scale feature with a scale different from that of the second image feature.

8. The semantic segmentation model training method according to claim 7, characterized in that, The scale of the third image feature is consistent with the scale of its corresponding training sample image.

9. The semantic segmentation model training method according to claim 5, characterized in that, The first initial processing submodule is a multi-scale attention module based on an attention mechanism, and the second initial processing submodule is a segmentation head module.

10. The semantic segmentation model training method according to claim 1, characterized in that, The first initial semantic segmentation module is a shared module that includes a backbone network.

11. The semantic segmentation model training method according to claim 1, characterized in that, If the training dataset to which the first training subset belongs includes multiple training subsets, each corresponding to a different semantic segmentation task, then the method further includes: The training sample images included in each of the training subsets are input into the initial semantic segmentation model to obtain the corresponding semantic segmentation results; Based on the semantic segmentation results corresponding to each of the semantic segmentation tasks, the first initial semantic segmentation module is trained to obtain the first target semantic segmentation module, and Based on the semantic segmentation results corresponding to each semantic segmentation task, the initial task-independent module corresponding to the semantic segmentation task is trained to obtain the target task-independent module, thereby obtaining a second target semantic segmentation module including the target task-independent module.

12. The semantic segmentation model training method according to claim 11, characterized in that, The training dataset includes a first training subset and a second training subset, where the first training subset corresponds to a first semantic segmentation task and the second training subset corresponds to a second semantic segmentation task. The second initial semantic segmentation module includes a first initial task-independent module and a second initial task-independent module, where the first initial task-independent module corresponds to the first semantic segmentation task and the second initial task-independent module corresponds to the second semantic segmentation task. Based on the semantic segmentation results corresponding to each semantic segmentation task, the initial task-independent module corresponding to each semantic segmentation task is trained to obtain a target task-independent module, including: Based on the semantic segmentation result corresponding to the first semantic segmentation task, the first initial task independent module is trained to obtain the first target task independent module; Based on the semantic segmentation result corresponding to the second semantic segmentation task, the second initial task independent module is trained to obtain the second target task independent module.

13. A semantic segmentation method applied to electronic devices, characterized in that, The method includes: The dataset to be classified includes images to be classified, which are then input into a target semantic segmentation model. The dataset to be classified includes at least one image to be classified. The target semantic segmentation model includes a first target semantic segmentation module and a second target semantic segmentation module. The second target semantic segmentation module includes multiple initial task-independent modules, each of which corresponds to a different semantic segmentation task. The multiple initial task-independent modules include a first target task-independent module, which has a corresponding semantic segmentation task. The target semantic segmentation model is obtained by the semantic segmentation model training method according to any one of claims 1-12. The first target semantic segmentation module performs fourth feature processing on the image to be classified to obtain fourth image features; The first target task independent module in the second target semantic segmentation module obtains the second semantic segmentation result based on the fourth image features.

14. A semantic segmentation model training device, characterized in that, include: The first input module is used to input the first training sample images included in the first training subset to the initial semantic segmentation model. The first training subset has a corresponding semantic segmentation task, and the first training subset includes at least one first training sample image, which includes at least one first category label. The initial semantic segmentation model includes a first initial semantic segmentation module and a second initial semantic segmentation module. The second initial semantic segmentation module includes multiple initial task independent modules, each of which corresponds to a different semantic segmentation task. The multiple initial task independent modules include the first initial task independent module, which has a corresponding semantic segmentation task. The semantic segmentation task corresponding to the first training subset is the same as the semantic segmentation task corresponding to the first initial task independent module. The initial semantic segmentation model module is used to perform a first feature processing on the first training sample image through the first initial semantic segmentation module in the initial semantic segmentation model included in the initial semantic segmentation model to obtain a first image feature, and the first initial task independent module in the second initial semantic segmentation module obtains a first semantic segmentation result based on the first image feature; The training module is used to train the initial semantic segmentation model based on the first semantic segmentation result to obtain the target semantic segmentation model.

15. A semantic segmentation device, characterized in that, include: The second input module is used to input the images to be classified from the dataset to be classified into the target semantic segmentation model. The dataset to be classified includes at least one image to be classified. The target semantic segmentation model includes a first target semantic segmentation module and a second target semantic segmentation module. The second target semantic segmentation module includes multiple initial task-independent modules, each of which corresponds to a different semantic segmentation task. The multiple initial task-independent modules include a first target task-independent module, which has a corresponding semantic segmentation task. The target semantic segmentation model is obtained by the semantic segmentation model training method according to any one of claims 1-12. The target semantic segmentation model module is used to perform fourth feature processing on the image to be classified through the first target semantic segmentation module in the target semantic segmentation model included in the target semantic segmentation model to obtain fourth image features, and the first target task independent module in the second target semantic segmentation module obtains a second semantic segmentation result based on the fourth image features.

16. A computing device cluster, characterized in that, The system includes at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the semantic segmentation model training method according to any one of claims 1-12, or to cause the cluster of computing devices to perform the semantic segmentation method according to claim 13.

17. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster performs the semantic segmentation model training method according to any one of claims 1-12, or performs the semantic segmentation method according to claim 13.

18. A computer-readable storage medium, characterized in that, The computer program instructions, when executed by a computing device cluster, enable the computing device cluster to perform the semantic segmentation model training method as described in any one of claims 1-12, or the computing device cluster to perform the semantic segmentation method as described in claim 13.

Images