Semantic segmentation method and system for learning image structure difficulty information, and storage medium

Abstract

The application provides a semantic segmentation method and system for learning image structure difficulty information and a storage medium. The method comprises the following steps: constructing a semantic segmentation model; training the model by the following steps: inputting an image into a backbone network; inputting the backbone network features into a segmentation head to output a segmentation result and obtain an initial loss map; inputting the backbone network features with truncated backpropagation gradients into a difficulty head to output a difficulty map; obtaining a final loss supervision function of the segmentation head and a final loss supervision function of the difficulty head according to the obtained initial loss map and difficulty map; inputting the image into the backbone network and the segmentation head of the trained semantic segmentation model for testing to obtain a semantic segmentation result of the image that passes the test. The application can output a more reasonable difficulty map by adding the difficulty head, predicting a pixel-level structure difficulty map for each input image, and comprehensively reflecting the segmentation difficulty of the image by using the historical loss supervision function information of each pixel.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of difficult sample mining in image semantic segmentation, in particular to a semantic segmentation method and system for learning image structure difficulty information and a storage medium. BACKGROUND

[0002] Semantic segmentation aims to assign a semantic label to each pixel in an image, and is widely used in automatic driving, remote sensing monitoring and other tasks.

[0003] The current mainstream approach is to regard segmentation as a dense pixel classification task. There are many works based on this mainstream approach. They mainly focus on designing various segmentation heads. The commonly used loss supervision function is the cross-entropy loss supervision function, which specifically calculates the loss of each pixel point and then averages to obtain the overall loss supervision function of the input image. This approach ignores the fact that different pixels in the picture have different segmentation difficulties. Existing difficult sample mining methods mainly focus on the target detection task. The most representative ones are Focal loss and OHEM, but these two methods do not perform well in the semantic segmentation task. There are also difficult sample mining methods designed specifically for medical image segmentation, but they are limited to medical images and cannot be applied to natural images. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings of the prior art and provide a semantic segmentation method and system for learning image structure difficulty information and a storage medium.

[0005] In a first aspect, the present application provides a semantic segmentation method for learning image structure difficulty information, comprising the following steps:

[0006] A semantic segmentation model is constructed, which includes a backbone network, a segmentation head and a difficulty head. The segmentation head and the difficulty head are connected after the backbone network, and the segmentation head and the difficulty head have the same network structure.

[0007] The constructed semantic segmentation model is trained by the following steps:

[0008] An input image is input to the backbone network to obtain the features output by the backbone network;

[0009] The backbone network features are input to the segmentation head to output segmentation results, and the backbone network features with truncated backpropagation gradients are input to the difficulty head to output a pixel-level structure difficulty map;

[0010] According to the obtained initial loss map and difficulty map, the final loss supervision function of the segmentation head of the semantic segmentation model and the final loss supervision function of the difficulty head are obtained.

[0011] The image is input into the backbone network and the segmentation head of the trained semantic segmentation model to test, and the semantic segmentation result of the qualified image is obtained.

[0012] According to the first aspect, in a first possible implementation manner of the first aspect, the step of inputting the backbone network feature with the truncated backpropagation gradient into the difficulty head and outputting a pixel-level structure difficulty map specifically includes the following steps:

[0013] The initial difficulty map is output through a Sigmoid function to limit the value to a range of 0 to 1, and the difficulty map after the value is limited is output.

[0014] The difficulty map after the value is limited is added by a constant to obtain a difficulty map after the constant is added.

[0015] Each value in the difficulty map after the constant is added is divided by the sum of the difficulty map to obtain a pixel-level structure difficulty map.

[0016] According to the first aspect, in a second possible implementation manner of the first aspect, the step of obtaining the final loss supervision function of the segmentation head and the final loss supervision function of the difficulty head of the trained semantic segmentation model according to the obtained segmentation result, the initial loss map and the difficulty map specifically includes the following steps:

[0017] According to the label, the loss value of each pixel of the input image is calculated to obtain an initial loss map.

[0018] According to the obtained initial loss map and the difficulty map, the final loss supervision function of the segmentation head is obtained.

[0019] According to the obtained initial loss map and the difficulty map, the final loss supervision function of the difficulty head is obtained.

[0020] According to the second possible implementation manner of the first aspect, in a third possible implementation manner of the first aspect, the step of obtaining the final loss supervision function of the segmentation head according to the obtained initial loss map and the difficulty map specifically includes the following steps:

[0021] The initial loss map is multiplied by the corresponding position of the difficulty map with the truncated gradient, and the sum is calculated as the final loss supervision function of the segmentation head part.

[0022] According to the second possible implementation manner of the first aspect, in a fourth possible implementation manner of the first aspect, the step of obtaining the final loss supervision function of the difficulty head according to the obtained initial loss map and the difficulty map specifically includes the following steps:

[0023] The initial loss map with the truncated gradient is multiplied by the corresponding position of the difficulty head, and the sum is taken as the final loss supervision function of the difficulty head.

[0024] According to a first aspect, in a fifth possible implementation manner of the first aspect, the step of inputting the image into the backbone network and the segmentation head of the trained semantic segmentation model to obtain a semantic segmentation result of the image that passes the test specifically includes the following steps:

[0025] inputting the image into the backbone network of the trained semantic segmentation model to obtain features output by the backbone network;

[0026] processing the features through the segmentation head of the trained semantic segmentation model to obtain a segmentation result of the image;

[0027] testing and verifying the obtained segmentation result of the image;

[0028] obtaining a semantic segmentation result of the image that passes the test.

[0029] According to the first aspect, in a sixth possible implementation manner of the first aspect, the method is suitable for a task with a pixel-level supervision function, including but not limited to depth estimation and crowd counting.

[0030] In a second aspect, the present application provides a semantic segmentation system for learning image structure difficulty information, including:

[0031] a model construction module, configured to construct a semantic segmentation model, the semantic segmentation model including a backbone network, a segmentation head and a difficulty head, the segmentation head and the difficulty head being connected after the backbone network, the segmentation head and the difficulty head having the same network structure;

[0032] a model training module, in communication connection with the model construction module, configured to train the constructed semantic segmentation model through the following steps, the model training module including:

[0033] a feature output unit, configured to input an image into the backbone network to obtain features output by the backbone network;

[0034] a feature processing unit, in communication connection with the feature output unit, configured to input the features into the segmentation head to output a segmentation result, and input the features of the backbone network with truncated backpropagation gradient into the difficulty head to output a pixel-level structure difficulty map;

[0035] a final loss supervision function obtaining unit, in communication connection with the feature processing unit, configured to obtain a final loss supervision function of the segmentation head and a final loss supervision function of the difficulty head of the trained semantic segmentation model according to the obtained segmentation result, an initial loss map and the difficulty map;

[0036] The model testing and segmentation result obtaining module, in communication connection with the model training module, is configured to input an image into the backbone network and the segmentation head of the trained semantic segmentation model for testing, and obtain a semantic segmentation result of the image after testing.

[0037] According to a second aspect, in a first possible implementation manner of the second aspect, the feature processing unit comprises:

[0038] The difficulty map value limiting subunit, in communication connection with the model training module, is configured to output an initial difficulty map through a Sigmoid function, limit the value of the initial difficulty map to a range of 0 to 1, and output a difficulty map after value limiting.

[0039] The difficulty map constant adding subunit, in communication connection with the difficulty head value limiting unit, is configured to add a constant to the output difficulty map after value limiting, and obtain a difficulty map after constant adding.

[0040] The pixel-level difficulty map obtaining subunit, in communication connection with the difficulty map constant adding unit, is configured to divide each value in the obtained difficulty map after constant adding by a sum of the difficulty map, and obtain a pixel-level structure difficulty map.

[0041] According to a third aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement all method steps of the semantic segmentation method for learning image structure difficulty information.

[0042] Compared with the prior art, the present application has the following advantages:

[0043] The semantic segmentation method for learning image structure difficulty information provided by the present application can output a more reasonable difficulty map by adding an additional output part, i.e., a difficulty head, to predict a pixel-level structure difficulty map for each input image, and can more comprehensively reflect the segmentation difficulty of an image by using historical loss supervision function information of each pixel. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is a method flowchart of the semantic segmentation method for learning image structure difficulty information according to an embodiment of the present application;

[0045] Figure 2 is a semantic segmentation model schematic diagram of the semantic segmentation method for learning image structure difficulty information according to an embodiment of the present application;

[0046] Figure 3 is another method flowchart of the semantic segmentation method for learning image structure difficulty information according to an embodiment of the present application;

[0047] Figure 4is a functional module block diagram of a semantic segmentation system learning image structure difficulty information according to an embodiment of the present application. DETAILED DESCRIPTION

[0048] Reference will now be made in detail to the specific embodiments of the application, examples of which are illustrated in the accompanying drawings. While the application will be described in conjunction with the specific embodiments, it will be understood that the application is not limited to the embodiments described. On the contrary, the application is intended to cover alternatives, modifications and equivalents, which are included within the spirit and scope of the application as defined by the appended claims. It should be noted that the steps of the methods described herein can all be implemented by any functional block or functional arrangement, and any functional block or functional arrangement can be implemented as a physical entity or a logical entity, or a combination of both.

[0049] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below in conjunction with the drawings and specific embodiments.

[0050] Note: The examples to be introduced next are only specific examples, and are not intended to limit the embodiments of the present application to the specific steps, values, conditions, data, sequences, etc. Those skilled in the art can use the concept of the present application to construct more embodiments not mentioned in the present specification by reading the present specification.

[0051] The conventional difficult sample mining method only uses the current loss information of each pixel, but the historical loss also contains a lot of information. Generally, a neural network will prefer to fit simple samples first and then fit difficult samples, so the pixels that maintain a high loss for a long time are more difficult than other pixels.

[0052] Therefore, the present application provides a semantic segmentation method for learning image structure difficulty information to solve the technical problem that the existing image semantic segmentation method performs poorly in the task of mining difficult samples.

[0053] In a first aspect, referring to Figure 1 The present application provides a semantic segmentation method for learning image structure difficulty information, comprising the following steps:

[0054] Step S1, a semantic segmentation model as shown in Figure 2 is constructed, the semantic segmentation model comprising a backbone network, a segmentation head and a difficulty head, the segmentation head and the difficulty head being connected after the backbone network, the segmentation head and the difficulty head having the same network structure, Figure 2Inverse to negative means flipping to negative, that is, turning every positive number into its opposite, that is, adding a negative sign, Pixel-wise production means multiplying the values of corresponding positions, that is, multiplying the corresponding positions of two shape equal matrices, multiplying the corresponding positions of two matrices to obtain a result matrix with the same shape as the input matrix, for example: A: [1, 2, 3], B: [1, 2, 3], the result is C: [1, 4, 9], Sum over all pixels means summing the values of all pixels, that is, adding all the values of the matrix to obtain a value, such as sum(A) = 1 + 2 + 3 = 6, Gradient detached means truncated gradient, gradient is the back propagation gradient in the theory of back propagation, truncated gradient means that the gradient will not be transmitted backward from this point. In other words, the stage gradient is only numerically reused, and there is no gradient correlation in the training process.

[0055] Step S2, the constructed semantic segmentation model is trained through the following steps:

[0056] Step S21, inputting an image to a backbone network to obtain features output by the backbone network;

[0057] Step S22, inputting the features to a segmentation head to output a segmentation result, and inputting the features of the backbone network with truncated back propagation gradient to a difficulty head to output a pixel-level structure difficulty map;

[0058] Step S23, obtaining a final loss supervision function of the segmentation head and a final loss supervision function of the difficulty head of the trained semantic segmentation model according to the obtained segmentation result, an initial loss map and the difficulty map, and preventing the output value of the difficulty map from being too large or too small through the final loss supervision function of the segmentation head and the final loss supervision function of the difficulty head;

[0059] Step S3, inputting an image to the backbone network and the segmentation head of the trained semantic segmentation model for testing to obtain a semantic segmentation result of the image that passes the test.

[0060] The semantic segmentation method for learning image structure difficulty information provided in the application can output a more reasonable difficulty map by adding an additional output part, the difficulty head, to predict the pixel-level structure difficulty map of each input image, compared with the traditional method. In addition, the historical loss supervision function information of each pixel can more comprehensively reflect the segmentation difficulty of the image. Only a slight cost is added in the training stage, and no cost is added in the testing stage, and significant and stable performance improvement is achieved on most mainstream semantic segmentation models.

[0061] In an embodiment, the step of inputting the backbone network feature with truncated backpropagation gradient into the difficulty head and outputting a pixel-level structure difficulty map includes the following steps:

[0062] In step S221, the initial difficulty map is output through a Sigmoid function to limit the value to the range of 0 to 1, and the difficulty map after limiting the output value prevents the difficulty map output value from being too large.

[0063] In step S222, the output value-limited difficulty map is added by a constant, for example, 0.1, to prevent too small values in the difficulty map, and the difficulty map after adding the constant is obtained.

[0064] In step S223, each value in the difficulty map after adding the constant is divided by the sum of the difficulty map to obtain a pixel-level structure difficulty map. The subsequent normalization operation achieves the purpose of learning relative difficulty by using the competition mechanism between pixels.

[0065] In an embodiment, the step of obtaining the final loss supervision function of the segmentation head and the final loss supervision function of the difficulty head of the trained semantic segmentation model according to the obtained segmentation result, the initial loss map, and the difficulty map includes the following steps:

[0066] According to the label, the loss value of each pixel of the input image is calculated to obtain an initial loss map.

[0067] According to the obtained initial loss map and the difficulty map, the final loss supervision function of the segmentation head is obtained.

[0068] According to the obtained initial loss map and the difficulty map, the final loss supervision function of the difficulty head is obtained.

[0069] In an embodiment, the step of obtaining the final loss supervision function of the segmentation head according to the obtained initial loss map and the difficulty map includes the following steps:

[0070] The initial loss map is multiplied by the corresponding position of the difficulty map with truncated gradient, and the sum is calculated as the final loss supervision function of the segmentation head part.

[0071] In an embodiment, the step of obtaining the final loss supervision function of the difficulty head according to the obtained initial loss map and the difficulty map includes the following steps:

[0072] The initial loss map with truncated gradient is multiplied by the corresponding position of the difficulty head, and the sum is taken as the final loss supervision function of the difficulty head.

[0073] In an embodiment, the step of inputting the image into the backbone network and the segmentation head of the trained semantic segmentation model for testing and obtaining the semantic segmentation result of the image that passes the test includes the following steps:

[0074] inputting the image into a backbone network of the trained semantic segmentation model to obtain features output by the backbone network;

[0075] processing the features through a segmentation head of the trained semantic segmentation model to obtain a segmentation result of the image;

[0076] testing and verifying the obtained segmentation result of the image;

[0077] obtaining a semantic segmentation result of the image after testing. The testing phase only needs to use the backbone network and the segmentation head, and therefore does not need to increase additional overhead

[0078] The semantic segmentation method for learning image structure difficulty information provided in the present application is verified on a large number of combinations of backbone networks and segmentation heads, and a large number of experiments are performed on four different domain data sets (street view images, natural images, remote sensing images, and text images). Significant and stable improvements are achieved. Specifically, in 25 groups of street view image experiments, the average improvement is 1.37 points. In 10 groups of natural image experiments, the average improvement is 0.66 points. In 4 groups of remote sensing image experiments, the average improvement is 1.20 points, and in 4 groups of text image experiments, the average improvement is 1.33 points.

[0079] In an embodiment, it is suitable for tasks with pixel-level supervision functions, including but not limited to depth estimation and crowd counting.

[0080] In a second aspect, based on the same inventive concept, please refer to Figure 4The application provides a semantic segmentation system for learning image structure difficulty information, comprising a model construction module 100, a model training module 200 and a model testing and segmentation result acquisition module 300, the model construction module 100 is used for constructing a semantic segmentation model, the semantic segmentation model comprises a backbone network, a segmentation head and a difficulty head, the segmentation head and the difficulty head are connected after the backbone network, and the segmentation head and the difficulty head have the same network structure; the model training module 200 is in communication connection with the model construction module and is used for training the constructed semantic segmentation model through the following steps, the model training module comprises a feature output unit 210, a feature processing unit 220 and a final loss supervision function acquisition unit 230, the feature output unit 210 is used for inputting an image to the backbone network to obtain features output by the backbone network; the feature processing unit 220 is in communication connection with the feature output unit and is used for inputting the features to the segmentation head to output a segmentation result, inputting the features of the backbone network with truncated backpropagation gradient to the difficulty head to output a pixel-level structure difficulty map; the final loss supervision function acquisition unit 230 is in communication connection with the feature processing unit and is used for acquiring a final loss supervision function of the segmentation head and a final loss supervision function of the difficulty head of the trained semantic segmentation model according to the acquired segmentation result, an initial loss map and the difficulty map; the model testing and segmentation result acquisition module 300 is in communication connection with the model training module and is used for inputting an image to the backbone network and the segmentation head of the trained semantic segmentation model to perform testing and acquire a semantic segmentation result of the image that passes the testing.

[0081] In an embodiment, the feature processing unit comprises:

[0082] A difficulty map value limiting subunit is in communication connection with the model training module and is used for outputting the initial difficulty map through a Sigmoid function to limit the value to a range of 0 to 1, outputting the difficulty map after value limiting to prevent excessively large values therein;

[0083] A difficulty map constant adding subunit is in communication connection with the difficulty head value limiting unit and is used for adding a constant to the output difficulty map after value limiting to acquire the difficulty map after adding the constant (for example, 0.1) to prevent excessively small values therein;

[0084] A pixel-level difficulty map acquisition subunit is in communication connection with the difficulty map constant adding unit and is used for dividing each value in the acquired difficulty map after adding the constant by the sum of the difficulty map to acquire a pixel-level structure difficulty map.

[0085] Based on the same inventive concept, the embodiments of the application also provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement all method steps or part of the method steps of the above method.

[0086] The present application implements all or part of the above method, and can also be completed by a computer program instructing related hardware. The computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of the above method embodiments can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable medium can include any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content of the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0087] Based on the same inventive concept, the embodiments of the present application also provide an electronic device, including a memory and a processor, the memory stores a computer program running on the processor, and the processor implements all or part of the method steps when executing the computer program.

[0088] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The processor is the control center of the computer device, and connects all parts of the computer device through various interfaces and lines.

[0089] The memory can be used to store computer programs and / or modules, and the processor realizes various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application program (such as a sound playing function, an image playing function, etc.) required by a function; and the data storage area can store data (such as audio data, video data, etc.) created according to the use of the mobile phone. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0090] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, a server or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer usable program codes.

[0091] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems), servers and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

[0092] These computer program instructions can also be stored in a computer readable memory capable of guiding a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer readable memory produce a product including instruction devices, which realize the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

[0093] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable devices provide processes for implementing the functions specified in the flowchart Figure 1 one or more flows and / or blocks Figure 1 one or more blocks or steps of the functions specified in the flowchart

[0094] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A semantic segmentation method for learning image structural difficulty information, characterized in that, Includes the following steps: A semantic segmentation model is constructed, which includes a backbone network, a segmentation head, and a difficulty head. The segmentation head and the difficulty head are both connected after the backbone network and have the same network structure. The constructed semantic segmentation model is trained through the following steps: Input the image into the backbone network and obtain the features output by the backbone network; Input the backbone network features into the segmentation head, output the segmentation result, and obtain the initial loss map; input the backbone network features with truncated backpropagation gradients into the difficulty head, and output the pixel-level structural difficulty map. Based on the obtained initial loss map and difficulty map, obtain the final loss supervision function of the segmentation head and the final loss supervision function of the difficulty head of the semantic segmentation model; The image is input into the backbone network and segmentation head of the trained semantic segmentation model for testing, and the semantic segmentation result of the image after passing the test is obtained. The step of inputting the backbone network features with truncated backpropagation gradients into the difficulty head and outputting a pixel-level structural difficulty map specifically includes the following steps: The initial difficulty map is passed through a Sigmoid function, which restricts its value to a range of 0 to 1, and the resulting difficulty map is output after the value restriction. Add a constant to the difficulty graph after limiting the output values ​​to obtain the difficulty graph with the constant added. Divide each value in the obtained difficulty map after adding a constant by the sum of the difficulty maps to obtain a pixel-level structural difficulty map.

2. The semantic segmentation method for learning image structural difficulty information as described in claim 1, characterized in that, The step of obtaining the final loss supervision function of the segmentation head and the final loss supervision function of the difficulty head of the semantic segmentation model based on the obtained initial loss map and difficulty map specifically includes the following steps: The loss value of each pixel in the input image is calculated based on the label to obtain the initial loss map; Based on the obtained initial loss map and difficulty map, obtain the final loss supervision function of the segmentation head; Based on the obtained initial loss map and difficulty map, obtain the final loss supervision function for the difficulty head.

3. The semantic segmentation method for learning image structural difficulty information as described in claim 2, characterized in that, The step of obtaining the final loss supervision function of the segmentation head based on the obtained initial loss map and difficulty map specifically includes the following steps: Multiply the initial loss map with the corresponding positions of the difficulty map with the truncated gradient, and calculate the sum as the final loss supervision function for the segmentation head part.

4. The semantic segmentation method for learning image structural difficulty information as described in claim 2, characterized in that, The step of obtaining the final loss supervision function of the difficulty head based on the obtained initial loss map and difficulty map specifically includes the following steps: Multiply the initial loss map with the truncated gradient by the corresponding position in the difficulty head, and take the negative 0.01 times the sum to get the final loss supervision function of the difficulty head.

5. The semantic segmentation method for learning image structural difficulty information as described in claim 1, characterized in that, The step of inputting the image into the backbone network and segmentation head of the trained semantic segmentation model for testing, and obtaining the semantic segmentation result of the image after passing the test, specifically includes the following steps: Input the image into the backbone network of the trained semantic segmentation model and obtain the features output by the backbone network. The segmentation result of the image is obtained by processing the segmentation head of the semantic segmentation model after the features are trained. The segmentation results of the acquired images were tested and verified. Obtain the semantic segmentation results of the image after the test is passed.

6. The semantic segmentation method for learning image structural difficulty information as described in claim 1, characterized in that, It is suitable for tasks with pixel-level supervision functions, including but not limited to depth estimation and crowd counting.

7. A semantic segmentation system for learning image structural difficulty information, characterized in that, include: The model building module is used to build a semantic segmentation model, which includes a backbone network, a segmentation head, and a difficulty head. The segmentation head and the difficulty head are both connected after the backbone network and have the same network structure. A model training module, communicatively connected to the model building module, is used to train the constructed semantic segmentation model through the following steps, wherein the model training module includes: The feature output unit is used to input an image into the backbone network and obtain the features output by the backbone network. The feature processing unit is communicatively connected to the feature output unit and is used to input features into the segmentation head and output segmentation results, and input backbone network features with truncated backpropagation gradients into the difficulty head and output pixel-level structural difficulty map. The final loss supervision function acquisition unit is communicatively connected to the feature processing unit and is used to acquire the final loss supervision function of the segmentation head and the final loss supervision function of the difficulty head of the trained semantic segmentation model based on the acquired segmentation results, the initial loss map, and the difficulty map. The model testing and segmentation result acquisition module is communicatively connected to the model training module. It is used to input images into the backbone network and segmentation head of the trained semantic segmentation model for testing and to obtain the semantic segmentation results of the images that have passed the test. The feature processing unit includes: The difficulty map value limiting subunit is communicatively connected to the model training module. It is used to pass the initial difficulty map output through a Sigmoid function to limit its value to the range of 0 to 1, and output the value-limited difficulty map. The difficulty map constant addition subunit is communicatively connected to the difficulty map value restriction unit, and is used to add a constant to the output difficulty map after value restriction to obtain the difficulty map with the constant added. The pixel-level difficulty map acquisition subunit is communicatively connected to the difficulty map constant addition unit, and is used to divide each value in the acquired difficulty map after constant addition by the sum of the difficulty maps to obtain a pixel-level structural difficulty map.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements all the method steps of the semantic segmentation method for learning image structural difficulty information as described in any one of claims 1 to 6.

Images