A multi-level semantic segmentation method, device, electronic device and storage medium

Abstract

The present disclosure relates to a multi-level semantic segmentation method, device, electronic device and storage medium, the method comprising: obtaining a set of images to be processed; obtaining multi-category labels of each image to be processed in the set of images to be processed; extracting feature maps of each image to be processed, inputting the feature maps into the classification head of the image segmentation network, and obtaining label information corresponding to each part of the image to be processed; inputting the multi-category labels and label information into the multi-level structured loss function to obtain a first loss value; and inputting the multi-category labels and label information into the triple loss function to obtain a second loss value; updating the classification head based on the first loss value and the second loss value to obtain a trained preset model, and using the trained preset model as a deep hierarchical real-time semantic segmentation model; inputting the image to be segmented into the deep hierarchical real-time semantic segmentation model to obtain the segmentation result of the image to be segmented. The technical problem of low overall accuracy of image segmentation results is solved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a multi-level semantic segmentation method, device, electronic device and storage medium. Background Art

[0002] Semantic segmentation is a technique for classifying images at the pixel level. Each pixel in the image is assigned to a corresponding category, and different categories are usually marked with different colors. Semantic segmentation technology has been widely used in parts inspection, medical image analysis, geographic information systems and other fields.

[0003] The current semantic segmentation algorithm has achieved a lot of significant research results, but there are still many problems that need to be solved. On the one hand, the overall accuracy of the image segmentation result is not high, and the original image information is lost. On the other hand, the segmentation efficiency is low, and the training and reasoning speed of the model is far from reaching the real-time level. Summary of the invention

[0004] The present invention provides a multi-level semantic segmentation method, device, electronic device and storage medium.

[0005] According to a first aspect of the present disclosure, a multi-level semantic segmentation method is provided, the method comprising:

[0006] Get the image set to be processed;

[0007] Acquire multiple category labels of each image to be processed in the image set to be processed; wherein each image to be processed includes multiple objects, and each object corresponds to a category label;

[0008] Extracting feature maps of each image to be processed, inputting the feature maps into a classification head of an image segmentation network, and obtaining label information corresponding to each part of the image to be processed;

[0009] Inputting the multi-category labels and the label information into a multi-level structured loss function to obtain a first loss value; and inputting the multi-category labels into a triple loss function to obtain a second loss value;

[0010] Update the classification head based on the first loss value and the second loss value to obtain a trained preset model, and use the trained preset model as a deep hierarchical real-time semantic segmentation model;

[0011] The image to be segmented is input into the deep hierarchical real-time semantic segmentation model to obtain a segmentation result of the image to be segmented.

[0012] According to a second aspect of the present disclosure, a multi-level semantic segmentation device is provided, characterized in that the device comprises:

[0013] A data acquisition module, used for acquiring a set of images to be processed;

[0014] The data acquisition module is further used to acquire multiple category labels of each image to be processed in the image set to be processed; wherein each image to be processed includes multiple objects, and each object corresponds to a category label;

[0015] A processing module, used for extracting feature maps of each image to be processed, inputting the feature maps into a classification head of an image segmentation network, and obtaining label information corresponding to each part of the image to be processed;

[0016] The processing module is further used to input the multi-category labels and the label information into a multi-level structured loss function to obtain a first loss value; and input the multi-category labels and the label information into a triple loss function to obtain a second loss value;

[0017] The processing module is further used to update the classification head based on the first loss value and the second loss value to obtain a trained preset model, and use the trained preset model as a deep hierarchical real-time semantic segmentation model;

[0018] The processing module is also used to input the image to be segmented into the deep hierarchical real-time semantic segmentation model to obtain a segmentation result of the image to be segmented.

[0019] According to a third aspect of the present disclosure, an electronic device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.

[0020] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above method of the present disclosure is implemented.

[0021] The multi-level semantic segmentation method, device, electronic device and storage medium provided by the embodiments of the present disclosure obtain multi-category labels of each image to be processed in the image set by obtaining a set of images to be processed. The feature map of each image to be processed is extracted, and the feature map is input into the classification head of the image segmentation network, so as to obtain the label information corresponding to each part of the image to be processed. Then, by calculating two loss values, back-propagating and updating the classification head, the deep hierarchical real-time semantic segmentation model is trained. Finally, the image to be segmented is input into the trained deep hierarchical real-time semantic segmentation model to obtain the segmentation result of the image to be segmented. The multi-level semantic segmentation method provided by the embodiments of the present disclosure greatly increases the overall segmentation accuracy and robustness of the model, achieving the technical effect of reducing the computational cost of semantic segmentation, improving computational efficiency, and improving segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0023] Figure 1 A schematic diagram showing an example system in which various methods described herein may be implemented according to an exemplary embodiment of the present disclosure;

[0024] Figure 2 A schematic diagram of a multi-level semantic segmentation method flow chart of an exemplary embodiment of the present disclosure is shown;

[0025] Figure 3 The segmentation result diagrams of the bearing images generated by using the existing model and the multi-level semantic segmentation method provided by the present disclosure are shown;

[0026] Figure 4 A schematic block diagram of functional modules of a multi-level semantic segmentation device provided by an exemplary embodiment of the present disclosure;

[0027] Figure 5 A schematic block diagram of a chip according to an exemplary embodiment of the present disclosure is shown;

[0028] Figure 6 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0030] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0031] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0032] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0033] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0034] Before introducing the embodiments of the present disclosure, the following definitions are given for the terms involved in the embodiments of the present disclosure:

[0035] MaskFormer

[16] ,MaskFormer

[16] is a Transformer-based computer vision model, mainly used for image segmentation tasks. It was proposed by the Facebook AI Research team in 2021 to solve the problems of current image segmentation models in processing images with occlusion and complex backgrounds.

[0036] DeepLabV3+

[10] ,DeepLabV3+

[10] is a semantic segmentation model based on deep learning proposed by Google. It is widely used in image segmentation, portrait segmentation, natural scene segmentation and other fields, and has achieved state-of-the-art results in many image segmentation competitions.

[0037] Optical Character Recognition Network (OCRNet), OCRNet was proposed by the Institute of Automation, Chinese Academy of Sciences in 2020. Different from traditional OCR methods, OCRNet

[98] uses a combination of convolutional neural networks and recurrent neural networks to achieve end-to-end image text recognition.

[0038] Deep Residual Network (ResNet), ResNet101 is a deep residual network (ResNet) model with 101 layers of depth. It was proposed by Kaiming He et al. in 2015. It is one of the ResNet series models and can be used to quickly build and train computer vision models such as image classification, object detection, and semantic segmentation.

[0039] High-Resolution Network (HRNet), HRNet is a multi-resolution feature fusion network structure proposed by Huazhong University of Science and Technology. HRNet-W48 is a variant of HRNet, where "W" represents the width of the convolution kernel used, and 48 represents the number of channels used in the network. It is used for computer vision tasks such as image classification, target detection, and key point detection. Compared with HRNet-W32 and HRNet-W64, HRNet-W48 has achieved a balance in network width and depth, and has better computational efficiency and accuracy performance.

[0040] It refers to mean Intersection over Union (mIoU), which is a commonly used indicator for evaluating the performance of image segmentation models. In image segmentation tasks, it is usually necessary to classify each pixel in the image into different objects or backgrounds. mIoU is used to measure the similarity between the segmentation results predicted by the model and the actual segmentation results.

[0041] The sigmoid activation function is often used as an activation function of a neural network to map variables between 0 and 1.

[0042] Semantic segmentation is a technique for classifying images at the pixel level. Each pixel in the image is assigned to a corresponding category, and different categories are usually marked with different colors. Semantic segmentation technology has been widely used in parts inspection, medical image analysis, geographic information systems and other fields.

[0043] The current semantic segmentation algorithm has achieved a lot of significant research results, but there are still many problems that need to be solved. On the one hand, the overall accuracy of the image segmentation result is not high, and the original image information is lost. On the other hand, the segmentation efficiency is low, and the training and reasoning speed of the model is far from reaching the real-time level.

[0044] Therefore, in order to solve the above problems, the embodiments of the present disclosure firstly provide a multi-level semantic segmentation method.

[0045] Figure 1 Schematic diagram of an example system in which various methods described herein may be implemented according to an exemplary embodiment of the present disclosure. Figure 1 As shown, the system 100 of the exemplary embodiment of the present disclosure may include: a user device 110 , a computing device 120 , and a data storage system 130 .

[0046] like Figure 1 As shown, the user equipment 110 can communicate with the computing device 120 through a communication network. The communication network can be a wired communication network or a wireless communication network. The wired communication network can be a communication network based on power line carrier technology, and the wireless communication network can be a local wireless network or a wide area wireless network. The local wireless network can be a WIFI wireless network, a Zigbee wireless network, a mobile communication network, a satellite communication network, etc.

[0047] like Figure 1 As shown, the user device 110 may include a computer, a mobile phone, or an intelligent terminal such as an information processing center. The user device 110 may serve as the initiator of the model training operation or the feature map extraction operation and initiate a request to the computing device 120. The computing device 120 may be a server with data processing functions such as a cloud server, a network server, an application server, and a management server, for implementing the training method and the generation method. A deep learning processor may be configured in the server, and the deep learning processor may be a neuron of a single-core deep learning processor (Deep Learning Processor-Singlecore, abbreviated as DLP-S) or a multi-core deep learning processor (Deep Learning Processor-Multicore, abbreviated as DLP-M). DLP-M is a multi-core extension based on DLP-S, and multiple DLP-S are interconnected, multicast, and inter-core synchronization protocols are used to communicate between cores through the on-chip network (Network-on-chip, abbreviated as Noc) to complete deep learning tasks and expression image generation tasks.

[0048] like Figure 1As shown, the above-mentioned data storage system 130 can be a general term, including local storage and databases storing historical data. The database can be on the computing device 120, on other network servers, or on the data storage system 130. The data storage system 130 can be separate from the computing device 120 or integrated in the computing device 120. The data storage system 130 can not only input data uploaded by the user device, but also store program instructions, neuron data, weight data, etc. These neuron data and weight data can be trained data or untrained data, which is determined by the method implemented by the computing device 120. In addition, the data storage system 130 can also store the processing results (such as feature maps, multi-category labels, distance margins, segmented images, intermediate processing results or image quality) obtained by the computing device 120 into the data storage system.

[0049] In practical applications, such as Figure 1 As shown, the user device 110 may have a data set acquisition function, so that the user device can not only initiate a request to the execution device through the interactive interface, but also collect or build a training image data set, and send the image data set to the computing device 120 through the communication network. Based on this, when the computing device 120 implements the training method, the sample images used can not only be obtained from the data storage system 130, but also from the user device 110 through the communication network. In addition, when the computing device 120 implements the generation method, the multi-category labels generated by it can not only be fed back to the user device 110 through the communication network, but also be stored in the data storage system 130 as sample data for further training the model.

[0050] For example, Figure 1 As shown, the user device 110 may have a display interface, and image processing software such as Photoshop, MATLAB, Open CV, and Metahuman Creator (Unreal Engine Face Pincher) may be installed locally on the user device. The user device 110 may respond to user operations, use these image processing software to pre-process the image data set offline, and then upload it to the computing device 120. Photoshop, MATLAB, Open CV, Metahuman Creator (Unreal Engine Face Pincher) and other image processing software may also be installed in the computing device 120. The user may respond to user operations, remotely log in to the execution device, and use these image processing software to process the image data set.

[0051] Figure 2 A schematic diagram of a multi-level semantic segmentation method flow chart of an exemplary embodiment of the present disclosure is shown. Figure 2 As shown, the method may include the following steps:

[0052] Step S210: Obtain a set of images to be processed. The set of images to be processed here refers to any picture data that needs to be semantically segmented, including but not limited to public image data sets and special part photo data.

[0053] In an optional manner, the image dataset can be used as a training dataset and a test dataset for the multi-level semantic segmentation method of the exemplary embodiment of the present disclosure, the training dataset is used to train the deep hierarchical real-time semantic segmentation model, and the test dataset is used to cross-validate the model and complete hyperparameter tuning. In practical applications, the training dataset and the test dataset maintain the same preprocessing logic.

[0054] Exemplarily, the preprocessing steps of the image dataset include but are not limited to:

[0055] According to the actual situation, the image dataset is selectively subjected to random horizontal and vertical scale cropping, random image flipping, or random image brightness change operations.

[0056] The processed image dataset images are uniformly output to a fixed size.

[0057] The processed image dataset is then randomly divided into a training dataset and a test dataset, which are used for training and testing of the deep hierarchical real-time semantic segmentation model, respectively.

[0058] The embodiment of the present application obtains a feature map by adopting a spatial information extraction path based on deep separable convolution. In the process of obtaining the feature map, there is no need to significantly crop the original image, thereby retaining a large amount of spatial information, and can greatly reduce the amount of weight parameters involved in the calculation during training and inference, thereby reducing the computational overhead of the model.

[0059] Step S220: Obtain multiple category labels of each image to be processed in the image set to be processed. Each image to be processed here includes multiple objects, and each object corresponds to a category label.

[0060] Exemplarily, when the image to be processed is a bearing image, the objects may be the outer ring, inner ring and ball of the bearing. When the object is the outer ring of the bearing, the category label may be the outer ring; when the object is the inner ring of the bearing, the category label may be the inner ring; when the object is the ball of the bearing, the category label may be the ball.

[0061] The object contains sub-objects. For example, when the object is the outer ring of the bearing, the sub-objects are the outer ring metal part and the outer ring edge part; when the object is the inner ring of the bearing, the sub-objects are the inner ring metal part and the inner ring edge part.

[0062] The category labels of each image to be processed are extracted, the category labels are converted into a multi-level structure, and the multi-category labels are obtained. The multi-level structure includes the relationship between the category labels of the objects and the relationship between the category labels of the objects and their sub-objects.

[0063] In an optional manner, the multi-level structure may be a tree structure. Exemplarily, a manually defined method may be used to transform and organize the category labels of the target semantic segmentation into a tree structure in a manner from large to small concept scope, wherein the parent node is a label with a large coverage scope and the child node is a label with a small coverage scope. A parent node may correspond to multiple child nodes, and a child node may correspond to only one of the parent nodes.

[0064] The parent node or the child node corresponding to the parent node is used as a multi-category label. Exemplarily, according to the category label of the pixel in the image, the corresponding multi-category label is obtained through the hierarchical structure of the tree structure. In the tree structure, when the child node is the category of the image to be processed, the parent node corresponding to the child node can be used as the label of the image to be processed. When the parent node is not the category of the image to be processed, the child node corresponding to the parent node cannot be used as the label of the image to be processed. According to this rule, the nodes that can be used as labels are marked as 1, and the nodes that cannot be used as labels are marked as 0, and the multi-category labels are obtained.

[0065] In an optional way, the process of obtaining multi-category labels is as follows:

[0066] For an input image, the original category label of each pixel is searched in the tree structure.

[0067] If the node corresponding to the label of the pixel is a leaf node (ie, represents a specific category), the node and all its ancestor nodes (ie, the parent node and the parent node of the parent node, etc.) are marked as 1.

[0068] If the node corresponding to the label of the pixel is a non-leaf node (i.e., a parent node representing a category), the node and all its descendant nodes (i.e., child nodes, child nodes of child nodes, etc.) are marked as 0.

[0069] The final label is a matrix of the same size as the input image pixel, where each element is 0 or 1, indicating whether the pixel is a label of a category.

[0070] Step S230: extracting feature maps of each image to be processed, inputting the feature maps into the classification head of the image segmentation network, and obtaining label information corresponding to each part of the image to be processed.

[0071] In an optional manner, a feature map of the image to be segmented can be extracted by a depthwise separable convolutional network. The depthwise separable convolutional network is composed of multiple depthwise separable convolutional layers. This step can be expressed by the following formula (1):

[0072]

[0073] Among them, N dw : Depth-wise separable convolutional network; i: the i-th depth-wise separable convolutional layer, there are I in total; represents a normal convolutional layer; n in 、n out , g: represent the number of input channels, output channels and channel groupings respectively; in the separable convolutional layer, Represents the number of groups; ∪: union symbol.

[0074] Input the image to be segmented x into the deep separable convolutional network and output the extracted feature map F. This step can be expressed by the following formula (2):

[0075] F=N dw (x) (2)

[0076] Among them, N dw : Depth-wise separable convolutional network; x: represents the image to be segmented; F: represents the feature map.

[0077] The present disclosure adopts a deep separable convolution method to extract feature maps containing spatial information. Compared with traditional convolution, the deep separable convolution method can avoid the problem of weakened model learning ability due to a substantial decrease in the amount of weight parameters. By combining the front-end convolution block and the back-end convolution block, it can be ensured that the model learning ability is not affected while the amount of parameters is greatly reduced. On the one hand, the memory occupied by the model will be reduced, and on the other hand, it will greatly empower the reasoning speed, and can effectively reduce the weight parameters of the above-mentioned weight parameter reduction amount, which can greatly reduce the amount of weight parameters involved in the calculation during training and reasoning. For lightweight models, there can also be up to twenty times the parameter reduction. This makes the computational overhead of the model smaller.

[0078] The feature map is input into the classification head of the image segmentation network to obtain the label information corresponding to each part of the image to be processed.

[0079] In an optional manner, the image segmentation network here can be a hierarchical semantic segmentation network, and the label information here is a multi-label classification score.

[0080] Exemplarily, in order to calculate the multi-level classification score of the image to be segmented, the single-label classification of each point in the image to be segmented can be converted into a multi-label classification problem, and the extracted feature map is input into the classification head of the hierarchical semantic segmentation network. The multi-label classification score is calculated for each point, where each label represents a node in the tree structure, thereby extracting the multi-level classification score. This step can be expressed by the following formula (3):

[0081] M=σ(N h (F)) (3)

[0082] Among them, N h : classification head of the hierarchical semantic segmentation network; σ: sigmoid activation function; M: output multi-label binary classification score map.

[0083] Step S240: input the multi-category labels and label information into the multi-level structured loss function to obtain a first loss value; and input the multi-category labels into the triple loss function to obtain a second loss value.

[0084] For example, the relationship between nodes in the tree structure is used to add classification score restrictions. Since the directly obtained multi-classification binary prediction scores have no relationship restrictions between nodes and cannot use tree structure information, the scores are updated based on the relationship between parent and child nodes on the basis of the multi-label binary classification score graph. This step can be expressed by the following formula (4):

[0085]

[0086] Among them, G t : represents the true value of the t-th class in the multi-class label, M a : represents the score of the predecessor node position of the tth class in the multi-label binary classification score graph, M c : represents the score of the descendant node position of the t-th class in the multi-label binary classification score graph, a: represents the ancestor node position of the t-th class, c: represents the descendant node position of the t-th class, P t : represents the score M of the original t node based on the relationship between the parent and child nodes of the tree structure t The score obtained after correction.

[0087] By adding classification score restrictions, it is ensured that if a node in the score tree is a positive class (G t =1), its parent node score is not lower than this node. If it is a negative class (G t =0), its child node score is not higher than that of the node.

[0088] Calculate the multi-level structured loss function. This step can be expressed by the following formula (5):

[0089]

[0090] Among them, M i，j，t ∈(0,1): the classification score of the t-th category of the pixel at position (i, j) in the multi-label binary classification score map; G i，j，t ∈{0,1}: label G corresponding to pixel (i,j) i，j The true value of the t-th class in; w, h: width and height of the image; |T|: number of nodes in the tree structure, P i，j，t ∈{0, 1}: represents the prediction score of the pixel at position (i, j) in space for the tth class, M i，j，a : represents the score of the predecessor node of the pixel point at position (i, j) in space, M i，j，c : Represents the score of the descendant node of the pixel point at position (i, j) in the space.

[0091] Exemplarily, since the number of categories may be unbalanced, an adjustment factor is added on the basis of the multi-level structured loss function L2 to reduce the relative loss of easy-to-classify pixels and increase the relative loss of difficult samples, and the first loss value L3 is calculated. This step can be expressed by the following formula (6):

[0092]

[0093] Where, λ: represents the adjustment factor, λ≥0; G i，j，t ∈{0,1}: label G corresponding to pixel (i,j) i，j The true value of the t-th class in; w, h: width and height of the image; |T|: number of nodes in the tree structure, P i，j，t ∈{0, 1}: represents the prediction score of the pixel at position (i, j) in space for the tth class, M i，j，a : represents the score of the predecessor node of the pixel point at position (i, j) in space, M i，j，c : Represents the score of the descendant node of the pixel point at position (i, j) in the space.

[0094] Exemplarily, the multi-category labels are input into the triplet loss function to obtain the second loss value L4, which further expands the representation distance between the positive and negative classes by enhancing the discriminability between the representations of the positive and negative classes, thereby optimizing the classification head N of the hierarchical semantic segmentation network. h , the loss calculation can be expressed by the following formula (7):

[0095] L4(F,F + , F - )=max{d(F,F + )-d(F,F - )+d * ,0} (7)

[0096] Among them, d(,): represents the distance between two representations, such as cosine distance, F, F + 、F - : represent node features, positive features and negative features respectively, d * : Indicates the distance margin value.

[0097] In an optional manner, the distance margin may be dynamically determined according to the structural hierarchy. This step may be expressed by the following formula group (8):

[0098] d * =d0+md r

[0099]

[0100] Among them, d0: fixed margin value; o(,): the position between two nodes in the tree structure; H: the depth of the tree structure; m: coefficient.

[0101] Representation discriminability is a function similar to feature screening, which can purposefully learn the features that are most helpful for segmentation accuracy and ignore or not learn those interfering features. The feature importance vector is automatically calculated based on the existing feature layer, which reduces the intervention of manual feature engineering, reduces the bias of the output, and greatly increases the overall segmentation accuracy and robustness of the model.

[0102] Step S250: updating the classification head based on the first loss value and the second loss value to obtain a trained preset model, and using the trained preset model as a deep hierarchical real-time semantic segmentation model;

[0103] In an optional approach, the classification head N of the hierarchical semantic segmentation network can be optimized by optimizing the loss values ​​L3 and L4. h Update to obtain the trained deep hierarchical real-time semantic segmentation model.

[0104] Step S260: input the image to be segmented into the deep hierarchical real-time semantic segmentation model to obtain the segmentation result of the image to be segmented.

[0105] The present disclosure proposes a multi-level semantic segmentation method, including: obtaining a set of images to be processed, organizing the category labels of the set of images to be processed into a tree structure, establishing the relationship between the category labels, and obtaining multi-category labels of the images to be processed. Extracting the feature map of the image to be processed, and inputting the feature map into the classification head of the semantic segmentation network, so as to obtain the label information corresponding to each part of the image to be processed. Then, by calculating two loss values, back-propagating and updating the classification head, a deep hierarchical real-time semantic segmentation model is trained.

[0106] Among them, the first feature map is obtained by processing the image to be segmented through the spatial information extraction path, and the pixel-by-pixel hierarchical segmentation learning strategy is used to supervise the segmentation prediction in the deep hierarchical real-time semantic segmentation model. The pixel-by-pixel hierarchical representation learning strategy is used to generate a hierarchical induced margin, thereby improving the prediction accuracy. At the same time, building a tree structure and multi-category labels can hierarchize different semantic labels, so that the algorithm can better understand and express different semantic information. This hierarchical structure enables the algorithm to better handle complex scenes and improve segmentation accuracy. At the same time, by effectively combining the tree structure with the neural network, the features that are most helpful to the segmentation accuracy can be purposefully learned when building the model, which greatly increases the overall segmentation accuracy and robustness of the model, achieving the technical effect of reducing the computational cost of semantic segmentation, improving computational efficiency, and improving segmentation accuracy.

[0107] In one embodiment provided by the present disclosure, two groups of experiments are set up, and the image data sets processed by the two groups of experiments are the same. In the first group, the image data set to be processed is first constructed with a multi-level semantic structure by a hierarchical semantic segmentation network, and then image segmentation is performed using different existing semantic segmentation models. In the second group, the image data set of the control group is not constructed with a multi-level semantic structure by a hierarchical semantic segmentation network, and image segmentation is performed directly using different existing semantic segmentation models, and the segmentation effects of the two groups are compared.

[0108] First, obtain a set of images to be processed and perform preprocessing, and the method adopts step S210 described in detail above. Obtain multi-category labels for each image to be processed in the set of images to be processed, and the method adopts step S220 described in detail above. Then, extract the feature map of each image to be processed, input the feature map into the classification head of the image segmentation network, and obtain the label information corresponding to each part of the image to be processed, and the method adopts step S230 described in detail above. Then, input the multi-category labels and label information into the multi-level structured loss function to obtain a first loss value; and input the multi-category labels into the triple loss function to obtain a second loss value, and the method adopts step S240 described in detail above. Update the classification head based on the first loss value and the second loss value to obtain a trained preset model, and use the trained preset model as a deep hierarchical real-time semantic segmentation model, and the method adopts step S250 described in detail above. Finally, input the image to be segmented into the deep hierarchical real-time semantic segmentation model to obtain the segmentation result of the image to be segmented, and the method adopts step S260 described in detail above.

[0109] Table 1 shows the results of different semantic segmentation models in processing the same dataset. In this embodiment, the semantic segmentation models used are: MaskFormer

[16] model, DeepLabV3+

[10] model and OCRNet

[98] model.

[0110] Table 1 Results of different semantic segmentation models on the same dataset

[0111]

[0112] As shown in Table 1, the segmentation effects of using a hierarchical semantic segmentation network plus different semantic segmentation models are better than those of using a semantic segmentation model alone. Among them, the image segmentation effect of using a hierarchical semantic segmentation network plus the MaskFormer

[16] model is 1.02% higher than that of using the MaskFormer

[16] model alone, the image segmentation effect of using a hierarchical semantic segmentation network plus the DeepLabV3+

[10] model is 1.03% higher than that of using the DeepLabV3+

[10] model alone, and the image segmentation effect of using a hierarchical semantic segmentation network plus the OCRNet

[98] model is 1.26% higher than that of using the OCRNet

[98] model alone.

[0113] It can be seen from the experimental results of the above embodiments that the combination of different semantic segmentation models and hierarchical semantic segmentation networks can significantly improve the segmentation effect. On the one hand, different semantic segmentation models have different advantages and limitations in processing different tasks and scenarios. By combining them, their respective advantages can be combined to improve the accuracy and robustness of segmentation. On the other hand, the hierarchical semantic segmentation network can effectively fuse and integrate information at different levels in the image, thereby improving the global consistency and detail retention of the segmentation.

[0114] In one embodiment provided in the present disclosure, the test data set is a self-built data set, and the main content of the data set is bearing pictures.

[0115] First, the image data set is obtained and preprocessed. Due to the low diversity of bearing images, salt and pepper noise, Gaussian noise and reflection half-point noise will be generated during the recording, imaging and transmission of the image, which will affect the image generation effect. Therefore, it is necessary to use geometric shape analysis and mathematical analysis methods to preprocess the bearing images.

[0116] First, the detection background of the bearing is estimated by geometric shape analysis, and then the digital analysis method is used for detection, that is, an image is controlled by one element, and the image is processed according to the relationship between the elements that constitute the image to improve the quality of the image. The specific methods include:

[0117] According to the imaging factors of the image, the linear least square method is used to measure its influence on the imaging, that is, according to the minimum mean square error, the fitting change background is determined. Therefore, the fitting calculation formula can be expressed by the following formula (9):

[0118] ω(x)=σ1x+σ2x+σ3x+σ4x (9)

[0119] Among them, σ1, σ2, σ3, σ4: represent the background information influencing factors in each direction respectively; x: represents the factor mutation value.

[0120] The average variance is calculated according to the following formula (10):

[0121]

[0122] Among them, X i : represents the gray value of the i-th column of the formed image; ω(x): represents the degree of fitting.

[0123] The calculation result at this time is the background information factor in the four positive directions. Due to the inaccurate positioning of the fault position, there is also a calculation deviation. Therefore, the information interference in other directions is corrected by information extraction, and the median filter is weighted and added to obtain a more accurate detection background. The corrected background estimation can be expressed by the following formula (11):

[0124] A=a1λ 2 +a2M (11)

[0125] Among them, a1 and a2: represent weighting coefficients, and a1+a2=1, M: represents median filtering, λ 2 : represents the mean square error.

[0126] First, obtain a set of images to be processed and perform preprocessing, and the method adopts step S210 described in detail above. Obtain multi-category labels for each image to be processed in the set of images to be processed, and the method adopts step S220 described in detail above. Then, extract the feature map of each image to be processed, input the feature map into the classification head of the image segmentation network, and obtain the label information corresponding to each part of the image to be processed, and the method adopts step S230 described in detail above. Then, input the multi-category labels and label information into the multi-level structured loss function to obtain a first loss value; and input the multi-category labels into the triple loss function to obtain a second loss value, and the method adopts step S240 described in detail above. Update the classification head based on the first loss value and the second loss value to obtain a trained preset model, and use the trained preset model as a deep hierarchical real-time semantic segmentation model, and the method adopts step S250 described in detail above. Finally, input the image to be segmented into the deep hierarchical real-time semantic segmentation model to obtain the segmentation result of the image to be segmented, and the method adopts step S260 described in detail above.

[0127] When performing semantic segmentation on the cross-sectional image data of the bearing, it is necessary to distinguish the outer ring, inner ring, ball, noise background, etc. of the bearing. The image dataset can be a dataset of cross-sectional images of the bearing collected by a camera or a video camera, in which the main objects include: bearings, balls, debris, shadow interference, defects and other categories. Figure 3 The segmentation result images of bearing images generated using the existing model and the multi-level semantic segmentation method provided by the present disclosure are shown, wherein the existing model refers to the prediction result using semantic labels that do not have any association. Figure 3 The left side shows the segmentation result obtained after using the existing model. Figure 3 The right side is a segmentation result diagram generated using the multi-level semantic segmentation method provided by the present disclosure. Figure 3 As shown, in the left figure, there are extra arcs in the upper left corner of the outer ring, and the edge of the ball is very blurred and the number is unclear. At the same time, there are extra arcs in the inner ring. It can be seen that the overall segmentation accuracy of the segmentation results obtained using the existing model is low, the outer ring and inner ring areas are misjudged, and the edge and number of the ball cannot be identified. In the right figure, the segmentation result image generated by the multi-level semantic segmentation method provided by the present disclosure accurately shows the outer ring, inner ring and ball area of ​​the bearing. It can be seen that the use of the multi-level semantic segmentation method can improve the segmentation accuracy and reduce the loss of the original image information.

[0128] In the case of dividing each functional module according to each function, an embodiment of the present disclosure provides a multi-level semantic segmentation system, including: a first obtaining unit, a first constructing unit, a first processing unit, a second obtaining unit, and a second processing unit.

[0129] The first acquisition unit is used to obtain an image data set as training data. The first construction unit is used to construct a hierarchical semantic segmentation network, wherein the hierarchical semantic segmentation network includes extracting feature maps based on spatial information extraction paths. The first processing unit is used to train and test the hierarchical semantic segmentation network using the image data set to obtain a trained deep hierarchical real-time semantic segmentation model.

[0130] The second obtaining unit is used to obtain the image to be segmented. The second processing unit is used to input the image to be segmented into a deep hierarchical real-time semantic segmentation model, which includes processing the image to be segmented through a spatial information extraction path to obtain a feature map, using a pixel-by-pixel hierarchical segmentation learning strategy for the segmentation prediction in the supervision model, and outputting a real-time semantic segmentation result of the image to be segmented.

[0131] In the case of dividing each functional module according to each function, an embodiment of the present disclosure provides a multi-level semantic segmentation device, which may be a server or a chip applied to a server. Figure 4A schematic block diagram of the functional modules of a multi-level semantic segmentation device provided by an exemplary embodiment of the present disclosure. Figure 4 As shown, the multi-level semantic segmentation device 400 includes: a data acquisition module 410 and a processing module 420.

[0132] The data acquisition module 410 is used to acquire a set of images to be processed;

[0133] The data acquisition module 410 is further used to acquire multiple category labels of each image to be processed in the image set to be processed; wherein each image to be processed includes multiple objects, each object corresponds to a category label; and the object includes sub-objects;

[0134] A processing module 420 is used to extract feature maps of each image to be processed, input the feature maps into a classification head of an image segmentation network, and obtain label information corresponding to each part of the image to be processed;

[0135] The processing module 420 is further configured to input the multi-category labels and the label information into a multi-level structured loss function to obtain a first loss value; and input the multi-category labels and the label information into a triple loss function to obtain a second loss value;

[0136] The processing module 420 is further used to update the classification head based on the first loss value and the second loss value, obtain the trained preset model, and use the trained preset model as the deep hierarchical real-time semantic segmentation model;

[0137] The processing module 420 is further used to input the image to be segmented into the deep hierarchical real-time semantic segmentation model to obtain a segmentation result of the image to be segmented.

[0138] The processing module 420 is further used to extract the category labels of each image to be processed, convert the category labels into a multi-level structure, and obtain the multi-category labels; the multi-level structure includes the relationship between the category labels of the objects, and the relationship between the category labels of the objects and their sub-objects. The multi-level structure includes a parent node and a child node, the parent node includes a plurality of corresponding child nodes, and the parent node or the child node corresponding to the parent node is used as the multi-category label.

[0139] The processing module 420 is further configured to input the feature map into a classification head of an image segmentation network, and calculate a multi-label classification score for each pixel in the image to be processed:

[0140] M=σ(N h (F)

[0141] Among them, N h: represents the classification head of the hierarchical semantic segmentation network, σ: represents the activation function, F: represents the feature map, M: represents the output multi-label binary classification score map.

[0142] The processing module 420 is further configured to update the scores according to the relationship between the parent and child nodes in the multi-level structure based on the multi-label classification scores, and calculate the multi-level structured loss function:

[0143]

[0144] Among them, G t : represents the true value of the t-th class in the multi-class label, M a : represents the score of the predecessor node position of the tth class in the multi-label binary classification score graph, M c : represents the score of the descendant node position of the t-th class in the multi-label binary classification score graph, a: represents the ancestor node position of the t-th class, c: represents the descendant node position of the t-th class;

[0145]

[0146]

[0147] Among them, M i，j，t ∈(0,1): the t-th classification score of the pixel at position (i, j) in the multi-label binary classification score map; C i，j，t ∈{0,1}: label G corresponding to pixel (i,j) i，j The true value of the t-th class in; w, h: width and height of the image; |T|: number of nodes in the multi-level structure, P i，j，t ∈{0, 1}: represents the prediction score of the pixel at position (i, j) in space for the tth class, M i，j，a : represents the score of the predecessor node of the pixel point at position (i, j) in space, M i，j，c : Represents the score of the descendant node of the pixel point at position (i, j) in the space.

[0148] The processing module 420 is further configured to add an adjustment factor based on the multi-level structured loss function to calculate the first loss value L3:

[0149]

[0150] Where, λ: represents the adjustment factor, λ≥0; G i，j，t ∈{0,1}: label G corresponding to pixel (i,j) i，j The true value of the t-th class in; w, h: width and height of the image; |T|: number of nodes in the multi-level structure, P i，j，t∈{0, 1}: represents the prediction score of the pixel at position (i, j) in space for the tth class, M i，j，a : represents the score of the predecessor node of the pixel point at position (i, j) in space, M i，j，c : Represents the score of the descendant node of the pixel point at position (i, j) in the space.

[0151] The processing module 420 is further configured to calculate the second loss value L4:

[0152] d * =d0+md r

[0153]

[0154] L4(F,F + , F - )=max{d(F,F + )-d(F,F - )+d * ,0}

[0155] Among them, d(,): represents the distance between two representations, such as cosine distance, F, F+, F-: represent node features, positive features and negative features respectively, d * : represents the distance margin value, d0: represents the fixed margin value, o(,): represents the position between two nodes in the multi-level structure, H: represents the depth of the multi-level structure, m: coefficient.

[0156] Figure 5 Schematic block diagram of a chip according to an exemplary embodiment of the present disclosure is shown. Figure 5 As shown, the chip 500 includes one or more (including two) processors 501 and a communication interface 502. The communication interface 502 can support the server to perform the data sending and receiving steps in the above method, and the processor 501 can support the server to perform the data processing steps in the above method.

[0157] Optional, such as Figure 5 As shown, the chip 500 also includes a memory 503, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory (NVRAM).

[0158] In some embodiments, Figure 5As shown, the processor 501 performs corresponding operations by calling the operation instructions stored in the memory (the operation instructions may be stored in the operating system). The processor 501 controls the processing operations of any one of the terminal devices, and the processor may also be called a central processing unit (CPU). The memory 503 may include a read-only memory and a random access memory, and provides instructions and data to the processor. A portion of the memory 503 may also include NVRAM. For example, in an application, the memory, the communication interface, and the memory are coupled together through a bus system, wherein the bus system may include a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, in Figure 5 Various buses are labeled as bus system 504 .

[0159] The method disclosed in the above-mentioned embodiment of the present disclosure can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above-mentioned method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above-mentioned processor may be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiment of the present disclosure can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above-mentioned method in conjunction with its hardware.

[0160] The exemplary embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present disclosure when executed by the at least one processor.

[0161] The exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the method according to the embodiments of the present disclosure.

[0162] The exemplary embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform the method according to the embodiments of the present disclosure.

[0163] refer to Figure 6 , a block diagram of an electronic device 600 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0164] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0165] A plurality of components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 may be any type of device capable of inputting information to the electronic device 600, and the input unit 606 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 607 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 608 may include, but is not limited to, a disk, an optical disk. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0166] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (PU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above. Each of the methods described above may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via the ROM 602 and / or the communication unit 609.

[0167] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0168] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0169] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0170] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0171] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0172] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0173] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a terminal, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD).

[0174] Although the present disclosure has been described in conjunction with specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present disclosure. Accordingly, this specification and the drawings are merely exemplary illustrations of the present disclosure as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present disclosure. Obviously, those skilled in the art may make various modifications and variations to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A multi-level semantic segmentation method, characterized in that: The method comprises: Get the image set to be processed; Acquire multiple category labels of each image to be processed in the image set to be processed; wherein each image to be processed includes multiple objects, and each object corresponds to a category label; Extracting feature maps of each image to be processed, inputting the feature maps into a classification head of an image segmentation network, and obtaining label information corresponding to each part of the image to be processed; Inputting the multi-category labels and the label information into a multi-level structured loss function to obtain a first loss value; and inputting the multi-category labels into a triple loss function to obtain a second loss value; Update the classification head based on the first loss value and the second loss value to obtain a trained preset model, and use the trained preset model as a deep hierarchical real-time semantic segmentation model; Inputting the image to be segmented into the deep hierarchical real-time semantic segmentation model to obtain a segmentation result of the image to be segmented; The step of inputting the feature map into the classification head of the image segmentation network to obtain label information corresponding to each part of the image to be processed includes: The feature map is input into the classification head of the image segmentation network, and the multi-label classification score is calculated for each pixel in the image to be processed: M=σ(N h (F)) Among them, N h : represents the classification head of the hierarchical semantic segmentation network, σ: represents the activation function, F: represents the feature map, M: represents the output multi-label binary classification score map; Inputting the multi-category labels and the label information into a multi-level structured loss function to obtain a first loss value includes: Based on the multi-label classification score, the score is updated according to the relationship between the parent and child nodes in the multi-level structure, and the multi-level structured loss function is calculated: Among them, G t : represents the true value of the t-th class in the multi-class label, M a : represents the score of the predecessor node position of the tth class in the multi-label binary classification score graph, M c : represents the score of the descendant node position of the t-th class in the multi-label binary classification score graph, a: represents the ancestor node position of the t-th class, c: represents the descendant node position of the t-th class; Among them, M i，j，t ∈(0,1): the classification score of the t-th category of the pixel at position (i, j) in the multi-label binary classification score map; G i，j，t ∈{0,1}: label G corresponding to pixel (i,j) i，j The true value of the t-th class in; w, h: width and height of the image; |T|: number of nodes in the multi-level structure, P i，j，t ∈{0, 1}: represents the prediction score of the pixel at position (i, j) in space for the tth class, M i，j，a : represents the score of the predecessor node of the pixel point at position (i, j) in space, M i,j，c : represents the score of the descendant node of the pixel point at position (i, j) in the space; An adjustment factor is added on the basis of the multi-level structured loss function to calculate the first loss value L3: Where, λ: represents the adjustment factor, λ≥0; G i,j,t ∈{0,1}: the label G corresponding to pixel (i,j) i，j The true value of the t-th class in; w, h: width and height of the image; |T|: number of nodes in the multi-level structure, P i，j，t ∈{0,1}: represents the prediction score of the pixel at position (i,j) in space for the tth class, M i，j，a : represents the score of the predecessor node of the pixel point at position (i, j) in space, M i，j，c : Represents the score of the descendant node of the pixel point at position (i, j) in the space.

2. The method according to claim 1, characterized in that The object includes sub-objects; the multi-category labels of each image to be processed in the image set to be processed are obtained; wherein each image to be processed includes multiple objects, each object corresponds to a category label, including: Extract the category labels of the images to be processed, convert the category labels into a multi-level structure, and obtain multiple category labels; the multi-level structure includes the relationship between the category labels of the objects, and the relationship between the category labels of the objects and sub-objects.

3. The method according to claim 2, characterized in that The multi-level structure includes a parent node and child nodes, the parent node includes a plurality of corresponding child nodes, and the parent node or the child node corresponding to the parent node is used as the multi-category label.

4. The method according to claim 1, characterized in that: The step of inputting the multi-category labels into a triplet loss function to obtain a second loss value comprises: Calculate the second loss value L4: d * =d0+md r , L4(F,F + ,F - )=max{d(F,F + )-d(F,F - )+d * ,0} Among them, d(,) represents the cosine distance between two representations, F, F + 、F - : represent node features, positive features and negative features respectively, d * : represents the distance margin value, d0: represents the fixed margin value, o(,): represents the position between two nodes in the multi-level structure, H: represents the depth of the multi-level structure, m: coefficient.

5. A multi-level semantic segmentation device, characterized in that: The device comprises: A data acquisition module, used for acquiring a set of images to be processed; The data acquisition module is further used to acquire multiple category labels of each image to be processed in the image set to be processed; wherein each image to be processed includes multiple objects, and each object corresponds to a category label; A processing module is used to extract the feature map of each image to be processed, input the feature map into the classification head of the image segmentation network, and obtain the label information corresponding to each part of the image to be processed; wherein, inputting the feature map into the classification head of the image segmentation network to obtain the label information corresponding to each part of the image to be processed includes: inputting the feature map into the classification head of the image segmentation network, and calculating the multi-label classification score for each pixel in the image to be processed: M=σ(N h (F)) Among them, N h : represents the classification head of the hierarchical semantic segmentation network, σ: represents the activation function, F: represents the feature map, M: represents the output multi-label binary classification score map; The processing module is further used to input the multi-category labels and the label information into a multi-level structured loss function to obtain a first loss value; and input the multi-category labels and the label information into a triple loss function to obtain a second loss value; wherein, inputting the multi-category labels and the label information into the multi-level structured loss function to obtain the first loss value includes: based on the multi-label classification score, updating the score according to the relationship between the parent and child nodes in the multi-level structure, and calculating the multi-level structured loss function: Among them, G t : represents the true value of the t-th class in the multi-class label, M a : represents the score of the predecessor node position of the i-th category in the multi-label binary classification score graph, M c : represents the score of the descendant node position of the t-th class in the multi-label binary classification score graph, a: represents the ancestor node position of the t-th class, c: represents the descendant node position of the t-th class; Among them, M i，j，t ∈(0,1): the classification score of the t-th category of the pixel at position (i, j) in the multi-label binary classification score map; G i，j，t ∈{0,1}: label G corresponding to pixel (i,j) i，j The true value of the i-th category in; w, h: width and height of the image; |T|: number of nodes in the multi-level structure, P i，j，t ∈{0, 1}: represents the prediction score of the pixel at position (i, j) in space for the tth class, M i，j，a : represents the score of the predecessor node of the pixel point at position (i, j) in space, M i，j，c : represents the score of the descendant node of the pixel point at position (i, j) in the space; An adjustment factor is added on the basis of the multi-level structured loss function to calculate the first loss value L3: Where, λ: represents the adjustment factor, λ≥0; G i，j，t ∈{0,1}: label G corresponding to pixel (i,j) i，j The true value of the t-th class in; w, h: width and height of the image; |T|: number of nodes in the multi-level structure, P i，j，t ∈{0, 1}: represents the prediction score of the pixel at position (i, j) in space for the tth class, M i，j，a : represents the score of the predecessor node of the pixel point at position (i, j) in space, M i，j，c : represents the score of the descendant node of the pixel point at position (i, j) in the space; The processing module is further used to update the classification head based on the first loss value and the second loss value to obtain a trained preset model, and use the trained preset model as a deep hierarchical real-time semantic segmentation model; The processing module is also used to input the image to be segmented into the deep hierarchical real-time semantic segmentation model to obtain a segmentation result of the image to be segmented.

6. An electronic device, characterized in that: include: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 4.

Images