Multispectral image segmentation method, device, electronic device and storage medium

Through multiple feature extraction and expansion of multi-level image segmentation model, the problems of low accuracy and large calculation amount of multi-spectral image segmentation are solved, and more efficient image segmentation and lower cost are achieved, and multi-spectral in complex backgrounds are suitable for multi-spectral in complex backgrounds Image segmentation.

CN113327256BActive Publication Date: 2024-11-05WEBANK (CHINA)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110591478.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-05-28
Publication Date
2024-11-05
Estimated Expiration
2041-05-28

AI Technical Summary

Technical Problem

The existing multispectral image segmentation method has low accuracy, large calculation amount, slow inference speed, and requires a large amount of data fitting, which increases the labeling time and training cost. The traditional method is poorly robust and the multispectral image segmentation accuracy is low.

Method used

Through the multi-hierarchical structure of the image segmentation model, including the first segmentation sub-model, the first feature extension sub-model and the second segmentation sub-model, multiple feature extraction and expansion are performed to gradually improve the image segmentation accuracy, and use feature overlay and pixel processing layers Enhanced feature representation, reducing the impact of non-realistic category pixels.

Benefits of technology

It improves the accuracy of multi-spectral image segmentation, reduces the amount of calculation, reduces the cost of labeling and training, enhances the adaptability and application value of the model, and is suitable for application scenarios with few samples.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113327256B_ABST
    Figure CN113327256B_ABST
Patent Text Reader

Abstract

The present application provides a multispectral image segmentation method, device, electronic device, and computer-readable storage medium; the method comprises: performing image segmentation on the multispectral image to be processed through a first segmentation submodel included in an image segmentation model to obtain a corresponding first segmentation feature image; based on the multispectral image and the first segmentation feature image, performing feature expansion on the multispectral image through a first feature extension submodel included in the image segmentation model to obtain a corresponding feature extension image; performing image segmentation on the feature extension image through a second segmentation submodel included in the image segmentation model to obtain a first target segmentation image corresponding to the multispectral image. Through the present application, the segmentation accuracy of the multispectral image can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to computer vision technology, and in particular to a multispectral image segmentation method, device, electronic device and computer-readable storage medium. Background Art

[0002] Remote sensing object information classification, also known as semantic segmentation, is a pixel-level classification that is used in many aspects such as autonomous driving, medical image diagnosis, and remote sensing object classification. Limited by complex backgrounds, such as edge noise, occlusion, and illumination perspective, semantic segmentation has low accuracy (such as average intersection over union). And with the embedding and design of complex network structures, semantic segmentation models tend to be computationally intensive, with slower reasoning speeds and difficulty in interpretation. They require a large amount of data fitting, which increases labeling time and training costs.

[0003] In the related technologies, for the segmentation methods of multispectral images, the public datasets of remote sensing images used for semantic segmentation applications are relatively scarce. Traditional segmentation methods generally use random forest or texture primitive forest methods to construct classifiers for semantic segmentation. Such methods require artificially created replicated features, have poor robustness, and have low segmentation accuracy for multispectral images. Summary of the invention

[0004] The embodiments of the present application provide a multispectral image segmentation method, device, electronic device and computer-readable storage medium, which can accurately segment the multispectral image.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a multispectral image segmentation method, including:

[0007] Performing image segmentation on the multispectral image to be processed by using a first segmentation sub-model included in the image segmentation model to obtain a corresponding first segmentation feature image;

[0008] Based on the multispectral image and the first segmented feature image, the multispectral image is feature expanded by a first feature expansion submodel included in the image segmentation model to obtain a corresponding feature expansion image;

[0009] The feature extended image is segmented by using the second segmentation sub-model included in the image segmentation model to obtain a first target segmentation image corresponding to the multispectral image.

[0010] The present application provides a multispectral image segmentation device, comprising:

[0011] A first segmentation module is used to perform image segmentation on the multispectral image to be processed by using a first segmentation sub-model included in the image segmentation model to obtain a segmentation feature image;

[0012] A first feature extension module is used to perform feature extension on the multispectral image based on the multispectral image and the first segmentation feature image through a first feature extension sub-model included in the image segmentation model to obtain a corresponding feature extension image;

[0013] The second segmentation module is used to perform image segmentation on the feature extension image through the second segmentation sub-model included in the image segmentation model to obtain a first target segmentation image corresponding to the multispectral image.

[0014] In the above scheme, the first feature extension submodel includes a first probability determination layer and a first pixel processing layer; the first feature extension module is also used to determine the probability that each pixel in the first segmented feature image belongs to each category in at least two categories through the first probability determination layer; through the first pixel processing layer, the pixel data of each pixel in the multispectral image is multiplied by the probability corresponding to the pixel at the corresponding position in the feature extension image to obtain a multi-band feature map corresponding to each of the categories; the obtained multi-band feature map corresponding to each of the categories is used as the feature extension image of the multispectral image.

[0015] In the above scheme, the second segmentation sub-model includes a first feature overlay layer and a first image segmentation layer; the second segmentation module is also used to perform feature overlay on the multi-band feature maps corresponding to each of the categories through the first feature overlay layer to obtain a first feature combination image; and perform image segmentation on the first feature combination image through the first image segmentation layer to obtain a first target segmentation image corresponding to the multispectral image.

[0016] In the above scheme, the first segmentation sub-model includes a single-band extraction layer and an initial image segmentation layer; the first segmentation module is also used to perform band extraction on the multispectral image through the single-band extraction layer to obtain a single-band image corresponding to each band in the multiple bands; through the initial image segmentation layer, the single-band image corresponding to each band is segmented respectively to obtain a segmented image corresponding to each band as the first segmentation feature image.

[0017] In the above scheme, the first feature expansion submodel also includes a second probability determination layer and a second pixel processing layer; the first feature expansion module is also used to determine the probability that each pixel in the segmented image corresponding to each band belongs to each category in at least two categories through the second probability determination layer; through the second pixel processing layer, the pixel data of each pixel in the multispectral image is multiplied by the probability corresponding to the pixel at the corresponding position in each segmented image to obtain a single-band feature map corresponding to each category; the obtained single-band feature map corresponding to each category is used as the feature expansion image of the multispectral image.

[0018] In the above scheme, the second segmentation sub-model also includes a second feature overlay layer and a second image segmentation layer; the second segmentation module is also used to perform feature overlay on the single-band feature maps corresponding to each of the categories through the second feature overlay layer to obtain a second feature combination image; and perform image segmentation on the second feature combination image through the second image segmentation layer to obtain a first target segmentation image corresponding to the multispectral image.

[0019] In the above scheme, the multispectral image segmentation device also includes a second feature expansion module and a third segmentation module; after obtaining the first target segmentation image corresponding to the multispectral image, the second feature expansion module is used to perform feature expansion on the multispectral image based on the multispectral image and the first target segmentation image through the second feature expansion sub-model to obtain a corresponding feature expansion image; the third segmentation module is used to perform image segmentation on the feature expansion image output by the second feature expansion sub-model through the third segmentation sub-model to obtain a second target segmentation image corresponding to the multispectral image.

[0020] In the above scheme, the multispectral image segmentation device also includes a training module; the multispectral image to be processed is subjected to image segmentation by the first segmentation module included in the multispectral image segmentation device to obtain the corresponding first segmentation feature image before the training module is used to obtain a multispectral image sample and a corresponding standard segmentation image; the multispectral image sample is subjected to image segmentation by the first segmentation sub-model included in the image segmentation model to obtain the corresponding initial segmentation feature image; based on the multispectral image sample and the initial segmentation feature image, the multispectral image is subjected to feature expansion by the first feature expansion sub-model included in the image segmentation model to obtain the corresponding target feature expansion image; the target feature expansion image is subjected to image segmentation by the second segmentation sub-model included in the image segmentation model to obtain the predicted segmentation image corresponding to the multispectral image sample; based on the difference between the predicted segmentation image and the standard segmentation image, the model parameters of the image segmentation model are updated.

[0021] In the above scheme, the training module is also used to obtain multispectral image samples and corresponding standard segmented images; perform image segmentation on the multispectral image samples through the first segmentation submodel included in the image segmentation model to obtain the corresponding initial segmentation feature image; obtain a first loss based on the difference between the initial segmentation feature image and the standard segmentation image; based on the multispectral image samples and the initial segmentation feature image, perform feature expansion on the multispectral image through the first feature expansion submodel included in the image segmentation model to obtain the corresponding target feature expansion image; perform image segmentation on the target feature expansion image through the second segmentation submodel included in the image segmentation model to obtain a predicted segmentation image corresponding to the multispectral image sample; obtain a second loss based on the difference between the predicted segmentation image and the standard segmentation image; and update the model parameters of the image segmentation model based on the first loss and the second loss.

[0022] An embodiment of the present application provides an electronic device, including:

[0023] A memory for storing executable instructions;

[0024] The processor is used to implement the multispectral image segmentation method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0025] The embodiment of the present application provides a computer-readable storage medium storing executable instructions for implementing the multispectral image segmentation method provided in the embodiment of the present application when executed by a processor.

[0026] An embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the multispectral image segmentation method provided in the embodiment of the present application.

[0027] The embodiments of the present application have the following beneficial effects:

[0028] Compared with image segmentation in related technologies, the present application performs initial segmentation of the multispectral image through a segmentation model of the multispectral image to obtain a first segmentation feature image, and then based on the first segmentation feature image, increases the feature dimension of the multispectral image during the segmentation process to obtain a corresponding feature extension image, and then performs image segmentation on the feature extension image to obtain a first target segmentation image. In this way, with the expansion of the feature dimension, the image information of the multispectral image to be processed is also more comprehensively represented, so that the image information of the multispectral image to be processed can be segmented more accurately, thereby improving the image segmentation accuracy of the multispectral image. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1This is an optional structural diagram of a multispectral image segmentation system provided in an embodiment of the present application;

[0030] Figure 2 is an optional structural diagram of an electronic device provided in an embodiment of the present application;

[0031] Figure 3 This is an optional flow chart of a multispectral image segmentation method provided in an embodiment of the present application;

[0032] Figure 4 It is an optional structural schematic diagram of a multispectral image segmentation model provided in an embodiment of the present application;

[0033] Figure 5 It is an optional structural schematic diagram of a multispectral image segmentation model provided in an embodiment of the present application;

[0034] Figure 6 This is an optional flowchart of the feature expansion method provided in the embodiment of the present application;

[0035] Figure 7 It is an optional structural schematic diagram of a multispectral image segmentation model provided in an embodiment of the present application;

[0036] Figure 8 This is an optional flow chart of a multispectral image segmentation method provided in an embodiment of the present application;

[0037] Fig. 9 This is an optional flow chart of a multispectral image segmentation method provided in an embodiment of the present application;

[0038] Fig.10 It is an optional structural schematic diagram of a multispectral image segmentation model provided in an embodiment of the present application;

[0039] Fig.11 This is an optional flow chart of a multispectral image segmentation method provided in an embodiment of the present application;

[0040] Fig.12 It is an optional structural schematic diagram of a multispectral image segmentation model provided in an embodiment of the present application;

[0041] Fig.13 This is an optional flowchart of the feature expansion method provided in the embodiment of the present application;

[0042] Fig.14 It is an optional structural schematic diagram of a multispectral image segmentation model provided in an embodiment of the present application;

[0043] Fig.15It is an optional structural schematic diagram of a multispectral image segmentation model provided in an embodiment of the present application;

[0044] Fig.16 It is an optional flow chart of the multispectral image segmentation model training method provided in the embodiment of the present application;

[0045] Fig.17 It is an optional flow chart of the multispectral image segmentation model training method provided in the embodiment of the present application;

[0046] Fig.18 This is an optional flow chart of a multispectral image segmentation method provided in an embodiment of the present application;

[0047] Fig.19 It is an optional flow chart of the multispectral image segmentation model training method provided in the embodiment of the present application;

[0048] Fig. 20A It is an optional schematic diagram of the overall framework of the multispectral image segmentation model provided in the embodiment of the present application;

[0049] Fig. 20B It is an optional schematic diagram of the overall framework of the multispectral image segmentation model provided in the embodiment of the present application;

[0050] Fig. 20C It is an optional schematic diagram of the overall framework of the multispectral image segmentation model provided in the embodiment of the present application;

[0051] Fig.21 A schematic diagram of the structure of a multispectral image segmentation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be described in detail in some embodiments in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0053] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0054] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0056] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0057] 1) Image segmentation: Image segmentation refers to the process of subdividing a digital image into multiple image sub-regions (collections of pixels) (also called superpixels). It is a technique and process for dividing an image into several specific regions with unique properties and proposing targets of interest. It is a key step from image processing to image analysis. The purpose of image segmentation is to simplify or change the representation of an image so that the image is easier to understand and analyze. Image segmentation is often used to locate objects and boundaries (lines, curves, etc.) in an image. More precisely, image segmentation is a process of labeling each pixel in an image, which makes pixels with the same label have some common visual characteristics.

[0058] 2) Autonomous driving, also known as unmanned driving, computer driving or wheeled mobile robot, is a cutting-edge technology that relies on computers and artificial intelligence technology to complete complete, safe and effective driving without human control.

[0059] 3) Remote sensing of objects: Remote sensing sensors receive reflected and emitted energy from objects to form images. Different objects have different spectral information, so based on analyzing the spectrum in the remote sensing image, we can infer the category of the object.

[0060] 4) Remote sensing image segmentation refers to the process of dividing an image into a set of non-overlapping regions, while maintaining a certain degree of homogeneity in each region and showing obvious differences between different regions.

[0061] The inventors found that the semantic segmentation methods of related multispectral images are limited by complex backgrounds, such as edge noise, occlusion of illumination angle, etc., and the semantic segmentation has low accuracy (such as average intersection over union ratio). In addition, with the embedding and design of complex network structures, semantic segmentation models tend to have large computational complexity, slow reasoning speed, and are difficult to explain. They require a large amount of data fitting, which increases annotation time and training costs.

[0062] Based on this, the embodiments of the present application provide a multispectral image segmentation method, device, electronic device and computer-readable storage medium, which can accurately segment the multispectral image.

[0063] First, the multispectral image segmentation system provided in the embodiment of the present application is described. Figure 1 An optional schematic diagram of a multispectral image segmentation system 100 provided in an embodiment of the present application, in order to realize supporting a multispectral image segmentation application, a multispectral image segmentation client 410 is provided on a terminal 400, and the terminal 400 is connected to a server 200 via a network 300, and the network 300 can be a wide area network or a local area network, or a combination of the two, and a wireless link is used to realize data transmission. In some embodiments, the terminal 400 can be a laptop, a tablet computer, a desktop computer, a smart phone, a dedicated messaging device, a portable gaming device, a smart speaker, a smart watch, etc., but is not limited thereto. The server 200 can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The network 300 can be a wide area network or a local area network, or a combination of the two. The terminal 400 and the server 200 can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiment of the present application.

[0064] The terminal 400 is used to, in response to an upload operation for the multispectral image to be processed, obtain the multispectral image to be processed, and in response to an image segmentation instruction for the multispectral image to be processed, send the multispectral image to be processed to the server.

[0065] The server 200 is used to perform image segmentation on the multispectral image to be processed through the first segmentation sub-model included in the image segmentation model to obtain a corresponding segmented feature image; based on the multispectral image and the segmented feature image, perform feature expansion on the multispectral image through the first feature expansion sub-model included in the image segmentation model to obtain a corresponding feature expansion image; perform image segmentation on the feature expansion image through the second segmentation sub-model included in the image segmentation model to obtain a target segmentation image corresponding to the multispectral image, and send the target segmentation image to the terminal 400.

[0066] The terminal 400 is further used to receive and output the target segmented image sent by the server 200 .

[0067] Next, the electronic device for implementing the multispectral image segmentation method provided in the embodiment of the present application is described. Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application. In practical applications, the electronic device 500 can be implemented as Figure 1 The terminal 400 or the server 200 in the embodiment of the present invention is an electronic device. Figure 1 Taking the server 200 as an example, an electronic device for implementing the multispectral image segmentation method according to an embodiment of the present application is described. Figure 2 The electronic device 500 shown includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 540 is not described in detail. Figure 2 Various buses are labeled as bus system 540 .

[0068] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0069] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0070] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices that are physically remote from the processor 510.

[0071] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0072] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0073] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0074] A network communication module 552, for reaching other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB);

[0075] a presentation module 553 for enabling presentation of information via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with the user interface 530 (e.g., a user interface for operating peripherals and displaying content and information);

[0076] The input processing module 554 is used to detect one or more user inputs or interactions from one of the one or more input devices 532 and translate the detected inputs or interactions.

[0077] In some embodiments, the multispectral image segmentation device provided in the embodiments of the present application can be implemented in a software manner. Figure 2The multispectral image segmentation device 555 stored in the memory 550 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a first segmentation module 5551, a first feature expansion module 5552, and a second segmentation module 5553. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0078] In other embodiments, the multi-spectral image segmentation device provided in the embodiments of the present application can be implemented in hardware. As an example, the multi-spectral image segmentation device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image interference removal method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0079] Next, the multispectral image segmentation method provided by the embodiment of the present application will be described in combination with the exemplary application and implementation of the server provided by the embodiment of the present application. In some embodiments, the spectral image segmentation method provided by the embodiment of the present application can be implemented by the terminal or the server alone, or by the terminal and the server in collaboration. Next, the server implementation is taken as an example. Figure 3 , Figure 3 is an optional flow chart of the multispectral image segmentation method provided in the embodiment of the present application, which is combined with Figure 3 The steps shown are explained.

[0080] Step 101: The server performs image segmentation on the multispectral image to be processed by using a first segmentation sub-model included in the image segmentation model to obtain a corresponding first segmentation feature image.

[0081] Here, the multispectral image to be processed is input as input information to the first segmentation sub-model in the trained image segmentation model to perform image segmentation processing.

[0082] For an explanation of the image segmentation model for multispectral images, see Figure 4 , Figure 4It is an optional structural diagram of the multispectral image segmentation model provided in the embodiment of the present application. In some embodiments, the image segmentation model includes a first segmentation sub-model, a first feature extension sub-model and a second segmentation sub-model. The multispectral image is input into the first segmentation sub-model in the image segmentation model for segmentation processing to obtain a corresponding first segmentation feature image.

[0083] Here, the sizes of the segmentation feature maps output by the first segmentation sub-model and the second segmentation sub-model may be the same. In addition, the first segmentation sub-model, the first feature expansion sub-model, and the second segmentation sub-model may also be common basic segmentation models, such as UNet, FPN, PSPNet, lightweight models BiseNet, ICNet, etc.

[0084] It should be noted that the server performs image segmentation on the multispectral image to determine the category corresponding to the image content contained in the multispectral image. Here, when the multispectral image is a multispectral image of the corresponding ground object, the category corresponding to the image content can be understood as different categories such as "arable land", "forest land", "grassland", "road", "building" and "water area".

[0085] In other embodiments, when the above-mentioned image segmentation process is applied to the autonomous driving process, the categories corresponding to the multispectral image content here can be understood as different categories such as "sidewalk", "pedestrian", "green belt", "traffic light" and "building".

[0086] Here, different segmentation categories may be set corresponding to different image segmentation application scenarios, and no specific restrictions are imposed on the image segmentation categories.

[0087] Step 102: Based on the multispectral image and the first segmented feature image, the multispectral image is feature expanded by using a first feature expansion submodel included in the image segmentation model to obtain a corresponding feature expansion image.

[0088] Here, the first segmentation feature image is input as input information to the first feature expansion sub-model in the trained image segmentation model to perform feature expansion processing.

[0089] For an explanation of the first feature extension sub-model, see Figure 5 , Figure 5 1 is an optional structural diagram of the first feature extension submodel provided in an embodiment of the present application. In some embodiments, the first feature extension submodel includes a first probability determination layer and a first pixel processing layer. The first segmented feature image is input into the first feature extension submodel for feature extension processing to obtain a corresponding feature extension image.

[0090] Further, the process of performing feature expansion on the first segmented feature image according to the first feature expansion sub-model is shown in Figure 6 , Figure 6 is an optional flow chart of the feature expansion method provided in the embodiment of the present application, based on Figure 3 , step 102 can also be implemented in the following manner:

[0091] Step 1021: The server determines the probability that each pixel in the first segmented feature image belongs to each category of at least two categories through the first probability determination layer.

[0092] In actual implementation, the softmax probability value of each pixel in the first segmented feature image belonging to each category is calculated through the first probability determination layer. Here, the softmax probability value of each pixel belonging to each category describes the response of the input image to each category, and the category corresponding to the maximum probability value can be considered to be the category corresponding to this pixel.

[0093] Exemplarily, when the multispectral image is a multispectral image corresponding to a ground object, the categories corresponding to the image content can be three target categories of "arable land", "grassland" and "building". Here, the probability of each pixel in the first segmentation feature image of the first probability determination layer corresponding to each category is determined. For example, the probability of a certain pixel corresponding to "arable land" is a, the probability of corresponding to "grassland" is b, and the probability of corresponding to "building" is c.

[0094] Step 1022, through the first pixel processing layer, multiply the pixel data of each pixel in the multispectral image by the probability corresponding to the pixel at the corresponding position in the feature expansion image to obtain a multi-band feature map corresponding to each category.

[0095] Here, through the first pixel processing layer, the pixel data of each pixel in the multispectral image is multiplied pixel by pixel by the softmax probability value of each pixel at the corresponding position determined by the first probability determination layer belonging to each category to obtain a multi-band feature map corresponding to each category.

[0096] Continuing with the above example, through the first pixel processing layer, the pixel data of the pixel is multiplied by the probabilities a, b and c respectively to obtain the pixel data of the pixel under the categories of "arable land", "grassland" and "building". In this way, each pixel in the multispectral image is processed similarly to obtain the pixel data of each pixel under the three categories, that is, to obtain three multi-band feature maps.

[0097] It should be noted that, considering the extreme case of binary classification, the probability of 0 is the background, and after multiplication, the background part becomes 0. The subsequent model structure does not calculate the feature value and does not react to it, which is equivalent to an attention mechanism or filtering mechanism. However, compared with general attention, it has stronger semantic relevance.

[0098] Step 1023, using the obtained multi-band feature images corresponding to each category as feature extension images of the multispectral image.

[0099] Here, the first segmented feature image is expanded into a plurality of multi-band feature maps corresponding to each category through the first feature expansion sub-model, thereby achieving the expansion of the feature dimension.

[0100] Step 103: Perform image segmentation on the feature expansion image by using the second segmentation sub-model included in the image segmentation model to obtain a first target segmentation image corresponding to the multispectral image.

[0101] Here, the feature expanded image is input as input information to the second segmentation sub-model in the trained image segmentation model to perform image segmentation processing.

[0102] For an explanation of the second segmentation sub-model, see Figure 7 , Figure 7 is an optional structural diagram of the second segmentation sub-model provided in an embodiment of the present application. In some embodiments, the second segmentation sub-model includes a first feature superposition layer and a first image segmentation layer. The feature expansion image is input into the second segmentation sub-model for image processing to obtain a first target segmentation image corresponding to the multispectral image.

[0103] Further, the processing process of performing image segmentation on the feature extension image according to the second segmentation sub-model is shown in Figure 8 , Figure 8 is an optional flow chart of the multispectral image segmentation method provided in the embodiment of the present application, based on Figure 3 , step 103 can also be implemented in the following manner:

[0104] Step 1031 , performing feature superposition on the multi-band feature images corresponding to each category through a first feature superposition layer to obtain a first feature combination image.

[0105] Here, a plurality of multi-band feature maps corresponding to each category output by the first feature expansion sub-model are input into the first feature superposition layer for feature superposition.

[0106] Here, the feature expansion map obtained by the aforementioned first feature expansion submodel can obtain a corresponding multi-band feature map for each target category corresponding to the multispectral image content, and the obtained multiple multi-band feature maps are superimposed to obtain a first feature combination image.

[0107] Step 1032: Perform image segmentation on the first feature combination image through the first image segmentation layer to obtain a first target segmentation image corresponding to the multispectral image.

[0108] In other embodiments, see Fig. 9 , Fig. 9 An optional flow chart of a multispectral image segmentation method provided in an embodiment of the present application is applied to a server or a terminal. Here, the image segmentation model includes a first segmentation sub-model, a first feature extension sub-model, and a second segmentation sub-model; the first segmentation sub-model includes a single-band extraction layer and an initial image segmentation layer, the first feature extension sub-model includes a second probability determination layer and a second pixel processing layer, and the second segmentation sub-model includes a second feature superposition layer and a second image segmentation layer. The method includes:

[0109] Step 201 : performing band extraction on the multispectral image through the single band extraction layer of the first segmentation sub-model to obtain a single band image corresponding to each band in the plurality of bands.

[0110] Step 202 , performing image segmentation on the single-band images corresponding to each band through the initial image segmentation layer of the first segmentation sub-model, and obtaining the segmented images corresponding to each band as the first segmented feature images.

[0111] Here, the first segmentation sub-model is explained, see Fig.10 , Fig.10 It is an optional structural diagram of the first segmentation sub-model provided in the embodiment of the present application. Here, the multispectral image to be segmented includes at least three different bands. The single-band image corresponding to each band is input as input information to the single-band extraction layer in the first segmentation sub-model for band extraction to obtain multiple single-band images, and then the multiple single-band images are segmented through the initial image segmentation layer to obtain segmented images corresponding to each band.

[0112] For example, see Fig.11 , Fig.11 It is an optional flow chart of the multispectral image segmentation method provided in an embodiment of the present application. The multispectral image here is a multispectral image of the corresponding ground object. The categories corresponding to the image content can be different categories such as "arable land", "grassland" and "building". The first segmentation sub-model extracts single-band images of the red (R), green (G,), and blue (B) bands of the multispectral image through the single-band extraction layer; through the initial image segmentation layer, the single-band images under the RGB three bands are segmented respectively, so as to obtain three different segmentation feature images.

[0113] Step 203 , through the second probability determination layer of the first feature expansion sub-model, respectively determine the probability that each pixel in the segmented image corresponding to each band belongs to each category in at least two categories.

[0114] Step 204, through the second pixel processing layer of the first feature expansion sub-model, the pixel data of each pixel in the multispectral image is multiplied by the probability corresponding to the pixel at the corresponding position in each segmented image to obtain a single-band feature map corresponding to each category.

[0115] Here, the first feature expansion sub-model is described, see Fig.12 , Fig.12 It is an optional structural diagram of the first feature extension submodel provided in the embodiment of the present application. Here, the first segmentation feature image corresponding to each band is input as input information to the second probability determination layer in the first feature extension submodel to determine the probability of each pixel in the segmentation image corresponding to each band belonging to each category, and then through the second pixel processing layer, the pixel data of each pixel in the multispectral image is multiplied by the probability corresponding to the pixel at the corresponding position in each segmentation image to obtain the single-band feature map corresponding to each category.

[0116] Continuing with the above example, the first feature expansion submodel determines the probability that each pixel on the three different segmentation feature images corresponds to the three categories through the second probability determination layer; through the second pixel processing layer, the pixel data of each pixel on the three different segmentation feature images is multiplied by the probability that the pixel at the same position on the same segmentation feature image corresponds to the three target categories, see Fig.13 , Fig.13 It is an optional flow chart of the feature expansion method provided in an embodiment of the present application. The probability that a certain pixel on the segmented feature image under the R band corresponds to "arable land", "grassland" and "building" is determined to be d, e and f through the second probability determination layer. Then, through the second pixel processing layer, the pixel data of the pixel is multiplied by the probabilities d, e and f respectively to obtain the pixel data of the three categories of "arable land", "grassland" and "building" corresponding to the pixel under the R band. In this way, each pixel on the segmented feature image under the R band is similarly processed to obtain the pixel data of each pixel under the three categories, that is, to obtain three R band feature maps. Referring to the above process, the segmented feature images under the G and B bands are similarly processed. In this way, nine single-band feature maps are obtained.

[0117] Step 205: Use the obtained single-band feature images corresponding to each category as feature extension images of the multispectral image.

[0118] Step 206 , performing feature superposition on the single-band feature maps corresponding to each category through the second feature superposition layer of the second segmentation sub-model to obtain a second feature combination image.

[0119] Step 207 , performing image segmentation on the second feature combination image through the second image segmentation layer of the second segmentation sub-model to obtain a first target segmentation image corresponding to the multispectral image.

[0120] Here, the second segmentation sub-model is explained, see Fig.14 , Fig.14 It is an optional structural diagram of the second segmentation sub-model provided in the embodiment of the present application. Here, the second segmentation sub-model includes a second feature superposition layer and a second image segmentation layer. Here, the feature extension image is input as input information to the second feature superposition layer in the second segmentation sub-model for feature superposition to obtain a second feature combination image, and then the second feature combination image is segmented through the second image segmentation layer to obtain a first target segmentation image corresponding to the multispectral image.

[0121] Continuing with the above example, the second segmentation sub-model performs feature superposition on the single-band feature maps under the same category through the second feature superposition layer. For example, three R-band feature maps are superimposed to obtain the intermediate feature image of the R band, and then the three G-band feature maps and the three B-band feature maps are processed similarly to obtain the intermediate feature images of the G and B bands respectively. Then, the three intermediate feature images are subjected to band synthesis operation to obtain the second feature combination image; the second feature combination image is then segmented through the second image segmentation layer to obtain the first target segmentation image corresponding to the multispectral image.

[0122] In some embodiments, the image segmentation model further includes a second feature expansion sub-model and a third segmentation sub-model, see Fig.15 , Fig.15 It is an optional structural diagram of the segmentation model of the multispectral image provided in the embodiment of the present application; after step 1032 and step 207, the first target segmented image is input as input information to the second feature expansion sub-model to obtain a corresponding feature expansion image, and then the feature expansion image is input as input information to the third segmentation sub-model for segmentation processing to obtain a corresponding second target segmented image. Specifically, based on the multispectral image and the first target segmented image, the second feature expansion sub-model is used to perform feature expansion on the multispectral image to obtain a corresponding feature expansion image; the third segmentation sub-model is used to perform image segmentation on the feature expansion image output by the second feature expansion sub-model to obtain a second target segmented image corresponding to the multispectral image.

[0123] It should be noted that in Fig.15 The image segmentation model is constructed by cascading on the basis of , and this embodiment does not impose any limitation on this.

[0124] In some embodiments, see Fig.16 , Fig.16 is an optional flow chart of a multispectral image segmentation model training method provided in an embodiment of the present application, based on Figure 3Before step 101, it is necessary to train the segmentation model of the multispectral image. Fig.16 The steps shown are explained.

[0125] Step 301: The server obtains a multispectral image sample and a corresponding standard segmented image.

[0126] Before constructing the band combination image sample, the collected original multispectral image needs to be preprocessed. The preprocessing of multispectral images includes radiation correction, orthorectification, panchromatic sharpening, etc., and the value is stretched (such as percentage stretching, logarithmic stretching) to the 0-255 natural scene image numerical space.

[0127] The preprocessed multispectral images form an image set, and the size of each image in the image set is BxHxW, where BxHxW means that the number of input bands is B, the image height is H, and the width is W. Set the number of target categories included in the segmentation result to C, and add labels to each image in the image set according to the target category.

[0128] Step 302: Perform image segmentation on the multispectral image sample by using the first segmentation sub-model included in the image segmentation model to obtain a corresponding initial segmentation feature image.

[0129] Here the size of each segmentation feature image is CxHxW, and the size of each segmentation feature image is the same as the size of each image in the image set.

[0130] Step 303 , based on the multispectral image sample and the initial segmentation feature image, the multispectral image is feature expanded by using the first feature expansion submodel included in the image segmentation model to obtain a corresponding target feature expansion image.

[0131] Continuing with the above example, here, the segmentation feature image of size CxHxW is input into the first feature expansion sub-model, and the softmax probability value of each pixel in the first segmentation feature image corresponding to the multispectral image sample belonging to each category is obtained, and then the data of the B bands of the multispectral image sample is multiplied pixel by pixel with the data probability value of the C bands of the corresponding first segmentation feature image to obtain the data of (B*C) bands. Here, the data of (B*C) bands (B*C)xHxW is the data size of the corresponding feature expansion image.

[0132] Step 304: perform image segmentation on the target feature extension image using the second segmentation sub-model included in the image segmentation model to obtain a predicted segmented image corresponding to the multispectral image sample.

[0133] Continuing with the above example, a feature extension image of size (B*C)xHxW is input into the second segmentation sub-model, and segmentation processing is performed on the feature extension image to obtain a predicted segmentation image corresponding to the multispectral image sample.

[0134] Step 305: updating the model parameters of the image segmentation model based on the difference between the predicted segmentation image and the standard segmentation image.

[0135] It should be noted that the first segmentation sub-model and the second segmentation sub-model here can be common basic segmentation models, such as UNet, FPN, PSPNet, lightweight models BiseNet, ICNet, etc. The embodiment of the present application does not limit the basic segmentation model.

[0136] In other embodiments, see Fig.17 , Fig.17 is another optional flow chart of the multispectral image segmentation model training method provided in the embodiment of the present application, based on Figure 3 , before step 101, combined Fig.17 The steps shown are explained.

[0137] Step 401: The server obtains a multispectral image sample and a corresponding standard segmented image.

[0138] Step 402: perform image segmentation on the multispectral image sample by using the first segmentation sub-model included in the image segmentation model to obtain a corresponding initial segmentation feature image.

[0139] Step 403: obtaining a first loss based on the difference between the initial segmentation feature image and the standard segmentation image.

[0140] Step 404 , based on the multispectral image sample and the initial segmentation feature image, the multispectral image is feature expanded by using the first feature expansion submodel included in the image segmentation model to obtain a corresponding target feature expansion image.

[0141] Step 405 , performing image segmentation on the target feature extension image by using the second segmentation sub-model included in the image segmentation model to obtain a predicted segmented image corresponding to the multispectral image sample.

[0142] Step 406: Obtain a second loss based on the difference between the predicted segmented image and the standard segmented image.

[0143] Step 407: Update the model parameters of the image segmentation model based on the first loss and the second loss.

[0144] In actual implementation, a joint loss of the first loss and the second loss can be constructed, and the model parameters of the image segmentation model can be updated based on the joint loss. Here, the joint loss can be constructed by directly adding the first loss and the second loss to obtain the joint loss; as another construction method, the first loss and the second loss can be multiplied by their respective corresponding weight coefficients based on their respective weights, and then added to obtain the joint loss. Here, there is no restriction on the method of constructing the joint loss.

[0145] It should be noted that the first segmentation sub-model and the second segmentation sub-model here can be common basic segmentation models, such as UNet, FPN, PSPNet, lightweight models BiseNet, ICNet, etc. The embodiment of the present application does not limit the basic segmentation model.

[0146] In some embodiments, before the entire image segmentation model is formally trained, the initial image segmentation model is pre-trained for a preset round. In the pre-training stage, the model parameters of the initial image segmentation model are not randomly set, and the weights of the ImageNet pre-trained model can be directly loaded for initialization, which can accelerate the convergence of the first segmentation sub-model. After the pre-training stage of the first segmentation sub-model is completed, the model parameters of the first segmentation sub-model (also called pre-training weights) are obtained, and the model parameters of the initial image segmentation model when it converges are saved.

[0147] After completing the pre-training of the first segmentation sub-model, the entire image segmentation model ( Figure 4 As shown in the figure, formal training of the image segmentation model is performed. During the formal training phase, the model parameters saved in the pre-training phase are directly obtained to initialize the model parameters of the image segmentation model. It can be understood that in the formal training phase, the model parameters of the image segmentation model are not randomly set or started from scratch, but continue to be trained based on the model parameters in the pre-training phase. This is conducive to accelerating the convergence of the image segmentation model and reducing the demand for server computing power and network communication volume. Finally, the pre-training weights output in the pre-training phase are loaded, the joint loss of the output results of the entire image segmentation model in the formal training phase is constructed, and network training is performed. When the model training reaches the convergence condition, the trained image segmentation model is obtained. In this way, by constructing pre-training weights, the requirements for the amount of data are reduced, which helps to accelerate the convergence of the image segmentation model and reduce the requirements of complex network models for the number of samples and computing power.

[0148] By applying the above-mentioned embodiments of the present application, the segmentation model of the multispectral image is initialized by using pre-trained weights, thereby reducing the demand for sample data volume in the formal training stage of the segmentation model of the multispectral image, accelerating the convergence of the segmentation model of the multispectral image, and enabling the segmentation model of the multispectral image to be suitable for application scenarios with few samples, thereby enhancing the adaptability of the model; by cascading, the segmentation accuracy is gradually improved and integrated into the process of automated post-processing; the probability value of the segmentation result is multiplied by the input image, thereby further reducing the influence of non-actual category pixels, highlighting the real category, and ultimately improving the accuracy of semantic segmentation and enhancing its application value.

[0149] Next, the multispectral image segmentation method provided in the embodiment of the present application is further introduced. Fig.18 This is an optional flow chart of the multispectral image segmentation method provided in the embodiment of the present application, see Fig.18 The multispectral image segmentation method provided in the embodiment of the present application is implemented collaboratively by the client and the server.

[0150] Step 501: In response to an upload operation for a multispectral image, the client obtains a plurality of multispectral image samples and corresponding standard segmented images.

[0151] Here, the client can be a multispectral image segmentation client set on the terminal. The multispectral image can be triggered by the user based on the human-computer interaction interface of the client, so that the client presents an image selection interface on the human-computer interaction interface. The user uploads the multispectral image from the terminal locally based on the image selection interface, so that the client obtains the uploaded multispectral image.

[0152] In some embodiments, the multispectral image may also be captured by a camera that is communicatively connected to the terminal. After capturing the multispectral image, the camera transmits the multispectral image to the terminal and the terminal automatically uploads the multispectral image to the client.

[0153] Step 502: The client sends a multispectral image sample and a corresponding standard segmented image to a server.

[0154] Step 503: The server constructs a training and verification data set based on the received multispectral image samples.

[0155] Step 504: input the training and verification data sets into the initial image segmentation model.

[0156] Step 505: output the segmentation feature image of each multispectral image sample in the training and verification data set.

[0157] Step 506: based on the difference between the segmented feature image and the corresponding standard segmented image, update the model parameters of the initial image segmentation model to obtain the image segmentation model.

[0158] Here, after updating the model parameters of the initial image segmentation model, the training process of steps 504-506 is continued to obtain pre-trained weights, and the parameters of the initial image segmentation model are set based on the pre-trained weights to obtain the image segmentation model. By constructing pre-trained weights, the requirements for the amount of data are reduced, which helps to accelerate the convergence of the multispectral image segmentation model and reduce the requirements of the complex network model for the number of samples and computing power.

[0159] Step 507: The server performs image segmentation on the multispectral image samples in the training and verification data set through the image segmentation model to obtain a predicted segmented image.

[0160] Here, firstly, the multispectral image samples in the training and verification data set are segmented by the first segmentation sub-model included in the image segmentation model to obtain the corresponding segmentation feature image, and then based on the multispectral image samples and the segmentation feature image, the multispectral image is feature expanded by the first feature extension sub-model included in the image segmentation model to obtain the corresponding feature extension image, and finally, the feature extension image is segmented by the second segmentation sub-model included in the image segmentation model to obtain the predicted segmentation image corresponding to the multispectral image sample.

[0161] Step 508: The server updates the model parameters of the image segmentation model based on the difference between the predicted segmented image and the corresponding standard segmented image.

[0162] Here, the server can use the Euclidean distance loss function to calculate the sum of the pixel differences between each pixel of the predicted segmented image and the corresponding pixel in the standard segmented image, obtain the value of the Euclidean distance loss function, and update the model parameters of the image segmentation model based on the value. The server iterates the above training process until the Euclidean distance loss function reaches convergence, completing the training of the image segmentation model.

[0163] In addition, other loss functions may also be used to calculate the difference between the predicted segmented image and the multispectral image to be processed to update the model parameters of the image segmentation model, which is not limited in this embodiment.

[0164] Step 509: The server generates a prompt message indicating that the image segmentation model training is completed.

[0165] Step 510: The server sends a prompt message to the client.

[0166] Step 511 : The client obtains the multispectral image to be processed in response to the upload operation on the multispectral image to be processed.

[0167] Here, the multispectral image to be processed may also be sent to the client by other devices that are communicatively connected to the terminal.

[0168] Step 512: the client sends the multispectral image to be processed to the server in response to the image segmentation instruction for the multispectral image to be processed.

[0169] Here, the image segmentation instruction can be automatically generated by the client after receiving the message that the image segmentation model training is completed, or it can be automatically generated by the client due to certain trigger conditions. For example, after the client obtains a multispectral image to be processed, it automatically generates an image segmentation instruction for the image to be processed. It can also be sent to the client by other devices connected to the terminal for communication, or it can be generated by the user after triggering the corresponding image segmentation function item based on the human-computer interaction interface of the client.

[0170] In step 513, the server inputs the received multispectral image to be processed into the image segmentation model, so that the image segmentation model performs image segmentation on the multispectral image to be processed to obtain a corresponding target segmented image.

[0171] Step 514: the server sends the target segmented image to the client.

[0172] Step 515: The client outputs the target segmented image.

[0173] Here, the client may present the target segmentation image in the human-computer interaction interface of the client, may save the target segmentation image locally in the terminal, may send the target segmentation image to other devices that are communicatively connected to the terminal, and so on.

[0174] By applying the above-mentioned embodiments of the present application, the segmentation model of the multispectral image is initialized by using pre-trained weights, thereby reducing the demand for sample data volume in the formal training stage of the segmentation model of the multispectral image, accelerating the convergence of the segmentation model of the multispectral image, and enabling the segmentation model of the multispectral image to be suitable for application scenarios with few samples, thereby enhancing the adaptability of the model; by cascading, the segmentation accuracy is gradually improved and integrated into the process of automated post-processing; the probability value of the segmentation result is multiplied by the input image, thereby further reducing the influence of non-actual category pixels, highlighting the real category, and ultimately improving the accuracy of semantic segmentation and enhancing its application value.

[0175] In some embodiments, when the multispectral image is a multispectral remote sensing image of the corresponding ground object, the multispectral remote sensing image of the corresponding ground object is segmented by a first segmentation sub-model included in an image segmentation model to obtain a segmentation feature image corresponding to the multispectral remote sensing image; further, the segmented feature image is feature expanded by a first feature extension sub-model included in the image segmentation model to obtain a corresponding feature extension image; then, the feature extension image is image segmented by a second segmentation sub-model included in the image segmentation model to obtain a target segmentation image; accordingly, the target image is a ground object segmentation image corresponding to the multispectral remote sensing image; further, the ground object segmentation image is subjected to ground object recognition to obtain ground object categories included in the multispectral remote sensing image, as well as the relative position relationship between different ground object categories.

[0176] In this way, the multispectral remote sensing image is segmented through the image segmentation model to improve the accuracy of semantic segmentation, and then the ground objects in the multispectral remote sensing image are identified to improve the accuracy of ground object identification.

[0177] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0178] Semantic segmentation is pixel-level classification, which is used in many aspects such as autonomous driving, medical image diagnosis, and remote sensing object classification. Limited by complex backgrounds, such as edge noise, occlusion, and illumination perspective, semantic segmentation has low accuracy (such as average intersection over union). And with the embedding and design of complex network structures, semantic segmentation models tend to be computationally intensive, with slower reasoning speeds and difficulty in interpretation. They require a large amount of data fitting, which increases annotation time and training costs.

[0179] Based on this, the embodiment of the present application provides a multispectral image segmentation method for semantic segmentation, and constructs an image segmentation model for multispectral image segmentation. Fig. 20B , Fig. 20B This is a schematic diagram of the overall framework of the multispectral image segmentation model provided in the embodiment of the present application, combined with Fig. 20B It can be seen that the image segmentation model includes a first segmentation sub-model, a first feature expansion sub-model and a second segmentation sub-model.

[0180] Before using the model to perform semantic segmentation on multispectral images, the model needs to be trained first.

[0181] It should be noted that the training process of the image segmentation model is divided into two stages. The first stage is the training stage of the initial image segmentation model; the second stage is the training stage of the image segmentation model. For the specific training process, see Fig.19 , Fig.19 is an optional flow chart of a multispectral image segmentation model training method provided in an embodiment of the present application, combined with Fig.19 as well as Figures 20A-20C Provide explanation.

[0182] Step 601, obtain N multispectral images with B bands, H image height, and W width and their corresponding standard segmentation images to construct a training and verification data set.

[0183] It should be noted that before constructing the training and validation dataset, each multispectral image needs to be preprocessed. The preprocessing of multispectral images includes radiation correction, orthorectification, panchromatic sharpening, etc., and the values ​​are stretched (such as percentage stretching, logarithmic stretching) to the 0-255 natural scene image numerical space. The preprocessed multispectral images are labeled according to the set target category to obtain a training and validation dataset of size H*W. The size of each sample in the training and validation dataset is BxHxW.

[0184] Step 602: input the training and verification data sets into the initial image segmentation model.

[0185] Here, the initial image segmentation model can be a common basic segmentation model, such as UNet, FPN, PSPNet, lightweight models BiseNet, ICNet, etc. For the training of commonly used basic segmentation models (also called classification models), the weights of the ImageNet pre-trained model can be directly loaded for initialization, which can speed up the convergence of the network. Because the shallow features (such as spectral features, texture features) of multispectral images are more obvious, the underlying network of the above-mentioned common classification model can be used to extract edge texture features. The embodiment of the present application does not limit the basic segmentation model.

[0186] Step 603: output the segmentation feature image of each sample in the training and verification data set.

[0187] Here, see Fig. 20A , Fig. 20A It is a schematic diagram of the overall framework of the initial segmentation model. Each sample in the training and verification data set is input into the initial image model to obtain the corresponding segmentation feature image. Here, the size of each segmentation feature image is CxHxW, and C is the image segmentation target category.

[0188] Step 604: based on the difference between the segmented feature image and the corresponding standard segmented image, update the model parameters of the initial image segmentation model to obtain a trained initial image segmentation model.

[0189] Here, after obtaining the trained initial image segmentation model, the training process of steps 601-604 is continued, and after a preset round of training, pre-training weights are obtained, and the parameters of the initial image segmentation model are set based on the pre-training weights. In this way, by constructing pre-training weights, the requirements for the amount of data are reduced, which helps to accelerate the convergence of the multispectral image segmentation model and reduce the requirements of the complex network model for the number of samples and computing power.

[0190] It should be noted that the above steps 601-604 are the training stage of the initial image segmentation model, that is, the first stage of the training process of the image segmentation model, and the image segmentation model is initialized based on the pre-trained weights obtained in the above process. Next, steps 605-609 are the second stage of the training process of the image segmentation model, that is, training the initialized image segmentation model.

[0191] Step 605: input each sample in the training and validation data set into the image segmentation model.

[0192] Here, refer to Fig. 20B As shown, the image segmentation model includes a first segmentation sub-model, a first feature expansion sub-model and a second segmentation sub-model.

[0193] Step 606, obtaining a segmentation feature image through the first segmentation sub-model in the image segmentation model.

[0194] Here, a multispectral image sample of size BxHxW is input into the first segmentation sub-model in the image segmentation model, and each sample in the training and verification data set is segmented by the first segmentation sub-model to obtain a segmentation feature image of size CxHxW.

[0195] Step 607, obtaining a feature expanded image through the first feature expansion sub-model in the image segmentation model.

[0196] Here, the segmentation feature image of size CxHxW is input into the first feature expansion submodel in the image segmentation model, and the feature expansion of the segmentation feature image is performed by the first feature expansion submodel. Specifically, based on the softmax probability value of each pixel in the segmentation feature image belonging to each category, the data of the B bands of each sample in the training and verification data sets are multiplied pixel by pixel with the data probability value of the pixels at the same position in the C bands of the segmentation feature image to obtain a feature expansion image of size (B*C)xHxW.

[0197] In actual implementation, the data of the B bands of each sample are the corresponding data of each pixel of each sample in different bands; the data probability values ​​of the pixels at the same position in the C bands of the segmented feature image are the C probability values ​​of each pixel of the segmented feature image in different bands corresponding to each target category.

[0198] It should be noted that the softmax probability value of each pixel belonging to each category describes the response of the input image to each category. In actual implementation, the category probability information of each pixel is obtained through the softmax probability value, and the category corresponding to the maximum probability value can be considered as the category of this pixel. Considering the extreme case of binary classification, the probability of 0 is the background. After multiplication, the background part becomes 0. The subsequent network structure does not calculate the feature value and does not react to it, which is equivalent to an attention mechanism or filtering mechanism. However, compared with general attention, it has stronger semantic relevance.

[0199] Step 608: Obtain a target segmented image through the second segmentation sub-model in the image segmentation model.

[0200] Here, the feature extension image of size (B*C)xHxW is input into the second segmentation sub-model in the image segmentation model, and the feature extension image is segmented by the second segmentation sub-model to obtain the segmentation result of the second sub-model, that is, the target segmentation image. Here, the size of each target segmentation image is CxHxW;

[0201] Step 609, load the pre-trained weights, and construct a joint loss based on the segmentation feature image and the target segmentation image.

[0202] Here, the joint loss of constructing the segmentation feature image and the target segmentation image includes:

[0203] Based on the difference between the segmented feature image in step 606 and the corresponding standard segmented image, a first loss is obtained; based on the difference between the target segmented image in step 608 and the corresponding standard segmented image, a second loss is obtained; based on the pre-trained weights, a joint loss of the first loss and the second loss is constructed, illustratively, that is, loss = a*loss_step1+loss_step2, wherein loss_step1 is the first loss, loss_step2 is the second loss, and a is a hyperparameter, a weight coefficient, which needs to be fine-tuned according to the training results.

[0204] Step 610, updating the model parameters of the image segmentation model based on the joint loss.

[0205] It should be noted that the image segmentation model can be cascaded again, see Fig. 20C , Fig. 20C It is a schematic diagram of the overall framework of the cascaded multispectral image segmentation model provided in an embodiment of the present application. Specifically, after step 608, step 607 is executed immediately, and after feature expansion is completed, steps 608-609 are continued to be executed.

[0206] In addition, you can Fig. 20CThe image segmentation model is constructed by cascading on the basis of , and this embodiment does not impose any limitation on this.

[0207] By applying the above-mentioned embodiments of the present application, the segmentation model of the multispectral image is initialized by using pre-trained weights, thereby reducing the demand for sample data volume in the formal training stage of the segmentation model of the multispectral image, accelerating the convergence of the segmentation model of the multispectral image, and enabling the segmentation model of the multispectral image to be suitable for application scenarios with few samples, thereby enhancing the adaptability of the model; by cascading, the segmentation accuracy is gradually improved and integrated into the process of automated post-processing; the probability value of the segmentation result is multiplied by the input image, thereby further reducing the influence of non-actual category pixels, highlighting the real category, and ultimately improving the accuracy of semantic segmentation and enhancing its application value.

[0208] The following is a description of an exemplary structure of a multispectral image segmentation device 555 provided in an embodiment of the present application implemented as a software module. In some embodiments, Fig.21 As shown, the software modules stored in the multispectral image segmentation device 555 of the memory 540 may include:

[0209] The first segmentation module 5551 is used to perform image segmentation on the multispectral image to be processed by using the first segmentation sub-model included in the image segmentation model to obtain a segmentation feature image;

[0210] A first feature expansion module 5552 is used to perform feature expansion on the multispectral image based on the multispectral image and the first segmented feature image through a first feature expansion sub-model included in the image segmentation model to obtain a corresponding feature expansion image;

[0211] The second segmentation module 5553 is used to perform image segmentation on the feature extension image by using the second segmentation sub-model included in the image segmentation model to obtain a first target segmentation image corresponding to the multispectral image.

[0212] In some embodiments, the first feature extension submodel includes a first probability determination layer and a first pixel processing layer; the first feature extension module is also used to determine the probability of each pixel in the first segmented feature image belonging to each category of at least two categories through the first probability determination layer; through the first pixel processing layer, the pixel data of each pixel in the multispectral image is multiplied by the probability corresponding to the pixel at the corresponding position in the feature extension image to obtain a multi-band feature map corresponding to each of the categories; the obtained multi-band feature map corresponding to each of the categories is used as the feature extension image of the multispectral image.

[0213] In some embodiments, the second segmentation sub-model includes a first feature overlay layer and a first image segmentation layer; the second segmentation module is further used to perform feature overlay on the multi-band feature maps corresponding to each of the categories through the first feature overlay layer to obtain a first feature combination image; and perform image segmentation on the first feature combination image through the first image segmentation layer to obtain a first target segmentation image corresponding to the multispectral image.

[0214] In some embodiments, the first segmentation sub-model includes a single-band extraction layer and an initial image segmentation layer; the first segmentation module is also used to perform band extraction on the multispectral image through the single-band extraction layer to obtain a single-band image corresponding to each band in the multiple bands; through the initial image segmentation layer, image segmentation is performed on the single-band image corresponding to each band, and a segmented image corresponding to each band is obtained as the first segmentation feature image.

[0215] In some embodiments, the first feature extension submodel also includes a second probability determination layer and a second pixel processing layer; the first feature extension module is also used to determine the probability of each pixel in the segmented image corresponding to each band belonging to each category of at least two categories through the second probability determination layer; through the second pixel processing layer, the pixel data of each pixel in the multispectral image is multiplied by the probability corresponding to the pixel at the corresponding position in each segmented image to obtain a single-band feature map corresponding to each category; the obtained single-band feature map corresponding to each category is used as the feature expansion image of the multispectral image.

[0216] In some embodiments, the second segmentation sub-model also includes a second feature overlay layer and a second image segmentation layer; the second segmentation module is also used to perform feature overlay on the single-band feature maps corresponding to each of the categories through the second feature overlay layer to obtain a second feature combination image; and perform image segmentation on the second feature combination image through the second image segmentation layer to obtain a first target segmentation image corresponding to the multispectral image.

[0217] In some embodiments, the multispectral image segmentation device also includes a second feature expansion module and a third segmentation module; after obtaining the first target segmentation image corresponding to the multispectral image, the second feature expansion module is used to perform feature expansion on the multispectral image based on the multispectral image and the first target segmentation image through the second feature expansion sub-model to obtain a corresponding feature expansion image; the third segmentation module is used to perform image segmentation on the feature expansion image output by the second feature expansion sub-model through the third segmentation sub-model to obtain a second target segmentation image corresponding to the multispectral image.

[0218] In some embodiments, the multispectral image segmentation device also includes a training module; the multispectral image to be processed is subjected to image segmentation by the first segmentation module included in the multispectral image segmentation device to obtain a corresponding first segmentation feature image before the training module is used to obtain a multispectral image sample and a corresponding standard segmentation image; the multispectral image sample is subjected to image segmentation by the first segmentation submodel included in the image segmentation model to obtain a corresponding initial segmentation feature image; based on the multispectral image sample and the initial segmentation feature image, the multispectral image is subjected to feature expansion by the first feature expansion submodel included in the image segmentation model to obtain a corresponding target feature expansion image; the target feature expansion image is subjected to image segmentation by the second segmentation submodel included in the image segmentation model to obtain a predicted segmentation image corresponding to the multispectral image sample; and the model parameters of the image segmentation model are updated based on the difference between the predicted segmentation image and the standard segmentation image.

[0219] In some embodiments, the training module is also used to obtain multispectral image samples and corresponding standard segmented images; perform image segmentation on the multispectral image samples through the first segmentation submodel included in the image segmentation model to obtain a corresponding initial segmentation feature image; obtain a first loss based on the difference between the initial segmentation feature image and the standard segmentation image; based on the multispectral image samples and the initial segmentation feature image, perform feature expansion on the multispectral image through the first feature expansion submodel included in the image segmentation model to obtain a corresponding target feature expansion image; perform image segmentation on the target feature expansion image through the second segmentation submodel included in the image segmentation model to obtain a predicted segmentation image corresponding to the multispectral image sample; obtain a second loss based on the difference between the predicted segmentation image and the standard segmentation image; and update the model parameters of the image segmentation model based on the first loss and the second loss.

[0220] It should be noted that the description of the device of the embodiment of the present application is similar to the description of the above-mentioned method embodiment, and has similar beneficial effects as the method embodiment, so it will not be repeated.

[0221] An embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the multispectral image segmentation method provided in the embodiment of the present application.

[0222] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the multispectral image segmentation method provided by the present application embodiment, such as Figure 3 The segmentation method of multispectral image is shown.

[0223] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0224] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0225] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0226] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0227] In summary, the multispectral image segmentation accuracy is improved through the embodiments of the present application.

[0228] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A multispectral image segmentation method, It is characterized in that The method comprises: The multispectral image to be processed is segmented by a first segmentation sub-model included in the image segmentation model to obtain a corresponding first segmentation feature image, wherein the first segmentation sub-model includes a single-band extraction layer and an initial image segmentation layer; Based on the multispectral image and the first segmented feature image, the multispectral image is feature expanded by a first feature expansion submodel included in the image segmentation model to obtain a corresponding feature expansion image; The feature extended image is segmented by a second segmentation sub-model included in the image segmentation model to obtain a first target segmentation image corresponding to the multispectral image; wherein the second segmentation sub-model includes a first feature superposition layer and a first image segmentation layer; The first feature extension submodel includes a first probability determination layer and a first pixel processing layer; the first feature extension submodel included in the image segmentation model is used to perform feature extension on the multispectral image to obtain a corresponding feature extension image, including: Determining, by the first probability determination layer, the probability that each pixel in the first segmented feature image belongs to each category of at least two categories; Through the first pixel processing layer, pixel data of each pixel in the multispectral image is multiplied by the probability corresponding to the pixel at the corresponding position in the feature extension image to obtain a multi-band feature map corresponding to each category; The obtained multi-band feature map corresponding to each of the categories is used as the feature extension image of the multispectral image.

2. The method according to claim 1, It is characterized in that The second segmentation sub-model includes a first feature superposition layer and a first image segmentation layer; The step of performing image segmentation on the feature extended image by using the second segmentation sub-model included in the image segmentation model to obtain a first target segmented image corresponding to the multispectral image includes: Through the first feature superposition layer, feature superposition is performed on the multi-band feature maps corresponding to each of the categories to obtain a first feature combination image; The first feature combination image is segmented through the first image segmentation layer to obtain a first target segmentation image corresponding to the multispectral image.

3. The method according to claim 1, It is characterized in that The first segmentation sub-model includes a single-band extraction layer and an initial image segmentation layer; The first segmentation sub-model included in the image segmentation model performs image segmentation on the multispectral image to be processed to obtain a corresponding first segmentation feature image, including: Performing band extraction on the multispectral image through the single band extraction layer to obtain a single band image corresponding to each band in the multiple bands; The initial image segmentation layer is used to perform image segmentation on the single-band images corresponding to each band, and the segmented images corresponding to each band are obtained as the first segmented feature images.

4. The method according to claim 3, It is characterized in that The first feature extension submodel includes a second probability determination layer and a second pixel processing layer; the first feature extension submodel included in the image segmentation model is used to perform feature extension on the multispectral image to obtain a corresponding feature extension image, including: Determining, by the second probability determination layer, the probability that each pixel in the segmented image corresponding to each band belongs to each category in at least two categories; Through the second pixel processing layer, pixel data of each pixel in the multispectral image is multiplied by the probability corresponding to the pixel at the corresponding position in each segmented image to obtain a single-band feature map corresponding to each category; The obtained single-band feature map corresponding to each of the categories is used as the feature extension image of the multispectral image.

5. The method according to claim 4, It is characterized in that The second segmentation sub-model includes a second feature superposition layer and a second image segmentation layer; Through the second feature superposition layer, feature superposition is performed on the single-band feature maps corresponding to each of the categories to obtain a second feature combination image; The second feature combination image is segmented by the second image segmentation layer to obtain a first target segmentation image corresponding to the multispectral image.

6. The method according to claim 1, It is characterized in that The image segmentation model also includes a second feature expansion sub-model and a third segmentation sub-model; After obtaining the first target segmentation image corresponding to the multispectral image, the method further includes: Based on the multispectral image and the first target segmentation image, the multispectral image is feature expanded by the second feature expansion sub-model to obtain a corresponding feature expansion image; The feature expansion image output by the second feature expansion sub-model is segmented by the third segmentation sub-model to obtain a second target segmentation image corresponding to the multi-spectral image.

7. The method according to claim 1, It is characterized in that Before performing image segmentation on the multispectral image to be processed by using the first segmentation sub-model included in the image segmentation model to obtain the corresponding first segmentation feature image, the method further includes: Obtain multispectral image samples and corresponding standard segmentation images; Performing image segmentation on the multispectral image sample by using a first segmentation sub-model included in the image segmentation model to obtain a corresponding initial segmentation feature image; Based on the multispectral image sample and the initial segmentation feature image, the multispectral image is feature expanded by a first feature expansion submodel included in the image segmentation model to obtain a corresponding target feature expansion image; Performing image segmentation on the target feature extension image by using a second segmentation sub-model included in the image segmentation model to obtain a predicted segmented image corresponding to the multispectral image sample; Based on the difference between the predicted segmented image and the standard segmented image, the model parameters of the image segmentation model are updated.

8. The method according to claim 7, It is characterized in that Before performing image segmentation on the multispectral image to be processed by using the first segmentation sub-model included in the image segmentation model to obtain the corresponding first segmentation feature image, the method further includes: Obtaining a first loss based on a difference between the initial segmentation feature image and the standard segmentation image; Obtaining a second loss based on a difference between the predicted segmented image and the standard segmented image; Based on the first loss and the second loss, model parameters of the image segmentation model are updated.

9. A multispectral image segmentation device, It is characterized in that The device comprises: A first segmentation module, used for performing image segmentation on the multispectral image to be processed by using a first segmentation sub-model included in the image segmentation model to obtain a first segmentation feature image, wherein the first segmentation sub-model includes a single-band extraction layer and an initial image segmentation layer; A first feature extension module is used to perform feature extension on the multispectral image based on the multispectral image and the first segmented feature image through the first feature extension submodel included in the image segmentation model to obtain a corresponding feature extension image; the first feature extension submodel includes a first probability determination layer and a first pixel processing layer; the first feature extension module is also used to determine the probability that each pixel in the first segmented feature image belongs to each category in at least two categories through the first probability determination layer; through the first pixel processing layer, multiply the pixel data of each pixel in the multispectral image with the probability corresponding to the pixel at the corresponding position in the feature extension image to obtain a multi-band feature map corresponding to each category; and use the obtained multi-band feature map corresponding to each category as the feature extension image of the multispectral image; The second segmentation module is used to perform image segmentation on the feature extended image through the second segmentation sub-model included in the image segmentation model to obtain a first target segmentation image corresponding to the multispectral image; wherein the second segmentation sub-model includes a first feature overlay layer and a first image segmentation layer.

10. An electronic device, It is characterized in that The electronic device comprises: A memory for storing executable instructions; A processor, configured to implement the method according to any one of claims 1 to 8 when executing the executable instructions stored in the memory.

11. A computer-readable storage medium, It is characterized in that Executable instructions are stored, and when executed by a processor, the method described in any one of claims 1 to 8 is implemented.

12. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.