Image segmentation method, electronic device, medium, and computer program product

Abstract

The present disclosure provides an image segmentation method, an electronic device, a medium and a computer program product. The image segmentation method of the present disclosure comprises: analyzing the text type of the instruction text input by the user to determine the target instruction text; performing feature extraction on the target instruction text to obtain the text features of the target instruction text; and based on the learnable parameter matrix containing the mapping relationship between the target category and the semantic features, searching for the category to which the target to be segmented in the target instruction text belongs to determine the semantic features of the target instruction text; linearly projecting the semantic features to a feature space of the same dimension as the text features to obtain the linearly projected semantic features; element by element fusing the text features and the linearly projected semantic features to obtain the fusion features of the target instruction text; and based on the fusion features of the target instruction text, detecting the target to be segmented in the image to be segmented to determine the mask image.

Description

Technical Field

[0001] This disclosure relates to an image segmentation method, electronic device, medium, and computer program product. Background Technology

[0002] Existing methods can only obtain discrete category labels in practical applications and lack continuous contextual semantic descriptions, which makes it impossible to construct high-dimensional cross-modal feature maps. Furthermore, there are significant distribution differences between general pre-trained text encoders and visual features in specific remote sensing scenarios, and direct mapping will lead to a decrease in alignment accuracy. Summary of the Invention

[0003] This disclosure provides an image segmentation method, an electronic device, a medium, and a computer program product.

[0004] According to one aspect of this disclosure, an image segmentation method is provided, comprising: analyzing the text type of user-input instruction text to determine target instruction text, wherein the target instruction text is a category label of a target to be segmented; extracting features from the target instruction text to obtain text features of the target instruction text; and retrieving the category to which the target to be segmented in the target instruction text belongs based on a learnable parameter matrix containing a mapping relationship between target category and semantic features to determine the semantic features of the target instruction text; linearly projecting the semantic features onto a feature space of the same dimension as the text features to obtain linearly projected semantic features; fusing the text features and the linearly projected semantic features element by element to obtain fused features of the target instruction text; and detecting the target to be segmented in the image to be segmented based on the fused features of the target instruction text to determine a mask image.

[0005] According to one technical solution, the basic word vectors of the target instruction text containing only category names input by the user are extracted, and the semantic features of the implicit context are retrieved from the learnable parameter matrix. Finally, residual semantic enhancement is achieved through linear projection and element-wise fusion. This fusion feature effectively makes up for the lack of information in the weakly supervised text, and can accurately locate and segment target objects in the image even without pixel-level annotation. It significantly improves the semantic understanding ability, spatial positioning accuracy and weakly supervised generalization performance of cross-modal remote sensing image segmentation, while maintaining computational efficiency and engineering practicality.

[0006] The image segmentation method according to at least one embodiment of the present disclosure further includes: analyzing the text type of a user-input instruction text to determine a instruction text to be processed, wherein the instruction text to be processed is text containing the category of the target to be segmented, the visual attributes of the target to be segmented, the scene spatial relationship of the target to be segmented in the image to be segmented, and / or the environmental information of the target to be segmented; extracting semantic features from the instruction text to be processed to obtain the semantic features of the instruction text to be processed; and detecting the target to be segmented in the image to be segmented based on the semantic features of the instruction text to be processed to determine a mask image.

[0007] According to at least one embodiment of the image segmentation method of the present disclosure, the target to be segmented in the image to be segmented is detected based on a first neural network. The first neural network is trained using the prediction result output by a second neural network as a supervision signal. The second neural network has the same network structure as the first neural network, but the parameter update method is different.

[0008] According to at least one embodiment of the image segmentation method of this disclosure, the first neural network updates its parameters using gradient descent, and the parameters of the second neural network are updated using an exponential moving average.

[0009] According to at least one embodiment of the image segmentation method of this disclosure, the training process of the first neural network includes: acquiring a sample image, fully supervised text, and weakly supervised text; extracting visual features from the sample image to obtain visual features of the sample image; fusing and decoding the visual features of the sample image, the fully supervised text, and the weakly supervised text based on the first neural network to obtain a first prediction probability map of the target to be segmented in the sample image; and fusing and decoding the visual features of the sample image and the weakly supervised text based on the second neural network to obtain a second prediction probability map of the target to be segmented in the sample image; and constructing a loss function based on the first prediction probability map and the second prediction probability map for training the first neural network.

[0010] According to at least one embodiment of the image segmentation method of this disclosure, a loss function of the first neural network is constructed based on the consistency loss between the first predicted probability map and the second predicted probability map, and the consistency loss is used to drive the update of the learnable parameter matrix.

[0011] According to at least one embodiment of the image segmentation method of this disclosure, during the pre-training of the first neural network, the learnable parameter matrix is ​​frozen, and the first neural network is trained using fully supervised text; during the training of the first neural network, the learnable parameter matrix is ​​unfrozen, and the first neural network is trained using fully supervised text and weakly supervised text.

[0012] According to another aspect of this disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, causing the processor to perform an image segmentation method according to any embodiment of this disclosure.

[0013] According to another aspect of this disclosure, a readable storage medium is provided, wherein executable instructions are stored therein, which, when executed by a processor, are used to implement the image segmentation method of any embodiment of this disclosure.

[0014] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the image segmentation method of any embodiment of this disclosure. Attached Figure Description

[0015] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0016] Figure 1 This is a schematic diagram of the overall process of an image segmentation method according to one embodiment of the present disclosure.

[0017] Figure 2 This is a schematic flowchart illustrating the process of determining a mask image in an image segmentation method according to one embodiment of the present disclosure.

[0018] Figure 3 This is a schematic diagram of the process of constructing a loss function in an image segmentation method according to one embodiment of the present disclosure.

[0019] Figure 4 This is a flowchart illustrating an image segmentation method according to one embodiment of the present disclosure.

[0020] Figure 5 This is a schematic diagram of the framework of an image segmentation method according to one embodiment of the present disclosure.

[0021] Figure 6 This is a schematic structural block diagram of an image segmentation apparatus according to one embodiment of the present disclosure.

[0022] Figure 7This is a schematic structural block diagram of an electronic device according to one embodiment of the present disclosure. Detailed Implementation

[0023] The present disclosure will now be described in further detail with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0024] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] In practical remote sensing image interpretation applications, users often only provide simple, vague, or unstructured instruction text (such as "find the airplane in the picture"), and cannot provide annotation text data accurate to the pixel level. At the same time, remote sensing images themselves have characteristics such as varying ground scale, complex backgrounds, and large differences in the appearance of similar targets, which makes traditional methods that rely on strongly supervised learning prone to problems such as positioning errors, blurred boundaries, or missed detections when faced with such instruction text.

[0026] To address this, this disclosure proposes an image segmentation method that intelligently distinguishes text types and constructs a unified semantic representation: for target instruction text containing only categories, its basic text features are first extracted, and then the implicit contextual semantic features associated with the category in the remote sensing scene are retrieved using a learnable parameter matrix. Furthermore, a fusion feature rich in environmental information is generated through linear projection and element-wise fusion. This method can effectively and accurately locate and segment target regions in the image to be segmented that conform to the semantic description, even without pixel-level annotations. This overcomes the problems of inaccurate positioning, unclear boundaries, or missed detections caused by ambiguous instructions and complex images, achieving high-precision pixel-level segmentation of weakly supervised instruction text in real-world application scenarios.

[0027] The image segmentation method disclosed herein can be deployed on cloud server clusters to provide batch automated target segmentation services for large-scale satellite / aerial image datasets. Users only need to input command text to quickly generate high-precision mask images, significantly reducing the cost of manual visual interpretation. It can also be deployed on edge computing terminal devices to achieve real-time or near real-time target detection and localization. Furthermore, it can be deployed in mobile apps as a professional remote sensing tool or a public science application, allowing non-professionals to upload aerial images and natural language descriptions, and instantly return segmentation results, greatly lowering the barrier to entry for remote sensing technology.

[0028] Figure 1 A schematic diagram illustrating the overall flow of an image segmentation method according to one embodiment of this disclosure is shown. Figure 1 The method M100 shown includes steps S110 to S150. This method can be executed by a server, terminal device, or mobile device.

[0029] In step S110, the text type of the instruction text input by the user is analyzed to determine the target instruction text, which is the category label of the target to be segmented.

[0030] The input instruction text is subjected to structured semantic discrimination. Based on whether the input instruction text contains additional descriptive information (such as color, location, spatial relationship and environmental background) in addition to the category label of the target to be segmented, the instruction text is divided into target instruction text containing only discrete category labels and instruction text to be processed containing rich contextual description.

[0031] In step S120, feature extraction is performed on the target instruction text to obtain the text features of the target instruction text; and based on the learnable parameter matrix containing the mapping relationship between target category and semantic features, the category to which the target to be segmented in the target instruction text belongs is retrieved to determine the semantic features of the target instruction text.

[0032] The target category mentioned above refers to a predefined discrete semantic entity used to guide the segmentation of the image to be segmented.

[0033] By transforming discrete target instruction text into structured feature vectors and combining them with learnable parameter matrices, the system automatically supplements the target instruction text with information about the environmental, appearance, or spatial relationships of the target category in a specific application scenario, thereby upgrading the originally information-poor keywords into conditional guidance signals rich in scenario semantics.

[0034] Preferably, each row of the learnable parameter matrix (i.e., the global implicit semantic dictionary) corresponds to a target category (such as "airplane" or "oil tank"), storing the implicit contextual prototype of that target category in a specific application scenario. Unlike traditional fixed text word vectors, the parameters in this learnable parameter matrix are automatically updated with the network gradient during training, used to dynamically capture the environmental commonalities of each category. For example, "boat" is usually accompanied by the feature of "water surface," and "vehicle" is usually accompanied by the feature of "road."

[0035] Preferably, the basic semantic vector of the target instruction text, i.e., text features, is extracted by a text encoder (BERT) to reflect the category meaning of the target instruction text in a common language.

[0036] In step S130, the semantic features are linearly projected onto a feature space of the same dimension as the text features to obtain the linearly projected semantic features.

[0037] Furthermore, semantic features are projected to the same feature space dimension as text features through trainable linear transformation layers (such as fully connected layers or 1×1 convolutions), ensuring that the two are consistent in numerical range, semantic granularity and spatial structure.

[0038] In step S140, the text features and the semantic features after linear projection are fused element by element to obtain the fused features of the target instruction text.

[0039] By structurally integrating textual features with linearly projected semantic features, and enhancing the expression by superimposing residual connections to preserve the original semantic information, a dense semantic vector that reflects both the literal meaning of words and the context of the scene is generated, namely, fused features.

[0040] Specifically, feature fusion is represented as in, This refers to the reconstructed enhanced semantic features (i.e., fused features). Based on basic category features (i.e., text features). For implicit contextual features (i.e. semantic features), Proj(·) represents a linear transformation.

[0041] In step S150, based on the fusion features of the target instruction text, the target to be segmented in the image to be segmented is detected, and the mask image is determined.

[0042] By using the fusion features of the target instruction text as a conditional signal, the target object that conforms to the semantic description is identified and located from the image to be segmented, and finally a pixel-level binary or category mask image is output.

[0043] Therefore, the image segmentation method disclosed herein automatically enhances the semantics of target instruction text containing only category labels by intelligently determining the semantic structure of the user-input instruction text. First, it extracts the basic word vectors, then retrieves and linearly projects implicit contextual semantic features from the learnable parameter matrix, and finally generates fused features rich in scene context information through feature fusion. This fused feature effectively compensates for the deficiency of single information in weakly supervised text, and can accurately understand the typical environment and visual attributes of the target in the image to be segmented even without pixel-level annotation. This achieves high-precision, adaptive cross-modal image segmentation, significantly improving semantic guidance capability, spatial localization accuracy, and weakly supervised generalization performance.

[0044] In some embodiments of this disclosure, it may include, for example Figure 2 Steps S210 to S230 are shown.

[0045] In step S210, the text type of the instruction text input by the user is analyzed to determine the instruction text to be processed. The instruction text to be processed is text containing the category of the target to be segmented, the visual attributes of the target to be segmented, the scene spatial relationship of the target to be segmented in the image to be segmented, and / or the environmental information of the target to be segmented.

[0046] In step S220, semantic features are extracted from the text to be processed to obtain the semantic features of the text to be processed.

[0047] Preferably, semantic features are extracted from the text of the instruction to be processed using a text encoder.

[0048] In step S230, based on the semantic features of the instruction text to be processed, the target to be segmented in the image to be segmented is detected, and the mask image is determined.

[0049] Step S210 is the same as step S110, and step S230 is the same as step S150, so they will not be described in detail here.

[0050] Therefore, it is possible to directly semantically encode the instruction text containing rich semantic information, extracting its complete semantic features that integrate category, color, spatial relationships, and environmental background without additional reconstruction or enhancement steps. The semantic features of the instruction text serve as strong guiding signals, accurately locating and segmenting target objects that conform to complex descriptions in the image to be segmented, thus improving the spatial accuracy and semantic completeness of the segmentation results.

[0051] In one specific embodiment, a first neural network is used to detect the target to be segmented in the image to be segmented. The first neural network is trained using the prediction results output by a second neural network as a supervision signal. The second neural network has the same network structure as the first neural network, but the parameter update method is different. Specifically, using the stable prediction results output by the second neural network as a supervision signal to guide the learning of the first neural network effectively suppresses noise interference and model oscillations caused by weak supervision data. Because the two networks have the same structure but different parameter update methods, the second neural network can provide smoother and more reliable prediction results. Therefore, without pixel-level annotation, the detection accuracy and boundary clarity of the target to be segmented in the image to be segmented by the first neural network are significantly improved, achieving stable convergence and high-performance segmentation under weak supervision conditions.

[0052] Preferably, the first neural network is a visual encoder-decoder network based on a convolutional neural network or a visual self-attention architecture. Its network structure includes a backbone visual encoder (such as ResNet, Swin Transformer, etc.) for extracting multi-scale features of the image to be segmented and a multimodal fusion decoder (such as a variant of U-Net or a Transformer decoder) for progressively upsampling and fusing textual semantic features. The second neural network is a visual encoder-decoder network with the exact same structure as the first neural network. Its backbone also adopts a convolutional neural network or a visual self-attention architecture and is equipped with the same multimodal decoder structure.

[0053] Preferably, the first neural network updates its parameters using gradient descent, and the parameters of the second neural network are updated using an exponential moving average (EMA) of the first neural network.

[0054] Therefore, by using gradient descent to update the parameters of the first neural network, it can quickly respond to the semantic and visual information in the training data, and achieve real-time and accurate prediction of the target in the image to be segmented. At the same time, the parameters of the first network are slowly passed to the second neural network through exponential moving average, so that the latter outputs a more stable and robust prediction result as a supervision signal, effectively suppressing noise interference and training oscillation caused by weak supervision data.

[0055] In some embodiments of this disclosure, the training process of the first neural network may include, for example: Figure 3 Steps S310 to S340 are shown.

[0056] In step S310, sample images, fully supervised text, and weakly supervised text are acquired.

[0057] Sample images are collected from the actual image dataset to be segmented, and paired with two types of instruction text: fully supervised text, which corresponds to the category label or natural language description of the pixel-level labeled mask; and weakly supervised text, which only contains the name of the target category or a simple attribute description, without relying on pixel-level labeling.

[0058] In step S320, visual features are extracted from the sample image to obtain the visual features of the sample image.

[0059] Low-level edge texture features, mid-level semantic region features, and high-level global context features are extracted layer by layer from the sample image, enabling them to be fused and interact with the semantic information expressed by the instruction text in a unified space.

[0060] In step S330, the visual features, fully supervised text and weakly supervised text of the sample image are fused and decoded based on the first neural network to obtain a first predicted probability map of the target to be segmented in the sample image; and the visual features and weakly supervised text of the sample image are fused and decoded based on the second neural network to obtain a second predicted probability map of the target to be segmented in the sample image.

[0061] The aforementioned prediction probability map is a two-dimensional matrix output by the decoder of the first or second neural network, with the same spatial resolution as the input sample image. The value of each pixel position represents the prediction confidence or probability value (in the range of 0 to 1) of the pixel belonging to the category of the target to be segmented.

[0062] The first neural network simultaneously receives visual features, fully supervised text, and weakly supervised text, and outputs a first predicted probability map through a multimodal fusion decoder. Its training objective is to approximate the fully supervised ground truth mask, while being guided by weakly supervised text to enhance semantic generalization ability. The second neural network only receives visual features and weakly supervised text, and outputs a second predicted probability map. Its parameters are slowly updated through exponential moving average, providing smoother and more stable pseudo-labels as supervision signals to constrain the learning process of the first neural network.

[0063] In step S340, a loss function is constructed based on the first prediction probability map and the second prediction probability map to train the first neural network.

[0064] Preferably, a loss function for the first neural network is constructed based on the consistency loss between the first and second predicted probability maps, and the consistency loss is used to drive the update of the learnable parameter matrix, so that the semantic features are more in line with the distribution of the real image, thereby improving the accuracy of cross-modal image segmentation.

[0065] The first predicted probability map generated by the first neural network is compared with the real mask to calculate the segmentation loss (such as cross-entropy or Dice loss) to ensure that the model learns the accurate spatial boundary. At the same time, the first predicted probability map is compared with the second predicted probability map generated by the second neural network to calculate the consistency loss, so as to force the output of the first neural network to move closer to the more stable and reliable second predicted probability map, thereby maintaining semantic consistency and spatial continuity even in the absence of complete annotation.

[0066] Therefore, it not only enhances the model's ability to generalize to unseen data, but also ensures the spatial continuity and semantic consistency of the segmentation results, significantly improving segmentation accuracy.

[0067] In one specific embodiment, during the pre-training of the first neural network, the learnable parameter matrix is ​​frozen, and the first neural network is trained using fully supervised text; during the training of the first neural network, the learnable parameter matrix is ​​unfrozen, and the first neural network is trained using fully supervised text and weakly supervised text, so that the first neural network prioritizes learning accurate spatial segmentation capabilities from fully supervised text.

[0068] In one specific embodiment, such as Figure 4 As shown, the user inputs the image to be segmented and natural language command text. The system then performs semantic analysis on the command text, determining whether it belongs to target command text (containing only category labels) or command text to be processed (containing rich attributes, location, or environmental descriptions). If it is target command text, its semantic structure is further reconstructed and enhanced to supplement the remote sensing scene context information. Finally, the reconstructed target command text or command text to be processed is used as a conditional signal to perform target detection and pixel-level localization on the image to be segmented, outputting the corresponding binary mask image, realizing end-to-end intelligent interpretation from language input to spatial segmentation results. The entire process is simple and efficient, supports both weakly supervised and strongly semantic input modes, and is suitable for diverse remote sensing application scenarios.

[0069] The technical solution of this disclosure will be further explained below with a specific implementation example.

[0070] like Figure 5 As shown, a global learnable parameter matrix, called the Global Implicit Semantic Dictionary (GLSD), is initialized in the network. Each row of this dictionary corresponds to a target category (such as "airplane" or "oil tank"), storing the implicit context prototype of that category in a specific remote sensing scene.

[0071] Unlike traditional fixed text word vectors, the parameters in this dictionary are automatically updated with the network gradient during training to dynamically capture the environmental commonalities of each category (e.g., "boat" is usually accompanied by "water surface" features, and "vehicle" is usually accompanied by "road" features).

[0072] When the input data contains only category names (weak labels, i.e., target instruction text), the system performs a semantic manifold reconstruction operation to complete the missing semantic information. Specifically, based on the input category index, the corresponding implicit context vector is retrieved from the global implicit semantic dictionary. The basic semantic vector of the category name extracted by the text encoder (i.e., text features) is fused with the retrieved implicit context vector (i.e., semantic features of the target instruction text), outputting the fused enhanced semantic manifold features (i.e., fused features). The fusion process is implemented through a residual projection module, whose core computational logic is as follows: in, This refers to the reconstructed enhanced semantic features (i.e., fused features). Based on basic category features (i.e., text features). For implicit contextual features (i.e. semantic features), Proj(·) represents a linear transformation.

[0073] Therefore, discrete category labels are reconstructed into dense features containing rich environmental information, making them equivalent to a complete descriptive statement in the feature space.

[0074] In addition, to suppress noise fluctuations from pseudo-labels during weakly supervised training, a dual-tower architecture is constructed, consisting of an online stream (i.e., the first neural network) and a target stream (i.e., the second neural network). The two streams have the exact same network structure (including a visual encoder, a GLSD module, and a decoder), but their parameter update methods are completely different.

[0075] The online flow employs standard gradient descent, updating parameters in real-time using backpropagation error. The target flow does not participate in gradient calculation; its parameters are derived from the exponential moving average (EMA) of the online flow parameters. The update formula is as follows: in, This represents the parameters of the target flow network at time step t. This represents the parameters of the target flow network at time step t-1. Represents the coefficient of the exponential moving average. This represents the parameters of the online stream network at time step t.

[0076] The target stream network is essentially a smoothed version of the online stream network in the time dimension. Its prediction results are more stable than those of the online stream network at the current moment, and it can effectively filter out high-frequency noise during the training process.

[0077] In addition, during training, a small amount of fully supervised text data (including masks) and a large amount of weakly supervised text data (without masks) are input simultaneously. The online streaming network is trained using the fully supervised text data, and the standard segmentation loss between the predicted mask and the ground truth mask is calculated to establish basic visual-text alignment capabilities. The images to be segmented (i.e., sample images) serve as the visual basis and are fed into a shared visual encoder (CNN / ViT) to extract multi-scale visual features, which are then input into the online streaming network and the target streaming network.

[0078] For weakly supervised text data, it is input into both the online stream network and the target stream network. The second predicted probability map output by the target stream network is used as a soft pseudo-label to supervise the prediction results of the online stream network (i.e., the first predicted probability map). A consistency loss is calculated between the two networks to force the prediction trajectory of the online stream network to converge towards the more stable target stream network. The consistency loss is: in, Indicates consistency loss. This represents the first predicted probability graph. This represents the second predicted probability plot.

[0079] The gradient generated by the aforementioned consistency loss not only optimizes the online streaming network, but is also fed back to the Global Implicit Semantic Dictionary (GLSD), driving the implicit parameters in the dictionary to be automatically adjusted in order to learn the contextual features that are most conducive to achieving prediction consistency.

[0080] In addition, to prevent the randomness of dictionary parameters in the early stages of training from corrupting the feature space, a two-stage training approach is adopted. In the warm-up stage, the global implicit semantic dictionary is frozen, and the shared visual encoder is pre-trained using only fully supervised text data. In the joint evolution stage, the dictionary parameters are unfrozen, a weakly supervised text data stream is introduced, and dual-stream collaborative training is initiated, enabling the dictionary parameters and network parameters to converge synchronously.

[0081] During the actual inference phase, only the trained online stream network and the global implicit semantic dictionary are retained. If the user inputs a complete sentence, conventional segmentation is performed directly. If the user only inputs category keywords (such as "search for ships in the port"), the semantic manifold reconstruction mechanism is automatically triggered, and high-precision positioning is performed after the semantics are completed using the dictionary.

[0082] Figure 6 This is a schematic block diagram of an image segmentation apparatus according to one embodiment of the present disclosure.

[0083] like Figure 6 As shown, the image segmentation device includes: The text analysis module 6002 analyzes the text type of the user-input command text to determine the target command text, which is the category label of the target to be segmented. The feature extraction module 6004 extracts features from the target instruction text to obtain the text features of the target instruction text; and based on a learnable parameter matrix containing the mapping relationship between target category and semantic features, it retrieves the category to which the target to be segmented in the target instruction text belongs and determines the semantic features of the target instruction text. The feature projection module 6006 linearly projects the semantic features onto a feature space of the same dimension as the text features to obtain the linearly projected semantic features. The feature fusion module 6008 fuses the text features and the semantic features after linear projection element by element to obtain the fused features of the target instruction text. The image segmentation module 6010 detects the target to be segmented in the image to be segmented based on the fusion features of the target instruction text and determines the mask image.

[0084] The image segmentation device described above can be in the form of computer software, and each module of the image segmentation device can be implemented through computer software modules.

[0085] The specific implementation process of the functions and roles of each module in the above image segmentation device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0086] This disclosure also provides an electronic device. Figure 7 A schematic diagram of the hardware implementation using the processing system is shown.

[0087] The hardware architecture of electronic devices can be implemented using a bus architecture. A bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripherals, voltage regulators, power management circuits, external antennas, etc. Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one connection line is used in this diagram, but this does not indicate that there is only one bus or one type of bus.

[0088] For ease of explanation, certain steps of the above method are described in relation to modules. It should be understood that the corresponding module performing one or more steps of the above method may be one or more hardware modules specifically configured to perform the corresponding step, or implemented by a processor configured to perform the corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination thereof.

[0089] According to an embodiment of this application, a vehicle is also provided. The vehicle includes the aforementioned electronic equipment.

[0090] This disclosure also provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0091] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, all or part of the processes or functions of this disclosure are performed.

[0092] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0093] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0094] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0095] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0096] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0097] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment / mode or example, which are included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0098] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0099] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. An image segmentation method, characterized in that, include: The text type of the user-input command text is analyzed to determine the target command text, which is the category label of the target to be segmented; Feature extraction is performed on the target instruction text to obtain the text features of the target instruction text; and based on a learnable parameter matrix containing the mapping relationship between target category and semantic features, the category to which the target to be segmented in the target instruction text belongs is retrieved to determine the semantic features of the target instruction text. The semantic features are linearly projected onto a feature space of the same dimension as the text features to obtain the linearly projected semantic features. The text features and the semantic features after linear projection are fused element by element to obtain the fused features of the target instruction text; as well as Based on the fusion features of the target instruction text, the target to be segmented in the image to be segmented is detected, and the mask image is determined.

2. The image segmentation method as described in claim 1, characterized in that, Also includes: The text type of the command text input by the user is analyzed to determine the command text to be processed. The command text to be processed is text containing the category of the target to be segmented, the visual attributes of the target to be segmented, the scene spatial relationship of the target to be segmented in the image to be segmented, and / or the environmental information of the target to be segmented. Semantic features are extracted from the text of the instruction to be processed to obtain the semantic features of the text of the instruction to be processed; Based on the semantic features of the instruction text to be processed, the target to be segmented in the image to be segmented is detected, and the mask image is determined.

3. The image segmentation method as described in claim 1, characterized in that, The target to be segmented in the image to be segmented is detected based on the first neural network. The first neural network is trained using the prediction result output by the second neural network as a supervision signal. The second neural network has the same network structure as the first neural network, but the parameter update method is different.

4. The image segmentation method as described in claim 3, characterized in that, The first neural network updates its parameters using gradient descent, and the parameters of the second neural network are updated using an exponential moving average.

5. The image segmentation method as described in claim 3, characterized in that, The training process of the first neural network includes: Acquire sample images, fully supervised text, and weakly supervised text; Visual features are extracted from the sample image to obtain the visual features of the sample image; Based on the first neural network, the visual features of the sample image, the fully supervised text, and the weakly supervised text are fused and decoded to obtain a first prediction probability map of the target to be segmented in the sample image; and based on the second neural network, the visual features of the sample image and the weakly supervised text are fused and decoded to obtain a second prediction probability map of the target to be segmented in the sample image. A loss function is constructed based on the first prediction probability map and the second prediction probability map to train the first neural network.

6. The image segmentation method as described in claim 5, characterized in that, Based on the consistency loss between the first predicted probability map and the second predicted probability map, a loss function for the first neural network is constructed, and the consistency loss is used to drive the update of the learnable parameter matrix.

7. The image segmentation method as described in claim 3, characterized in that, During the pre-training of the first neural network, the learnable parameter matrix is ​​frozen, and the first neural network is trained using fully supervised text; during the training of the first neural network, the learnable parameter matrix is ​​unfrozen, and the first neural network is trained using fully supervised text and weakly supervised text.

8. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the image segmentation method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that, The readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the image segmentation method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image segmentation method according to any one of claims 1 to 7.

Images