Image segmentation method and device, equipment and storage medium

Abstract

The invention provides an image segmentation method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence. In some embodiments of the present disclosure, multi-modal data is obtained, where the multi-modal data includes: image data and prompt data; inputting the multi-modal data into a trained image segmentation model to obtain a segmentation mask; wherein the image segmentation model comprises an image encoder, a prompt encoder and a mask decoder, the image encoder is used for encoding the image data into an image embedding vector, the prompt encoder is used for encoding the prompt data into a prompt embedding vector, and the mask decoder is used for decoding the prompt embedding vector. The mask decoder is used for performing mask prediction according to the image embedding vector and the prompt embedding vector to obtain the segmentation mask; according to the invention, image segmentation is carried out by using the multi-modal data, the multi-granularity processing capability of the image segmentation model is improved, and the image segmentation precision of the image segmentation model is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to an image segmentation method, device, equipment and storage medium. Background Art

[0002] Image segmentation is the technology and process of dividing an image into several specific regions with unique properties and proposing objects of interest. It is a key step from image processing to image analysis. Existing image segmentation methods are mainly divided into the following categories: threshold-based segmentation methods, region-based segmentation methods, edge-based segmentation methods, and segmentation methods based on specific theories. From a mathematical point of view, image segmentation is the process of dividing a digital image into non-intersecting regions. The image segmentation process is also a labeling process, that is, pixels belonging to the same region are assigned the same number.

[0003] Visual AI learning methods that are usually applied to small sample scenarios mostly use deep learning methods, which have poor generalization capabilities when faced with new and unseen data, and cannot handle multiple tasks and multiple data types at the same time. It is difficult to comprehensively learn information from multiple perspectives and multiple modalities, and to perform joint reasoning and decision-making in a multi-task environment. At the same time, traditional methods lack the ability to process multiple granularities, and it is difficult to segment and identify data from details to the whole. This makes the model perform poorly when processing complex scenes and multi-scale data.

[0004] As mentioned above, the data used in current image segmentation is in a relatively simple form, and the accuracy of image segmentation is relatively low. Summary of the invention

[0005] The present disclosure provides an image segmentation method, apparatus, device and storage medium to at least solve the problem that the data used in the existing image segmentation is in a relatively single format and the image segmentation accuracy is low.

[0006] The technical solution of the present disclosure is as follows:

[0007] The present disclosure provides an image segmentation method, including:

[0008] Acquiring multimodal data, wherein the multimodal data includes: image data and prompt data;

[0009] The multimodal data is input into a trained image segmentation model to obtain a segmentation mask; wherein the image segmentation model includes: an image encoder, a prompt encoder and a mask decoder, the image encoder is used to encode the image data into an image embedding vector, the prompt encoder is used to encode the suggestive data into a prompt embedding vector, and the mask decoder is used to perform mask prediction based on the image embedding vector and the prompt embedding vector to obtain the segmentation mask.

[0010] Optionally, inputting the multimodal data into a trained image segmentation model to obtain a segmentation mask includes:

[0011] Within the image segmentation model, the image data is input into the image encoder, and feature dimension reduction is performed on the image data to obtain an image embedding vector;

[0012] Inputting the suggestive data into the hint encoder to encode the suggestive data into the hint embedding vector;

[0013] The image embedding vector and the hint embedding vector are input into the mask decoder to perform mask prediction to obtain the segmentation mask.

[0014] Optionally, the image segmentation model is used to decouple semantic learning of object classification and component classification.

[0015] Optionally, the image segmentation model adopts multi-granularity learning.

[0016] Optionally, the method further comprises:

[0017] Small sample data from actual application scenarios are used to fine-tune the mask decoder to obtain a fine-tuned image segmentation model.

[0018] Optionally, the image data includes: at least one of a picture, a video and a point cloud; the suggestive data includes: at least one of a background point, a mask, a bounding box and text.

[0019] The present disclosure also provides an image segmentation device, including:

[0020] An acquisition module, used for acquiring multimodal data, wherein the multimodal data includes: image data and prompt data;

[0021] A segmentation module is used to input the multimodal data into a trained image segmentation model to obtain a segmentation mask; wherein the image segmentation model includes: an image encoder, a prompt encoder and a mask decoder, the image encoder is used to encode the image data into an image embedding vector, the prompt encoder is used to encode the suggestive data into a prompt embedding vector, and the mask decoder is used to perform mask prediction based on the image embedding vector and the prompt embedding vector to obtain the segmentation mask.

[0022] Optionally, when the segmentation module inputs the multimodal data into a trained image segmentation model to obtain a segmentation mask, it is used to:

[0023] Within the image segmentation model, the image data is input into the image encoder, and feature dimension reduction is performed on the image data to obtain an image embedding vector;

[0024] Inputting the suggestive data into the hint encoder to encode the suggestive data into the hint embedding vector;

[0025] The image embedding vector and the hint embedding vector are input into the mask decoder to perform mask prediction to obtain the segmentation mask.

[0026] Optionally, the image segmentation model is used to decouple semantic learning of object classification and component classification.

[0027] Optionally, the image segmentation model adopts multi-granularity learning.

[0028] Optionally, the segmentation module can also be used to:

[0029] Small sample data from actual application scenarios are used to fine-tune the mask decoder to obtain a fine-tuned image segmentation model.

[0030] Optionally, the image data includes: at least one of a picture, a video and a point cloud; the suggestive data includes: at least one of a background point, a mask, a bounding box and text.

[0031] The present disclosure also provides an electronic device, including:

[0032] processor;

[0033] a memory for storing instructions executable by the processor;

[0034] The processor is configured to execute the instructions to implement each step in the above method.

[0035] The embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, each step in the above method is implemented.

[0036] The embodiment of the present disclosure further provides a computer program product, including a computer program / instruction, which implements each step of the above method when executed by a processor.

[0037] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0038] In some embodiments of the present disclosure, multimodal data is acquired, wherein the multimodal data includes: image data and suggestive data; the multimodal data is input into a trained image segmentation model to obtain a segmentation mask; wherein the image segmentation model includes: an image encoder, a hint encoder and a mask decoder, the image encoder is used to encode the image data into an image embedding vector, the hint encoder is used to encode the suggestive data into a hint embedding vector, and the mask decoder is used to perform mask prediction based on the image embedding vector and the hint embedding vector to obtain the segmentation mask; the present disclosure uses multimodal data for image segmentation, improves the multi-granularity processing capability of the image segmentation model, and improves the accuracy of the image segmentation model in performing image segmentation.

[0039] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0041] Figure 1 A flowchart of an image segmentation method provided for an exemplary embodiment of the present disclosure;

[0042] Figure 2 A schematic diagram of the structure of an image segmentation model provided for an exemplary embodiment of the present disclosure;

[0043] Figure 3 A schematic diagram of the structure of an image segmentation device provided by an exemplary embodiment of the present disclosure;

[0044] Figure 4 A schematic structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0045] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0046] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure.

[0047] It should be noted that the user information involved in this disclosure includes but is not limited to: user device information and user personal information; the collection, storage, use, processing, transmission, provision and disclosure of user information in this disclosure comply with the relevant laws and regulations and do not violate public order and good morals.

[0048] Image segmentation is the technology and process of dividing an image into several specific regions with unique properties and proposing objects of interest. It is a key step from image processing to image analysis. Existing image segmentation methods are mainly divided into the following categories: threshold-based segmentation methods, region-based segmentation methods, edge-based segmentation methods, and segmentation methods based on specific theories. From a mathematical point of view, image segmentation is the process of dividing a digital image into non-intersecting regions. The image segmentation process is also a labeling process, that is, pixels belonging to the same region are assigned the same number.

[0049] Visual AI learning methods that are usually applied to small sample scenarios mostly use deep learning methods, which have poor generalization capabilities when faced with new and unseen data, and cannot handle multiple tasks and multiple data types at the same time. It is difficult to comprehensively learn information from multiple perspectives and multiple modalities, and to perform joint reasoning and decision-making in a multi-task environment. At the same time, traditional methods lack the ability to process multiple granularities, and it is difficult to segment and identify data from details to the whole. This makes the model perform poorly when processing complex scenes and multi-scale data.

[0050] The synthetic data generated by the adversarial network may be different from the real data. Although the generated data can cover the range of potential data distribution, they are not collected from real scenes and cannot fully reflect the characteristics of real scenes.

[0051] Existing image segmentation model architectures are dominated by a single-input-single-output pipeline that removes any ambiguity. While this pipeline is prevalent in both anchor-based CNN architectures and query-based Transformer architectures and has demonstrated superior performance in semantic, instance, and panoptic segmentation tasks, it inherently limits the model to predict multi-granularity segmentation masks in an end-to-end manner. While clustering post-processing techniques can generate multiple masks for a single object query, they are neither an efficient nor an effective solution for granularity-aware segmentation models.

[0052] In response to the above technical problems, in some embodiments of the present disclosure, multimodal data is obtained, wherein the multimodal data includes: image data and suggestive data; the multimodal data is input into a trained image segmentation model to obtain a segmentation mask; wherein the image segmentation model includes: an image encoder, a hint encoder and a mask decoder, the image encoder is used to encode the image data into an image embedding vector, the hint encoder is used to encode the suggestive data into a hint embedding vector, and the mask decoder is used to perform mask prediction based on the image embedding vector and the hint embedding vector to obtain a segmentation mask; the present disclosure uses multimodal data for image segmentation, improves the multi-granularity processing capability of the image segmentation model, and improves the accuracy of the image segmentation model in performing image segmentation.

[0053] The technical solutions provided by various embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings.

[0054] Figure 1 FIG. 1 is a flow chart of an image segmentation method provided by an exemplary embodiment of the present disclosure. Figure 1 As shown, the method includes:

[0055] S101: Acquire multimodal data, wherein the multimodal data includes: image data and prompt data;

[0056] S102: Input the multimodal data into the trained image segmentation model to obtain a segmentation mask; wherein the image segmentation model includes: an image encoder, a prompt encoder and a mask decoder, the image encoder is used to encode the image data into an image embedding vector, the prompt encoder is used to encode the suggestive data into a prompt embedding vector, and the mask decoder is used to perform mask prediction based on the image embedding vector and the prompt embedding vector to obtain a segmentation mask.

[0057] In this embodiment, the execution subject of the above method may be a server or a terminal device.

[0058] Among them, the terminal device includes but is not limited to a mobile station (MS), a mobile terminal, a mobile phone, a handset and a portable equipment, etc. The terminal device can communicate with one or more core networks via a radio access network (RAN). For example, the terminal device can be a mobile phone (or a "cellular" phone), a computer with a wireless communication function, etc. The terminal device can also be a computer with a wireless transceiver function, a virtual reality (VR) terminal device, an AR terminal device, a wireless terminal in industrial control, a wireless terminal in self driving, a wireless terminal in remote medical, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, etc., and the operating system installed on the terminal device includes but is not limited to: IOS, Android, windows, linux, Mac OS and other operating systems. In different networks, the terminal may be called different names, such as user equipment, mobile station, subscriber unit, station, cellular phone, personal digital assistant, wireless modem, wireless communication device, handheld device, laptop computer, cordless phone, wireless local loop station, television, etc. For the convenience of description, it is referred to as terminal equipment in this embodiment.

[0059] In this embodiment, the implementation form of the server is not limited. For example, the server can be a conventional server, a cloud server, a cloud host, a virtual center, or other server devices. The server mainly includes a processor, a hard disk, a memory, a system bus, etc., and a general computer architecture type.

[0060] The present disclosure may adopt small sample data of actual application scenarios, for example, small sample data of power plants, oil fields, etc.

[0061] The present disclosure mainly targets algorithm scenarios for equipment status, instrument detection, and personnel safety monitoring in small sample scenarios such as power plants and oil fields. The above scenarios have high security requirements and high confidentiality, and have problems such as difficulty in obtaining data, difficulty in labeling, and multiple data type dimensions. Traditional deep learning methods have poor generalization capabilities when faced with new and unseen data, and cannot handle multiple tasks and multiple data types at the same time. This technical solution aims to solve the problems of small data and poor model generalization capabilities in small sample scenarios, and to improve the accuracy and robustness of the algorithm. Among them, the business logic of scenarios such as power plants and oil fields is complex, and it is necessary to comprehensively learn information from multiple perspectives and multiple modalities, and perform joint reasoning and decision-making in a multi-task environment. Traditional methods lack the ability to process at multiple granularities, and it is difficult to segment and identify data from details to the whole. The model performs poorly when processing complex scenarios and multi-scale data. The present disclosure improves the multi-modal, multi-scale, and multi-granularity processing capabilities of power plant scenario algorithms.

[0062] Figure 2 FIG. 1 is a schematic diagram of a structure of an image segmentation model provided by an exemplary embodiment of the present disclosure. Figure 2 As shown, the image segmentation model includes: an image encoder, a prompt encoder and a mask decoder. The image data is input into the image encoder, and the image encoder outputs an image embedding vector; the prompt data is input into the prompt encoder, and the prompt encoder outputs a prompt embedding vector; the image embedding vector and the prompt embedding vector are input into the mask decoder, and mask prediction is performed to obtain a segmentation mask. The present disclosure adopts the structure of the image encoder, the prompt encoder and the mask decoder, which can more efficiently process multimodal data and predict masks.

[0063] In some embodiments of the present disclosure, multimodal data is input into a trained image segmentation model to obtain a segmentation mask. One achievable method is to input image data into an image encoder within the image segmentation model, perform feature dimension reduction on the image data, and obtain an image embedding vector; input suggestive data into a hint encoder, and encode the suggestive data into a hint embedding vector; input the image embedding vector and the hint embedding vector into a mask decoder, perform mask prediction, and obtain a segmentation mask.

[0064] Among them, the image encoder performs feature embedding, converting the data into a fixed-size feature representation, where the data is converted to reduce the dimension of the image data, and the feature representation is a vector for easy processing and calculation, such as finding the distance. The main purpose of embedding is to reduce the dimension of sparse features. Its dimensionality reduction method can be compared to a fully connected layer, which reduces the dimension through the weight matrix calculation of the embedding layer. At the highest level, the image encoder generates a one-time image embedding that can be applied before the prompt model.

[0065] The prompt encoder encodes background points, masks, bounding boxes, or text into an embedding vector in real time. Two sets of prompts are considered: sparse (points, boxes, text) and dense (masks). Points and boxes are represented by positional encodings, and learned embeddings are added for each prompt type. Free-form text prompts are represented by an off-the-shelf text encoder from CLIP. Dense prompts, such as masks, are embedded in convolutions and summed element-wise with the image embedding. The multimodal prompt design of the present disclosure adopts prompts such as points, boxes, masks, and text, which improves adaptability to different application scenarios.

[0066] The mask decoder predicts a segmentation mask based on the embeddings from the image and hint encoders. It maps the image embedding, hint embedding, and output token to the mask. All embeddings are updated by the decoder block, which works in two directions: from hint to image embedding and back, using hint self-attention and cross-attention. The masks are annotated and used to update the model weights. This layout enhances the dataset and allows the model to learn and improve over time, making it efficient and flexible.

[0067] It should be noted that the image segmentation model disclosed in the present invention may be a SAM model.

[0068] In some exemplary embodiments of the present disclosure, in the scenarios of boiler status, instrumentation detection, and personnel safety monitoring in power plants, the present disclosure improves boiler operation efficiency, optimizes the accuracy of instrumentation detection, and ensures personnel safety. The specific steps include:

[0069] Step 1: Collect relevant data on power plant boiler status, instrumentation detection, and personnel safety monitoring, including images, videos, sensor data, etc. Label and preprocess the collected data, including image segmentation and labeling of various boiler components, such as water-cooled walls and superheaters; process video data, extract key frames and label them; clean and preprocess sensor data to eliminate noise and outliers. Use the processed data to train the SAM model and build a data engine.

[0070] Step 2: Training the SAM model: Determine the training objectives and tasks, such as detection and identification of boiler components, fault detection of instruments and meters, identification and classification of personnel behavior, etc. Train the SAM model on a large number of diverse masks, and optimize the model parameters through the back propagation algorithm to improve the generalization ability of the model. During the training process, adjust the model's hyperparameters and optimization strategies, such as learning rate, batch size, optimizer, etc., based on the characteristics of the application scenario.

[0071] Step 3: Apply the SAM model for real-time monitoring and early warning: Apply the trained SAM model to the real-time monitoring and early warning system, and process and analyze the input images, videos, and sensor data through the data engine. According to different tasks and scenarios, design appropriate prompts and effective output masks. For example, for the detection and recognition tasks of boiler components, you can design prompts for different components, such as "Is there a leak in the water-cooled wall?", "Is the superheater working properly?", etc. Use the SAM model to classify and predict the input data, extract key information and generate early warning signals, detect abnormal situations in time and notify relevant personnel to handle them. According to the actual application effect, the SAM model is continuously optimized and adjusted to improve the accuracy and stability of the model.

[0072] In some embodiments of the present disclosure, an image segmentation model is used to decouple semantic learning of object classification and component classification. In particular, in a power plant scenario, semantic learning of decoupling object classification and component classification can be applied to improve the recognition and understanding of boiler components, instruments and meters, and personnel behavior. Construct a data set containing object and component classification. For each object category, such as boiler components, instruments and meters, etc., collect multiple images with different perspectives, lighting, and postures, and annotate the location and category of the components in the image. Train an object classifier: Use a conventional convolutional neural network to train an object classifier for classifying objects in an image. A pre-trained model or training from scratch can be used. Train a component classifier: Use a model similar to an object classifier to train a component classifier for classifying components in an image. During the training process, the prediction results of the object classifier can be used as input to help the component classifier better learn the semantics of the component. Decouple object classification from component classification: During the application process, the object classifier and the component classifier are decoupled to classify the input image separately. The object classifier can be used to identify the object category in the image, while the component classifier can be used to identify the location and category of the component in the image. The decoupled object classification and component classification are applied to power plant boiler status, instrumentation detection, and personnel safety monitoring scenarios. The object classifier identifies the categories of objects such as boiler components and instruments, and the component classifier identifies the location and category of the components.

[0073] In some disclosed embodiments, the image segmentation model uses multi-granularity learning. The present disclosure extracts information from different regions for learning based on different granularity levels, decouples object classification and component classification, and has better effects on component-level classification and prediction tasks.

[0074] For example, in a power plant scenario, Many-to-Many multi-granularity learning can be applied to improve the understanding and segmentation effect of image information at different granularities. Construct a dataset containing multi-granularity segmentation information. For each granularity level, collect multiple images with different perspectives, lighting, and postures, and mark the segmentation areas in the image. Perform supervised learning through multi-granularity segmentation information. Different deep learning models can be selected as the basic model. During the application process, the trained model is applied to the real-time monitoring and early warning system to perform multi-granularity segmentation on the input image. According to different granularity levels, information of different regions is extracted and classified and predicted. Fuse multi-granularity information and fuse the segmentation information at different granularities to obtain a more comprehensive and accurate segmentation result. Different fusion strategies can be selected according to actual needs, such as taking the maximum value, average value, or weighted summation. The fused multi-granularity segmentation information is applied to scenarios such as boiler status, instrumentation detection, and personnel safety monitoring in power plants. The status and behavior of different regions can be more accurately identified through segmentation information.

[0075] In some other embodiments of the present disclosure, a mask decoder is fine-tuned using small sample data of actual application scenarios to obtain a fine-tuned image segmentation model.

[0076] For example, extract the basic parts of the SAM model, including the image encoder, the prompt encoder, and the mask decoder. Fine-tune the mask decoder. Extract the weight parameters of the image encoder, the prompt encoder, and the mask decoder. Enable gradient calculation in the mask decoder part so that it can be fine-tuned. Prepare an application scenario dataset for fine-tuning, including drawing segmented images, ground truth masks, and prompt information (such as a bounding box) for the input model. Set a suitable loss function, such as cross entropy loss or Dice loss, to measure the difference between the predicted mask and the true mask. Train the mask decoder using the fine-tuning dataset, and monitor the loss value and performance indicators during the training process. Perform multiple iterative fine-tuning as needed until a satisfactory performance level is achieved. After completing the fine-tuning, save the model weight parameters for use in actual application scenarios. The present disclosure uses small sample data of actual scenes to perform SAM model fine-tuning, retains the architecture of the image encoder and the prompt encoder, optimizes the effect of the mask decoding encoder, and solves the data problem of small sample scenes in a different way.

[0077] In the above method embodiment of the present disclosure, multimodal data is obtained, wherein the multimodal data includes: image data and suggestive data; the multimodal data is input into a trained image segmentation model to obtain a segmentation mask; wherein the image segmentation model includes: an image encoder, a hint encoder and a mask decoder, the image encoder is used to encode the image data into an image embedding vector, the hint encoder is used to encode the suggestive data into a hint embedding vector, and the mask decoder is used to perform mask prediction based on the image embedding vector and the hint embedding vector to obtain the segmentation mask; the present disclosure uses multimodal data for image segmentation, improves the multi-granularity processing capability of the image segmentation model, and improves the accuracy of the image segmentation model in performing image segmentation.

[0078] Figure 3 FIG. 3 is a schematic diagram of the structure of an image segmentation device 30 provided by an exemplary embodiment of the present disclosure. Figure 3 As shown, the image segmentation device 30 includes: an acquisition module 31 and a segmentation module 32.

[0079] The acquisition module 31 is used to acquire multimodal data, wherein the multimodal data includes: image data and prompt data;

[0080] The segmentation module 32 is used to input the multimodal data into the trained image segmentation model to obtain a segmentation mask; wherein the image segmentation model includes: an image encoder, a prompt encoder and a mask decoder, the image encoder is used to encode the image data into an image embedding vector, the prompt encoder is used to encode the suggestive data into a prompt embedding vector, and the mask decoder is used to perform mask prediction based on the image embedding vector and the prompt embedding vector to obtain a segmentation mask.

[0081] Optionally, when the segmentation module 32 inputs the multimodal data into the trained image segmentation model to obtain the segmentation mask, it is used to:

[0082] Inside the image segmentation model, the image data is input into the image encoder, and the image data is subjected to feature dimension reduction to obtain the image embedding vector;

[0083] Inputting the prompt data into a prompt encoder to encode the prompt data into a prompt embedding vector;

[0084] The image embedding vector and the hint embedding vector are input into the mask decoder for mask prediction to obtain the segmentation mask.

[0085] Optionally, an image segmentation model is used to decouple semantic learning of object classification from part classification.

[0086] Optionally, the image segmentation model adopts multi-granularity learning.

[0087] Optionally, the segmentation module 32 may also be used to:

[0088] Small sample data from actual application scenarios are used to fine-tune the mask decoder to obtain a fine-tuned image segmentation model.

[0089] Optionally, the image data includes: at least one of a picture, a video and a point cloud; the suggestive data includes: at least one of a background point, a mask, a bounding box and text.

[0090] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0091] Figure 4 FIG. 1 is a schematic diagram of a structure of an electronic device provided by an exemplary embodiment of the present disclosure. Figure 4 As shown, the electronic device includes: a memory 41 and a processor 42. In addition, the electronic device also includes a power supply component 43 and a communication component 44.

[0092] The memory 41 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application program or method operating on the electronic device.

[0093] The memory 41 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0094] The communication component 44 is used for data transmission with other devices.

[0095] The processor 42 can execute computer instructions stored in the memory 41 to: obtain multimodal data, wherein the multimodal data includes: image data and suggestive data; input the multimodal data into a trained image segmentation model to obtain a segmentation mask; wherein the image segmentation model includes: an image encoder, a hint encoder and a mask decoder, the image encoder is used to encode the image data into an image embedding vector, the hint encoder is used to encode the suggestive data into a hint embedding vector, and the mask decoder is used to perform mask prediction based on the image embedding vector and the hint embedding vector to obtain a segmentation mask.

[0096] Optionally, when the processor 42 inputs the multimodal data into the trained image segmentation model to obtain the segmentation mask, it is used to:

[0097] Inside the image segmentation model, the image data is input into the image encoder, and the image data is subjected to feature dimension reduction to obtain the image embedding vector;

[0098] Inputting the prompt data into a prompt encoder to encode the prompt data into a prompt embedding vector;

[0099] The image embedding vector and the hint embedding vector are input into the mask decoder for mask prediction to obtain the segmentation mask.

[0100] Optionally, an image segmentation model is used to decouple semantic learning of object classification from part classification.

[0101] Optionally, the image segmentation model adopts multi-granularity learning.

[0102] Optionally, the processor 42 may also be configured to:

[0103] Small sample data from actual application scenarios are used to fine-tune the mask decoder to obtain a fine-tuned image segmentation model.

[0104] Optionally, the image data includes: at least one of a picture, a video and a point cloud; the suggestive data includes: at least one of a background point, a mask, a bounding box and text.

[0105] Accordingly, the present disclosure also provides a computer-readable storage medium storing a computer program. When the computer-readable storage medium stores the computer program and the computer program is executed by one or more processors, the one or more processors execute Figure 1 Each step in the method embodiment.

[0106] Accordingly, the present disclosure also provides a computer program product, the computer program product includes a computer program / instruction, the computer program / instruction is executed by a processor Figure 1 Each step in the method embodiment.

[0107] Above Figure 4 The communication component in is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0108] Above Figure 4 The power supply component in the device provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0109] The electronic device also includes a display screen and an audio component.

[0110] The display screen includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0111] The audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0112] In the above-mentioned apparatus, device, storage medium and program product embodiments of the present disclosure, multimodal data is obtained, wherein the multimodal data includes: image data and suggestive data; the multimodal data is input into a trained image segmentation model to obtain a segmentation mask; wherein the image segmentation model includes: an image encoder, a hint encoder and a mask decoder, the image encoder is used to encode the image data into an image embedding vector, the hint encoder is used to encode the suggestive data into a hint embedding vector, and the mask decoder is used to perform mask prediction based on the image embedding vector and the hint embedding vector to obtain the segmentation mask; the present disclosure uses multimodal data for image segmentation, improves the multi-granularity processing capability of the image segmentation model, and improves the accuracy of the image segmentation model in performing image segmentation.

[0113] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0115] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0117] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0118] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0119] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0120] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0121] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image segmentation method, characterized in that: include: Acquiring multimodal data, wherein the multimodal data includes: image data and prompt data; The multimodal data is input into a trained image segmentation model to obtain a segmentation mask; wherein the image segmentation model includes: an image encoder, a prompt encoder and a mask decoder, the image encoder is used to encode the image data into an image embedding vector, the prompt encoder is used to encode the suggestive data into a prompt embedding vector, and the mask decoder is used to perform mask prediction based on the image embedding vector and the prompt embedding vector to obtain the segmentation mask.

2. The method according to claim 1, characterized in that: Inputting the multimodal data into a trained image segmentation model to obtain a segmentation mask includes: Within the image segmentation model, the image data is input into the image encoder, and feature dimension reduction is performed on the image data to obtain an image embedding vector; Inputting the suggestive data into the hint encoder to encode the suggestive data into the hint embedding vector; The image embedding vector and the hint embedding vector are input into the mask decoder to perform mask prediction to obtain the segmentation mask.

3. The method according to claim 1, characterized in that The image segmentation model is used to decouple semantic learning of object classification and component classification.

4. The method according to claim 1, characterized in that: The image segmentation model adopts multi-granularity learning.

5. The method according to claim 1, characterized in that The method further comprises: Small sample data from actual application scenarios are used to fine-tune the mask decoder to obtain a fine-tuned image segmentation model.

6. The method according to claim 1, characterized in that The image data includes: at least one of a picture, a video and a point cloud; the suggestive data includes: at least one of a background point, a mask, a bounding box and a text.

7. An image segmentation device, characterized in that it comprises: An acquisition module, used for acquiring multimodal data, wherein the multimodal data includes: image data and prompt data; A segmentation module is used to input the multimodal data into a trained image segmentation model to obtain a segmentation mask; wherein the image segmentation model includes: an image encoder, a prompt encoder and a mask decoder, the image encoder is used to encode the image data into an image embedding vector, the prompt encoder is used to encode the suggestive data into a prompt embedding vector, and the mask decoder is used to perform mask prediction based on the image embedding vector and the prompt embedding vector to obtain the segmentation mask.

8. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement each step in the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps in the method according to any one of claims 1 to 6 are implemented.

Images