Foreground segmentation method, device and electronic equipment

Abstract

The application discloses a foreground segmentation method and device and electronic equipment, and relates to the field of computer vision. The foreground segmentation method comprises the following steps: acquiring an image to be segmented; inputting the image to be segmented into a preset foreground extraction model to acquire initial mask prompt information of the image to be segmented, wherein the preset foreground extraction model is a model obtained by adding a hierarchical attention mechanism in a UNet network; inputting the initial mask prompt information and the segmented image into an image segmentation large model to obtain a foreground image of the image to be segmented.

Description

Technical Field

[0001] This application relates to the field of computer vision, and more particularly to a foreground segmentation method, apparatus, and electronic device. Background Technology

[0002] Foreground segmentation is a fundamental task in computer vision, aiming to accurately extract foreground objects from images and generate pixel-level masks. It is widely used in fields such as image synthesis, video editing, and augmented reality.

[0003] Foreground segmentation in related technologies includes foreground segmentation based on convolutional neural networks and interactive segmentation based on large models. However, the first segmentation method has poor generalization and low segmentation accuracy due to the small number of model parameters and limited training data. The second segmentation method relies on the quality of human prompts and is difficult to meet the user's demand for intelligent foreground segmentation. Summary of the Invention

[0004] This application provides a foreground segmentation method, apparatus, and electronic device to at least solve the problem of low segmentation accuracy in related technologies.

[0005] In a first aspect, embodiments of this application provide a foreground segmentation method, including: Obtain the image to be segmented; The image to be segmented is input into a preset foreground extraction model to obtain the initial mask cue information of the image to be segmented. The preset foreground extraction model is a model obtained by adding a hierarchical attention mechanism to the UNet network. The initial mask hint information and the segmented image are input into the large image segmentation model to obtain the foreground image of the image to be segmented.

[0006] Secondly, embodiments of this application provide a foreground segmentation apparatus, comprising: The first acquisition module is used to acquire the image to be segmented; The second acquisition module is used to input the image to be segmented into a preset foreground extraction model to obtain the initial mask prompt information of the image to be segmented. The preset foreground extraction model is a model obtained by adding a hierarchical attention mechanism to the UNet network. The segmentation module is used to input the initial mask prompt information and the segmented image into the large image segmentation model to obtain the foreground image of the image to be segmented.

[0007] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method described in the first aspect.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0009] Fifthly, embodiments of this application provide a computer program product stored in a computer-readable storage medium, which is executed by at least one processor to implement the steps of the method described in the first aspect.

[0010] In this embodiment, the image to be segmented is first acquired, then input into a preset foreground extraction model to obtain initial mask prompts. The preset foreground extraction model is a model obtained by adding a hierarchical attention mechanism to the UNet network. Finally, the initial mask prompts and the segmented image are input into a large image segmentation model to obtain the foreground image of the image to be segmented. This embodiment perceives foreground information by combining the UNet network and the hierarchical attention mechanism, which can replace the manual prompting step, reduce manual intervention, and achieve automated segmentation of the image foreground. Compared with related technologies that use user-specified bounding boxes or points, it can improve the model's segmentation accuracy. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of a foreground segmentation method provided in one embodiment of this application; Figure 2 This is a detailed flowchart of a foreground segmentation method provided in one embodiment of this application; Figure 3 This is a schematic diagram of the structure of a foreground segmentation device provided in one embodiment of this application; Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0012] The embodiments of this application will now be described in detail. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0013] The terms "first" and "second" in the specification and claims of this application may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise stated, "multiple" means two or more. Furthermore, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0014] Foreground segmentation methods based on convolutional neural networks (CNNs) in related technologies mainly utilize deep learning frameworks to perform pixel-level segmentation of images. They automatically learn feature representations of the image through multiple convolutional and pooling layers of the CNN. These features are then used to predict the category of each pixel in the image, thereby achieving foreground segmentation. However, due to the limited number of model parameters and training data, these methods suffer from poor generalization and low segmentation accuracy.

[0015] Interactive segmentation methods based on large models, such as SAM (Segment Anything Model), are large models for interactive segmentation that are trained on millions of images and over a billion masks, returning accurate and effective segmentation masks. They segment specific objects by allowing users to specify bounding boxes / points, such as a set of foreground / background points, a coarse bounding box or mask, or free-form text. Their high segmentation accuracy has been applied to image matting techniques. However, these models lack the ability to automatically segment foregrounds, relying excessively on manual prompts, and coarse bounding box / point prompts can lead to inaccurate segmentation results. Because the segmentation effect of these methods depends on the quality of manual prompts, they struggle to meet users' needs for intelligent foreground segmentation.

[0016] Therefore, embodiments of this application provide a foreground segmentation method, apparatus, and electronic device that automatically generates coarse-grained foreground mask prompts for large model segmentation through a foreground extraction network, replacing the manual prompting step, reducing manual intervention, and realizing automated segmentation of image foreground; and compared with the prompting method of user-specified bounding boxes and specified points, coarse-grained mask prompts can further improve the model segmentation accuracy.

[0017] The following is in conjunction with the appendix Figures 1 to 4 As shown, a foreground segmentation method, apparatus, and electronic device provided in this application will be described in detail through specific embodiments and application scenarios.

[0018] Figure 1 This is a flowchart illustrating a foreground segmentation method provided in an embodiment of this application. Figure 1 As shown, the foreground segmentation method may include the contents shown in steps 101 to 103.

[0019] In S101, the image to be segmented is obtained.

[0020] The image to be segmented includes a foreground image and a background image.

[0021] In S102, the image to be segmented is input into the preset foreground extraction model to obtain the initial mask cue information of the image to be segmented. The preset foreground extraction model is a model obtained by adding a hierarchical attention mechanism to the UNet network.

[0022] Among them, UNet (U-shaped Network) is a semantic segmentation architecture based on convolutional neural networks.

[0023] This embodiment uses a preset foreground extraction model to generate coarse-grained foreground mask prompts for large models, enabling accurate segmentation of large models without manual prompts.

[0024] In S103, the initial mask prompt information and the segmented image are input into the large image segmentation model to obtain the foreground image of the image to be segmented.

[0025] Among them, the large-scale image segmentation model can be SAM.

[0026] In this embodiment, the image to be segmented is first acquired, then input into a preset foreground extraction model to obtain initial mask prompts. The preset foreground extraction model is a model obtained by adding a hierarchical attention mechanism to the UNet network. Finally, the initial mask prompts and the segmented image are input into a large image segmentation model to obtain the foreground image of the image to be segmented. This embodiment perceives foreground information by combining the UNet network and the hierarchical attention mechanism, which can replace the manual prompting step, reduce manual intervention, and achieve automated segmentation of the image foreground. Compared with related technologies that use user-specified bounding boxes or points, it can improve the model's segmentation accuracy.

[0027] In one possible implementation of this application, the construction of the preset foreground extraction model may include: using the UNet network as the base network; adding a hierarchical attention mechanism to the base network to obtain the preset foreground extraction model, wherein the preset foreground extraction model is used to guide the low-level semantic feature layer to predict pixel-level segmentation results using a high-level semantic feature layer, and generating initial mask prompt information.

[0028] The pre-defined foreground extraction model in this embodiment is constructed using UNet as the backbone network, with a hierarchical attention mechanism incorporated into the network. A high-level semantic feature layer guides the low-level feature layer to predict pixel-level segmentation results. By employing a model combining UNet and a hierarchical attention mechanism to perceive foreground information, coarse-grained foreground mask cues can be accurately obtained, thereby improving the accuracy of foreground extraction.

[0029] In one possible implementation of this application, the training process of the preset foreground extraction model may include: dividing pre-acquired data into a training set and a test set, and modifying the semantic segmentation annotation information, wherein the annotation information of background pixels is the first annotation information, and the annotation information of foreground pixels is the second annotation information; inputting the training set into the initial foreground extraction model for training, and obtaining a first foreground extraction model under preset conditions; testing the first foreground extraction model using the test set, and obtaining the preset foreground extraction model when the similarity between the model output result and the annotation information of the test set is greater than a similarity threshold.

[0030] The data acquired beforehand is publicly available data, such as the publicly available dataset PASCAL VOC 2012, which is used as the training and testing set for the model. Background pixels can be set to a category of 0, and foreground pixels to a category of 1. Other annotation information can also be set in other embodiments, depending on the actual application; this embodiment does not impose any limitations.

[0031] The similarity threshold can be set according to the actual application, and this embodiment does not impose any limitations.

[0032] This application uses a training set for training and a test set for testing the trained model in order to fine-tune the model and finally obtain a preset foreground extraction model to achieve intelligent segmentation of image foreground. This solves the problem of excessive reliance on manual prompts in the SAM model and significantly improves the segmentation accuracy of the model through dense mask prompts.

[0033] In one possible implementation of this application, the step of inputting the training set into the initial foreground extraction model for training may include: based on a first image, downsampling the first image using the UNet network to obtain a downsampled image; extracting the highest-level semantic feature map of the downsampled image using the UNet network, and using the highest-level semantic feature map as the query vector Q value of the attention mechanism; based on the first image, upsampling the first image using the UNet network to obtain an upsampled image; extracting the low-level feature map of the upsampled image using the UNet network, and using the low-level feature map as the key vector K value and value vector V value of the attention mechanism; determining the foreground-aware feature vector of the first image based on the Q value, the K value, and the V value; and performing a convolution operation on the foreground-aware feature vector to obtain pixel-level segmentation mask prompt information.

[0034] Among them, Q value, K value, and V value are the query vector, key vector, and value vector, respectively.

[0035] In one instance, the foreground-aware feature vector is determined by the following formula:

[0036] in, For attention calculation, This is the highest-level semantic feature map. This is a low-level feature map.

[0037] In this embodiment, for images Extracting images from the Unet network The highest-level semantic feature map after downsampling The Q-value, used as the attention mechanism, is used to extract the upsampled low-level feature map. K and V values, serving as attention mechanisms, are used to calculate foreground perception features. Finally, the foreground-aware feature map Convolutional operations are performed to predict pixel-level segmentation masks.

[0038] Figure 2 A detailed flowchart of a foreground segmentation method provided for one embodiment of this application is shown. Figure 2 As shown, it includes the following steps: Coarse-grained mask cues generated based on a foreground extraction model: Coarse-grained foreground mask cues are generated for large models using a foreground extraction model. The training process for the foreground extraction model is as follows: Training data preparation: The public dataset PASCAL VOC 2012 was used as the training and testing set for the model. The semantic segmentation annotation information was modified so that the pixel category of the background was 0 and the pixel category of the foreground was 1.

[0039] Foreground extraction network construction: Using UNet as the backbone network, a hierarchical attention mechanism is added to the network, utilizing high-level semantic feature layers to guide low-level feature layers in predicting pixel-level segmentation results. Specifically, for images... Extracting images from the Unet network The highest-level semantic feature map after downsampling The Q-value, used as the attention mechanism, is used to extract the upsampled low-level feature map. K and V values, serving as attention mechanisms, are used to calculate foreground perception features. .in, Dimensions and The formula is the same, as follows:

[0040] in, This involves attention calculation, and finally, the foreground-aware feature map. Convolutional operations are performed to predict pixel-level segmentation masks.

[0041] Model Training and Evaluation: Set the model training parameters, use the cross-entropy loss function to train the network, adopt the AdamW method as the optimizer, use the cosine annealing learning rate adjustment method to adjust the learning rate, and use the Intersection over Union (IoU) (also known as the Jacobian coefficient) as the evaluation metric. This metric checks the similarity between the predicted results and the true labels to determine how much area overlaps between the two masks.

[0042] Foreground segmentation in a large model based on coarse-grained mask cues: Input the coarse-grained mask cues and the original image into the SAM large model to achieve foreground segmentation and matting.

[0043] This application's embodiments design and train a foreground extraction network based on a hierarchical attention mechanism to obtain coarse-grained foreground segmentation masks. This network uses UNet as its backbone and incorporates a hierarchical attention mechanism, utilizing a high-level semantic feature layer to guide the low-level feature layer in predicting pixel-level segmentation results. Since the high-level semantic feature layer better reflects the foreground information of the image, this method can better perceive the foreground region, thus achieving more accurate foreground extraction. The foreground extraction network automatically generates coarse-grained foreground mask prompts for large-scale model segmentation, replacing manual prompts, reducing human intervention, and achieving automated foreground segmentation. Furthermore, compared to user-specified bounding boxes or points, coarse-grained mask prompts can further improve the model's segmentation accuracy.

[0044] This application also provides a foreground segmentation apparatus. For example... Figure 3 The diagram shown is a structural schematic of a foreground segmentation device provided in an embodiment of this application. Figure 3 As shown, the foreground segmentation device may include: a first acquisition module 301, a second acquisition module 302, and a segmentation module 303.

[0045] The first acquisition module 301 is used to acquire the image to be segmented; the second acquisition module 302 is used to input the image to be segmented into a preset foreground extraction model to acquire the initial mask prompt information of the image to be segmented, wherein the preset foreground extraction model is a model obtained by adding a hierarchical attention mechanism to the UNet network; and the segmentation module 303 is used to input the initial mask prompt information and the segmented image into the large image segmentation model to obtain the foreground image of the image to be segmented.

[0046] In this embodiment, the first acquisition module 301 first acquires the image to be segmented. Then, the second acquisition module 302 inputs the image to be segmented into a preset foreground extraction model to obtain initial mask prompts for the image to be segmented. The preset foreground extraction model is a model obtained by adding a hierarchical attention mechanism to the UNet network. Finally, the segmentation module 303 inputs the initial mask prompts and the segmented image into a large image segmentation model to obtain the foreground image of the image to be segmented. This embodiment perceives foreground information by combining the UNet network and the hierarchical attention mechanism, which can replace the manual prompting step, reduce manual intervention, and achieve automated segmentation of the image foreground. Compared with the method of user-specified bounding boxes and points in related technologies, it can improve the model segmentation accuracy.

[0047] In one possible implementation of this application, the second acquisition module 302 is used to: use the UNet network as the base network; add a hierarchical attention mechanism to the base network to obtain a preset foreground extraction model, the preset foreground extraction model is used to guide the low-level semantic feature layer to predict the pixel-level segmentation result using the high-level semantic feature layer, and generate initial mask prompt information.

[0048] In one possible implementation of this application, the second acquisition module 302 is configured to: divide the pre-acquired data into a training set and a test set, and modify the semantic segmentation annotation information, wherein the annotation information of the background pixels is the first annotation information and the annotation information of the foreground pixels is the second annotation information; input the training set into the initial foreground extraction model for training, and obtain the first foreground extraction model under the condition of satisfying the preset conditions; test the first foreground extraction model using the test set, and obtain the preset foreground extraction model when the similarity between the model output result and the annotation information of the test set is greater than the similarity threshold.

[0049] In one possible implementation of this application, the second acquisition module 302 is configured to: downsample the first image using a UNet network to obtain a downsampled image; extract the highest-level semantic feature map of the downsampled image using a UNet network, and use the highest-level semantic feature map as the query vector Q value of the attention mechanism; upsample the first image using a UNet network to obtain an upsampled image; extract the low-level feature map of the upsampled image using a UNet network, and use the low-level feature map as the key vector K value and value vector V value of the attention mechanism; determine the foreground-aware feature vector of the first image based on the Q value, K value, and V value; and perform a convolution operation on the foreground-aware feature vector to obtain pixel-level segmentation mask prompt information.

[0050] The foreground segmentation device in this application embodiment can be a system, or a component, integrated circuit, or chip in a terminal. The system can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0051] The foreground segmentation device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0052] The foreground segmentation device provided in this application embodiment can achieve... Figures 1 to 2 The various processes implemented in the method embodiments achieve the same technical effect, and will not be described again here to avoid repetition.

[0053] Figure 4 This is a schematic diagram of the hardware structure of an electronic device according to various embodiments of this application. The electronic device 400 includes, but is not limited to, components such as: a radio frequency unit 401, a network module 402, an audio output unit 403, an input unit 404, a sensor 405, a display unit 406, a user input unit 407, an interface unit 408, a memory 409, a processor 410, and a power supply 411. Those skilled in the art will understand that... Figure 4 The electronic device structures shown are not intended to limit the electronic device. An electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. In the embodiments of this application, the electronic device includes, but is not limited to, mobile phones, tablets, laptops, PDAs, in-vehicle terminals, wearable devices, and pedometers.

[0054] The processor 410 can be used to: acquire an image to be segmented; input the image to be segmented into a preset foreground extraction model to acquire initial mask cue information of the image to be segmented, wherein the preset foreground extraction model is a model obtained by adding a hierarchical attention mechanism to the UNet network; and input the initial mask cue information and the segmented image into a large image segmentation model to obtain the foreground image of the image to be segmented.

[0055] In this embodiment, the image to be segmented is first acquired, then input into a preset foreground extraction model to obtain initial mask prompts. The preset foreground extraction model is a model obtained by adding a hierarchical attention mechanism to the UNet network. Finally, the initial mask prompts and the segmented image are input into a large image segmentation model to obtain the foreground image of the image to be segmented. This embodiment perceives foreground information by combining the UNet network and the hierarchical attention mechanism, which can replace the manual prompting step, reduce manual intervention, and achieve automated segmentation of the image foreground. Compared with related technologies that use user-specified bounding boxes or points, it can improve the model's segmentation accuracy.

[0056] It should be understood that, in this embodiment, the radio frequency unit 401 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink data from the base station and processes it with the processor 410; additionally, it transmits uplink data to the base station. Typically, the radio frequency unit 401 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier, a duplexer, etc. Furthermore, the radio frequency unit 401 can also communicate with networks and other devices through a wireless communication system.

[0057] The electronic device provides users with wireless broadband internet access through network module 402, such as helping users send and receive emails, browse web pages, and access streaming media.

[0058] The audio output unit 403 can convert audio data received by the radio frequency unit 401 or the network module 402 or stored in the memory 409 into audio signals and output them as sound. Furthermore, the audio output unit 403 can also provide audio output related to specific functions performed by the electronic device 400 (e.g., call signal reception sound, message reception sound, etc.). The audio output unit 403 includes a speaker, a buzzer, and a receiver, etc.

[0059] Input unit 404 is used to receive audio or video signals. Input unit 404 may include a graphics processing unit (GPU) 4041 and a microphone 4042. The GPU 4041 processes image data of still images or videos acquired by an image capture device (such as a camera) in video capture mode or image capture mode. The processed image frames can be displayed on display unit 406. The image frames processed by GPU 4041 can be stored in memory 409 (or other storage media) or transmitted via radio frequency unit 401 or network module 402. Microphone 4042 can receive sound and process such sound into audio data. The processed audio data can be converted into a format that can be transmitted to a mobile communication base station via radio frequency unit 401 in telephone call mode.

[0060] The electronic device 400 also includes at least one sensor 405, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 4061 according to the ambient light level, and the proximity sensor can turn off the display panel 4061 and / or backlight when the electronic device 400 is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used to identify the posture of the electronic device (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc. The sensor 405 may also include a fingerprint sensor, pressure sensor, iris sensor, molecular sensor, gyroscope, barometer, hygrometer, thermometer, infrared sensor, etc., which will not be described in detail here.

[0061] The display unit 406 is used to display information input by the user or information provided to the user. The display unit 406 may include a display panel 4061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0062] User input unit 407 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of electronic devices. Specifically, user input unit 407 includes a touch panel 4071 and other input devices 4072. Touch panel 4071, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near touch panel 4071). Touch panel 4071 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 410, which receives and executes commands from the processor 410. In addition, touch panel 4071 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. Besides touch panel 4071, user input unit 407 may also include other input devices 4072. Specifically, other input devices 4072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, joysticks, etc., which will not be described in detail here.

[0063] Furthermore, the touch panel 4071 can cover the display panel 4061. When the touch panel 4071 detects a touch operation on or near it, it transmits the information to the processor 410 to determine the type of touch event. Subsequently, the processor 410 provides corresponding visual output on the display panel 4061 based on the type of touch event. Although in Figure 4 In this embodiment, the touch panel 4071 and the display panel 4061 are two independent components to realize the input and output functions of the electronic device. However, in some embodiments, the touch panel 4071 and the display panel 4061 can be integrated to realize the input and output functions of the electronic device. The specific implementation is not limited here.

[0064] Interface unit 408 serves as an interface for connecting external devices to electronic device 400. For example, external devices may include a wired or wireless headphone port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, and so on. Interface unit 408 can be used to receive input from external devices (e.g., data, power, etc.) and transmit the received input to one or more components within electronic device 400, or it can be used to transmit data between electronic device 400 and external devices.

[0065] The memory 409 can be used to store software programs and various data. The memory 409 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback, image playback, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). Furthermore, the memory 409 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0066] The processor 410 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 409, and by calling data stored in the memory 409, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. The processor 410 may include one or more processing units; preferably, the processor 410 may integrate an application processor and a modem processor. The application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 410.

[0067] The electronic device 400 may also include a power supply 411 (such as a battery) that supplies power to various components. Preferably, the power supply 411 can be logically connected to the processor 410 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system.

[0068] In addition, the electronic device 400 includes some functional modules not shown, which will not be described in detail here.

[0069] This application also provides an electronic device, including a processor 410, a memory 409, and a computer program stored in the memory 409 and executable on the processor 410. When the computer program is executed by the processor 410, it implements the various processes of the above-described foreground segmentation method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here.

[0070] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the foreground segmentation method embodiments described above and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0071] This application provides a computer program product stored in a computer-readable storage medium. The computer program product is executed by at least one processor to implement the various processes of the above-described foreground segmentation method embodiments and achieve the same technical effects. To avoid repetition, it will not be described again here.

[0072] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described foreground segmentation method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0073] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0074] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0075] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0076] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A foreground segmentation method, characterized in that, include: Obtain the image to be segmented; The image to be segmented is input into a preset foreground extraction model to obtain the initial mask hint information of the image to be segmented. The preset foreground extraction model is a model obtained by adding a hierarchical attention mechanism to the UNet network. The initial mask hint information and the segmented image are input into the large image segmentation model to obtain the foreground image of the image to be segmented.

2. The method according to claim 1, characterized in that, The construction of the preset foreground extraction model includes: Use the UNet network as the base network; A hierarchical attention mechanism is added to the basic network to obtain a preset foreground extraction model. The preset foreground extraction model is used to guide the low-level semantic feature layer to predict the pixel-level segmentation results using the high-level semantic feature layer and generate initial mask prompt information.

3. The method according to claim 1, characterized in that, The training process of the preset foreground extraction model includes: The pre-acquired data is divided into training set and test set, and the semantic segmentation annotation information is modified. The annotation information of background pixels is the first annotation information, and the annotation information of foreground pixels is the second annotation information. The training set is input into the initial foreground extraction model for training, and a first foreground extraction model is obtained when the preset conditions are met. The first foreground extraction model is tested using the test set. If the similarity between the model output and the labeled information in the test set is greater than a similarity threshold, a preset foreground extraction model is obtained.

4. The method according to claim 3, characterized in that, The step of inputting the training set into the initial foreground extraction model for training includes: Based on the first image, the first image is downsampled using the UNet network to obtain a downsampled image; The highest-level semantic feature map of the downsampled image is extracted using the UNet network, and the highest-level semantic feature map is used as the query vector Q value of the attention mechanism; Based on the first image, the first image is upsampled using the UNet network to obtain an upsampled image; The UNet network is used to extract low-level feature maps of the upsampled image, and these low-level feature maps are used as the key vector K and value vector V of the attention mechanism. Based on the Q value, the K value, and the V value, the foreground-aware feature vector of the first image is determined; The foreground-aware feature vectors are convolved to obtain pixel-level segmentation mask prompts.

5. A foreground segmentation device, characterized in that, include: The first acquisition module is used to acquire the image to be segmented; The second acquisition module is used to input the image to be segmented into a preset foreground extraction model to obtain the initial mask prompt information of the image to be segmented. The preset foreground extraction model is a model obtained by adding a hierarchical attention mechanism to the UNet network. The segmentation module is used to input the initial mask prompt information and the segmented image into the large image segmentation model to obtain the foreground image of the image to be segmented.

6. The apparatus according to claim 5, characterized in that, The second acquisition module is used for: Use the UNet network as the base network; A hierarchical attention mechanism is added to the basic network to obtain a preset foreground extraction model. The preset foreground extraction model is used to guide the low-level semantic feature layer to predict the pixel-level segmentation results using the high-level semantic feature layer and generate initial mask prompt information.

7. The apparatus according to claim 5, characterized in that, The second acquisition module is used for: The pre-acquired data is divided into training set and test set, and the semantic segmentation annotation information is modified. The annotation information of background pixels is the first annotation information, and the annotation information of foreground pixels is the second annotation information. The training set is input into the initial foreground extraction model for training, and a first foreground extraction model is obtained when the preset conditions are met. The first foreground extraction model is tested using the test set. If the similarity between the model output and the labeled information in the test set is greater than a similarity threshold, a preset foreground extraction model is obtained.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 4.

9. A computer-readable storage medium, characterized in that, include: A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 4.

10. A computer program product, characterized in that, The computer program product is stored in a computer-readable storage medium, and the computer program product is executed by at least one processor to implement the steps of the method as described in any one of claims 1 to 4.

Images