Image processing method, image segmentation model training method and related device

Abstract

The embodiment of the invention provides an image processing method, an image segmentation model training method and related devices, and relates to the technical field of terminals. The image processing method comprises the steps of obtaining a to-be-processed image; masks of one or more objects in the to-be-processed image are obtained according to the image segmentation model; wherein the image segmentation model is deployed in the electronic equipment, when the image segmentation model is trained, a comparative learning loss function is adopted in an edge region of a sample, the edge region comprises edges of one or more target objects in the sample, and setting of the comparative learning loss function meets the condition that features of the edges are close to features of the one or more target objects. The edge features are far away from the background features; the category of one or more target objects in the sample comprises the category of one or more objects in the to-be-processed image. Based on the image segmentation model, more accurate image segmentation can be performed on the to-be-identified image, and a mask with relatively high precision is obtained.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to image processing methods, image segmentation model training methods, and related devices. Background Technology

[0002] Image segmentation is a task in image processing, aiming to extract the region of interest from an image. After segmentation, results such as masks can be obtained. These results can be used for image processing tasks such as image enhancement, image blurring, and image correction.

[0003] However, existing implementations still suffer from poor image segmentation quality, resulting in low mask accuracy after image segmentation. Summary of the Invention

[0004] The image processing method, image segmentation model training method, and related apparatus provided in this application can improve the segmentation quality of images and enhance the accuracy of masks.

[0005] In a first aspect, the image processing method provided in the embodiments of this application is applied to an electronic device, and the method includes:

[0006] The process involves: acquiring the image to be processed; obtaining a mask for one or more objects in the image to be processed based on the image segmentation model; wherein the image segmentation model is deployed on an electronic device; during training the image segmentation model, a contrastive learning loss function is used in the edge regions of the samples; the edge regions include the edges of one or more target objects in the samples; the contrastive learning loss function is set to ensure that the features of the edges are close to the features of one or more target objects, and that the features of the edges are far from the background features; the categories of one or more target objects in the samples include the categories of one or more objects in the image to be processed.

[0007] Thus, during the training of this image segmentation model, a contrastive learning loss function is employed in the edge regions of the samples. This loss function is set to ensure that the features of the target object's edges are close to the target object's features and far removed from the background features. Therefore, after training, this image segmentation model has a relatively accurate ability to extract image features, with the extracted image features showing that the object's edge features are closer to the target object's features. Based on these image features, it can more accurately distinguish whether pixels in the edge regions are object pixels or background pixels, thereby reducing the number of misclassified pixels and improving the mask's accuracy. Therefore, this image segmentation model can perform more accurate image segmentation on the image to be recognized, obtaining a higher-precision mask, which helps improve the accuracy of image processing.

[0008] In one possible implementation, obtaining a mask for one or more objects in the image to be processed based on an image segmentation model includes: processing the image to be processed from a first size to a second size, where the second size is larger than the first size; segmenting the second-sized image to be processed into multiple image regions with overlapping areas using a sliding window; encoding the multiple image regions with overlapping areas according to the encoder in the image segmentation model, merging the encoded features, and processing the merged features according to the decoder in the image segmentation model to obtain a mask for one or more objects in the image to be processed. In this way, the image to be processed can be processed quickly using a sliding window inference method, shortening the inference time of the image segmentation model and obtaining the mask more quickly.

[0009] In one possible implementation, the image segmentation model is trained using multiple positive and multiple negative samples, with the number of negative samples exceeding the number of positive samples. Positive samples are the intersections of edge regions with one or more target object labels, while negative samples are the intersections of edge regions with background labels. This allows for more calculations of the contrastive learning loss function, improving the training effect of contrastive learning, increasing the inference accuracy of the image segmentation model, and consequently improving the accuracy of the mask.

[0010] In one possible implementation, the edge region includes multiple edge pixels calculated using an average pooling algorithm, where each edge pixel is a pixel whose pooling result is not an integer. This allows for convenient calculation of multiple edge pixels and faster determination of the edge region.

[0011] In one possible implementation, the image segmentation model includes N encoders and M decoders. The encoders downsample the input image features, and the decoders upsample the input image features. The encoders are equipped with modules for capturing multi-scale features in the image. In this way, the encoders can acquire multi-scale features of the image to be processed, improving the representation strength of the image features, and the decoders can decode the upsampled image features to obtain a high-precision mask.

[0012] In one possible implementation, each encoder includes at least one block, which is obtained by replacing the attention layer in the transformer block with an inception module. Thus, based on the inception module, the encoder can reduce its computational load while capturing a wider range of local features, improving the representational power of image features and saving computational resources. This, in turn, can improve the accuracy of the mask and reduce the power consumption of the electronic device.

[0013] In one possible implementation, some convolutional kernels in the encoder employ depthwise separable convolution. This way, by using partially depthwise separable convolutional kernels, the computational load during model runtime can be reduced, saving computational resources and power consumption, while minimizing the impact of feature fusion during convolution operations.

[0014] In one possible implementation, the image segmentation model further includes convolutional blocks, pyramid pooling modules, and a mapping head. Training the image segmentation model involves the following steps: inputting the target image into the convolutional blocks to obtain image features with a resolution lower than the target image; inputting the image features sequentially into N encoders for downsampling; inputting the output features of the last encoder into the pyramid pooling module for multi-scale feature fusion, and then inputting them together with the output features of the penultimate encoder into the decoders, which are then processed by M decoders to obtain target image features with the same resolution as the target image; processing the target image features through the mapping head to obtain masks for one or more objects; and adjusting the parameters of the image segmentation model during training according to the contrastive learning loss function until the value of the contrastive learning loss function converges. This allows for effective training of the model using contrastive learning, enabling the trained image segmentation model to classify pixels more accurately, thus improving image segmentation quality and obtaining higher-precision masks.

[0015] Secondly, embodiments of this application provide an image segmentation model training method. The image segmentation model is obtained based on a training model, which includes convolutional blocks, N encoders, M decoders, a pyramid pooling module, and a mapping head. The method includes:

[0016] The target image is input into a convolutional block to obtain image features with a resolution lower than that of the target image. These image features are then sequentially input into N encoders for downsampling. The output features of the last encoder are input into a pyramid pooling module for multi-scale feature fusion, and then combined with the output features of the penultimate encoder before being input into a decoder. M decoders process these features to obtain target image features with the same resolution as the target image. A mapping head processes these target image features to obtain masks for one or more objects. The parameters of the model to be trained are adjusted according to the contrastive learning loss function until the value of the contrastive learning loss function converges, resulting in an image segmentation model. This allows for effective training of the model using contrastive learning, enabling the trained image segmentation model to classify pixels more accurately, thus improving image segmentation quality and yielding higher-precision masks.

[0017] In one possible implementation, when adjusting the parameters of the model to be trained based on the contrastive learning loss function, multiple positive samples and multiple negative samples are used, with the number of negative samples greater than the number of positive samples. Positive samples are the intersections of the edge regions of the target image and one or more target object labels in the target image, while negative samples are the intersections of the edge regions of the target image and the background labels of the target image. This allows for more calculations of the contrastive learning loss function, improving the training effect of contrastive learning, increasing the inference accuracy of the image segmentation model, and consequently improving the accuracy of the mask.

[0018] In one possible implementation, the edge region includes multiple edge pixels calculated using an average pooling algorithm, where each edge pixel is a pixel whose pooling result is not an integer. This allows for convenient calculation of multiple edge pixels and faster determination of the edge region.

[0019] In one possible implementation, each encoder includes at least one block, which is obtained by replacing the attention layer in the transformer block with an inception module. Thus, based on the inception module, the encoder can reduce its computational load while capturing a wider range of local features, improving the representational power of image features and saving computational resources. This, in turn, can improve the accuracy of the mask and reduce the power consumption of the electronic device.

[0020] In one possible implementation, some convolutional kernels in the encoder employ depthwise separable convolution. This way, by using partially depthwise separable convolutional kernels, the computational load during model runtime can be reduced, saving computational resources and power consumption, while minimizing the impact of feature fusion during convolution operations.

[0021] Thirdly, embodiments of this application provide an image processing apparatus, which may be an electronic device, a chip or chip system within an electronic device. The apparatus may include a processing unit. The processing unit is used to implement any processing-related method executed by the electronic device in the first aspect or any possible implementation of the first aspect. When the apparatus is an electronic device, the processing unit may be a processor. The apparatus may further include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to cause the electronic device to implement the method described in the first aspect or any possible implementation of the first aspect. When the apparatus is a chip or chip system within an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit to cause the electronic device to implement the method described in the first aspect or any possible implementation of the first aspect. The storage unit may be a storage unit within the chip (e.g., a register, cache, etc.), or a storage unit located outside the chip within the electronic device (e.g., a read-only memory, random access memory, etc.).

[0022] For example, a processing unit is used to acquire an image to be processed; obtain a mask of one or more objects in the image to be processed according to an image segmentation model; wherein, the image segmentation model is deployed on an electronic device, and when training the image segmentation model, a contrastive learning loss function is used in the edge region of the sample, the edge region includes the edge of one or more target objects in the sample, and the contrastive learning loss function is set to satisfy that the features of the edge are close to the features of one or more target objects, and the features of the edge are far away from the background features; the category of one or more target objects in the sample includes the category of one or more objects in the image to be processed.

[0023] In one possible implementation, the processing unit is used to process the image to be processed from a first size to a second size, the second size being larger than the first size; to segment the image to be processed at the second size into multiple image regions with overlapping areas using a sliding window; to encode the multiple image regions with overlapping areas according to the encoder in the image segmentation model, and to merge the encoded features; and to process the merged features according to the decoder in the image segmentation model to obtain a mask of one or more objects in the image to be processed.

[0024] In one possible implementation, multiple positive samples and multiple negative samples are used when training the image segmentation model, and the number of negative samples is greater than the number of positive samples. The positive samples are the intersection of the edge region and one or more target object labels, and the negative samples are the intersection of the edge region and the background label.

[0025] In one possible implementation, the edge region includes multiple edge pixels calculated using an average pooling algorithm, wherein the edge pixels are pixels whose pooling result is not an integer.

[0026] In one possible implementation, the image segmentation model includes N encoders and M decoders. The encoders downsample the input image features, and the decoders upsample the input image features. The encoders are equipped with modules for capturing multi-scale features in the image.

[0027] In one possible implementation, each encoder includes at least one block, which is obtained by changing the attention layer in the transformer block to the inception module.

[0028] In one possible implementation, some convolutional kernels in the encoder employ depth-separable convolution.

[0029] In one possible implementation, the image segmentation model further includes convolutional blocks, pyramid pooling modules, and a mapping head. Training the image segmentation model involves the following steps: inputting the target image into the convolutional blocks to obtain image features with a resolution lower than the target image; inputting the image features sequentially into N encoders for downsampling; inputting the output features of the last encoder into the pyramid pooling module for multi-scale feature fusion, and then inputting them together with the output features of the penultimate encoder into the decoders, which are then processed by M decoders to obtain target image features with the same resolution as the target image; processing the target image features through the mapping head to obtain masks for one or more objects; and adjusting the parameters of the image segmentation model during training according to the contrastive learning loss function until the value of the contrastive learning loss function converges. This allows for effective training of the model using contrastive learning, enabling the trained image segmentation model to more accurately classify pixels, thus improving image segmentation quality and obtaining higher-precision masks.

[0030] Fourthly, embodiments of this application provide an image segmentation model training apparatus. This apparatus can be an electronic device, or a chip or chip system within an electronic device. The apparatus may include a processing unit. The processing unit is used to implement any processing-related method executed by the electronic device in the second aspect or any possible implementation of the second aspect. When the apparatus is an electronic device, the processing unit may be a processor. The apparatus may also include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to cause the electronic device to implement the method described in the second aspect or any possible implementation of the second aspect. When the apparatus is a chip or chip system within an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit to cause the electronic device to implement the method described in the second aspect or any possible implementation of the second aspect. The storage unit may be a storage unit within the chip (e.g., a register, cache, etc.), or a storage unit located outside the chip within the electronic device (e.g., read-only memory, random access memory, etc.).

[0031] For example, the image segmentation model is obtained based on a training model, which includes a convolutional block, N encoders, M decoders, a pyramid pooling module, and a mapping head. A processing unit is used to input the target image into the convolutional block to obtain image features with a resolution lower than the target image. These image features are then sequentially input into the N encoders for downsampling. The output features of the last encoder are input into the pyramid pooling module for multi-scale feature fusion, and then combined with the output features of the penultimate encoder and input into the decoders. The M decoders then process these features to obtain target image features with the same resolution as the target image. The mapping head processes the target image features to obtain masks for one or more objects. The parameters of the training model are adjusted according to the contrastive learning loss function until the value of the contrastive learning loss function converges, thus obtaining the image segmentation model.

[0032] In one possible implementation, when adjusting the parameters of the model to be trained according to the contrastive learning loss function, multiple positive samples and multiple negative samples are used, and the number of negative samples is greater than the number of positive samples. The positive samples are the intersection of the edge region of the target image and one or more target object labels of the target image, and the negative samples are the intersection of the edge region of the target image and the background label of the target image.

[0033] In one possible implementation, the edge region includes multiple edge pixels calculated using an average pooling algorithm, wherein the edge pixels are pixels whose pooling result is not an integer.

[0034] In one possible implementation, each encoder includes at least one block, which is obtained by changing the attention layer in the transformer block to the inception module.

[0035] In one possible implementation, some convolutional kernels in the encoder employ depth-separable convolution.

[0036] Fifthly, embodiments of this application provide an electronic device including a processor and a memory, the memory for storing code instructions, and the processor for running the code instructions to perform the methods described in the first aspect or any possible implementation of the first aspect, or to perform the methods described in the second aspect or any possible implementation of the second aspect.

[0037] Sixthly, embodiments of this application provide a computer-readable storage medium storing a computer program or instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect or any possible implementation thereof, or cause the computer to perform the method described in the second aspect or any possible implementation thereof.

[0038] In a seventh aspect, embodiments of this application provide a computer program product including a computer program, which, when run on a computer, causes the computer to perform the method described in the first aspect or any possible implementation thereof, or causes the computer to perform the method described in the second aspect or any possible implementation thereof.

[0039] Eighthly, this application provides a chip or chip system including at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a circuit. The at least one processor is used to run computer programs or instructions to perform the methods described in the first aspect or any possible implementation thereof, or to perform the methods described in the second aspect or any possible implementation thereof. The communication interface in the chip can be an input / output interface, pins, or circuits, etc.

[0040] In one possible implementation, the chip or chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).

[0041] It should be understood that the third to eighth aspects of this application correspond to the technical solutions of the first and / or second aspects of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0043] Figure 2 A software structure block diagram of an electronic device provided in an embodiment of this application;

[0044] Figure 3 A schematic diagram of the structure of a model provided in an embodiment of this application;

[0045] Figure 4 A schematic diagram illustrating the training process of the model to be trained provided in an embodiment of this application;

[0046] Figure 5 A schematic diagram of sliding window inference provided in an embodiment of this application;

[0047] Figure 6 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0048] Figure 7 A flowchart illustrating an image segmentation model training method provided in an embodiment of this application;

[0049] Figure 8 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0050] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0051] 1. Image Segmentation: Image segmentation refers to separating one or more objects from an image. The object is the target to be segmented from the image; it can also be called foreground, region of interest, region of concern, or target, etc. Object categories can include various types, such as people, animals, plants, vehicles, buildings, sky, or water. Regions outside the objects in the image are non-objects; these can also be called background, non-region of interest, non-target, etc. After image segmentation, a segmentation result is obtained, which can be used to perform downstream image processing tasks, such as image enhancement, image blurring, and image correction.

[0052] Mask: A mask is a segmentation result obtained after image segmentation. It identifies which pixels in an image belong to objects and which do not. Masks are also called segmentation masks, segmentation maps, or mask images. After obtaining the mask for an image, downstream tasks can perform image processing such as enhancement, blurring, or correction based on the mask.

[0053] 2. Terminology

[0054] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with substantially the same function and purpose. For example, "first chip" and "second chip" are used only to distinguish different chips and do not limit their order of execution. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0055] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0056] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.

[0057] 3. Electronic equipment

[0058] The electronic device in this application embodiment can also be any form of terminal device. For example, the electronic device may include: mobile phone, tablet computer, handheld computer, laptop computer, mobile internet device (MID), virtual reality (VR) device, augmented reality (AR) device, wireless terminal in industrial control, wireless terminal in self-driving, wireless terminal in remote medical surgery, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, cellular phone, cordless phone, session initiation protocol (SIP) phone, wireless local loop (WLL) station, personal digital assistant (PDA), handheld device with wireless communication function, computing device or other processing device connected to a wireless modem, vehicle-mounted device, electronic device in 5G network, or future evolved public land mobile communication network. The embodiments of this application do not limit the scope of electronic devices in a network (PLMN).

[0059] By way of example and not limitation, in this embodiment, the electronic device can also be a wearable device. Wearable devices, also known as wearable smart devices, are a general term for devices that utilize wearable technology to intelligently design and develop everyday wearables, such as glasses, gloves, watches, clothing, and shoes. Wearable devices are portable devices that are worn directly on the body or integrated into the user's clothing or accessories. Wearable devices are not merely hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are feature-rich, large in size, and can achieve complete or partial functions without relying on a smartphone, such as smartwatches or smart glasses, as well as those that focus on a specific type of application function and require the use of other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0060] Furthermore, in this application embodiment, the electronic device can also be an electronic device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0061] The electronic equipment in the embodiments of this application may also be referred to as: user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.

[0062] In this embodiment, the electronic device or various network devices include a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on top of the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, instant messaging software, cameras, and / or image processing.

[0063] Some electronic devices support image processing tasks such as taking photos, recording videos, and / or enhancing images stored on the device. Different image processing scenarios may involve image segmentation, which can include portrait segmentation, skin segmentation, and scene semantic segmentation.

[0064] For example, when a user takes a photo with their phone, they may be offered portrait mode and / or professional mode. These modes often require image processing such as portrait enhancement, portrait blurring, artificial intelligence dynamic range correction (AIDRC), and / or background blurring. Therefore, in this scenario, portrait segmentation is necessary to perform post-processing on the segmented areas to obtain the final photo. Alternatively, users can select photos from their gallery and apply image processing such as portrait enhancement, portrait blurring, and / or background blurring to the selected photos. This also requires portrait segmentation of the image and processing based on the segmented areas.

[0065] In one implementation, image segmentation yields a corresponding mask. Higher segmentation accuracy results in a more precise mask, while lower segmentation accuracy leads to a less precise mask. The precision of the mask can affect the image processing performance of downstream tasks.

[0066] For example, in image processing tasks involving portrait enhancement, if the mask obtained after portrait segmentation contains many misclassified pixels at the junction of the portrait and the surrounding background (e.g., pixels belonging to the portrait are classified as background, or vice versa), the mask accuracy at the junction of the portrait and the background will be low. During portrait enhancement, pixel enhancement based on a low-precision mask will result in portrait pixels that should be enhanced not being enhanced, while background pixels that should not be enhanced are enhanced, leading to poor portrait enhancement results. Therefore, the accuracy of the mask has a significant impact on the image processing performance of downstream tasks.

[0067] In related implementations, a machine learning neural network model can be used to segment the image to be processed, thereby obtaining a mask for the image. For example, based on a training set of sample images, an initial image segmentation model can be trained to obtain an image segmentation model capable of segmenting images. Inputting the image to be processed into the trained image segmentation model can output a mask for that image. However, during the training of the initial image segmentation model, no targeted training was performed on the edge regions of objects in the sample images. Therefore, the mask obtained based on this model may contain many misclassified pixels among the edge pixels of objects, resulting in low mask accuracy.

[0068] In view of this, embodiments of this application provide an image processing method, an image segmentation model training method, and related apparatus. In the image processing method, a mask for the image to be processed is obtained based on the trained image segmentation model. During the training of the image segmentation model, a contrastive learning loss function is used in the edge regions of the samples. This contrastive learning loss function is set to ensure that the features of the target object's edges are close to the target object's features and far from the background features. Therefore, after training, the image segmentation model has the ability to accurately extract image features, and the extracted image features show that the features of the object's edges are closer to the object's features. Based on these image features, it is possible to more accurately distinguish whether pixels in the edge regions are object pixels or background pixels, thereby reducing the number of misclassified pixels and improving the accuracy of the mask. Therefore, based on this image segmentation model, more accurate image segmentation can be performed on the image to be recognized, resulting in a higher-precision mask, which helps to improve the accuracy of image processing.

[0069] The image processing method provided in this application can be implemented using an electronic device. To facilitate understanding of the image processing method provided in this application, the hardware structure of the electronic device provided in this application is described below.

[0070] For example, Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0071] The electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0072] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may include hardware, software, or a combination of software and hardware.

[0073] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0074] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0075] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the aforementioned memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0076] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface, and / or a USB interface, etc.

[0077] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a limitation on the structure of the electronic device. In other embodiments of this application, the electronic device may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0078] The camera 193 can be used to capture still images or videos. In some embodiments, the electronic device may include at least one camera 193.

[0079] For example, in this embodiment of the application, at least one camera 193 can be used to capture images to obtain at least one image to be processed for image segmentation. For any image to be processed, the image to be processed is input into an image segmentation model, which can segment one or more objects in the image to be processed, obtaining a mask corresponding to the image to be processed. Downstream image processing tasks can also be performed based on the mask.

[0080] Internal memory 121 can be used to store executable program code, including instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as image capture, image editing, etc.). The data storage area may store data created during the use of the electronic device (such as image data, video streams, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of the electronic device by running instructions stored in internal memory 121 and / or instructions stored in memory located within the processor.

[0081] For example, in this embodiment of the application, by running executable program code related to the image processing method provided in this embodiment of the application through processor 110, image segmentation of the image to be processed can be achieved to obtain a mask corresponding to the image to be processed. The mask may include one or more objects. When a mask includes multiple objects, the objects may be objects of the same category or objects of different categories. For example, when a mask includes objects of multiple categories, some objects may be people, and some objects may be vehicles, etc.

[0082] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. In some embodiments, an electronic device may include at least one display screen 194. The electronic device implements display functions through a GPU, display screen 194, and application processor, etc. The GPU is a microprocessor for image processing, connected to display screen 194 and application processor.

[0083] The electronic device implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 194 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information. The electronic device can implement shooting functions through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0084] Figure 2This is a software architecture block diagram of an electronic device provided in an embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android operating system can be divided into five layers, from top to bottom: applications, application framework, hardware abstraction layer (HAL), kernel, and hardware layer.

[0085] The application layer can include a series of application packages. The application layer runs applications by calling the application programming interface (API) provided by the application framework layer. For example... Figure 2 As shown, the application package can include applications such as camera and gallery. Applications can include system applications and third-party applications.

[0086] The application framework layer provides APIs and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example... Figure 2 As shown, the application framework layer can include a camera access interface and a view system. The camera access interface provides the application programming interface and framework for the camera application. The view system is responsible for rendering the application's interface and handling events. The view system can include visual controls, such as various controls for taking photos.

[0087] like Figure 2 As shown, the HAL layer may include a camera hardware abstraction layer and a camera algorithm library. The camera hardware abstraction layer can provide virtual hardware for the camera device. The camera algorithm library may include runtime code and data implementing the image processing methods provided in the embodiments of this application. For example, the image segmentation model of the embodiments of this application can be deployed in the camera algorithm library for image segmentation processing of the image to be processed, to obtain masks of one or more objects in the image to be processed.

[0088] The kernel layer is the layer between hardware and software. For example... Figure 2 As shown, this kernel layer may include one or more of the following: camera device driver, digital signal processor driver, and graphics processor driver. The camera device driver is used to drive the camera's sensor to acquire images. The digital signal processor driver is used to drive the digital signal processor to process images. The graphics processor driver is used to drive the graphics processor to process images.

[0089] The hardware layer may include hardware such as cameras, displays, digital signal processors, and graphics signal processors.

[0090] It should be understood that in some embodiments, layers that perform the same function may be called by other names, or layers that can perform the functions of multiple layers may be considered as one layer, or layers that can perform the functions of multiple layers may be divided into multiple layers. This application does not impose such limitations.

[0091] It should be noted that the embodiments of this application are only illustrated using the Android operating system as an example. In other operating systems (such as Windows operating system, iOS operating system, etc.), as long as the functions implemented by each functional module are similar to those in the embodiments of this application, the solution of this application can also be implemented.

[0092] The following is in conjunction with the above. Figure 2 The software structure shown below provides a detailed description of the implementation of the image processing method in the photography scenario in this application embodiment:

[0093] In response to a user's action of opening the camera application, such as clicking the camera application icon, the camera application calls the camera access interface of the application framework layer to launch the camera application, and then sends a command to start the camera by calling the camera hardware abstraction layer. The camera hardware abstraction layer sends this command to the camera device driver in the kernel layer. The camera device driver can then start the corresponding camera to acquire images. After processing such as image stabilization and fusion, a color image can be obtained. By inputting the color image into the image segmentation model deployed in the camera algorithm library, the image processing method provided in this application embodiment can be executed on the color image to obtain the mask corresponding to the color image.

[0094] In some implementations, the camera algorithm library may also include image processing algorithms capable of blurring, enhancing, and correcting images. After obtaining the mask corresponding to the color image based on the image segmentation model, the corresponding image processing algorithms in the camera algorithm library can be executed to perform image processing such as blurring, enhancing, and correcting on the color image based on the mask. The camera algorithm library can transmit the processed image to the camera hardware abstraction layer. The camera hardware abstraction layer can then transmit the processed image to the display screen for display.

[0095] It is understood that, in the embodiments of this application, in addition to the camera algorithm library, the electronic device may also deploy the image segmentation model in other locations, without limitation.

[0096] The image segmentation model in this application embodiment can be a model obtained after training a training model. The training model and the image segmentation model have the same structure, the main difference being the different model parameters. The parameters of the training model are the initial parameters, while the parameters of the image segmentation model are the parameters obtained after iterative optimization of the initial parameters. The parameters can be understood as the weight parameters of each module and each network layer in the model, such as convolutional kernel weights.

[0097] For example, Figure 3 This is a schematic diagram of the structure of a model provided in an embodiment of this application, such as... Figure 3 As shown, the model's structure is based on an encoder-decoder architecture, employing a structure similar to the U-net convolutional network (Unet). Both the training model and the image segmentation model can adopt this structure.

[0098] The following description uses the training process from the model to be trained to the image segmentation model as an example to illustrate the modules and network layers in the model structure. It can be understood that the modules and network layers in the model to be trained and the image segmentation model have the same structure, function and effect.

[0099] The model to be trained may include convolutional blocks, an encoder, a decoder, a pyramid pooling module (PPM), and a projection head. The encoder and decoder can also be called encoder modules and decoder modules, respectively.

[0100] After inputting the target image into the model to be trained, features can be extracted from the target image using convolutional blocks to obtain the image features corresponding to the target image. The target image can be any sample used for training the model; a sample can also be understood as a sample image.

[0101] In one possible implementation, several sample images can be obtained from an open-source image set to construct a training set. When any sample image from the training set is input into the model to be trained, a target image is input. The image features corresponding to the target image can be features that describe the image information of the target image; these image features can be understood as the feature map of the target image.

[0102] For example, a convolutional block can be two contiguous convolutional blocks. The target image is input into the first convolutional block, where convolutional layers perform convolution operations on the target image to extract pixel features, resulting in the image features output by the first convolutional block. These image features are then input into the second convolutional block, where convolutional layers further extract pixel features, resulting in the image features output by the second convolutional block. These image features can be the image features corresponding to the target image.

[0103] In one possible implementation, the model parameters of two contiguous convolutional blocks can be the same or different. For example, the stride of the convolutional layers in the first convolutional block can be 2, and the stride of the convolutional layers in the second convolutional block can be 1. In this way, when the first convolutional block extracts features through convolution operations, it can downsample the target image once, obtaining image features with a resolution half that of the target image. Assuming the target image resolution is 1000*1000 pixels, the resolution of the image features obtained after this convolutional block will be half that of the target image; this can be understood as the output resolution becoming half the input resolution.

[0104] Image resolution can also be understood as the image's size or dimensions, representing the number of pixels in the image. Image resolution can also correspond to the resolution of the image features, which represents the number of pixels corresponding to those features.

[0105] In this embodiment, downsampling of the target image based on two convolutional blocks connected end to end can reduce the computational load of subsequent network layers in the model when performing image feature operations, thus saving computing resources. Furthermore, feature extraction during downsampling can achieve the fusion of spatial features of the target image, thereby improving the representational strength of image features.

[0106] Since the span of the convolutional layer in the subsequent convolutional block is 1, it does not change the resolution of the image features during feature extraction. It should be understood that if the span of the convolutional layer in the subsequent convolutional block is set to 2, then the subsequent convolutional block will also perform downsampling during feature extraction. The resolution of the image features output from this convolutional block will then be 1 / 4 of the target image's resolution. Because the resolution of these image features is relatively small, the number of times the encoder can encode these image features will be limited.

[0107] In one possible implementation, the number of convolutional blocks can be one. This fewer number simplifies the structure of the model being trained, resulting in a lightweight image segmentation model. Alternatively, the number of convolutional blocks can be any integer greater than two. Increasing the number of convolutional blocks increases the number of model parameters, improving the modeling ability and segmentation accuracy. Multiple convolutional blocks can extract image features multiple times from the target image, resulting in more powerful image features.

[0108] In one possible implementation, the convolutional block may also include a batch normalization layer and an activation layer. The batch normalization layer can normalize the feature values ​​of the image features obtained after the convolution operation along the batch dimension. The activation layer can perform non-linear processing on the feature values ​​of the batch-normalized image features.

[0109] In one possible implementation, before inputting the target image into the convolutional block, image preprocessing operations such as resizing and standardization can be performed on the target image. Resizing the target image allows its size to be adapted to the input image size requirements of the model being trained. Standardization involves subtracting the mean from the pixel values ​​and then dividing by the variance, thus narrowing the pixel values ​​from a large range to a smaller range. Inputting the standardized target image into the model for training can improve the stability of the model training.

[0110] like Figure 3 As shown, after obtaining the image features of the target image, these features can be sequentially input into five encoders (encoders 1 through 5) for downsampling. For example, each encoder can downsample the image features to half the input resolution. For instance, the output image features of the next convolutional block have a resolution of half the target image resolution. Inputting this image feature into encoder 1 for downsampling results in an output feature with a resolution of 1 / 4 of the target image resolution. Inputting this output feature into encoder 2 for downsampling results in an output feature with a resolution of 1 / 8 of the target image resolution. Inputting this output feature into encoder 3 for downsampling results in an output feature with a resolution of 1 / 16 of the target image resolution. Inputting this output feature into encoder 4 for downsampling results in an output feature with a resolution of 1 / 32 of the target image resolution. Inputting this output feature into encoder 5 for downsampling results in an output feature with a resolution of 1 / 64 of the target image resolution. After five encoders, five different resolution output features are obtained, which are used for the decoding operation of the decoder.

[0111] In one possible implementation, each encoder may include n blocks, which can also be understood as encoder layers, where n is any integer greater than or equal to 1. An encoder layer can be a network structure capable of encoding. For example, n can take any value from n1 to n5, where n1 to n5 gradually increase. For example, n1 = 2, n5 = 8, etc. The number of encoder layers in any two encoders can be the same or different. Figure 3 In the diagram, encoder 1 includes n1 encoder layers, encoder 2 includes n2 encoder layers, encoder 3 includes n3 encoder layers, and so on, which will not be elaborated further.

[0112] It is evident that the number of encoder layers in later encoders can be greater than that in earlier encoders. This is understandable because the resolution of the output features gradually decreases as they are passed from earlier to later encoders. Having more encoder layers in earlier encoders increases their computational load; therefore, earlier encoders can control this load by using fewer encoder layers. While the output features of later encoders have lower resolution, having more encoder layers only slightly increases computational load. However, increasing the number of encoder layers improves encoding capability, enhancing the representational power of the output features from later encoders, which is beneficial for improving the accuracy of model predictions, i.e., improving the accuracy of the obtained mask.

[0113] In one possible implementation, the encoder layer in the encoder can be an encoder layer that includes a transformer model. Alternatively, the encoder layer in the encoder can be an encoder layer that includes a modified transformer. Alternatively, encoders within the same model can include encoder layers of the same or different types, without limitation.

[0114] For example, the attention layer in a transformer architecture performs computationally intensive matrix operations, resulting in high computational overhead. Furthermore, the attention layer includes some activation layers that are not quantization-friendly, which may lead to a decrease in prediction accuracy. Therefore, the attention layer in the transformer architecture can be replaced with an inception module, resulting in an improved transformer encoder layer. The inception module can include a compression and activation network employing a channel attention mechanism.

[0115] For example, the inception module can capture features at different scales by using convolutional kernels of different sizes (e.g., 1×1, 3×3, 5×5) and pooling operations (e.g., 3×3 max pooling) in parallel.

[0116] For example, an Inception module might include the following branches: a 1×1 convolutional kernel for reducing dimensionality and computational complexity; a 3×3 convolutional kernel for capturing medium-scale features; a 5×5 convolutional kernel for capturing larger-scale features; and 3×3 max pooling for capturing local spatial information. The outputs of these branches are concatenated along the channel dimension to obtain the output of the Inception module. This Inception module can extract features at different scales by using convolutional kernels of different sizes in parallel. The 1×1 convolutions reduce dimensionality and computational complexity.

[0117] The Inception module offers high flexibility by allowing users to adjust the parameters and structure of different branches according to specific tasks and dataset requirements. For example, the number of branches in a structure within the Inception module can be preset. For instance, the number of branches in a structure within the Inception module can be set to four. Optionally, the receptive field of the convolution corresponding to each branch can be set based on the resolution of the input image. For example, a larger receptive field can be set if the input image resolution is high, and a smaller receptive field can be set if the input image resolution is low. The Inception module can also be versions of Inception, such as Inception v2, Inception v3, and Inception v4.

[0118] Based on this improved transformer encoder layer, the encoder can reduce the computational cost of the encoder while capturing a wider range of local features. It can also reduce the impact of activation layers that are not quantized well. Therefore, based on this improved transformer encoder layer, the overall computational cost of the model can be reduced and the accuracy of the output mask can be improved.

[0119] like Figure 3 As shown, after the last encoder (encoder 5) encodes the output features, the output features can be input into the pyramid pooling module for multi-scale feature fusion to obtain the pooled output features.

[0120] In one possible implementation, the pyramid pooling module can set multiple pooling windows of different scales to perform multi-scale pooling operations. For example, pooling windows of different sizes such as 1×1, 2×2, 3×3, and 5×5 can be used to pool the output features obtained from the last encoder, resulting in pooled features corresponding to each size pooling window. This can be understood as each pooling window generating a corresponding feature map. The feature maps generated by pooling at different scales are then upsampled (e.g., by bilinear interpolation) to the same size as the output features of the last encoder. The upsampled feature maps are then concatenated along the channel dimension to achieve feature fusion of feature maps of different granularities, forming a feature representation containing multi-scale information. This concatenated feature representation can be input into subsequent network layers or other processing modules for further feature extraction and processing. Here, the concatenated feature representation can be understood as the pooled output features.

[0121] Through the processing of the pyramid pooling module, pooling operations can be performed at different scales to capture global and local information of the target image, while satisfying the control of overall semantic characteristics and detailed semantic features, thereby improving the model's receptive field and feature expression ability, and improving the accuracy of image segmentation.

[0122] The image segmentation model in this application is an improvement on the transformer, and combines network units such as depthwise separable convolution, compression and activation networks, and pyramid pooling modules with the transformer network structure. After training, an image segmentation network with strong feature fitting ability and low computational cost can be obtained.

[0123] like Figure 3 As shown, after obtaining the pooled output features, the pooled output features and the output features of the penultimate encoder (encoder 4) can be input into the decoder, and decoder 5 to decoder 1 will perform decoding processing one by one until the target image features with the same resolution as the target image are obtained.

[0124] In one possible implementation, each decoder may include m blocks, where each block can be understood as a decoder layer, and m is any integer greater than or equal to 1. A decoder layer can be a network structure capable of decoding. For example... Figure 3 In the process, each of decoders 5 to decoder 1 can include one decoder layer. Since the number of decoder layers is small, the structure of the decoder can be simplified and the amount of computation during decoding can be reduced.

[0125] For example, the structure of the decoder layer can be similar to that of the encoder layer described above. For instance, the decoder layer can be a decoder layer that includes a transformer and / or a modified transformer. The decoder layer with the modified transformer can be similar to the encoder layer with the modified transformer described above.

[0126] For example, the pooled output features and the output features of encoder 4 are input into decoder 5. The decoder layer in decoder 5 performs feature fusion on the two input features to achieve the decoding processing of the encoder's output features and obtain the corresponding decoded features.

[0127] The resolution of the pooled output features may differ from that of the encoder 4 output features. Therefore, before feature fusion, the pooled output features can be transformed and upsampled in decoder 5 to achieve the same resolution as the encoder 4 output features. Then, feature fusion is performed to obtain the decoded features output by decoder 5. The feature transformation performed on the pooled output features can be an internal feature fusion operation, such as a convolution operation.

[0128] After obtaining the decoded features output by decoder 5, these features, along with the output features of encoder 3, can be input into decoder 4. The decoder layer in decoder 4 then performs feature fusion on the two input features to obtain the decoded features output by decoder 4. During this process, a feature transformation and upsampling can be performed on the decoded features output by decoder 5 to make them have the same resolution as the output features of encoder 3. Other decoders follow a similar process and will not be described further.

[0129] The decoding features output by each decoder and the output features output by the corresponding encoder are fused in the same manner as described above until the target image features with the same resolution as the target image are obtained.

[0130] It should be noted that, as Figure 3 As shown, after decoder 2 outputs the corresponding decoded features, it can be input into decoder 1 along with the image features output by the next convolutional block for feature fusion. Decoder 1 can include two connected decoder layers. One decoder layer can fuse the decoded features output by decoder 2 with the image features output by the next convolutional block to obtain fused features. The other decoder layer can upsample the fused features to make their resolution the same as the target image. Therefore, decoder 1 can output target image features with the same resolution as the target image.

[0131] For example, the pooled output features (resolution of 1 / 64 of the target image resolution) input to decoder 5 are transformed and upsampled (resolution changed to 1 / 32 of the target image resolution) and then fused with the output features (resolution of 1 / 32 of the target image resolution) of encoder 4 to obtain the decoding features (resolution of 1 / 32 of the target image resolution) of decoder 5.

[0132] The decoding features of decoder 5 are input into decoder 4. After feature transformation and upsampling (the resolution is changed to 1 / 16 of the target image resolution), they are fused with the output features of encoder 3 (the resolution is 1 / 16 of the target image resolution) to obtain the decoding features of decoder 4 (the resolution is 1 / 16 of the target image resolution).

[0133] The decoding features of decoder 4 are input into decoder 3. After feature transformation and upsampling (the resolution is changed to 1 / 8 of the target image resolution), they are fused with the output features of encoder 2 (the resolution is 1 / 8 of the target image resolution) to obtain the decoding features of decoder 3 (the resolution is 1 / 8 of the target image resolution).

[0134] The decoding features of decoder 3 are input into decoder 2. After feature transformation and upsampling (the resolution is changed to 1 / 4 of the target image resolution), they are fused with the output features of encoder 1 (the resolution is 1 / 4 of the target image resolution) to obtain the decoding features of decoder 2 (the resolution is 1 / 4 of the target image resolution).

[0135] In one decoding layer of decoder 1, the decoded features from decoder 2 are input. After feature transformation and upsampling (the resolution is changed to half the target image resolution), these features are fused with the image features output from the next convolutional block (with a resolution of half the target image resolution) to obtain fused features (with a resolution of half the target image resolution). In another decoding layer of decoder 1, this fused feature is upsampled to the same resolution as the target image. Thus, decoder 1 outputs target image features with the same resolution as the target image after a four-fold upsampling.

[0136] In one possible implementation, the image segmentation model can have N encoders, where N is any integer greater than or equal to 1. The image segmentation model can also have M decoders, where M is any integer greater than or equal to 1. N and M can be equal or unequal.

[0137] The number of encoders and / or decoders can be determined based on the resolution of the input image to be processed in the application scenario. For example, if the resolution of the input image to be processed is low, a smaller number of encoders and / or decoders can be set; if the resolution of the input image to be processed is high, a larger number of encoders and / or decoders can be set. For instance, if the image segmentation model is applied in an image preview scenario, the resolution of the input image to be processed is usually low, so a smaller number of encoders and / or decoders can be set in the training model for this application scenario. If the image segmentation model is applied in a mobile phone photography scenario, the resolution of the input image to be processed is relatively high, so a larger number of encoders and / or decoders can be set in the training model for this application scenario.

[0138] In one possible implementation, the number of encoders can be determined based on the resolution of the output features after downsampling. Furthermore, determining the number of encoders based on the resolution of the output features after downsampling can be based on whether the resolution of the downsampled output features is sufficient to capture the overall semantic features.

[0139] For example, if a resolution of less than or equal to 10*10 (e.g., 7*7) is sufficient to capture the overall semantic features, then the number of encoders can be set to k, where k can be any integer greater than or equal to 1, based on the premise that an output feature with a resolution of less than or equal to 10*10 can be obtained after downsampling by k encoders. For instance, the resolution of the output feature after downsampling by each encoder is not limited to half the resolution of the input.

[0140] In one possible implementation, each convolutional kernel in the encoder and / or decoder can at least partially employ depthwise separable convolution. For example, the encoder and / or decoder may use depthwise separable convolution for larger kernels and not for smaller kernels. Using depthwise separable convolution for larger kernels can reduce the computational cost during model execution. Using depthwise separable convolution for smaller kernels results in a smaller reduction in computational cost and may even decrease the accuracy of model predictions. Therefore, using depthwise separable convolution for larger kernels and not for smaller kernels can reduce the computational cost during model execution, saving computational resources and power consumption, while minimizing the impact of feature fusion during convolution operations.

[0141] After obtaining the target image features, these features can be input into a mapping head for feature dimension transformation. The output is a mask containing one or more objects, thus determining the segmentation category. The mapping head can be a network structure that maps the dimensions of the target image features to the required dimensions. The mapping head can include at least one network layer, such as a fully connected layer, an activation function layer, and a normalization layer.

[0142] For example, when the image segmentation model is used in a binary classification task to segment objects of a single category, such as portrait segmentation (portraits are foreground, non-portraits are background), the mask dimension is two-dimensional. If the target image features are high-dimensional (e.g., 100-dimensional), the target image features are input into the mapping head for feature dimension transformation, which outputs two-dimensional mapped target image features. After normalizing these mapped target image features, it can be determined whether each pixel in the target image is a portrait pixel or a background pixel, thus obtaining the mask corresponding to the target image.

[0143] For example, the model to be trained can be tailored to the specific task in the application scenario, enabling the trained image segmentation network to predict the category corresponding to the task and output a mask for that category. For instance, a mask can include one or more objects. For example, the mask could include four objects: two people, one vehicle, and one dog.

[0144] For example, specific tasks in application scenarios could be intensive prediction tasks such as portrait segmentation, skin segmentation, monocular depth estimation, and scene semantic segmentation. During model training, the prediction head dimension can be changed according to the number of categories required for the task, in order to achieve image segmentation for one or more categories of objects.

[0145] For example, in a specific application scenario involving image segmentation, sample images containing human figures and their corresponding ground truth labels can be used to train a training model. This yields an image segmentation model capable of segmenting human figures within the image to be processed. When the input image to this model contains at least one human figure, it can output a mask that includes at least one human figure.

[0146] For example, in a specific application scenario where the task is image segmentation of cats and / or dogs, sample images including cats and / or dogs, along with their corresponding ground truth labels, can be used to train a training model. This will produce an image segmentation model capable of segmenting cats and / or dogs in the image to be processed. When the input image to this image segmentation model includes at least one cat and / or dog, it can output a mask that includes at least one cat and / or dog.

[0147] In this embodiment, the mask can be a binary image that represents the pixel value of a pixel using two different values. For example, when represented by 0 and 1, the pixel value of the foreground pixel can be represented as 1, and the pixel value of the background pixel can be represented as 0. The mask of the binary image can also be represented as a black and white image, where the pixel corresponding to 1 can be represented as white, and the pixel corresponding to 0 can be represented as black. The mask can also be represented as a matrix or other forms. The size of the mask can be the same as the size of the image to be processed; for example, each pixel in the mask corresponds to one pixel in the image to be processed.

[0148] In one possible implementation, when training the model to obtain the image segmentation model, a contrastive learning loss function can be used to train the model. This improves the model's ability to recognize edge pixels of objects in the image, reduces the probability of misclassifying edge pixels, and improves the accuracy of the mask. The contrastive learning loss function can be any loss function that helps the model learn a feature space by bringing features closer together between similar samples and distancing features between dissimilar samples. Examples of contrastive learning loss functions include the information noise-contrastive estimation loss (InfoNCELoss) or a modified version of InfoNCELoss.

[0149] Figure 4 This is a schematic diagram of the training process of the model to be trained provided in the embodiments of this application, as shown below. Figure 4 As shown, a preprocessing operation to resize a target image with height H and width W can be performed to obtain a target image with height h1 and width w1. h1 can be equal to H. The h1×w1 target image is then input into the model to be trained for image feature extraction, yielding the corresponding image features. These image features are then processed by the encoder and decoder in the model to obtain the target image features. This target image feature is then input into a mapping head for feature dimension transformation, resulting in the mapped target image features. These mapped target image features can be understood as unactivated output values ​​(logits), which are the raw output values ​​before activation function processing. After normalization, these mapped target image features yield a mask.

[0150] In one possible implementation, to improve the accuracy of edge segmentation, during the training phase of the model, a contrastive learning loss function optimized for edge accuracy is used to train the model, helping it learn a better parameter distribution. The contrastive learning loss function is set to satisfy the following conditions: the features of the target object's edges are close to the features of the target object itself, while the features of the target object's edges are far from the background features.

[0151] In this context, the target object can be understood as any object in the target image. There can be one or more target objects, which can belong to the same or different categories. The edge region can be understood as the boundary between the target object and the background in the target image, and the area surrounding it. The edge region can include pixels from both the target object and the background. The edge of the target object can be understood as the region within the edge region that includes the pixels of the target object. The features of the target object's edge can be understood as image features used to characterize the image information of the region within the edge region that includes the pixels of the target object. The features of the target object can be understood as image features characterizing the image information of the region excluding the background and edge regions. The background features can be understood as image features characterizing the image information of the region excluding the target object and edge regions. The features of the target object's edge, the features of the target object, and the background features can all be obtained based on the features of the target image.

[0152] Based on this, by training the model through contrastive learning, the edge features of the target object in the output image features of the target image after iterating the parameters are closer to the features of the target object. This increases the probability that the pixels of the target object in the edge region will be classified as the target object, thereby improving the segmentation accuracy of the target object edge in the mask.

[0153] Taking portrait segmentation as an example, during model training, an additional contrastive learning loss is added to the real edge regions, making the features of the portrait edge pixels closer to the portrait prototype features and farther away from the background prototype features. Here, the portrait prototype features are the features of the target object, and the background prototype features are the background features. When determining the features of the target object, they can be based on the maximum, minimum, median, and / or average values ​​of each feature value of the target object in the target image features. Similarly, when determining the background features, they can be based on the maximum, minimum, median, and / or average values ​​of each background feature value in the target image features.

[0154] Taking determination based on the average value as an example, such as Figure 3 and Figure 4 As shown, in the target image features output by decoder 1, the average value of the features of the real human portrait region is calculated, and this average value is determined as the human portrait prototype feature. The real human portrait region is the area in the target image after removing the background and edge regions, which can be understood as the pure human portrait image region. Similarly, in the target image features output by decoder 1, the average value of the features of the real background region is calculated, and this average value is determined as the background prototype feature. The real background region is the area in the target image after removing the human portrait and edge regions, which can be understood as the pure background image region.

[0155] When determining the features and background features of a target object based on maximum, minimum, and / or intermediate values, the maximum, minimum, or intermediate value among the features of the real portrait region can be determined as the portrait prototype feature, or the average value of the maximum, minimum, and intermediate values ​​can be calculated and determined as the portrait prototype feature. Similarly, the maximum, minimum, or intermediate value among the features of the real background region can be determined as the background prototype feature, or the average value of the maximum, minimum, and intermediate values ​​can be calculated and determined as the background prototype feature. Based on this, the features and background features of the target object can be conveniently determined. Alternatively, other methods can be used, which are not limited in this embodiment.

[0156] When training a model using a contrastive learning loss function, average pooling can be used to calculate multiple edge pixels. Edge pixels can be understood as pixels within an edge region; they can be pixels of the target object or pixels of the background. After determining multiple edge pixels, the edge region can be identified.

[0157] In one possible implementation, when labeling the true edge regions of the target image, each pixel can be assigned a label value of 0 or 1. For example, pixels of the target object can be labeled as 1, and pixels of the background as 0. When calculating the true edge regions in the target image using the average pooling algorithm, pixels in edge regions will receive non-integer label values. Based on this, to determine the true edge regions in the target image, an average pooling algorithm can be performed on each pixel of the target image using a pooling kernel of any size, and pixels whose pooling result is not an integer can be identified as edge pixels. The pooling window can be, for example, a 3×3 or 2×2 pooling window, where a 3×3 pooling window is larger than a 2×2 pooling window. Alternatively, other sizes of pooling windows can be chosen for the calculation; there is no limitation.

[0158] It should be understood that if the pooling window is large, the range of the determined edge region is wide. This can be understood as including not only the pixels at the boundary between the target object and the background, but also the pixels in a large surrounding area. Therefore, when determining positive and / or negative samples for contrastive learning based on a wide edge region, the selected range is large, which may reduce the effect of contrastive learning and may lead to lower accuracy of model prediction.

[0159] If the pooling window is small, the range of the determined edge region is narrow. This can be understood as the edge region containing fewer edge pixels. Therefore, when determining positive and / or negative samples for contrastive learning based on the narrow edge region, the selection range is small, which is not conducive to selecting more positive and / or negative samples. When the number of positive and / or negative samples used for training is small, it may reduce the effect of contrastive learning and affect the prediction accuracy.

[0160] Based on this, average pooling calculations can be performed by setting an appropriately sized pooling kernel to determine edge regions of suitable range, thereby improving the model training effect.

[0161] In this embodiment, a positive sample can be understood as the intersection of an edge region and one or more target object labels, where the target object label can be understood as the true label of the target object. A positive sample may include one or more pixels. A negative sample can be understood as the intersection of an edge region and a background label, where the background label can be understood as the true label of the background. A negative sample may include one or more pixels. All positive samples of the target image can constitute a positive sample set, and all negative samples of the target image can constitute a negative sample set. The true label can be obtained through manual annotation, machine annotation, and / or other methods, and is not limited thereto.

[0162] For example, one or more edge regions can be identified in the target image. After identifying the edge regions, their intersection with the ground truth labels is calculated to obtain positive and / or negative samples. If the number of positive and / or negative samples is large, using all positive and / or negative samples for training a contrastive learning model may result in a longer training time and increased training costs due to the large number of positive and / or negative samples involved in the training.

[0163] In one possible implementation, to reduce training time and cost, random sampling can be performed on each of the obtained positive and / or negative samples to obtain a predetermined number of positive and / or negative samples. This results in a feature set for calculating the contrastive learning loss, which includes the feature values ​​of each of the randomly sampled positive and / or negative samples. The number of randomly sampled positive and / or negative samples can be determined based on the computing power of the electronic device used to train the model.

[0164] Alternatively, other sampling methods can be used to obtain an appropriate number of positive and / or negative samples. For example, sampling rules can be preset, and sampling can be performed according to these rules. These rules could be, for example, rules for filtering positive and / or negative samples. This application does not limit the sampling method, sampling quantity, or sampling rules.

[0165] In one possible implementation, to improve the training effect of contrastive learning, a larger number of negative samples can be used during training. For example, the number of negative samples is greater than the number of positive samples, such as ten times or about ten times the number of positive samples, or other proportions are not limited.

[0166] In one possible implementation, if the number of sampled positive samples and / or negative samples does not meet the preset requirements, the contrastive learning loss function may not be calculated for the target image. For example, if the number of positive samples participating in training is set to be no less than 100, and the total number of positive samples is less than 100, or if it is impossible to determine no less than 100 positive samples from all positive samples according to the sampling rules, then the training requirements for contrastive learning cannot be met, and in this case, the contrastive learning loss function may not be calculated for the target image. The case for negative samples is similar and will not be elaborated further.

[0167] Taking human image segmentation as an example, if the target image contains a small human figure or no human figure at all, then the human figure has fewer or no pixels, resulting in fewer or no identified edge regions. In this case, the number of positive samples is insufficient, which may not meet the requirements for the number of positive samples during contrastive learning. Conversely, if the target image contains many human figures and a small or nonexistent background, then the human figures have more pixels, while the background has fewer or no pixels, resulting in fewer or no identified edge regions. In this case, the number of negative samples is insufficient, which may not meet the requirements for the number of negative samples during contrastive learning.

[0168] When using InfoNCELoss for contrastive learning training, increasing the number of positive samples can improve InfoNCELoss, resulting in an improved version. This enhances the optimization of object edges, improving image segmentation accuracy. It's important to note that the contrastive learning loss function is only used to optimize model parameters during training and not during application, thus minimizing computational overhead during inference.

[0169] In some implementations, in addition to using the contrastive learning loss function for training, joint training can be performed by combining the cross-entropy loss function and the DICE similarity coefficient loss (DICE). For example... Figure 4As shown, after the model to be trained outputs the mapped target image features corresponding to the target image, the cross-entropy loss function and the DICE loss function can be calculated based on the mapped target image features and the true labels of the target image. Based on the target image features, the features of the target object and background features are obtained, and the contrastive learning loss function for positive and / or negative samples is calculated based on the feature set obtained after sampling. Based on the values ​​of the three types of loss functions, the parameters of the model to be trained are optimized and adjusted through gradient optimization and other methods to obtain the optimized parameters. After multiple rounds of training in this way, the model parameters can be iterated. Training can be stopped when the values ​​of the three types of loss functions converge to their respective preset conditions. The parameters obtained at this point are the optimized, better parameters, and the trained image segmentation model can be obtained based on these better parameters.

[0170] After obtaining the image segmentation model, the acquired image to be processed can be input into the image segmentation model for processing to obtain a mask that includes one or more objects. The stage of using the model to process the image to be processed and obtain the mask can be understood as the inference application stage.

[0171] In one possible implementation, during the inference application phase, to shorten the model's inference time, a sliding window inference method can be used for inference prediction to quickly obtain the mask. Sliding window inference can be understood as using a sliding window to segment the image to be processed into at least two image regions with overlapping areas, and then inputting the segmented image regions into the image segmentation model in batch processing to obtain a mask with the same size as the image to be processed.

[0172] In this context, a sliding window can be a virtual window. Moving the sliding window across the image to be processed divides the image into multiple image regions, with overlapping areas between them. An image region can be understood as a region composed of a portion of the pixels in the image to be processed; an image region can also be called a patch. Because the image regions overlap, no pixels in the image to be processed will be missed during inference.

[0173] The size of the sliding window can be preset to be the same as the size of the target image input to the model during training. Alternatively, the size of the sliding window can also be preset according to the size of the image to be processed; for example, the height of the sliding window can be equal to the height of the image to be processed, and the width of the sliding window can be smaller than the width of the image to be processed. Alternatively, the size of the sliding window can also be preset according to the computing power of the electronic device. This application does not limit how the sliding window size is set.

[0174] In a possible implementation, before performing image region segmentation on the image to be processed using a sliding window, the image to be processed can be resized from a first size to a second size, where the second size is larger than the first size. Resizing the image to be processed can ensure that the image to be processed can be segmented into at least two different image regions and that there are overlapping regions between the image regions, so that the inference result can be smoother at the boundaries between different image regions.

[0175] Figure 5 The figure shows a schematic diagram of the sliding window inference provided by an embodiment of this application. As Figure 5 shown, in the inference stage, the image to be processed can be segmented into two image regions using a sliding window. For an image to be processed with a size of H×W, its size can be first adjusted to h1×w2, where w1 < w2 < w1×2, that is, w2 is greater than w1 and less than twice w1. h1 is the height of the sliding window, and w1 is the width of the sliding window. h1 can be the same as the image height of the target image input to the model during training, and w1 can be the same as the image width of the target image input to the model during training. h1 can be equal to H. Segmenting the image to be processed with a size of h1×w2 using a sliding window with a size of h1×w1 can not only segment out two different image regions but also make the two image regions have overlapping regions, making the inference result smoother at the boundary between the two image regions. Moreover, according to the computing power of the electronic device deploying the image segmentation model, segmenting the image to be processed into two image regions using a sliding window for sliding window inference can reduce the computing amount during model inference and save the power consumption of the electronic device.

[0176] As Figure 5 shown, during sliding window inference, the image to be processed with a size of h1×w2 is divided into 2 patches with overlapping regions through the sliding window, and the size of each patch is h1×w1. The 2 patches can be input into the image segmentation model in batches for processing to obtain a mask with the same size as the image to be processed, that is, a mask with a size of h1×w2.

[0177] Exemplarily, an image can be composed of multiple channel images with different channels. The channels of the image can be represented by the symbol c, and c can be any integer. For example, in a red green blue (RGB) image, an RGB image can be composed of channel images with three channels, so the channel c is 3. Therefore, an image to be processed with a size of h1×w2 is an image of 1*ch1w2. When it is input into the image segmentation model for processing in the form of a batch size of 2, it can be understood as inputting an image of 2*ch1w1.

[0178] It is understandable that if the sliding window segments the image to be processed into q image regions, then the batch size is q. A batch can input all q image regions into the image segmentation model for processing, where q can be any positive integer.

[0179] If the value of q is large, the image to be processed will be segmented into more image regions, and the image segmentation model will process more image regions, which may increase the computational load of the image segmentation model. Therefore, in application, a suitable sliding window size and / or adjusting the first size of the image to be processed to a more suitable second size can be set according to the performance and power consumption requirements of the electronic device in the application scenario, so as to obtain an appropriate number of image regions, effectively control the number of segmented image regions, reduce the computational load of the image segmentation model during inference, improve the efficiency of model inference, and reduce the power consumption of the electronic device.

[0180] like Figure 5 As shown, after inputting two ch1w1 patches into the image segmentation model, each encoder can encode the two patches separately, obtaining two encoded features. Before inputting the two encoded features into the corresponding decoder for decoding, the two encoded features can be merged to obtain merged features. After merging the two encoded features, the resolution of the merged features can be restored to the size h1×w2 of the image to be processed before sliding window segmentation. The decoder corresponding to the encoder can perform decoding operations on the merged features. The features decoded by each decoder are passed through a mapping head to obtain the mask corresponding to the image to be processed, completing the inference.

[0181] In one possible implementation, when processing q image regions in the same batch using any encoder, the encoded features of the q image regions can be merged by calculating an average value to obtain the merged features. For example, for any overlapping region among the q image regions, the average feature value of any pixel in the overlapping region corresponding to the encoded features of each image region can be calculated, and this average value can be determined as the feature value of the merged feature for that pixel. Based on this, feature merging can be performed on each pixel in the overlapping region. For any pixel in a non-overlapping region, the feature value corresponding to that pixel in the encoded features can be determined as the feature value in the merged feature.

[0182] For example, feature merging during batch processing can be achieved without changing the original network structure of the image segmentation model, or without adding a network layer for feature merging. It can be achieved by adding a feature merging computation operation during the inference process of the image segmentation model.

[0183] The image processing method of this application will be described in detail below through specific embodiments. The following embodiments can be implemented in combination with each other or independently, and the same or similar concepts or processes may not be described again in some embodiments.

[0184] Figure 6 This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. The method is applied to electronic devices, such as... Figure 6 As shown, the method includes:

[0185] S601. Obtain the image to be processed.

[0186] In this embodiment of the application, the image to be processed can be any image. The method of obtaining the image to be processed may include, for example... Figure 1 The described embodiments are obtained by taking pictures using at least one camera, or by selecting from a gallery, etc. This application does not limit the embodiments in this way.

[0187] S602. Obtain a mask for one or more objects in the image to be processed based on the image segmentation model; wherein, the image segmentation model is deployed on an electronic device, and when training the image segmentation model, a contrastive learning loss function is used in the edge region of the sample, the edge region includes the edge of one or more target objects in the sample, and the contrastive learning loss function is set to satisfy that the features of the edge are close to the features of one or more target objects, and the features of the edge are far away from the background features; the category of one or more target objects in the sample includes the category of one or more objects in the image to be processed.

[0188] In this embodiment, the object can be any one or more categories of objects, such as at least one of people, cats, dogs, trees, houses, and other objects. The electronic device can be an edge-side electronic device, such as a mobile phone, computer, or camera. The electronic device can also be a cloud-based electronic device, such as a cloud server.

[0189] For example, an image segmentation model can be deployed on an edge electronic device, on a cloud electronic device, or both on the edge and in the cloud.

[0190] In one possible implementation, the image segmentation model is deployed on an edge electronic device. The device can acquire the image to be processed based on images captured or stored by the device, and input the acquired image to the image segmentation model to obtain a corresponding mask. Furthermore, based on the obtained mask, the electronic device can also perform image enhancement and other processing on the image to be processed using algorithms from downstream image processing tasks. This allows for the differentiation and adjustment of any object and / or non-object image regions in the image to achieve better image processing results.

[0191] In one possible implementation, an electronic device with an image segmentation model deployed in the cloud can acquire the image to be processed through data transmission and / or accessing data stored in the cloud. For example, a user can upload the image to be processed to a cloud server via their mobile phone for image processing. After receiving the image processing instructions and the image to be processed, the cloud server can process the image using the deployed image segmentation model to obtain a corresponding mask. After obtaining the mask, it can send the mask to the user's mobile phone; alternatively, the electronic device in the cloud can also deploy downstream image processing algorithms, complete the downstream image processing tasks based on the mask, and send the processing results to the user's mobile phone.

[0192] In this application embodiment, the image segmentation model may include any one of the image segmentation models in the above embodiments. For details, please refer to the description in the above embodiments, which will not be repeated here.

[0193] In this embodiment, a mask for the image to be processed can be obtained based on the trained image segmentation model. During the training of this image segmentation model, a contrastive learning loss function is used in the edge regions of the samples. This loss function is set to ensure that the features of the target object's edges are close to the target object's features and far from the background features. Therefore, after training, the image segmentation model has the ability to accurately extract image features, and the extracted image features show that the edge features of the object are closer to the object's features. Based on these image features, it is possible to accurately distinguish whether pixels in the edge regions are object pixels or background pixels, thereby reducing the number of misclassified pixels and improving the mask's accuracy. Therefore, based on this image segmentation model, more accurate image segmentation can be performed on the image to be recognized, resulting in a higher-precision mask, which helps improve the accuracy of image processing.

[0194] Optionally, a mask for one or more objects in the image to be processed is obtained based on the image segmentation model, including:

[0195] The image to be processed is changed from a first size to a second size, which is larger than the first size. A sliding window is used to segment the second-sized image to be processed into multiple image regions with overlapping areas. The encoder in the image segmentation model encodes the multiple image regions with overlapping areas, and the encoded features are merged. The decoder in the image segmentation model processes the merged features to obtain a mask of one or more objects in the image to be processed.

[0196] For example, the first size can be the original size of the image to be processed. The second size can be a size larger than the first size; for example, the second size can be any size between the first size and twice the first size. When processing the image from the first size to the second size, any method of image size change can be used, for example, image scaling.

[0197] In one possible implementation, when using a sliding window to segment the image to be processed, adjusted to the second size, this can be achieved by minimizing the number of image regions. Minimizing the number of image regions can be understood as ensuring that, when segmenting the image to be processed at the second size, the number of image regions is the minimum achievable while still satisfying the conditions that the segmented image regions have overlapping areas and no missing areas. Based on this, redundant image regions can be reduced, thereby reducing redundant computation during sliding window inference and improving the speed of model inference.

[0198] In this embodiment, a sliding window inference method is used to process the image region according to the image segmentation model. When obtaining a mask for one or more objects in the image to be processed, it can be referred to... Figure 5 The descriptions in the corresponding embodiments will not be repeated. In this way, by using sliding window inference, the image to be processed can be processed quickly, shortening the inference time of the image segmentation model and obtaining the mask more quickly.

[0199] Optionally, when training the image segmentation model, multiple positive samples and multiple negative samples are used, and the number of negative samples is greater than the number of positive samples. Positive samples are the intersection of the edge region and one or more target object labels, and negative samples are the intersection of the edge region and the background label.

[0200] For example, multiple positive samples can be obtained from the positive sample set of the target image through random sampling, and multiple negative samples can be obtained from the negative sample set of the target image through random sampling. The number of negative samples can be greater than the number of positive samples. When the number of positive and negative samples does not meet the training requirements, the contrastive learning loss function can be omitted from the target image calculation. For specific implementation details, please refer to... Figure 4 The descriptions in the corresponding embodiments will not be repeated. This allows for more calculations of the contrastive learning loss function, improving the training effect of contrastive learning, increasing the inference accuracy of the image segmentation model, and consequently improving the accuracy of the mask.

[0201] Optionally, the edge region includes multiple edge pixels calculated using an average pooling algorithm, wherein the edge pixels are pixels whose pooling result is not an integer.

[0202] In this embodiment, when using the average pooling algorithm, a pooling kernel with an arbitrary size pooling window can be used to perform average pooling calculations on each pixel in the target image, and pixels whose pooling result is not an integer are identified as edge pixels. For example, a pooling kernel with a 3×3 pooling window can be used to calculate to obtain edge regions of appropriate range. For specific implementation details, please refer to [reference needed]. Figure 4 The descriptions in the corresponding embodiments will not be repeated here. This allows for the convenient calculation of multiple edge pixels, enabling faster determination of edge regions.

[0203] Optionally, the image segmentation model includes N encoders and M decoders. The encoders downsample the input image features, and the decoders upsample the input image features. The encoders are equipped with modules for capturing multi-scale features in the image.

[0204] In this embodiment, any encoder may include a network layer or module capable of downsampling image features; any decoder may include a network layer or module capable of upsampling image features. The number of encoders in the image segmentation model can be N, where N can be any integer greater than or equal to 1. The number of decoders in the image segmentation model can be M, where M can be any integer greater than or equal to 1. N and M may be equal or unequal. For specific implementation details, please refer to... Figure 3 The descriptions in the corresponding embodiments will not be repeated. In this way, the encoder can acquire multi-scale features of the image to be processed, thereby improving the representation strength of image features, and the decoder can upsample and decode the image features to obtain a mask with higher accuracy.

[0205] In some applications, when deploying image segmentation models on electronic devices, it's necessary to balance segmentation performance with power consumption. Focusing too much on power consumption can limit segmentation performance, leading to reduced accuracy. To better balance segmentation performance and power consumption, the encoder structure can be optimized to suit the application requirements.

[0206] Optionally, each encoder includes at least one block, which is obtained by changing the attention layer in the transformer block to the inception module.

[0207] In this embodiment, the blocks included in the encoder can also be understood as encoder layers. Replacing the attention layer in the transformer block with an inception module yields the blocks in the encoder. The inception module can process image features at different scales, capturing semantic features of multi-scale features, and its computational cost is lower than that of the attention layer, thus reducing computational overhead. For specific implementation details, please refer to... Figure 3 The descriptions in the corresponding embodiments will not be repeated. Thus, based on the Inception module, the encoder can reduce its computational load while capturing a wider range of local features, improving the representational power of image features and saving computational resources. This, in turn, can improve the accuracy of the mask and reduce the power consumption of the electronic device.

[0208] Optionally, some convolutional kernels in the encoder may employ depth-separable convolution.

[0209] In the embodiments of this application, the convolutional kernels in the encoder and / or decoder vary in size; for example, a 5×5 convolutional kernel is larger than a 3×3 convolutional kernel. The encoder can use depthwise separable convolution for the larger kernels, which can reduce the computational load during model runtime. The decoder follows a similar approach. For specific implementation details, please refer to [reference needed]. Figure 3 The descriptions in the corresponding embodiments will not be repeated. In this way, by using convolution kernels that are partially depthwise separable, the computational load during model operation can be reduced, saving computational resources and power consumption, while minimizing the impact of feature fusion during the convolution operation.

[0210] Optionally, the image segmentation model also includes convolutional blocks, pyramid pooling modules, and a mapping head. Training the image segmentation model involves the following steps: inputting the target image into the convolutional blocks to obtain image features with a resolution lower than that of the target image; inputting the image features sequentially into N encoders for downsampling; inputting the output features of the last encoder into the pyramid pooling module for multi-scale feature fusion, and then inputting them together with the output features of the penultimate encoder into the decoders, which are then processed by M decoders to obtain target image features with the same resolution as the target image; processing the target image features through the mapping head to obtain masks for one or more objects; and adjusting the parameters of the image segmentation model during training according to the contrastive learning loss function until the value of the contrastive learning loss function converges.

[0211] For example, a convolutional block can be a convolutional block containing one convolutional layer or a convolutional block containing multiple convolutional layers; one or more convolutional blocks can be set. Convolutional blocks can be used to extract image features from a target image, obtaining the image features corresponding to the target image.

[0212] The pyramid pooling module can set multiple pooling windows of different scales to perform multi-scale pooling operations on the features encoded by the encoder. The mapping head can be a network structure that maps the dimensions of the target image features to the required dimensions. For example, the mapping head can be composed of fully connected layers, activation function layers, and normalization layers to perform feature mapping on the target image features, complete pixel classification, and obtain the corresponding mask.

[0213] For details on training the image segmentation model, please refer to... Figure 3 and Figure 4 The descriptions in the corresponding embodiments will not be repeated. In this way, the model to be trained can be effectively trained using a contrastive learning method, enabling the trained image segmentation model to classify pixels more accurately, thus giving it higher image segmentation capabilities, improving image segmentation quality during image processing, and obtaining a higher-precision mask.

[0214] The image segmentation model training method of this application will be described in detail below through specific embodiments. The following embodiments can be implemented in combination with each other or independently, and the same or similar concepts or processes may not be described again in some embodiments.

[0215] Figure 7 This is a flowchart illustrating an image segmentation model training method provided in an embodiment of this application. The image segmentation model is obtained based on a training model, which includes convolutional blocks, N encoders, M decoders, a pyramid pooling module, and a mapping head; as shown... Figure 7 As shown, the method includes:

[0216] S701. Input the target image into the convolution block to obtain image features with a resolution lower than that of the target image.

[0217] S702. Input the image features into N encoders sequentially for downsampling.

[0218] S703. The output features of the last encoder are input into the pyramid pooling module for multi-scale feature fusion, and then input together with the output features of the penultimate encoder into the decoder. The M decoders then process the features to obtain the target image features with the same resolution as the target image.

[0219] S704. The target image features are processed by the mapping head to obtain a mask for one or more objects.

[0220] S705. Adjust the parameters of the model to be trained according to the contrastive learning loss function until the value of the contrastive learning loss function converges to obtain the image segmentation model.

[0221] The model to be trained can be deployed on an electronic device, such as a computer, server, or other electronic device with model training capabilities. The target image can be any sample image used to train the model. By inputting the target image into the convolutional block, feature extraction and downsampling can be performed on the target image to obtain image features with a resolution lower than that of the target image. These image features can be understood as feature maps, etc.

[0222] The process of training the model can be found in [reference needed]. Figure 3 and Figure 4 The descriptions in the corresponding embodiments will not be repeated. In this way, the model to be trained can be effectively trained using a contrastive learning method, enabling the trained image segmentation model to classify pixels more accurately, thus giving it higher image segmentation capabilities, improving image segmentation quality during image processing, and obtaining a higher-precision mask.

[0223] In this embodiment, the model to be trained is based on a contrastive learning loss function, and targeted training is performed on the edge regions of the samples, realizing a model training process based on edge optimization loss. Therefore, the image segmentation model obtained after training has high segmentation accuracy in the edge regions of the image to be processed, and can obtain masks with high edge accuracy. This is beneficial for more accurately distinguishing and adjusting image regions of different categories of objects in the image to be processed in downstream image processing tasks, resulting in better image processing effects.

[0224] Optionally, when adjusting the parameters of the model to be trained according to the contrastive learning loss function, multiple positive samples and multiple negative samples are used, and the number of negative samples is greater than the number of positive samples. The positive samples are the intersection of the edge region of the target image and one or more target object labels of the target image, and the negative samples are the intersection of the edge region of the target image and the background label of the target image.

[0225] The specific implementation can be referred to the description in the corresponding embodiments above, and will not be repeated here. In this way, more calculations of the contrastive learning loss function can be performed, improving the training effect of contrastive learning, increasing the inference accuracy of the image segmentation model, and thus improving the accuracy of the mask.

[0226] Optionally, the edge region includes multiple edge pixels calculated using an average pooling algorithm, wherein the edge pixels are pixels whose pooling result is not an integer.

[0227] The specific implementation can be referred to the description in the corresponding embodiments above, and will not be repeated here. In this way, multiple edge pixels can be calculated conveniently, and the edge region can be determined quickly.

[0228] Optionally, each encoder includes at least one block, which is obtained by changing the attention layer in the transformer block to the inception module.

[0229] The specific implementation can be referred to the description in the corresponding embodiments above, and will not be repeated here. In this way, based on the Inception module, the encoder can reduce the computational load of the encoder, improve the representational power of image features and save computational load while capturing a larger range of local features, thereby improving the accuracy of the mask and reducing the power consumption of electronic devices.

[0230] Optionally, some convolutional kernels in the encoder may employ depth-separable convolution.

[0231] The specific implementation can be referred to the description in the corresponding embodiments above, and will not be repeated here. In this way, by using convolution kernels that are partially depthwise separable, the amount of computation during model operation can be reduced, saving computing power and energy consumption, while minimizing the impact of feature fusion during the convolution operation.

[0232] It should be noted that the module names involved in the embodiments of this application can all be defined as other names, as long as they can achieve the function of each module, and no specific restrictions are placed on the module names.

[0233] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0234] The foregoing primarily describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the aforementioned functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the method steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0235] This application embodiment can divide the apparatus for implementing the method into functional modules based on the above method examples. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0236] Figure 8 This is a schematic diagram of a chip structure provided in an embodiment of this application. The chip 800 includes one or more processors 801, communication lines 802, communication interfaces 803, and memory 804.

[0237] In some implementations, memory 804 stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof.

[0238] The methods described in the embodiments of this application can be applied to or implemented by processor 801. Processor 801 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuit in the hardware of processor 801 or by instructions in software form. The processor 801 may be a general-purpose processor (e.g., a microprocessor or conventional processor), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates, transistor logic devices, or discrete hardware components. Processor 801 can implement or execute the various processing-related methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0239] The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in mature storage media in the art, such as random access memory, read-only memory, programmable read-only memory, or electrically erasable programmable read-only memory (EEPROM). This storage medium is located in memory 804, and processor 801 reads the information in memory 804 and, in conjunction with its hardware, completes the steps of the above method.

[0240] The processor 801, memory 804 and communication interface 803 can communicate with each other through communication line 802.

[0241] In the above embodiments, the instructions stored in the memory for execution by the processor can be implemented in the form of a computer program product. This computer program product can be pre-written into the memory, or it can be downloaded and installed into the memory as software.

[0242] This application also provides a computer program product comprising one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from a website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. For example, available media may include magnetic media (e.g., floppy disk, hard disk, or magnetic tape), optical media (e.g., digital versatile disc (DVD)), or semiconductor media (e.g., solid-state disk (SSD)).

[0243] This application also provides a computer-readable storage medium. The methods described in the above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. The computer-readable medium may include computer storage media and communication media, and may also include any medium capable of transferring a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0244] In one possible implementation, a computer-readable medium may include a compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM, or other optical disc storage; a computer-readable medium may also include a disk storage device or other disk storage device. Furthermore, any connecting cable may also be appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disks and optical discs include optical discs (CD), laser discs, optical discs, digital versatile discs (DVD), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers.

[0245] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0246] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image processing method, characterized by, The method is applied to an electronic device, and the method comprises: obtaining a to-be-processed image; obtaining a mask of one or more objects in the to-be-processed image according to an image segmentation model; wherein the image segmentation model is deployed on the electronic device, and when the image segmentation model is trained, a contrast learning loss function is used in an edge region of a sample, the edge region comprises an edge of one or more target objects in the sample, the contrast learning loss function is set to make the features of the edge close to the features of the one or more target objects, and the features of the edge are far away from the features of the background; and the categories of the one or more target objects in the sample comprise the categories of the one or more objects in the to-be-processed image.

2. The method of claim 1, wherein, The method comprises: processing the to-be-processed image from a first size to a second size, the second size being greater than the first size; segmenting the to-be-processed image of the second size into a plurality of image regions with overlapping areas by using a sliding window; encoding the plurality of image regions with overlapping areas according to an encoder in the image segmentation model, merging the encoded features, and processing the merged features according to a decoder in the image segmentation model to obtain the mask of the one or more objects in the to-be-processed image.

3. The method according to claim 1 or 2, characterized in that, When the image segmentation model is trained, a plurality of positive samples and a plurality of negative samples are used, and the number of the negative samples is greater than the number of the positive samples, the positive samples are the intersection of the edge region and the label of the one or more target objects, and the negative samples are the intersection of the edge region and the background label.

4. The method of claim 3, wherein, The edge region comprises a plurality of edge pixel points calculated by using an average pooling algorithm, wherein the edge pixel points are pixel points whose pooling results are not integers.

5. The method according to any one of claims 1 to 4, characterized in that, The image segmentation model comprises N encoders and M decoders, the encoder performs down-sampling on the input image features, the decoder performs up-sampling on the input image features, and the encoder is provided with a module for capturing multi-scale features in an image.

6. The method of claim 5, wherein, Any one of the encoders comprises at least one block, the block is obtained by changing an attention layer in a transformer block into an inception module.

7. The method according to claim 5 or 6, characterized in that, Part of the convolution kernels in the encoder use depth separable convolution.

8. The method according to any one of claims 5-7, characterized in that, The image segmentation model further comprises a convolution block, a pyramid pooling module and a mapping head; when the image segmentation model is trained, the following steps are included: inputting a target image into the convolution block to obtain image features with a resolution lower than that of the target image; inputting the image features into the N encoders in sequence for down-sampling; inputting the output features of the last encoder into the pyramid pooling module for multi-scale feature fusion, and then inputting the output features of the penultimate encoder and the output features of the last encoder into the decoder together to obtain target image features with the same resolution as the target image by the M decoders; processing the target image features by the mapping head to obtain the mask of the one or more objects. The parameters of training the image segmentation model are adjusted according to the contrast learning loss function until the value of the contrast learning loss function converges.

9. An image segmentation model training method, characterized in that, The image segmentation model is obtained based on a trained model, and the trained model comprises a convolution block, N encoders, M decoders, a pyramid pooling module, and a mapping head. The target image is input into the convolution block to obtain image features with a resolution lower than that of the target image. The image features are sequentially input into the N encoders for down-sampling. The output features of the last encoder are input into the pyramid pooling module for multi-scale feature fusion, and then the output features of the penultimate encoder and the output features of the last encoder are jointly input into the decoder, and then the M decoders are used to process to obtain target image features with the same resolution as the target image. The mapping head is used to process the target image features to obtain the mask of the one or more objects. The parameters of the trained model are adjusted according to the contrast learning loss function until the value of the contrast learning loss function converges, thereby obtaining the image segmentation model.

10. The method of claim 9, wherein, When adjusting the parameters of the trained model according to the contrast learning loss function, a plurality of positive samples and a plurality of negative samples are used, and the number of negative samples is greater than the number of positive samples, the positive sample is the intersection of the edge region of the target image and the label of one or more target objects of the target image, and the negative sample is the intersection of the edge region of the target image and the background label of the target image.

11. The method of claim 10, wherein, The edge region comprises a plurality of edge pixel points calculated by using an average pooling algorithm, wherein the edge pixel points are pixel points whose pooling results are not integers.

12. The method according to any one of claims 9-11, characterized in that, Each of the encoders comprises at least one block, and the block is obtained by changing an attention layer in a transformer block into an inception module.

13. The method according to any one of claims 9-12, characterized in that, Some convolution kernels in the encoders are deep separable convolution kernels.

14. An electronic device, comprising: The memory and the processor are configured to execute the computer program to perform the method of any one of claims 1-8, or to perform the method of any one of claims 9-13. The computer readable storage medium stores instructions which, when executed, cause a computer to perform the method of any one of claims 1-8, or cause a computer to perform the method of any one of claims 9-13.

15. A computer-readable storage medium, characterized in that, The computer readable storage medium stores instructions which, when executed, cause a computer to perform the method of any one of claims 1-8, or cause a computer to perform the method of any one of claims 9-13.

16. A computer program product, characterised in that, ​

Images