Image processing method and electronic equipment

Abstract

The invention provides an image processing method and electronic equipment, and relates to the technical field of image processing. The electronic equipment can determine a target cutting parameter and a target connected domain area threshold matched with a to-be-processed image based on the to-be-processed image. Afterwards, the electronic device can cut the to-be-processed image by using the target cutting parameter to obtain a cut image with a proper proportion of the target object. Afterwards, the electronic device can input the cut image into a segmentation network model, so that the segmentation network model carries out semantic segmentation on the cut image to determine a target object area on the cut image, and a first mask is obtained. As the proportion of the target object in the cut image is appropriate, the segmentation network model can accurately perform semantic segmentation on the cut image. Afterwards, the electronic device can continuously use the target connected domain area threshold to remove the false positive target object region in the first mask, thereby ensuring the precision of semantic segmentation.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and electronic device. Background Art

[0002] Semantic segmentation is the process of assigning each pixel in an image to a predefined semantic category. It is widely used in different scenarios, such as autonomous driving, medical image analysis, and photographic image processing (such as portrait enhancement, background blur, and image clipping).

[0003] Currently, electronic devices can first perform semantic segmentation on an image to identify objects in the image that belong to a target category, i.e., the target object. The electronic device can then perform relevant processing on the target object, such as the aforementioned portrait enhancement. Low semantic segmentation accuracy can result in poor image processing results. Therefore, improving semantic segmentation accuracy has become an urgent issue. Summary of the Invention

[0004] In view of this, the present application provides an image processing method and an electronic device, which can improve the accuracy of semantic segmentation of images, thereby improving the semantic segmentation effect.

[0005] In a first aspect, the present application provides an image processing method, where an electronic device can determine target cropping parameters and a target connected domain area threshold that match the image to be processed based on the image to be processed.

[0006] Afterwards, the electronic device may crop the image to be processed based on the target cropping parameters to obtain a cropped image.

[0007] Afterwards, the electronic device may perform semantic segmentation processing on the cropped image to obtain a first mask; wherein the first mask is a binary image of the cropped image, and the first mask includes at least one first target object region and a first non-target object region; the first target object region refers to a region including the target object, and the first non-target object region refers to a region not including the target object;

[0008] Based on the target connected domain area threshold, a connected domain analysis is performed on the first mask to determine the target mask corresponding to the image to be processed; wherein the connected domain analysis indicates that the area that is incorrectly semantically segmented as the first target object area is updated (i.e., corrected) to the target object area.

[0009] In the present application, the electronic device can determine target cropping parameters and target connected domain area thresholds that match the image to be processed based on the actual situation of the target object in the image to be processed, that is, determine target cropping parameters and target connected domain area thresholds that are suitable for the image to be processed, thereby achieving adaptive determination of the cropping parameters and connected domain area thresholds. The electronic device then uses the target cropping parameters to crop the image to be processed to obtain a cropped image, ensuring that the proportion of the target object in the cropped image is appropriate, thereby ensuring that the electronic device performs semantic segmentation processing on the cropped image with high precision, that is, ensuring high semantic segmentation accuracy, and further ensuring high accuracy of the target object in the obtained first mask. The electronic device then uses the target connected domain area threshold to correct the first target object area in the first mask to a first non-target object area, thereby correcting the area that was mistakenly determined to include the target object in the semantic segmentation process to an area that does not include the target object, thereby obtaining a target mask that does not include false positive areas, ensuring the accuracy of the target object in the target mask, further ensuring high accuracy of semantic segmentation, and good semantic segmentation effect, thereby ensuring subsequent image processing effects and improving user satisfaction.

[0010] The target cropping parameters that match the image to be processed may refer to cropping parameters that match the distribution information (e.g., distribution position and / or proportion) of the target object (i.e., the object to be segmented) on the image to be processed. Similarly, the target connected domain area threshold that matches the image to be processed may refer to a connected domain area threshold that matches the distribution information of the target object on the image to be processed.

[0011] In a possible design, the first target object area is a first color, the first non-target object area is a second color, the first color is white or black, the second color is black or white, and the first color and the second color are different.

[0012] Optionally, when the color of a pixel is white, the value of the pixel may be 255 (or 1). When the color of a pixel is black, the value of the pixel may be 0.

[0013] In one possible design, the first mask may be determined by the electronic device using a target segmentation network model. Specifically, the electronic device may input the cropped image into the target segmentation network model to obtain a predicted probability map corresponding to the cropped image. The predicted probability map may include a first probability corresponding to each pixel in the cropped image, where the first probability corresponding to the pixel represents the probability that the pixel belongs to the target object.

[0014] The electronic device may then determine a first mask based on the predicted probability map, wherein the first probability corresponding to the pixels in the first target object region in the first mask is greater than a first preset probability value, and the first probability corresponding to the pixels in the first non-target object region in the first mask is less than the first preset probability value.

[0015] In this application, the electronic device can use the target segmentation network model to perform semantic segmentation processing on the cropped image to achieve fast semantic segmentation processing. Afterwards, the electronic device can use the probability corresponding to each pixel point in the cropped image determined by the semantic segmentation processing (i.e., the first probability) to determine whether the pixel point belongs to the target object, thereby determining the binary image corresponding to the cropped image and achieving accurate semantic segmentation of the cropped image.

[0016] In one possible design, before determining the first mask, that is, after performing semantic segmentation processing on the cropped image, the electronic device can first determine whether the accuracy of this semantic segmentation processing is high, that is, whether the first number of pixel points whose first probability is equal to the first preset probability value in the above-mentioned predicted probability map obtained by the electronic device is less than the first number threshold.

[0017] When it is determined that the first number is less than the first number threshold, it indicates that the accuracy of this semantic segmentation process is high, and the electronic device may continue to determine the first mask.

[0018] When it is determined that the first number is greater than or equal to the first number threshold, it indicates that the accuracy of this semantic segmentation processing is low, and the electronic device does not need to continue to determine the first mask. The electronic device can return to the step of determining the target cropping parameters and the target connected domain area threshold that match the image to be processed based on the image to be processed, so as to re-determine the target cropping parameters that match the image to be processed, so as to obtain appropriate target cropping parameters, thereby ensuring the accuracy of semantic segmentation.

[0019] In one possible design, the electronic device can first input the image to be processed into a global neural convolutional network model to obtain the global features corresponding to the image to be processed, thereby compressing the image. The electronic device can then determine the target cropping parameters and target connected domain area threshold that match the image to be processed based on the global features. This reduces the complexity of image data calculations, enables rapid determination of the target cropping parameters and target connected domain area threshold, and thus improves the overall efficiency of semantic segmentation processing.

[0020] In one possible design, the process of determining the target mask may include: the electronic device may first update the first target object region in the first mask whose area is smaller than the target connected domain area threshold to a first non-target object region, thereby obtaining a second mask to perform connected domain analysis, thereby accurately removing false positive regions on the first mask. The electronic device may then determine the target mask based on the second mask to ensure the accuracy of semantic segmentation of the target mask.

[0021] In one possible design, the process of the electronic device determining the target mask based on the second mask may include:

[0022] The electronic device can extend the second mask based on the size of the image to be processed to obtain a target mask. The size of the target mask is the same as the size of the image to be processed, and the values of the pixel points in the extended area in the target mask are the same as the values of the pixel points in the first non-target object area. This can avoid the occurrence of false positive areas in the position area corresponding to the extended area on the image to be processed when semantic segmentation processing is performed on the entire image to be processed, due to the presence of objects that are easily mistakenly identified as target objects in the position area.

[0023] In one possible design approach, the electronic device can input the image to be processed into the target strategy network model to obtain the target cropping parameters and the target connected domain area threshold, thereby quickly determining the target cropping parameters and the target connected domain area threshold.

[0024] In one possible design, the target policy network model can be obtained through training. The training process may include: first, the electronic device may obtain a training image sample set. The training image sample set includes at least one training image and the true value corresponding to each training image in the at least one training image. Each training image is an image including a target object, and the true value corresponding to the training image represents the actual mask corresponding to the training image, that is, the actual binary image. Afterwards, the electronic device may train the initial policy network model based on the training image sample set to obtain the target policy network model. Based on this, the electronic device uses the training images and the actual binary images corresponding to the training images to accurately train the initial policy network model, so that the target cropping parameters and target connected domain area thresholds that can accurately determine the matching with the image can be obtained, thereby ensuring that the semantic segmentation of the image is highly accurate.

[0025] In one possible design, the training process may include:

[0026] For each training image, the electronic device may input the training image into an initial policy network model to obtain initial cropping parameters and an initial connected domain area threshold that match the training image. The electronic device may then crop the training image based on the initial cropping parameters to obtain a cropped training image. The electronic device may then perform semantic segmentation on the cropped training image to obtain a first training mask, wherein the first training mask includes at least one second target object region and a second non-target object region. An overlap value may then be determined based on the ground truth values corresponding to the first training mask and the training image. The greater the overlap value, the greater the degree of overlap between the second target object region in the first training mask and the ground truth.

[0027] Afterwards, the electronic device can determine whether the degree of overlap is less than the preset overlap threshold to determine the accuracy of this semantic segmentation, that is, to determine the matching degree between the initial cropping parameters and the initial connected domain area threshold determined by the initial strategy network model and the training image, that is, to determine whether the accuracy of the initial strategy network model meets the requirements.

[0028] When the overlap value is less than the preset overlap threshold, it indicates that the accuracy of this semantic segmentation is low, that is, the initial cropping parameters and initial connected domain area threshold determined by the initial policy network model are poorly matched with the training image, that is, it is determined that the accuracy of the initial policy network model does not meet the requirements. The electronic device can return to the step of inputting the training image into the initial policy network model to obtain the initial cropping parameters and initial connected domain area threshold that match the training image, so as to continue the next training.

[0029] When the overlap value is greater than or equal to the preset overlap threshold, it indicates that the accuracy of this semantic segmentation is high, that is, the initial cropping parameters and the initial connected domain area threshold determined by the initial policy network model are highly matched with the training image, that is, the accuracy of the initial policy network model meets the requirements, and there is no need to continue the next training. The electronic device can determine that the initial policy network model is the target policy network model.

[0030] In a possible design approach, the process of determining the overlap value may include:

[0031] For each training image, the electronic device may first perform a connected domain analysis on a first training mask corresponding to the training image based on an initial connected domain area threshold that matches the training image, and determine a target training mask corresponding to the training image. The electronic device may then determine the degree of overlap between the target training mask corresponding to the training image and the ground truth corresponding to the training image. Based on this, the electronic device determines the prediction accuracy, and therefore the training precision, of the initial policy network model based on the degree of overlap between the predicted target training mask and the ground truth corresponding to the training image.

[0032] Optionally, the process of performing connected component analysis on the first training image may include:

[0033] First, the electronic device can update the first target object region in the first training mask whose area is smaller than the target connected component area threshold to a first non-target object region, thereby obtaining a second training mask to remove false positive regions in the first training mask. The second training mask is then extended based on the size of the training image to obtain a target training mask corresponding to the training image, thereby ensuring the accuracy of semantic segmentation of the obtained target training mask.

[0034] In another possible design manner, the process of determining the overlap value may include:

[0035] For each training image, the electronic device can directly determine the degree of overlap between the second target object area in the first training mask corresponding to the training image and the second target object area in the true value corresponding to the training image, so as to use the degree of overlap to determine the accuracy of the initial policy network model.

[0036] In one possible design, the image to be processed may include an image of a person. The image of the person may be a preview image displayed by the electronic device. That is, before determining target cropping parameters and target connected domain area thresholds that match the image to be processed based on the image to be processed, the electronic device displays a shooting interface; wherein the shooting interface includes the image of the person.

[0037] Accordingly, after determining the target mask corresponding to the image to be processed, the electronic device can perform a first processing on the person image based on the target mask corresponding to the image to be processed to obtain a target person image. The first processing includes at least one or more of portrait enhancement processing, image cutout processing, filter processing, and background blur processing.

[0038] Afterwards, the electronic device can display the target person image to process the person in the person image and ensure a better processing effect.

[0039] In a second aspect, the present application provides a model training method, which is applied to a first device, wherein the first device obtains a training image sample set; wherein the training image sample set includes at least one training image and a true value corresponding to each training image; each training image is an image including a target object; and the true value corresponding to the training image represents an actual mask corresponding to the training image;

[0040] Based on the training image sample set, the initial policy network model is trained to obtain the target policy network model; wherein the target policy network model outputs target cropping parameters and target connected domain area thresholds that match the image.

[0041] In a third aspect, the present application provides a chip system, which is applied to an electronic device that executes the image processing method as described in any one of the first aspects above.

[0042] In a fourth aspect, the present application provides a chip system, which is applied to a first device, and the first device executes the model training method as described in any one of the above-mentioned second aspects.

[0043] In a fifth aspect, the present application provides an electronic device, comprising a memory and one or more processors; the memory and the processor are coupled; the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device performs the image processing method as described in any one of the above-mentioned first aspects.

[0044] In the sixth aspect, the present application provides an electronic device, which can be used as the above-mentioned first device, wherein the first device includes a memory and one or more processors; the memory and the processor are coupled; the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the first device executes the model training method as described in any one of the above-mentioned second aspects.

[0045] In a seventh aspect, the present application provides a computer-readable storage medium comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the image processing method as described in any one of the above-mentioned first aspects.

[0046] In an eighth aspect, the present application provides a computer-readable storage medium comprising computer instructions, which, when executed on a first device, causes the first device to execute the model training method as described in any one of the second aspects above.

[0047] In a ninth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the image processing method as described in any one of the first or second aspects above.

[0048] It can be understood that the beneficial effects that can be achieved by the model training method described in the second aspect, the chip system described in the third and fourth aspects, the electronic device described in the fifth and sixth aspects, the computer storage medium described in the seventh and eighth aspects, and the computer program product described in the ninth aspect can refer to the beneficial effects in the first aspect and any possible design method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1A Schematic diagram 1 of a semantic segmentation process provided in an embodiment of the present application;

[0050] Figure 1B Schematic diagram 2 of a semantic segmentation process provided in an embodiment of the present application;

[0051] Figure 2A A schematic diagram of a semantic segmentation process provided in an embodiment of the present application Figure 3 ;

[0052] Figure 2B A schematic diagram of a semantic segmentation process provided in an embodiment of the present application Figure 4 ;

[0053] Figure 3 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0054] Figure 4 A schematic diagram of a semantic segmentation process provided in an embodiment of the present application Figure 5 ;

[0055] Figure 5 Schematic diagram 1 of a flow chart of an image processing method provided in an embodiment of the present application;

[0056] Figure 6 A schematic diagram of a training sample and its corresponding true value provided in an embodiment of the present application;

[0057] Figure 7A Schematic diagram 1 of a cropping parameter provided in an embodiment of the present application;

[0058] Figure 7B Schematic diagram 2 of a cropping parameter provided in an embodiment of the present application;

[0059] Figure 8 A schematic diagram of a semantic segmentation process provided in an embodiment of the present application Figure 6 ;

[0060] Figure 9 Schematic diagram 2 of a flow chart of an image processing method provided in an embodiment of the present application;

[0061] Figure 10A Schematic diagram 1 of an image processing scenario provided in an embodiment of the present application;

[0062] Figure 10B Schematic diagram 2 of an image processing scenario provided in an embodiment of the present application;

[0063] Figure 10C A schematic diagram of an image processing scenario provided in an embodiment of the present application Figure 3 ;

[0064] Figure 10D A schematic diagram of an image processing scenario provided in an embodiment of the present application Figure 4 ;

[0065] Figure 10E A schematic diagram of an image processing scenario provided in an embodiment of the present application Figure 5 ;

[0066] Figure 10F A schematic diagram of an image processing scenario provided in an embodiment of the present application Figure 6 . DETAILED DESCRIPTION

[0067] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0068] To facilitate understanding of the embodiments of the present application, some terms involved in the embodiments of the present application are introduced below.

[0069] Semantic segmentation: refers to the assignment of pixels in an image to predefined semantic categories. Pixels in the image are given a label indicating the semantic category to which they belong, such as people, vehicles, roads, trees, and so on.

[0070] A global convolutional neural network (CNN) typically consists of multiple convolutional and pooling layers, used to extract and classify features from input data. Compared to conventional CNNs, a global CNN utilizes a global pooling layer instead of the fully connected layers. This global pooling layer not only reduces the dimensionality and compresses the feature maps output by the convolutional layers, but also preserves more positional and spatial structural information. This reduces the number of model parameters, improves computational efficiency, and effectively avoids overfitting.

[0071] A policy network is a key concept in reinforcement learning. A policy π represents the probability distribution of different actions taken under a given state. In this embodiment, the state can be understood as the proportion of the target object (or object to be segmented) in the image, and the action can be understood as the cropping parameters and connected domain area threshold corresponding to the image.

[0072] A mask is used to represent a binary image. Each pixel in a binary image has a value of 0 or 255. In semantic segmentation tasks, a pixel value of 0 indicates that the pixel does not belong to the object to be segmented, while a pixel value of 255 indicates that the pixel belongs to the object to be segmented.

[0073] The ground truth (GT) generally refers to an image that has been manually annotated. In the embodiment of the present application, the ground truth may refer to a binary image that has been manually annotated with target object areas and non-target object areas.

[0074] IoU (intersection over union) is an evaluation metric used in semantic segmentation tasks. It is used to compare the degree of overlap between the predicted image and the true image to measure the accuracy of semantic segmentation. In embodiments of the present application, in relevant models (such as policy network models and segmentation network models), the IoU value between the target mask corresponding to the predicted training image and the true value corresponding to the training image can be measured to determine whether to stop training the relevant model.

[0075] With the development of semantic segmentation technology, it has been widely used in various scenarios, such as autonomous driving, medical image analysis, and image processing (such as cropping, portrait enhancement, background blur, depth of field adjustment, smart filters, and automatic data labeling). Electronic devices generally use semantic segmentation models to perform semantic segmentation tasks to achieve semantic segmentation of images.

[0076] In some embodiments, the execution process of the above-mentioned semantic segmentation task may be that, first, the electronic device may crop the image to determine a screenshot including the object to be segmented. Afterwards, the electronic device inputs the screenshot into the target segmentation network model so that the target segmentation network model performs semantic segmentation processing on the screenshot to obtain a mask 1 corresponding to the screenshot, that is, a mask 1 corresponding to the image. The mask 1 is a binary image, which includes at least one object area to be segmented and at least one non-object area to be segmented. Specifically, the value of the pixel in the object area to be segmented may be 255, and the value of the pixel in the non-object area to be segmented may be 0. Wherein, the object area to be segmented and the non-object area to be segmented are both connected domains.

[0077] Afterwards, the electronic device can continue to perform connected domain analysis on Mask 1 using the connected domain area threshold to determine whether the area of the object to be segmented on Mask 1 is a false positive area, thereby correcting the false positive area on Mask 1 to a non-object to be segmented area, thereby obtaining Mask 2. A false positive area refers to an area that does not include the image of the object to be segmented, that is, an area that is actually a non-object to be segmented area but is mistakenly determined by the target segmentation network model to be an area to be segmented.

[0078] Specifically, for each of the at least one object region to be segmented, the electronic device may determine whether the area of the object region to be segmented is greater than or equal to a connected domain area threshold. If the area of the object region to be segmented is less than the connected domain area threshold, it indicates that the initial object region to be segmented is a false positive region, i.e., the initial object to be segmented does not include the object to be segmented. The electronic device may correct the object region to be segmented to a non-object region to be segmented.

[0079] When the area of the object region to be segmented is greater than or equal to the connected domain area threshold, it indicates that the object region to be segmented is not a false positive region, that is, the initial object region to be segmented includes the object to be segmented, and the electronic device may not need to correct the object region to be segmented to a non-object region to be segmented.

[0080] Afterwards, the electronic device may determine the target mask corresponding to the image based on the mask 2 corresponding to the image. Afterwards, the electronic device may perform subsequent processing (such as portrait enhancement, background blur, smart filter, etc.) on the image based on the mask 2 corresponding to the image.

[0081] However, the proportion of the object to be segmented in the screenshot (or the cropping ratio) will affect the accuracy of the mask 1 output by the target segmentation network model, thereby affecting the accuracy of the target mask, that is, affecting the accuracy of the semantic segmentation of the image. Figure 1A As shown in the figure, when the cropping ratio is not appropriate, such as when the cropping ratio is too large, the accuracy of the mask 1 output by the target segmentation network model is low, and the object to be segmented on the mask 1 is quite different from the object to be segmented on the original image (such as the shape of the person's shoes changes), which leads to low accuracy of the segmented object and poor semantic segmentation effect, which in turn causes poor subsequent image processing effect. Figure 1B As shown in the figure, when the cropping ratio is appropriate, the mask 1 output by the target segmentation network model has high accuracy, and the difference between the object to be segmented on the mask 1 and the object to be segmented on the original image is small (such as the shape of the person's shoes is the same as the actual shape), so that the accuracy of the segmented object to be segmented is high, the semantic segmentation effect is better, and the effect of subsequent image processing is also better.

[0082] In addition, the size of the connected domain area threshold also affects the accuracy of the target mask, that is, the accuracy of semantic segmentation. Figure 2A As shown in the figure, when the connected domain area threshold is small, the false positive area in mask 1 cannot be accurately identified, resulting in the existence of false positive areas on mask 2, which leads to low accuracy of the target mask, that is, low accuracy of the segmented object and low semantic segmentation accuracy.

[0083] And as Figure 2BAs shown in the figure, when the connected domain area threshold is appropriate, the false positive area in mask 1 can be accurately identified, so that there is a false positive area on mask 2, thereby obtaining a target mask with higher accuracy, realizing accurate segmentation of the object to be segmented, and ensuring the accuracy of semantic segmentation.

[0084] Therefore, in response to the above problems, the present application proposes an adaptive semantic segmentation framework, which starts from the perspective of image pre- and post-processing. For the image to be processed, the electronic device can determine the target cropping parameters and target connected domain area threshold that match the image to be processed, and realize the adaptive determination of the pre- and post-processing parameters. Afterwards, the electronic device can use the target cropping parameters to crop the image to be processed to obtain a screenshot with a suitable cropping ratio, and realize adaptive pre-processing of the image to be processed, so that the proportion of the object to be segmented in the image to be processed is appropriate. Afterwards, the electronic device uses the target segmentation network model to perform semantic segmentation processing on the screenshot to determine the semantic category to which the pixel points in the screenshot belong, and outputs the mask 1 corresponding to the screenshot. The mask 1 includes at least one object area to be segmented and at least one non-object area to be segmented. Afterwards, the electronic device can use the target connected domain area threshold to perform connected domain analysis on Mask 1, correcting the to-be-segmented object regions in Mask 1 that are smaller than the target connected domain area threshold to non-to-be-segmented object regions, thereby implementing adaptive post-processing of the processed image. This results in a highly accurate target mask, avoids false positive regions in the target mask, accurately segments the to-be-segmented object, ensures semantic segmentation, and ultimately, guarantees the effectiveness of subsequent image processing. Furthermore, by performing pre- and post-processing on the processed image, the electronic device can achieve high segmentation accuracy with minimal computational overhead, reducing waste of electronic device resources and providing a high user experience.

[0085] For example, the electronic devices in the embodiments of the present application may be mobile phones, drones, smart cameras, wearable devices (such as smart watches and smart bracelets), tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), media players and other devices with data processing capabilities. The embodiments of the present application do not impose any special restrictions on the specific form of the electronic devices.

[0086] For example, Figure 3 Schematic diagram of the structure of the electronic device 200 is shown. Figure 3As shown, the electronic device 200 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 211, a power management module 212, a battery 213, an antenna 1, an antenna 2, a mobile communication module 240, a wireless communication module 250, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc.

[0087] It should be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0088] The processor 210 may include one or more processing units. For example, the processor 210 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0089] The controller may be the nerve center and command center of the electronic device 200. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0090] Processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 210 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 210. If processor 210 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 210 latency, and thus improves system efficiency.

[0091] In some embodiments, the processor 210 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0092] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0093] While the charging management module 211 charges the battery 213 , it can also provide power to the electronic device through the power management module 212 .

[0094] The wireless communication function of the electronic device 200 can be implemented through the antenna 1, the antenna 2, the mobile communication module 240, the wireless communication module 250, the modem processor and the baseband processor.

[0095] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals.

[0096] Mobile communication module 240 can provide wireless communication solutions, including 2G / 3G / 4G / 5G, for electronic device 200. Mobile communication module 240 can receive electromagnetic waves through antenna 1, filter and amplify the received electromagnetic waves, and transmit them to the modem processor for demodulation. Mobile communication module 240 can also amplify the signals modulated by the modem processor and convert them into electromagnetic waves for radiation via antenna 1.

[0097] The modulation and demodulation processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal.

[0098] The wireless communication module 250 can provide wireless communication solutions including wireless local area networks (WLAN) (such as Wi-Fi (wireless fidelity) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., applied to the electronic device 200. The wireless communication module 250 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and filters it, and sends the processed signal to the processor 210. The wireless communication module 250 can also receive the signal to be sent from the processor 210, modulate the frequency of the signal, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0099] The electronic device 200 implements the display function through the GPU, the display screen 294 , and the application processor, etc. In some embodiments, the electronic device 200 may include 1 or N display screens 294 , where N is a positive integer greater than 1.

[0100] The electronic device 200 can realize the shooting function through the ISP, camera 293, video codec, GPU, display screen 294 and application processor, etc. The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 200.

[0101] The internal memory 221 can be used to store computer executable program codes, which include instructions. The processor 210 executes various functional applications and data processing of the electronic device 200 by running the instructions stored in the internal memory 221. The internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function, etc. The data storage area can store data created during the use of the electronic device 200, etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0102] The electronic device 200 can implement audio functions such as music playback and recording through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the headphone jack 270D, and the application processor.

[0103] The buttons 290 include a power button, a volume button, etc. The indicator 292 can be used to indicate the charging status or a message. The sensor module 280 can include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0104] This application proposes an image processing method. Figure 4As shown, first, the electronic device can input the image to be processed into a global convolutional neural network model to determine the global features corresponding to the image to be processed. The height of the image to be processed is H, and the width is W. The electronic device can then input the global features corresponding to the image to be processed into a target policy network model, so that the target policy network model determines target cropping parameters and a target connected domain area threshold that match the image to be processed. The target cropping parameters can serve as pre-processing parameters, and the target connected domain area threshold can serve as a post-processing parameter. The electronic device can then use the target cropping parameters to crop the image to be processed, obtaining a screenshot 1 corresponding to the image to be processed, where the proportion of the objects to be segmented in the screenshot 1 is appropriate, thereby achieving accurate pre-processing of the image to be processed and ensuring the accuracy of the subsequent semantic segmentation performed using the screenshot 1. The height of the screenshot 1 is h, and the width is w. The electronic device can then input the screenshot 1 into the target segmentation network model, so that the target segmentation network model performs semantic segmentation on the screenshot 1 and generates and outputs a mask 1 corresponding to the screenshot 1. Among them, mask 1 is the binary image of screenshot 1, and the values of the pixels in the area of the object to be segmented on mask 1 are different from the values of the pixels in the area not to be segmented. Afterwards, the electronic device can perform a connected domain analysis on mask 1, that is, compare the area of the area of the object to be segmented in mask 1 with the connected domain area threshold, so as to correct the false positive area of the object to be segmented (i.e., the false positive area) to the area not to be segmented, and obtain mask 2, thereby removing the false positive area on mask 1, thereby achieving accurate post-processing of the image to be processed, and thus ensuring the semantic segmentation accuracy of the image to be processed. Among them, the height of mask 1 and mask 2 are both h, and the width is also w.

[0105] Afterwards, the electronic device can restore mask 2 to restore the size of mask 2 to the size of the image to be processed, and obtain a target mask corresponding to the image to be processed, so that the electronic device can continue to perform relevant processing on the image to be processed based on the target mask, thereby obtaining an image that meets the requirements.

[0106] Among them, the above-mentioned target segmentation network model and target policy network model are obtained through training. The electronic device can first train the initial policy network model and the initial segmentation network model based on the training image and its corresponding true value to obtain a target policy network model that can accurately determine the cropping parameters and connected domain area thresholds that match the image, as well as a target segmentation network model that can realize accurate semantic segmentation.

[0107] The following will take the above-mentioned electronic device as a mobile phone and the object to be segmented as a person as an example, and the image processing method provided by the embodiment of the present application will be divided into two parts for introduction. One part is used to introduce the training process of the initial policy network model and the initial segmentation network model, that is, to introduce the training process based on a general adaptive segmentation training framework. The other part is used to introduce the process of semantic segmentation using the target policy network model and the target segmentation network obtained by training, that is, to introduce the process of semantic segmentation based on a general adaptive segmentation reasoning framework.

[0108] The following will first introduce Figure 5 , introduces the training process based on an adaptive segmentation training framework. Figure 5 As shown, the image processing method provided in the embodiment of the present application may include S301-S315.

[0109] S301: The mobile phone obtains a training image sample set, wherein the training image sample set includes at least one training image and a true value corresponding to each training image in the at least one training image.

[0110] Each of the above training images includes an object to be segmented, which can be a person. The true value corresponding to the training image represents the standard binary image corresponding to the training image, which can be obtained by relevant personnel by accurately segmenting the person in the training image. For example, Figure 6 Shown is a training image and its corresponding ground truth.

[0111] The height of the training image is H, and the width is W. The height of the true value corresponding to the training image is also H, and the width is also W. In addition, the sizes (i.e., height and width) between the training images can be the same or different, and this application does not limit them.

[0112] Optionally, the size of the training image may be the same as or different from the size of the image to be processed.

[0113] S302: For each training image, the mobile phone inputs the training image into the global neural network model to obtain the global features corresponding to the training image.

[0114] In the embodiment of the present application, in order to improve the image processing efficiency and improve the training efficiency of the initial network model (such as the initial strategy network model and the initial segmentation network model), for each training image, the mobile phone can input the training image into the global convolutional neural network model, so that the global convolutional neural network model performs a rough processing on the training image, that is, performs dimensionality reduction and compression processing, and obtains the global feature e corresponding to the training image. G In other words, e G =f G (i). Where i represents the training image.G () represents the global convolutional neural network model.

[0115] Optionally, the global features corresponding to the training images may be one-dimensional features.

[0116] In some embodiments, the global convolutional neural network model can be a lightweight convolutional network (such as MobileNet-V2) model, or a shallow neural network model to reduce computational complexity and thus improve the training efficiency of the initial network model.

[0117] In some embodiments, the above S302 is an optional step, that is, the mobile phone may not determine the global features corresponding to the training image, but directly use the training image to train the initial network model. Simply put, the operations performed on the global features corresponding to the training image described below can be replaced by descriptions of related operations on the training image.

[0118] S303. The mobile phone trains an initial policy network model based on the global features corresponding to each training image and the true value corresponding to each training image.

[0119] S304: After the training is completed, for each training image, the mobile phone obtains the initial cropping parameters and initial connected domain area threshold corresponding to the training image output by the initial strategy network model.

[0120] Among them, the initial cropping parameters and initial connected domain area threshold corresponding to the training image represent the initial cropping parameters and initial connected domain area threshold predicted by the initial policy network model that match the training image.

[0121] S305 : The mobile phone crops the training image based on the initial cropping parameters to obtain a screenshot 2 corresponding to the training image.

[0122] In an embodiment of the present application, the initial policy network model updates the parameters of the initial policy network model by learning the global features corresponding to the training image and the true value corresponding to the training image, so that the initial policy network model can output pre- and post-processing parameters suitable for the training image, thereby realizing the training of the initial policy network model. In addition, the training may include multiple rounds of training (i.e., multiple trainings). After a round of training, for each training image, the mobile phone can obtain the initial cropping parameters and initial connected domain area thresholds corresponding to the training image predicted by the initial policy network model during the current round of training (or described as after the current round of training), so as to determine the training effect of the initial policy network model using the initial cropping parameters and the initial connected domain area threshold, thereby determining whether it is necessary to continue training the initial policy network model.

[0123] The above-mentioned initial cropping parameters can be determined by the initial strategy network model based on the constraints corresponding to the pre-processing parameters. For example, the initial cropping parameters corresponding to the training image include w1, w2, h1 and h2 corresponding to the training image. In one case, Figure 7A As shown, w1 represents the distance between the left boundary of the character boundary area on screenshot 2 (herein may be described as a training image or a true value) and the left boundary of screenshot 2. w2 represents the distance between the right boundary of the character boundary area on screenshot 2 and the right boundary of screenshot 2. h1 represents the distance between the upper boundary of the character boundary area on screenshot 2 and the upper boundary of screenshot 2. h2 represents the distance between the lower boundary of the character boundary area on screenshot 2 and the lower boundary of screenshot 2. Accordingly, the above constraints can be as follows st:

[0124]

[0125] Wherein, the above x1 can be the horizontal coordinate of the upper left corner vertex of the character boundary area on the training image (here or replaced by the true value), and y1 can be the vertical coordinate of the upper left corner vertex of the character boundary area on the training image. The above x2 can be the horizontal coordinate of the lower right corner vertex of the character boundary area on the training image, and y2 can be the vertical coordinate of the lower right corner vertex of the character boundary area on the training image. Figure 7A or Figure 7B As shown, the top left vertex of the training image (or the true value here) can be the coordinate origin (0, 0). Correspondingly, x1 < x2, y1 < y2. Of course, using the top left vertex as the coordinate origin is only an example, and other vertices can also be used as the coordinate origin, and this application is not limited to this.

[0126] In some embodiments, the position information of the character boundary area in the above-mentioned initial policy network model (such as the coordinates of the upper left vertex and the lower right vertex of the above-mentioned character boundary area) can be determined by the initial policy network model. On the one hand, the initial policy network model has the ability to detect the position information of the character. Among them, the initial policy network model can obtain image area information related to the task (such as the position information of the above-mentioned character boundary area). Simply put, it is equivalent to adding a detection module to the initial policy network model, and taking the global features corresponding to the image (such as the training image) as the input of the initial policy network model to determine the image area information related to the task, reduce the redundancy of the image information, and reduce the interference of irrelevant image information. On the other hand, the training image or the true value corresponding to the training image is marked with the character boundary area, so that the initial policy network model can directly obtain the position information of the character boundary area through the training image or the true value.

[0127] Of course, the above-mentioned method of determining the location information of the boundary area of the character by the initial policy network model is only an example. The mobile phone (such as the initial policy network model) can also be determined based on other methods. For example, the initial policy network model uses the global features corresponding to the training image to determine it, or the initial policy network model uses the value of the pixel point on the true value (i.e., 255 or 0) to determine it.

[0128] It should be noted that the operations performed by the network model in the embodiment of the present application (such as the above-mentioned initial strategy network model) are actually performed by the mobile phone. In addition, in this case, the mobile phone can be based on the position information of the character boundary area on the training image. For each training image, the mobile phone can be based on the position information of the character boundary area on the training image, combined with the initial cropping parameters, to crop the training image to obtain a screenshot 2 including the character boundary area (or called a cropped image after training), reducing the probability of cropping the character boundary area on the training image, and realizing accurate cropping of the training image. Among them, the height of screenshot 2 can be h, and the width can be w, where h=h1+h2+y2-y1, and w=w1+w2+x2-x1. For example, combined with Figure 7A For example, first, the phone can determine the position information of the left boundary of screenshot 2 based on the position information of the person boundary area and w1. Then, based on the boundary information of the left boundary of screenshot 2, the phone can crop the image portion between the left boundary of screenshot 2 and the left boundary of the training image. Similarly, the phone can crop the image portion between the right boundary of screenshot 2 and the right boundary of the training image. The phone can crop the image portion between the upper boundary of screenshot 2 and the upper boundary of the training image. The phone can crop the image portion between the lower boundary of screenshot 2 and the lower boundary of the training image.

[0129] In another case, if Figure 7B As shown, w1 represents the distance between the left boundary of the training image and the left boundary of screenshot 2. w2 represents the distance between the right boundary of the training image and the right boundary of screenshot 2. h1 represents the distance between the upper boundary of the training image and the upper boundary of screenshot 2. h2 represents the distance between the lower boundary of the training image and the lower boundary of screenshot 2. Accordingly, the above constraints can be as follows:

[0130]

[0131] In addition, in this other case, for each training image, the mobile phone can crop the training image based on the initial cropping parameters to obtain a screenshot 2 including the character boundary area, thereby reducing the probability of cropping the character boundary area on the training image and achieving accurate cropping of the training image. Figure 7BFor example, first, the phone can determine the position information of the left boundary of screenshot 2 based on the position information of the character boundary area and w1. Then, based on the boundary information of the left boundary of screenshot 2, the phone can crop the image portion between the left boundary of screenshot 2 and the left boundary of the training image. Similarly, the phone can crop the image portion between the right boundary of screenshot 2 and the right boundary of the training image. The phone can crop the image portion between the upper boundary of screenshot 2 and the upper boundary of the training image. The phone can crop the image portion between the lower boundary of screenshot 2 and the lower boundary of the training image.

[0132] It can be understood that in this other case, the mobile phone does not need to use the position information of the character boundary area to crop the training image. For example, when the character in the training image is located on the left side of the training image, the left boundary of the character boundary area coincides with the left boundary of the training image. Therefore, w1 in the initial cropping parameter determined by the initial strategy network model can be 0.

[0133] In some embodiments, the initial policy network model may determine a constraint range based on the above constraint conditions, so as to select a trimming parameter from the constraint range and determine it as the initial trimming parameter.

[0134] Among them, optionally, the initial policy network model can randomly select cropping parameters from the constraint range. Alternatively, the cropping parameters are started from the boundary value, that is, starting from the maximum value, and gradually reducing the value (it can be simply understood that the size of screenshot 2 is the same as the size of the training image at the beginning, and then the size of screenshot 2 begins to gradually decrease). Alternatively, the value can be taken from 1 / 2 of the value range, that is, w1 can be taken from x1 / 2, w2 can be taken from (W-x2) / 2, h1 can be taken from y1 / 2, and h2 can be taken from (H-y2) / 2, thereby accelerating the convergence speed of the initial policy network model, and then accelerating the convergence speed of the overall adaptive segmentation training framework, and improving training efficiency.

[0135] It should be noted that, in practical applications, there may be multiple distribution situations (such as distribution position, proportion) of the objects to be segmented in the image to be semantically segmented (i.e., the above-mentioned image to be processed). For example, the objects to be segmented on some images to be processed are located on the left side of the image to be processed, and the objects to be segmented on some images to be processed are located on the right side of the image to be processed. Therefore, in order to better train the initial policy network model so that the trained initial policy network model can accurately determine the target cropping parameters and target connected domain area threshold that match the distribution situation (or called distribution information) of the objects to be segmented in the input image to be processed, the above-mentioned training image sample set may include training images with different distribution situations.

[0136] The above introduces the training process of the initial policy network model. The following will continue to introduce the process of training the initial segmentation network.

[0137] S306 : The mobile phone trains the initial segmentation network model based on the screenshot 2 corresponding to each training image and the true value corresponding to each training image.

[0138] S307. After the training is completed, for each screenshot 2, the mobile phone obtains the training mask 1 corresponding to the screenshot 2 output by the initial segmentation network model.

[0139] Among them, the initial segmentation network model is used to perform semantic segmentation on the image.

[0140] The training mask 1 (or the first training mask) corresponding to the above-mentioned screenshot 2 (or the cropped training image) refers to the binary image corresponding to the screenshot 2 obtained by the initial segmentation network model segmentation, which can be understood as the binary image predicted by the initial segmentation network model. The training mask 1 includes at least one person area (i.e., the object area to be segmented) and at least one non-person area (i.e., the non-object area to be segmented). Each person area in the at least one person area (or can be replaced by being described as the second target object area) includes a person. Each non-person area in the at least one non-person area (or can be replaced by being described as the second non-target object area) does not include a person.

[0141] The value of the pixel in the above-mentioned human area can be 255, and accordingly, the color of the human area can be white. The value of the pixel in the above-mentioned non-human area can be 0, and the color of the non-human area can be black. Of course, the value of the pixel in the human area can also be 0, and the value of the pixel in the non-human area can be 255, that is, the value of the pixel in the human area can be 255 or 0, and the value of the pixel in the non-human area can be 0 or 255, as long as the value of the pixel in the human area is different from the value of the pixel in the non-human area.

[0142] In some embodiments, the above-mentioned training mask 1 can be determined based on a prediction probability map. The prediction probability map represents the probability corresponding to each pixel point on the screenshot, and the probability corresponding to the pixel point represents the probability that the pixel point belongs to a person. The higher the probability corresponding to the pixel point, the higher the probability that the pixel point belongs to a person. The present application can set a binarization threshold. When the probability corresponding to the pixel point is greater than the binarization threshold, the probability that the pixel point belongs to a person is very high, and the mobile phone can regard the pixel point as the foreground. When the probability corresponding to the pixel point is less than the binarization threshold, the probability that the pixel point does not belong to a person is very low, and the mobile phone can regard the pixel point as the background.

[0143] The probability corresponding to the above pixel points is between 0 and 1. The above binarization threshold can be set according to actual needs, such as 0.5, 0.4 or 0.6, etc., as long as it is a value greater than 0 and less than or equal to 1.

[0144] In some embodiments, the initial segmentation network model can perform semantic segmentation on the screenshot 2 corresponding to the training image and output a predicted probability map 1 corresponding to the training image. The mobile phone can then identify pixels on the predicted probability map 1 with probabilities greater than a binarization threshold as pixels in the person area, and pixels with probabilities less than or equal to the binarization threshold as pixels in the non-target area, thereby obtaining the corresponding training mask 1.

[0145] In other embodiments, the initial segmentation network model first performs semantic segmentation on the screenshot 2 corresponding to the training image to obtain a predicted probability map 1 corresponding to the training image. Afterwards, the initial segmentation network model uses the pixels on the predicted probability map with a probability greater than the binarization threshold as pixels in the person area, and uses the pixels with a probability less than or equal to the binarization threshold as pixels in the non-target area, thereby obtaining and outputting the corresponding training mask 1. In other words, the initial segmentation network model can directly output the training mask 1, i.e., m crop =S(i crop ). Among them, m crop Represents training mask 1, i crop Indicates the screenshot corresponding to the training image 2. S represents the initial segmentation network model.

[0146] Optionally, in this embodiment, the initial segmentation network model can not only output a training mask 1 corresponding to the training image, but also output a prediction probability map 1.

[0147] It should be noted that when the probability corresponding to a pixel point on the predicted probability map is equal to the binarization threshold, the mobile phone can use the pixel point as a pixel point in the person area, or use the pixel point as a pixel point in the non-person area. This application does not limit it.

[0148] In an embodiment of the present application, the initial segmentation network model is based on the screenshot 2 corresponding to the training image, combined with the true value corresponding to the training image, to update the parameters of the initial segmentation model, so that the initial segmentation model can accurately predict the character area and non-character area on the screenshot 2, and realize the semantic segmentation of the screenshot 2, thereby realizing the training of the initial segmentation network model. In addition, the training may include multiple rounds of training (i.e., multiple trainings). After a round of training, for each training image, the mobile phone can obtain the training mask 1 corresponding to the screenshot 2 predicted by the initial segmentation network model during the current round of training (or described as after the current round of training), so as to use the training mask 1 to determine the training effect of the initial segmentation network model, thereby determining whether it is necessary to continue training the initial segmentation network model.

[0149] The above describes the training process of the initial segmentation network model. The following will continue to describe how to determine whether to stop the training of the initial segmentation network model and the initial policy network model.

[0150] S308 : For each person region in the training mask 1 , the mobile phone determines whether the area of the person region is greater than or equal to the initial connected domain area threshold.

[0151] In the embodiment of the present application, generally speaking, the proportion of people in the image is large, and correspondingly, the area of the person region is also large. Therefore, for each person region on the training mask 1 corresponding to screenshot 2, the mobile phone determines whether the area of the person region is greater than or equal to the initial connected domain area threshold to determine whether the person region is a false positive region.

[0152] When the area of the person region is greater than or equal to the initial connected domain area threshold, it indicates that the area of the person region is large and the person region is not a false positive region, and the mobile phone may execute S309.

[0153] When the area of the person region is smaller than the initial connected domain area threshold, it indicates that the area of the person region is small and the person region may not actually be a person region. The person region is a false positive region, and the mobile phone may execute S310.

[0154] S309: The mobile phone does not correct the person area on the training mask 1.

[0155] In the case where the person region on the training mask 1 is not a false positive region, the mobile phone does not process the person region.

[0156] S310: The mobile phone corrects the person area to a non-person area, and obtains a training mask 2.

[0157] Correcting the human area to a non-human area may refer to updating the pixel values of the human area to the pixel values of the non-human area, such as updating from 255 to 0. For example, the mobile phone corrects all human areas on training mask 1 whose area is smaller than the initial connected domain area threshold to non-human areas. That is, after the connected domain analysis is completed, the training mask 2 (i.e., m thr ) to remove the false positive areas on the training mask 1 and further improve the semantic segmentation accuracy.

[0158] S311 . The mobile phone restores the training mask 2 based on the size of the training image to obtain a target training mask corresponding to the training image.

[0159] The size of the target training mask corresponding to the training image is the same as the size of the training image.

[0160] In an embodiment of the present application, for each training image, the mobile phone can restore (or describe as extend) based on the training mask 2 (or called the second training mask) corresponding to the training image to obtain a target training mask with the same size as the training image, thereby restoring the original image size, so as to facilitate the subsequent determination of the training efficiency of the initial network model by comparing the target training mask corresponding to the training image with the true value corresponding to the training image.

[0161] Among them, the target training mask corresponding to the above training image can be understood as the true value obtained by the prediction corresponding to the training image, that is, the true value obtained by semantic segmentation through the initial network model.

[0162] In some embodiments, the restoration process may include: the mobile phone may use training mask 2 corresponding to the training image as a reference, and extend background pixels above, below, left, and right of training mask 2, respectively, until a target training mask having a height equal to the height of the training image and a width equal to the width of the training image is obtained. The values of the pixels in the extended portion are the same as the values of the pixels in the non-person area.

[0163] The extended portion may include a left extended portion, which represents the portion from the left boundary of training mask 2 to the left boundary of the training image. For example, if w1, w2, h1, and h2 in the initial cropping parameters are w1, w2, h1, and h2 in the above case, the extended distance corresponding to the left extended portion may be x1-w1.

[0164] The extended portion may include a right extended portion, which represents the portion from the right boundary of the training mask 2 to the right boundary of the training image. The extension distance corresponding to the right extended portion may be W-x2-w2.

[0165] The extended portion may include an upper extended portion, which represents a portion from an upper boundary of the training mask 2 to an upper boundary of the training image. The extended distance corresponding to the upper extended portion may be y1-h1.

[0166] The extended portion may include a lower extended portion, which represents the portion from the lower boundary of the training mask 2 to the lower boundary of the training image. The extended distance corresponding to the lower extended portion may be H-y2-h2.

[0167] In an embodiment of the present application, the mobile phone extends the training mask 2 to obtain the target training mask, so that there is no false positive area in the extended part of the target training mask relative to the training mask 2, thereby reducing segmentation false positives and avoiding segmentation of the entire image. When there is an object in the extended part of the image that is easily segmented into a human object, the object in the image is mistakenly segmented as a human object, resulting in the occurrence of semantic segmentation errors, which can effectively improve the accuracy of semantic segmentation.

[0168] In some embodiments, after a round of training, the mobile phone can determine the semantic segmentation accuracy by comparing the similarity between the predicted true value and the actual true value corresponding to the training image. The mobile phone can then use this semantic segmentation accuracy to determine whether the accuracy of the initial network model after this round of training meets the requirements, thereby determining whether to stop training the initial network model. The following describes the process of determining whether to stop training the initial network model, using the example of similarity being an overlap value, in conjunction with S312-S315.

[0169] S312. The mobile phone calculates the overlap value between the target training mask corresponding to the training image and the true value corresponding to the training image.

[0170] The greater the overlap value, the higher the accuracy of the initial network model. For example, the overlap value may be an IoU value.

[0171] In some embodiments, S312 introduces only one possible method for calculating the degree of overlap. The mobile phone can also determine the degree of overlap according to other methods, such as calculating the degree of overlap based on the above-mentioned training mask 1 and the true value. Specifically, the degree of overlap between the person area in the training mask 1 and the person area in the true value can be calculated.

[0172] S313: The mobile phone determines whether the overlap degree value is greater than or equal to a preset overlap threshold.

[0173] In an embodiment of the present application, when the overlap degree value is greater than or equal to the preset overlap threshold, it indicates that the overlap between the training mask 1 corresponding to the training image predicted by the initial segmentation network model and the true value corresponding to the training image is high, that is, the accuracy of the training mask 1 obtained based on the segmentation of the initial segmentation network model is high, that is, the semantic segmentation accuracy of the initial segmentation network model is high. In other words, the screenshot 2 obtained by cropping based on the initial cropping parameters output by the initial strategy network model can enable the semantic segmentation of the initial segmentation network model to achieve a higher accuracy. Therefore, the accuracy of the initial strategy network model is also high, and it can output appropriate cropping parameters and connected domain area thresholds, and the mobile phone can execute S314.

[0174] When the overlap value is less than the preset overlap threshold, it indicates that the overlap between the training mask 1 corresponding to the training image predicted by the initial segmentation network model and the true value corresponding to the training image is low, that is, the accuracy of the training mask 1 obtained based on the segmentation of the initial segmentation network model is low, that is, the semantic segmentation accuracy of the initial segmentation network model is low. In other words, the screenshot 2 obtained by cropping based on the initial cropping parameters output by the initial strategy network model cannot enable the semantic segmentation of the initial segmentation network model to achieve a higher accuracy. Therefore, the accuracy of the initial strategy network model is also low, and it is unable to output appropriate cropping parameters and connected domain area thresholds. The mobile phone can execute S315.

[0175] In some embodiments, the preset overlap threshold can be pre-set or adjusted by the mobile phone as the loss function of the segmentation network model changes. As the number of training rounds increases, the segmentation accuracy of the initial segmentation network model increases, and accordingly, the loss function value of the initial segmentation network model decreases. As the loss function value decreases, the preset overlap threshold can be increased.

[0176] It is understandable that the loss function of the initial segmentation network model does not always decrease. As the number of training rounds increases, the initial segmentation network model will gradually converge. Correspondingly, the loss function of the initial segmentation network model usually shows a downward trend and then tends to stabilize. Therefore, the above-mentioned preset overlap threshold can be gradually increased as the loss function decreases. Then, when the loss function tends to stabilize, the preset overlap threshold also tends to stabilize, so that the convergence speed of the policy network model and the segmentation network model is consistent, so that the overall framework (pipeline) of adaptive segmentation training can achieve the best effect at the fastest speed and lowest computational overhead.

[0177] In some embodiments, the loss function value of the above-mentioned initial segmentation network model can be the above-mentioned overlap value. Of course, the loss function value can also be determined by other means, such as calculating the overlap value between the screenshot 2 corresponding to the training image and the true value corresponding to the training image, and using it as the loss function value.

[0178] S314. The mobile phone uses the initial policy network model as the target policy network model and the initial segmentation network model as the target segmentation network model.

[0179] In an embodiment of the present application, the mobile phone stops training the initial policy network model and the initial segmentation network model, and uses the initial policy network model obtained in the latest round of training as the target policy network model, and uses the initial segmentation network model obtained in the latest round of training as the target segmentation network model.

[0180] S315: The mobile phone returns to the above S303.

[0181] For example, when the above overlap value is less than the preset overlap threshold, it indicates that the accuracy of the initial network model obtained in this round of training is still low and semantic segmentation cannot be accurately achieved. Therefore, the mobile phone needs to continue the next round of training for the initial network model.

[0182] For example, if Figure 8 As shown, the mobile phone inputs the training image into the global convolutional neural network model to obtain the global features corresponding to the training image, so that the mobile phone can begin multiple rounds of training of the initial training network model based on the training image, the global features corresponding to the training image, and the true value corresponding to the training image. First round of training: The mobile phone inputs the global features corresponding to the training image and the true value corresponding to the training image into the initial policy network model, so that the initial policy network model determines and outputs the initial cropping parameters and initial connected domain area threshold corresponding to the training image based on policy p1. Afterwards, the mobile phone can crop the training image based on the initial cropping parameters to obtain screenshot 2. Afterwards, the mobile phone can input screenshot 2 into the initial segmentation network model so that the initial segmentation network model performs semantic segmentation on screenshot 2 to obtain training mask 1.

[0183] The phone then performs a connected domain analysis and restoration operation on training mask 1 based on the initial connected domain area threshold to obtain the target training mask corresponding to the training image. The phone can then calculate the IoU value between training mask 2 corresponding to the training image and the target training mask corresponding to the training image.

[0184] The phone then calculates whether the IoU value is greater than or equal to a preset overlap threshold a. If so, training is complete, and the phone stops training, using the initial policy network model after this round of training as the target policy network model, and the initial segmentation network model after this round of training as the target segmentation network model. Optionally, the phone can also output the policy p1 so that the target policy network model can subsequently use it to determine cropping parameters.

[0185] If not, the mobile phone can perform a second round of training. The process of the second round of training is similar to that of the first round of training and will not be described in detail here.

[0186] Optionally, the initial strategy network model is a cyclic network model that can cyclically output different cropping parameters.

[0187] In some embodiments, the above-mentioned training images may also include training images with different segmentation difficulties. The segmentation difficulty indicates the difficulty of semantic segmentation of the object to be segmented on the image. For example, if there is an object in the training image that is easily mis-segmented as the object to be segmented, then the segmentation difficulty of the training image is relatively high. Based on this, for training images with a high segmentation difficulty, the mobile phone may need to train the initial network model for multiple rounds to meet the requirements, while for training images with a low segmentation difficulty, the mobile phone may only need to train for a few rounds to meet the requirements, thereby achieving adaptive training and dynamically allocating computing power, thereby improving overall training efficiency. In addition, the trained target strategy network model is able to adaptively determine the target cropping parameters corresponding to the images to be processed with different segmentation difficulties, and the trained target segmentation network model is able to perform accurate semantic segmentation on the screenshots corresponding to the images to be processed with different segmentation difficulties, thereby ensuring the segmentation accuracy of different images to be processed.

[0188] In some embodiments, during the training process, the overall training computational overhead of the initial network model can be adjusted by changing the size of the threshold (such as the preset overlap threshold and the target connected domain area threshold mentioned above). For example, for situations where different segmentation accuracy requirements are required, the mobile phone does not need to retrain the initial network model. By adjusting the size of the threshold, the required segmentation accuracy can be dynamically achieved with minimal computational overhead, or all available computing resources can be maximized in real time to improve segmentation accuracy and ensure semantic segmentation effect.

[0189] In some embodiments, the initial policy network model can be modeled as an agent, and the initial policy network model is trained using a reinforcement learning algorithm, wherein the agent's action is the initial cropping parameters (such as the above-mentioned w1, w2, h1, and h2) and the connected domain area threshold, and the state can be the proportion of the character area in screenshot 2 corresponding to the training image to the training image. The discounted reward function can be the IoU value obtained based on the current action. When the IoU shows an upward trend, a positive reward is given. The initial policy network model is getting better and better, that is, the output initial cropping parameters are increasingly matched with the training image, thereby finding the most appropriate cropping parameters. The optimization goal is to maximize the discounted reward function. In addition, the above-mentioned IoU value greater than or equal to a can be the stop condition for the cyclic training of the initial policy network. Optionally, the positive reward can be understood as increasing the IoU value based on the actual calculated IoU value.

[0190] In some embodiments, for a trained target segmentation network model, if the distribution of people in the image to be processed (i.e., target domain data) is different from that in the training image (i.e., source domain data) (e.g., the distribution position and proportion of people), this may result in reduced semantic segmentation accuracy. When the semantic segmentation accuracy decreases, the mobile phone can train the target strategy network model using only the target domain data without having to train the target segmentation network model again, thereby achieving the required semantic segmentation accuracy without changing the target segmentation network model, thereby achieving rapid semantic segmentation of the target domain data.

[0191] The above describes the process of training the initial network model to obtain a target network model with higher accuracy (such as a target segmentation network model and a target policy network model). After obtaining the target network model, the mobile phone can use the target network model for semantic segmentation. Figure 9 , continue to introduce the process of semantic segmentation using the application target network model.

[0192] S401: The mobile phone receives a user's operation to start a camera application.

[0193] S402: In response to the above-mentioned start-up operation, the mobile phone displays a shooting interface, wherein the shooting interface includes a preview area, and the preview area displays a person image captured by the camera on the mobile phone.

[0194] The shooting interface is the interface displayed when the phone is recording a video or taking a photo. The shooting interface may include a preview area and a shooting button. The preview area is used to display the image captured by the camera. The following will take the phone in the photo state as an example to introduce the semantic segmentation process. For example, if the user clicks Figure 10A The camera application shown in FIG. 1 is used by the mobile phone to respond to a click operation on the camera application, start the camera application, and enter the photo taking state (such as Figure 10B As shown), the mobile phone displays the image of the person captured by the mobile phone camera. Here, the image of the person refers to the image to be processed.

[0195] Among them, the above shooting interface can also include the following Figure 10B The portrait control shown is used to trigger the phone to enter portrait mode.

[0196] S403: The mobile phone receives a user click operation on the portrait control on the shooting interface.

[0197] S404: In response to the user clicking the portrait control, the mobile phone inputs the portrait image into the global convolutional neural network model to obtain global features corresponding to the portrait image.

[0198] For example, when the user wants to enhance the person in the preview image displayed on the mobile phone, the user can click on the Figure 10B In response to the user's click operation on the portrait control, the mobile phone enters the portrait mode (such as Figure 10C As shown), the process of enhancing the portrait begins, such as steps S405-S417 introduced below.

[0199] S405 , the mobile phone inputs the global features corresponding to the person image into the target strategy network model to obtain the target cropping parameters and target connected domain area threshold corresponding to the person image.

[0200] S406 : The mobile phone crops the person image based on the target cropping parameters to obtain a screenshot 1 corresponding to the person image.

[0201] The target policy network model is used to output target cropping parameters and target connected domain area thresholds that match the person image. The target cropping parameters are used to crop the person image so that the person occupies an appropriate proportion in the cropped person image. This ensures that the mask obtained from semantic segmentation based on the cropped person image is highly accurate, thereby ensuring good semantic segmentation results.

[0202] It should be noted that when a mobile phone crops a person image based on the target cropping parameters, it does not crop the person from the image. For example, the target policy network model can determine the location information of the person boundary area in the person image, and thus determine the appropriate target cropping parameters and target connected domain area threshold based on the location information of the person boundary area. The target policy network model can then output the target cropping parameters, the target connected domain area threshold, and the location information of the person boundary area, so that the mobile phone can use the location information of the person boundary and the target cropping parameters to perform cropping. For another example, the mobile phone can first input the person image into a human body detection model to identify the person boundary area on the person image and determine the location information of the person boundary area. The mobile phone can then input the person image and the location information of the person boundary area on the person image into the target policy network model, so that the target policy network model determines and outputs the appropriate target cropping parameters and target connected domain area threshold. The mobile phone can then use the location information of the person boundary and the target cropping parameters to perform cropping.

[0203] Optionally, the human body detection model may be a small human body detection model, such as a lightweight human body detection model, so as to identify a person for rapid detection.

[0204] In an embodiment of the present application, the mobile phone enters portrait mode in response to a user clicking on a portrait control on a shooting interface to perform portrait enhancement processing on a person in a person image. In order to achieve portrait enhancement processing, the mobile phone needs to first perform a semantic segmentation task to determine the person area on the person image. The mobile phone can then input the person image into a global convolutional neural network model to obtain the global features corresponding to the person image, so as to use the global features corresponding to the person image to determine appropriate pre- and post-processing parameters, thereby using the target cropping parameters and the target connected domain area threshold to achieve accurate semantic segmentation of the person image. In addition, since the pre- and post-processing parameters are determined using the global features corresponding to the person image, the amount of computation can be reduced and the efficiency of the pre- and post-processing parameters can be improved.

[0205] In some embodiments, similar to S302, S404 is also an optional step. Accordingly, without executing S404, S405 can be replaced with the description of the mobile phone inputting the person image into the target policy network model to obtain the target cropping parameters and target connected domain area threshold corresponding to the person image.

[0206] In some embodiments, the mobile phone uses the target strategy network model to determine the target connected domain area threshold and target cropping parameters that match the person image. This is only one possible implementation method for determining the target connected domain area threshold and target cropping parameters that match the person image. The mobile phone can also use other methods to determine the target connected domain area threshold and target cropping parameters that match the person image, such as the mobile phone determines the distribution of people in the person image, determines the matching target connected domain area threshold and target cropping parameters, and the specific determination process is similar to the process of the above-mentioned target strategy network model determining the matching target connected domain area threshold and target cropping parameters based on the distribution of the person image.

[0207] In some embodiments, the above-mentioned user click operation on the portrait control is only one possible trigger operation for triggering the mobile phone to perform the semantic segmentation task. The trigger operation can also be other types of trigger operations. For example, in response to the user clicking on the portrait control Figure 10D Click the Gallery app, and the phone will display the following Figure 10E The gallery interface 20 shown. Afterwards, the mobile phone receives the user's Figure 10E Clicking the displayed image 21 displays Figure 10FThe interface including the image is shown. Among them, the interface 21 also includes controls for image processing, such as edit, more, and other controls. In response to the user's click operation on the edit control, the mobile phone can display more edit controls, such as blur, filter, and other controls. After that, when the user wants to add a filter to the image 21, the user can click on the filter. Here, the image 21 can be used as the above-mentioned person image 1, and the click on the filter can be the above-mentioned trigger operation that can trigger the mobile phone to perform a semantic segmentation task. For another example, the trigger operation can be the user's operation to turn on the AI mode on the shooting interface, etc.

[0208] S407. The mobile phone inputs the screenshot 1 corresponding to the person image into the target segmentation network model, so that the target segmentation network model performs semantic segmentation on the screenshot 1 and outputs the predicted probability map corresponding to the screenshot 1.

[0209] In the embodiment of the present application, the person image is cropped using the target cropping parameters matching the screenshot 1 (or the cropped image) for the target segmentation network model to perform semantic segmentation using the cropped person image, which can effectively reduce the computational overhead of the target segmentation network model and improve the segmentation accuracy when the proportion of the person on the person image is small, ensuring the semantic segmentation effect.

[0210] S408. The mobile phone counts the number of pixel points on the predicted probability map whose probability is equal to the binarization threshold.

[0211] S409. The mobile phone determines whether the number is greater than or equal to a preset number threshold.

[0212] In the embodiment of the present application, when the probability of the pixel point is equal to 0.5, the pixel point may belong to the person or may not belong to the person. Based on this, when the number of pixel points on the predicted probability map whose probability is equal to the binarization threshold is large, that is, when the number (or the first number) is greater than or equal to the preset number threshold (or the first number threshold), it indicates that the number of pixel points whose belonging to the person is uncertain is large, that is, it indicates that the semantic segmentation accuracy of the target segmentation network model for the screenshot 1 is low, and the proportion of the person on the screenshot 1 affects the semantic segmentation accuracy. Therefore, the above-mentioned target cropping parameters may not be suitable for the person image, so the mobile phone needs to re-determine the cropping parameters to crop the person image, and then the mobile phone can execute S410.

[0213] When the number of pixel points on the predicted probability image whose probability is equal to the binarization threshold is small, that is, when the number is less than the preset number threshold, it indicates that the number of pixel points whose belonging to the person is uncertain is small, that is, the semantic segmentation accuracy is high. In other words, the above-mentioned target cropping parameters are suitable for the person image and are matched with the person image. Therefore, the mobile phone does not need to re-crop the person image, and then the mobile phone can execute S411.

[0214] S410: The mobile phone returns to the above S405.

[0215] In an embodiment of the present application, the mobile phone re-inputs the person image into the target policy network model to determine new target cropping parameters and a new target connected domain area threshold corresponding to the person image. The new target cropping parameters are different from the previously determined target cropping parameters. However, the new target connected domain threshold may be the same as or different from the previously determined target connected domain threshold, and this application is not limited thereto.

[0216] S411. The mobile phone determines mask 1 corresponding to the person image based on the predicted probability map.

[0217] Among them, mask 1 (or called the first mask) is similar to the above-mentioned training mask 1, which may include at least one person area (or can be alternatively described as the first target object area) and at least one non-person area (or called the first non-target object area).

[0218] In some embodiments, the probability corresponding to the pixel in the above-mentioned human area is greater than the binarization threshold. The probability corresponding to the pixel in the non-human area is less than or equal to the binarization threshold. In addition, when the probability corresponding to the pixel on the predicted probability map is equal to the binarization threshold, the mobile phone can determine that the pixel belongs to the human area, or it can determine that the pixel does not belong to the human area, and this application is not limited to this.

[0219] In some embodiments, the above-mentioned mask 1 can be determined by the mobile phone based on the predicted probability image, or it can be directly output by the target segmentation network model. That is, the mobile phone outputs the screenshot 1 to the target segmentation network model, and the target segmentation network model outputs the predicted probability map and mask 1.

[0220] In addition, the above S408-S410 are optional steps. The mobile phone may not determine whether the semantic segmentation accuracy is high by judging whether the number of pixel points on the predicted probability map whose probability is equal to the binarization threshold is greater than the preset number threshold. The mobile phone can directly execute the above S411 to determine mask 1.

[0221] In some embodiments, the mask 1 may be output by the segmentation network model. Accordingly, if the above steps S408-S410 are optional, then it does not matter whether the target segmentation network model outputs a predicted probability map.

[0222] Although using the target segmentation network model to perform semantic segmentation on screenshot 1, which is cropped based on the target cropping parameters, can achieve good semantic segmentation results, the resulting mask 1 may still contain false positive areas. Therefore, the phone can perform connected domain analysis on mask 1 using the target connected domain area threshold to remove false positive areas in mask 1, thereby further improving semantic segmentation accuracy. The connected domain analysis process will be described in detail below in conjunction with steps S412-S414.

[0223] In some embodiments, the use of a target segmentation network model by a mobile phone to determine mask 1 corresponding to a person image is merely an example. The mobile phone may also directly determine mask 1 corresponding to a person image based on a semantic segmentation algorithm. The determination process is similar to the process of determining mask 1 using the target segmentation network model described above, and this application does not limit this. Similarly, during the training phase, the mobile phone may also train only the target strategy network model without training the target segmentation network model. Accordingly, the mobile phone may also perform semantic segmentation on the cropped training image without using the initial segmentation network model.

[0224] S412 : For each person region on mask 1 , the mobile phone determines whether the area of the person region is greater than or equal to the target connected domain area threshold.

[0225] In an embodiment of the present application, for each person region on mask 1, if the area of the person region is greater than or equal to the target connected domain area threshold, it indicates that the area of the person region is large, and the person region is not a false positive area, then the mobile phone can execute S413.

[0226] When the area of the person region is smaller than the target connected domain area threshold, it indicates that the area of the person region is small and the person region is a false positive region, and the mobile phone may execute S414.

[0227] S413: The mobile phone does not correct the person area on mask 1.

[0228] S414: The mobile phone corrects the person area on mask 1 to obtain mask 2.

[0229] The process of connected domain analysis introduced in S412-S414 above can refer to the relevant contents of S308-S310 above, and will not be repeated here.

[0230] The above describes the process by which the mobile phone obtains Mask 2, which excludes false positive areas. After obtaining Mask 2, in order to better process the person image, the mobile phone can restore Mask 2 to obtain a target mask of the same size as the person image. For details, please refer to the relevant content of S415 below.

[0231] S415 : The mobile phone restores mask 2 based on the size of the person image to obtain a target mask corresponding to the person image.

[0232] The size of the target mask corresponding to the person image is the same as that of the person image, that is, the height and width are the same.

[0233] In this embodiment of the present application, the mobile phone can use the mask obtained by performing connected domain analysis on Mask 1 as Mask 2. The mobile phone can then restore the size of Mask 2 (or the second mask) to the size of the person image to obtain a target mask of the same size as the person image, thereby facilitating subsequent editing operations on the person image using the target mask, i.e., image processing. Furthermore, the mobile phone can also directly use Mask 2 as the target mask corresponding to the aforementioned person image, enabling rapid determination of the target mask.

[0234] The process of restoring the mask 2 by the mobile phone can refer to the process of restoring the training mask 2 by the mobile phone, which will not be described in detail here.

[0235] S416. The mobile phone performs portrait enhancement processing on the person image based on the target mask to obtain an enhanced person image.

[0236] S417. The mobile phone displays the enhanced image of the person.

[0237] For example, the mobile phone can perform portrait enhancement processing on the person in the person image based on the target mask to improve the quality of the person image. For example, in response to the user clicking the above-mentioned portrait control, the mobile phone displays an enhanced person image (or referred to as the target person image), and the person in the enhanced person image is clearer.

[0238] It should be noted that the above-mentioned portrait enhancement processing is only one possible processing method for the portrait image. The mobile phone can also perform other types of processing on the portrait enhancement processing, such as cutout processing, filter processing (or smart filter), background blur processing, etc. In other words, the mobile phone can perform a first processing on the portrait image based on the target mask, and the first processing can include one or more of portrait enhancement processing, cutout processing, filter processing, and background blur processing.

[0239] In some embodiments, during the application process, the overall computational overhead of the target network model can be adjusted by changing the threshold value (such as the preset number threshold value described above). For example, when the segmentation accuracy requirement is low, the mobile phone will set a smaller threshold value, thereby achieving the required segmentation accuracy with a lower computational overhead.

[0240] It should be noted that the embodiment of the present application is to introduce the solution of the present application by taking the device for training the initial network model and the device for applying the target network model obtained by training as the same device (such as a mobile phone). Of course, the device for training the initial network model and the device for applying the target network model obtained by training may not be the same device. For example, the server trains the initial network model, and the device that needs to perform semantic segmentation can install the trained target network model. In addition, the number of objects to be segmented on the above-mentioned image to be processed or training image can be one or more. The number of objects to be segmented shown in Figure 1 is only an example, and this application does not limit it.

[0241] In some embodiments, the aforementioned people as the objects to be segmented are merely examples. The image processing method provided in this application can be applied to any single-category semantic segmentation task (e.g., segmenting target objects such as sky, pets, and plants) or multi-category semantic segmentation tasks. However, if applied to a multi-category semantic segmentation task, the mobile phone can first segment the target objects of each category within the multi-category task.

[0242] In some embodiments, the present application provides a chip system, which can be applied to the above-mentioned electronic device, and the electronic device executes the image processing method as described above.

[0243] In some embodiments, the present application provides a computer storage medium comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the image processing method as described above.

[0244] In some embodiments, the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the image processing method described above.

[0245] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0246] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0247] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0248] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0249] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0250] The above content is only a specific embodiment of this application, but the scope of protection of this application is not limited to this. Any changes or replacements within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method includes: Determining target cropping parameters and target connected domain area thresholds that match the image to be processed based on the image to be processed; Cropping the image to be processed based on the target cropping parameter to obtain a cropped image; Performing semantic segmentation processing on the cropped image to obtain a first mask; wherein the first mask is a binary image of the cropped image, and the first mask includes at least one first target object region and a first non-target object region; the first target object region refers to a region including the target object, and the first non-target object region refers to a region not including the target object; Based on the target connected domain area threshold, a connected domain analysis is performed on the first mask to determine a target mask corresponding to the image to be processed; wherein the connected domain analysis indicates that the area that is incorrectly semantically segmented as the first target object area is updated to the first non-target object area.

2. The method according to claim 1, characterized in that The step of performing semantic segmentation on the cropped image to obtain a first mask includes: Inputting the cropped image into a target segmentation network model to obtain a prediction probability map; wherein the prediction probability map includes a first probability corresponding to each pixel in the cropped image; the first probability corresponding to the pixel represents the probability that the pixel belongs to the target object; Based on the predicted probability map, the first mask is determined; wherein the first probability corresponding to the pixel points in the first target object area in the first mask is greater than a first preset probability value, and the first probability corresponding to the pixel points in the first non-target object area in the first mask is less than the first preset probability value.

3. The method according to claim 2, characterized in that Before determining the first mask, the method further includes: Obtaining a first number of pixel points in the predicted probability map whose first probability is equal to the first preset probability value; It is determined that the first quantity is less than a first quantity threshold.

4. The method according to claim 3, characterized in that The method further comprises: Determine whether the first number is greater than or equal to a first number threshold, and return to the step of determining, based on the image to be processed, a target cropping parameter and a target connected domain area threshold that match the image to be processed.

5. The method according to any one of claims 1 to 4, characterized in that The determining, based on the image to be processed, target cropping parameters and target connected domain area thresholds that match the image to be processed includes: Inputting the image to be processed into a global neural convolutional network model to obtain global features corresponding to the image to be processed; Based on the global features corresponding to the image to be processed, a target cropping parameter and a target connected domain area threshold that match the image to be processed are determined.

6. The method according to any one of claims 1 to 5, characterized in that The performing a connected domain analysis on the first mask based on the target connected domain area threshold to determine a target mask corresponding to the image to be processed includes: updating the first target object region in the first mask whose area is smaller than the target connected domain area threshold to the first non-target object region to obtain a second mask; Based on the second mask, the target mask is determined.

7. The method according to claim 6, characterized in that The determining the target mask based on the second mask includes: Based on the size of the image to be processed, the second mask is extended to obtain the target mask; wherein the size of the target mask is the same as the size of the image to be processed, and the values of the pixel points in the extended area of the target mask are the same as the values of the pixel points in the first non-target object area.

8. The method according to any one of claims 1 to 7, characterized in that The determining, based on the image to be processed, target cropping parameters and target connected domain area thresholds that match the image to be processed includes: The image to be processed is input into the target strategy network model to obtain the target cropping parameters and the target connected domain area threshold.

9. The method according to claim 8, characterized in that The method further comprises: Obtaining a training image sample set; wherein the training image sample set includes at least one training image and a true value corresponding to each training image; each training image is an image including the target object; and the true value corresponding to the training image represents an actual mask corresponding to the training image; Based on the training image sample set, the initial policy network model is trained to obtain the target policy network model.

10. The method according to claim 9, characterized in that The step of training the initial policy network model based on the training image sample set to obtain the target policy network model includes: For each of the training images, inputting the training image into the initial strategy network model to obtain initial cropping parameters matching the training image; Cropping the training image based on the initial cropping parameters to obtain a cropped training image; Performing semantic segmentation processing on the cropped training image to obtain a first training mask; wherein the first training mask includes at least one second target object region and a second non-target object region; determining an overlap degree value based on the first training mask and the true value corresponding to the training image; wherein a larger overlap degree value indicates a higher overlap degree between the second target object region in the first training mask and the second target object region in the true value; When the overlap degree value is less than a preset overlap threshold, returning to the step of inputting the training image into the initial strategy network model to obtain initial cropping parameters matching the training image; When the overlap degree value is greater than or equal to the preset overlap threshold, the initial policy network model is determined to be the target policy network model.

11. The method according to claim 10, characterized in that The initial strategy network model is further configured to output an initial connected domain area threshold; and determining an overlap value based on the first training mask and the true value corresponding to the training image, comprising: Performing a connected domain analysis on the first training mask based on the initial connected domain area threshold to determine a target training mask corresponding to the training image; Determine an overlap value between a target training mask corresponding to the training image and a true value corresponding to the training image.

12. The method according to claim 10, characterized in that The determining of the overlap degree value based on the first training mask and the true value corresponding to the training image includes: Determine an overlap value between the second target object region in the first training mask and the second target object region in the true value.

13. The method according to any one of claims 1 to 12, characterized in that The image to be processed includes a person image; before determining the target cropping parameters and target connected domain area threshold that match the image to be processed based on the image to be processed, the method further includes: Displaying a shooting interface; wherein the shooting interface includes the image of the person; After determining the target mask corresponding to the image to be processed, the method further includes: Display the target person image; wherein, the target person image is obtained by performing a first processing on the person image based on the target mask corresponding to the image to be processed; wherein, the first processing includes at least one or more of portrait enhancement processing, cutout processing, filter processing and background blur processing.

14. A chip system, characterized in that: The chip system is applied to an electronic device, and the electronic device executes the method according to any one of claims 1 to 13.

15. An electronic device, characterized in that: The electronic device includes a memory and one or more processors; the memory and the processor are coupled; the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes the method as described in any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 13.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images