Image processing method and electronic device

Abstract

The application provides an image processing method and an electronic device, and relates to the technical field of image processing. The electronic device can determine a target cropping parameter and a target connected domain area threshold value matched with a to-be-processed image based on the to-be-processed image. Then, the electronic device can crop the to-be-processed image by using the target cropping parameter to obtain a cropped image in which the proportion of a target object is appropriate. Then, the electronic device can input the cropped image into a segmentation network model to enable the segmentation network model to perform semantic segmentation on the cropped image to determine a target object region on the cropped image and obtain a first mask. Since the proportion of the target object in the cropped image is appropriate, the segmentation network model can accurately perform semantic segmentation on the cropped image. Then, the electronic device can continue to remove a target object region that is a false positive in the first mask by using the target connected domain area threshold value, thereby ensuring the accuracy of semantic segmentation.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method and electronic device. Background Technology

[0002] Semantic segmentation assigns each pixel in an image to a predefined semantic category. It is widely used in various scenarios, such as autonomous driving, medical image analysis, and image processing (e.g., portrait enhancement, background blurring, image matting).

[0003] Currently, electronic devices can first perform semantic segmentation on images to identify objects within the target category, i.e., the target objects. Then, the electronic device can perform relevant processing on these target objects, such as the aforementioned image enhancement. However, low semantic segmentation accuracy can lead to poor image processing results. Therefore, improving the accuracy of semantic segmentation is a pressing issue that needs to be addressed. Summary of the Invention

[0004] In view of this, this application provides an image processing method and an electronic device that can improve the accuracy of semantic segmentation of images, thereby improving the semantic segmentation effect.

[0005] In a first aspect, this application provides an image processing method in which an electronic device can determine, based on an image to be processed, target cropping parameters and target connected region area thresholds that match the image to be processed.

[0006] Then, the electronic device can crop the image to be processed based on the target cropping parameters to obtain the cropped image.

[0007] Subsequently, the electronic device can perform semantic segmentation processing on the cropped image to obtain a first mask; wherein, the first mask is a binarized image of the cropped image, and the first mask includes at least one first target object region and a first non-target object region; the first target object region refers to the region that includes the target object, and the first non-target object region refers to the region that does not include the target object;

[0008] Based on the target connected component area threshold, connected component analysis is performed on the first mask to determine the target mask corresponding to the image to be processed; wherein, the connected component analysis indicates that the region that was incorrectly semantically segmented into the first target object region is updated (i.e. corrected) into the target object region.

[0009] In this application, the electronic device can determine the target cropping parameters and target connected component area threshold that match the image to be processed based on the actual situation of the target object in the image to be processed. This means determining the target cropping parameters and target connected component area threshold suitable for the image to be processed, achieving adaptive determination of the cropping parameters and connected component area threshold. Then, the electronic device uses the target cropping parameters to crop the image to obtain a cropped image, ensuring that the proportion of the target object in the cropped image is appropriate. This ensures high accuracy in semantic segmentation of the cropped image, thus guaranteeing high semantic segmentation accuracy and consequently high accuracy of the target object in the obtained first mask. Next, the electronic device uses the target connected component area threshold to correct the first target object region in the first mask to a first non-target object region. This corrects regions that were mistakenly identified as including the target object in semantic segmentation to regions that do not include the target object, thereby obtaining a target mask that does not include false positive regions. This ensures the accuracy of the target object in the target mask, further guaranteeing high accuracy and good semantic segmentation results, thus ensuring the subsequent image processing effect and improving user satisfaction.

[0010] The target cropping parameter matched with the image to be processed can refer to a cropping parameter that matches the distribution information (such as distribution location and / or proportion) of the target object (i.e., the object to be segmented) on the image to be processed. Similarly, the target connected component area threshold matched with the image to be processed can refer to a connected component area threshold that matches the distribution information of the target object on the image to be processed.

[0011] In one possible design approach, the first target object area is a first color, and the first non-target object area is a second color. The first color is either white or black, and the second color is either black or white. The first color and the second color are different.

[0012] Optionally, when a pixel is white, its value can be 255 (or 1). When a pixel is black, its value can be 0.

[0013] In one possible design approach, the aforementioned first mask can be determined by the electronic device using a target segmentation network model. Specifically, the electronic device can input the cropped image into the target segmentation network model to obtain a predicted probability map corresponding to the cropped image. This predicted probability map can include a first probability corresponding to each pixel in the cropped image, where the first probability represents the probability that the pixel belongs to the target object.

[0014] Subsequently, the electronic device can determine a first mask based on the predicted probability map. In this first mask, the first probability corresponding to the pixels in the first target object region is greater than a first preset probability value, and the first probability corresponding to the pixels in the first non-target object region is less than the first preset probability value.

[0015] In this application, the electronic device can utilize a target segmentation network model to perform semantic segmentation processing on the cropped image to achieve fast semantic segmentation. Subsequently, the electronic device can use the probability (i.e., the first probability) corresponding to each pixel in the cropped image determined by the semantic segmentation processing to determine whether the pixel belongs to the target object, thereby determining the binarized image corresponding to the cropped image and achieving accurate semantic segmentation of the cropped image.

[0016] In one possible design approach, before determining the first mask, that is, after performing semantic segmentation processing on the cropped image, the electronic device can first determine whether the accuracy of the current semantic segmentation processing is high, that is, determine whether the first number of pixels with a first probability equal to a first preset probability value in the predicted probability map obtained by the electronic device is less than a first number threshold.

[0017] If the first quantity is less than the first quantity threshold, it indicates that the accuracy of this semantic segmentation process is relatively high, and the electronic device can continue to determine the first mask.

[0018] If the first quantity is greater than or equal to the first quantity threshold, it indicates that the accuracy of the current semantic segmentation process is low. The electronic device does not need to continue to determine the first mask. The electronic device can return to the step of determining the target cropping parameters and the target connected component area threshold that match the image to be processed based on the image to be processed, so as to re-determine the target cropping parameters that match the image to be processed, so as to obtain appropriate target cropping parameters, thereby ensuring the accuracy of semantic segmentation.

[0019] In one possible design approach, the electronic device can first input the image to be processed into a global neural convolutional network model to obtain the global features corresponding to the image, thereby compressing the image. Then, based on the global features, the electronic device can determine the target cropping parameters and the target connected component area threshold that match the image, thus reducing the computational complexity of the image data and enabling rapid determination of the target cropping parameters and the target connected component area threshold, thereby improving the overall semantic segmentation processing efficiency.

[0020] In one possible design approach, the process of determining the target mask may include: the electronic device first updates a first target object region in the first mask whose area is less than a target connected component area threshold to a first non-target object region, obtaining a second mask to perform connected component analysis, thereby accurately removing false positive regions on the first mask. Then, the electronic device can determine the target mask based on the second mask, ensuring the accuracy of semantic segmentation of the target mask.

[0021] In one possible design approach, the process of determining the target mask based on the second mask in the aforementioned electronic device may include:

[0022] The electronic device can extend the second mask based on the size of the image to be processed to obtain a target mask. The size of the target mask is the same as the size of the image to be processed. The pixel values ​​of the extended region in the target mask are the same as the pixel values ​​of the first non-target object region. This avoids the occurrence of false positive regions in the extended region of the image to be processed when performing semantic segmentation on the entire image. This is because the extended region of the image to be processed contains objects that are easily misidentified as target objects.

[0023] In one possible design approach, the electronic device can input the image to be processed into the target policy network model to obtain the target cropping parameters and the target connected component area threshold, thereby achieving rapid determination of the target cropping parameters and the target connected component area threshold.

[0024] In one possible design approach, the aforementioned target policy network model can be trained. The training process may include: First, the electronic device can acquire a training image sample set. This training image sample set includes at least one training image and a ground truth value corresponding to each training image. Each training image is an image containing the target object, and the ground truth value corresponding to the training image represents the actual mask corresponding to the training image, i.e., the actual binarized image. Then, the electronic device can train the initial policy network model based on the training image sample set to obtain the target policy network model. Based on this, the electronic device uses the training images and the corresponding actual binarized images to accurately train the initial policy network model, enabling the accurate determination of the target cropping parameters and target connected component area thresholds that match the image, thereby ensuring high accuracy in semantic segmentation of the image.

[0025] In one possible design approach, the above training process may include:

[0026] For each training image, the electronic device can input the training image into an initial policy network model to obtain initial cropping parameters and initial connected component area thresholds that match the training image. Then, the electronic device can crop the training image based on the initial cropping parameters to obtain a cropped training image. Next, the electronic device can perform semantic segmentation processing on the cropped training image to obtain a first training mask; wherein the first training mask includes at least one second target object region and a second non-target object region. Then, based on the ground truth values ​​corresponding to the first training mask and the training image, an overlap value is determined; wherein, the larger the overlap value, the higher the degree of overlap between the second target object region in the first training mask and the second target object region in the ground truth value.

[0027] Afterwards, the electronic device can determine whether the degree of overlap is less than a preset overlap threshold in order to determine the accuracy of this semantic segmentation, that is, to determine the matching degree between the initial cropping parameters and the initial connected component area threshold determined by the initial policy network model and the training image, which means determining whether the accuracy of the initial policy network model meets the requirements.

[0028] If the overlap value is less than the preset overlap threshold, it indicates that the accuracy of this semantic segmentation is low. That is, the initial cropping parameters and initial connected component area threshold determined by the initial policy network model have a low matching degree with the training image. In other words, the accuracy of the initial policy network model does not meet the requirements. In this case, the electronic device can return to the step of inputting the training image into the initial policy network model to obtain the initial cropping parameters and initial connected component area threshold that match the training image, so as to continue the next training.

[0029] If the overlap value is greater than or equal to the preset overlap threshold, it indicates that the accuracy of this semantic segmentation is relatively high. That is, it is determined that the initial cropping parameters and initial connected component area threshold determined by the initial policy network model have a high degree of matching with the training image. In other words, it is determined that the accuracy of the initial policy network model meets the requirements. Therefore, there is no need to continue the next training. The electronic device can determine the initial policy network model as the target policy network model.

[0030] In one possible design approach, the process of determining the aforementioned overlap value may include:

[0031] For each training image, the electronic device can first perform connected component analysis on the first training mask corresponding to the training image based on an initial connected component area threshold that matches the training image, thereby determining the target training mask corresponding to the training image. Then, the electronic device can determine the degree of overlap between the target training mask and the ground truth value corresponding to the training image. Based on this, the electronic device determines the prediction accuracy, i.e., the training precision of the initial policy network model, based on the predicted degree of overlap between the target training mask and the ground truth value corresponding to the training image.

[0032] Optionally, the process of performing connected component analysis on the first training image may include:

[0033] First, the electronic device can update the first target object region in the first training mask whose area is smaller than the target connected component area threshold to a first non-target object region, thus obtaining a second training mask to remove false positive regions on the first training mask. Then, based on the size of the training image, the second training mask is extended to obtain the target training mask corresponding to the training image, ensuring the accuracy of semantic segmentation of the obtained target training mask.

[0034] In another possible design approach, the process of determining the aforementioned overlap value may include:

[0035] For each training image, the electronic device can directly determine the degree of overlap between the second target object region in the first training mask corresponding to the training image and the second target object region in the ground truth corresponding to the training image, so as to use the degree of overlap to determine the accuracy of the initial policy network model.

[0036] In one possible design, the image to be processed may include an image of a person. This image of a person may be a preview image displayed on the electronic device. That is, before determining the target cropping parameters and target connected component area thresholds that match the image to be processed, the electronic device displays a shooting interface; wherein the shooting interface includes the image of a person.

[0037] Accordingly, after determining the target mask corresponding to the image to be processed, the electronic device can perform a first processing on the image of the person based on the target mask to obtain the target image of the person. The first processing includes at least one or more of the following: portrait enhancement processing, image cutout processing, filter processing, and background blurring processing.

[0038] Afterwards, the electronic device can display the image of the target person to process the person in the image and ensure better processing results.

[0039] Secondly, this application provides a model training method applied to a first device, wherein the first device acquires a training image sample set; wherein the training image sample set includes at least one training image and a ground truth value corresponding to each training image; each training image is an image including a target object; the ground truth value corresponding to the training image represents the actual mask corresponding to the training image;

[0040] Based on the training image sample set, the initial policy network model is trained to obtain the target policy network model; the target policy network model outputs the target cropping parameters and the target connected component area threshold that match the image.

[0041] Thirdly, this application provides a chip system applied to an electronic device, the electronic device performing the image processing method as described in any of the first aspects above.

[0042] Fourthly, this application provides a chip system applied to a first device, the first device performing the model training method as described in any of the second aspects above.

[0043] Fifthly, this application provides an electronic device, the electronic device including a memory and one or more processors; the memory and the processors are coupled; the memory is used to store computer program code, the computer program code including computer instructions; when the processor executes the computer instructions, the electronic device performs the image processing method as described in any of the first aspects above.

[0044] In a sixth aspect, this application provides an electronic device that can serve as the first device described above. The first device includes a memory and one or more processors; the memory and the processors are coupled; the memory is used to store computer program code, the computer program code including computer instructions; when the processor executes the computer instructions, the first device performs the model training method as described in any of the second aspects above.

[0045] In a seventh aspect, this application provides a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the image processing method as described in any of the first aspects above.

[0046] Eighthly, this application provides a computer-readable storage medium including computer instructions that, when executed on a first device, cause the first device to perform the model training method as described in any of the second aspects above.

[0047] Ninthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the image processing method as described in any one of the first or second aspects above.

[0048] Understandably, the beneficial effects that can be achieved by the model training method described in the second aspect above, the chip system described in the third and fourth aspects, the electronic device described in the fifth and sixth aspects, the computer storage medium described in the seventh and eighth aspects, and the computer program product described in the ninth aspect can be referred to the beneficial effects in the first aspect and any of its possible design methods, and will not be repeated here. Attached Figure Description

[0049] Figure 1A A schematic diagram of a semantic segmentation process provided for an embodiment of this application;

[0050] Figure 1B A second schematic diagram illustrating a semantic segmentation process provided in an embodiment of this application;

[0051] Figure 2A A schematic diagram of a semantic segmentation process provided in this application embodiment. Figure 3 ;

[0052] Figure 2B A schematic diagram of a semantic segmentation process provided in this application embodiment. Figure 4 ;

[0053] Figure 3 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0054] Figure 4 A schematic diagram of a semantic segmentation process provided in this application embodiment. Figure 5 ;

[0055] Figure 5 A flowchart illustrating an image processing method provided in this application embodiment is shown below;

[0056] Figure 6 A schematic diagram of a training sample and its corresponding ground truth provided in an embodiment of this application;

[0057] Figure 7A A schematic diagram of a cutting parameter provided for an embodiment of this application;

[0058] Figure 7B A second schematic diagram illustrating a cutting parameter provided in an embodiment of this application;

[0059] Figure 8 A schematic diagram of a semantic segmentation process provided in this application embodiment. Figure 6 ;

[0060] Figure 9 A schematic flowchart of an image processing method provided in an embodiment of this application is shown below;

[0061] Figure 10A A schematic diagram of an image processing scenario provided in an embodiment of this application;

[0062] Figure 10B Schematic diagram 2 of an image processing scenario provided for an embodiment of this application;

[0063] Figure 10C This application provides an illustration of an image processing scenario. Figure 3 ;

[0064] Figure 10D This application provides an illustration of an image processing scenario. Figure 4 ;

[0065] Figure 10E This application provides an illustration of an image processing scenario. Figure 5 ;

[0066] Figure 10F This application provides an illustration of an image processing scenario. Figure 6 . Detailed Implementation

[0067] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0068] To facilitate understanding of the embodiments of this application, some terms involved in the embodiments of this application will be introduced below.

[0069] Semantic segmentation refers to assigning pixels in an image to predefined semantic categories. Each pixel in the image is given a label indicating its semantic category, such as people, vehicles, roads, trees, etc.

[0070] A global convolutional neural network (CNN) typically consists of multiple convolutional and pooling layers, used for feature extraction and classification of input data. Compared to a regular CNN, the main characteristic of a global CNN is that it uses global pooling layers instead of the fully connected layers in a conventional CNN. Global pooling layers not only reduce the dimensionality and compress the feature maps output by the convolutional layers, but also preserve more location and spatial structure information. This reduces the number of model parameters, improves computational efficiency, and effectively avoids overfitting.

[0071] A policy network is an important concept in reinforcement learning. The policy π represents the probability distribution of different actions taken in a given state. In this embodiment, the state can be understood as the proportion of the target object (or object to be segmented) region in the image, and the action can be understood as the cropping parameters and connected component area thresholds corresponding to the image.

[0072] A mask is used to represent a binary image. Each pixel in a binary image has a value of either 0 or 255. In semantic segmentation tasks, a pixel value of 0 indicates that the pixel does not belong to the object to be segmented, while a pixel value of 255 indicates that the pixel belongs to the object to be segmented.

[0073] Ground truth (GT) typically refers to an image that has been manually annotated. In this embodiment, ground truth can refer to a binarized image with manually annotated target and non-target regions.

[0074] IoU (intersection over union) is an evaluation metric used in semantic segmentation tasks. It compares the degree of overlap between predicted and ground truth images to measure the accuracy of semantic segmentation. In this embodiment, the relevant model (such as a policy network model or a segmentation network model) can determine whether to stop training by measuring the IoU value between the predicted target mask and the ground truth value of the training image.

[0075] With the development of semantic segmentation technology, it has been widely applied in various scenarios, such as autonomous driving, medical image analysis, and image processing (e.g., image matting, portrait enhancement, background blurring, depth-of-field adjustment, smart filters, and automatic data annotation). Electronic devices generally utilize semantic segmentation models to perform semantic segmentation tasks to achieve semantic segmentation of images.

[0076] In some embodiments, the execution process of the semantic segmentation task described above may involve the following steps: First, the electronic device may crop an image to determine a screenshot containing the object to be segmented. Then, the electronic device inputs this screenshot into a target segmentation network model, enabling the target segmentation network model to perform semantic segmentation processing on the screenshot, obtaining a mask 1 corresponding to the screenshot, i.e., a mask 1 corresponding to the image. This mask 1 is a binarized image that includes at least one region containing the object to be segmented and at least one region containing the object not to be segmented. Specifically, the pixel values ​​in the region containing the object to be segmented can be 255, and the pixel values ​​in the region not to be segmented can be 0. Both the region containing the object to be segmented and the region not to be segmented are connected components.

[0077] Subsequently, the electronic device can continue to perform connected component analysis on mask 1 using the connected component area threshold to determine whether the region to be segmented on mask 1 is a false positive region, thereby correcting the false positive regions on mask 1 to be non-segmented objects, resulting in mask 2. A false positive region refers to a region that does not include the image of the object to be segmented, i.e., it is actually a region that is not the object to be segmented, but is mistakenly identified as a region to be segmented by the target segmentation network model.

[0078] Specifically, for each of the at least one object regions to be segmented, the electronic device can determine whether the area of ​​the object region to be segmented is greater than or equal to the connected component area threshold. If the area of ​​the object region to be segmented is less than the connected component area threshold, it indicates that the initial object region to be segmented is a false positive region, that is, the initial object to be segmented does not include the object to be segmented, and the electronic device can correct the object region to be segmented to a non-object region.

[0079] If the area of ​​the object to be segmented is greater than or equal to the area threshold of the connected region, it indicates that the object to be segmented is not a false positive region, that is, the initial object to be segmented includes the object to be segmented. In this case, the electronic device does not need to correct the object to be segmented to a non-object to be segmented region.

[0080] Then, the electronic device can determine the target mask corresponding to the image based on the mask 2 corresponding to the image. After that, the electronic device can perform subsequent processing on the image (such as portrait enhancement, background blurring, smart filters, etc.) based on the mask 2 corresponding to the image.

[0081] However, the proportion of the object to be segmented in the screenshot (or the cropping ratio) will affect the accuracy of the mask 1 output by the target segmentation network model, thus affecting the accuracy of the target mask and consequently the semantic segmentation accuracy of the image. For example, as Figure 1A As shown, when the cropping ratio is inappropriate, such as being too large, the accuracy of mask 1 output by the target segmentation network model is low. The object to be segmented on mask 1 differs significantly from the object to be segmented in the original image (e.g., the shape of a person's shoes changes), resulting in lower accuracy of the segmented object, poor semantic segmentation, and consequently, poor subsequent image processing performance. However, as... Figure 1B As shown, when the cropping ratio is appropriate, the mask 1 output by the target segmentation network model has high precision. The difference between the object to be segmented on mask 1 and the object to be segmented on the original image is small (such as the shape of the person's shoes being the same as the actual shape), which makes the segmented object more accurate and the semantic segmentation effect better, thus making the subsequent image processing effect better.

[0082] Furthermore, the size of the connected component area threshold also affects the accuracy of the target mask, which in turn affects the accuracy of semantic segmentation. For example, as... Figure 2A As shown, when the area threshold of the connected component is small, it is impossible to accurately identify the false positive region in mask 1, resulting in the existence of false positive regions on mask 2. This leads to a lower accuracy of the target mask, which in turn leads to a lower accuracy of the segmented object and a lower semantic segmentation accuracy.

[0083] And such Figure 2BAs shown, when the connected component area threshold is appropriate, false positive regions in mask 1 can be accurately identified, resulting in false positive regions on mask 2. This leads to a high-precision target mask, enabling accurate segmentation of the object to be segmented and ensuring semantic segmentation accuracy.

[0084] Therefore, to address the aforementioned problems, this application proposes an adaptive semantic segmentation framework. From the perspective of image pre- and post-processing, for the image to be processed, the electronic device can determine the target cropping parameters and the target connected component area threshold that match the image, achieving adaptive determination of pre- and post-processing parameters. Then, the electronic device can use the target cropping parameters to crop the image to obtain a screenshot with an appropriate cropping ratio, achieving adaptive pre-processing of the image to ensure that the proportion of the object to be segmented in the image is appropriate. Next, the electronic device uses a target segmentation network model to perform semantic segmentation processing on the screenshot to determine the semantic category of the pixels in the screenshot and output a mask 1 corresponding to the screenshot. This mask 1 includes at least one region of the object to be segmented and at least one region of the object not to be segmented. Subsequently, the electronic device can perform connected component analysis on mask 1 using the target connected component area threshold. Regions in mask 1 with areas smaller than the target connected component area threshold are corrected to non-segmented regions, achieving adaptive post-processing of the image. This results in a high-precision target mask, avoiding false positives and ensuring accurate segmentation of the target object, thus guaranteeing semantic segmentation and subsequent image processing performance. Furthermore, by performing pre- and post-processing on the image, the electronic device can achieve high segmentation accuracy with minimal computational overhead, reducing resource waste and improving user experience.

[0085] For example, the electronic device in the embodiments of this application may be a mobile phone, drone, smart camera, wearable device (such as smartwatch, smart bracelet), tablet computer, desktop, laptop, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), media player and other devices with data processing capabilities. The embodiments of this application do not impose special restrictions on the specific form of the electronic device.

[0086] For example, Figure 3 A schematic diagram of the structure of electronic device 200 is shown. For example... Figure 3As shown, the electronic device 200 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 211, a power management module 212, a battery 213, an antenna 1, an antenna 2, a mobile communication module 240, a wireless communication module 250, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headphone jack 270D, a sensor module 280, buttons 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc.

[0087] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 200. In other embodiments of this application, the electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0088] Processor 210 may include one or more processing units, such as application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0089] The controller can be the nerve center and command center of the electronic device 200. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0090] The processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. This memory can store instructions or data that the processor 210 has just used or that are used repeatedly. If the processor 210 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0091] In some embodiments, the processor 210 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0092] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 200. In other embodiments of this application, the electronic device 200 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0093] While charging the battery 213, the charging management module 211 can also supply power to electronic devices through the power management module 212.

[0094] The wireless communication function of electronic device 200 can be implemented through antenna 1, antenna 2, mobile communication module 240, wireless communication module 250, modem processor, and baseband processor.

[0095] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals.

[0096] The mobile communication module 240 can provide wireless communication solutions, including 2G / 3G / 4G / 5G, for use on the electronic device 200. The mobile communication module 240 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 240 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1.

[0097] A modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal.

[0098] The wireless communication module 250 can provide solutions for wireless communication applications on the electronic device 200, including wireless local area networks (WLAN) (such as Wi-Fi), Bluetooth, Global Navigation Satellite System (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR). The wireless communication module 250 receives electromagnetic waves via antenna 2, modulates and filters the electromagnetic wave signal, and sends the processed signal to processor 210. The wireless communication module 250 can also receive signals to be transmitted from processor 210, modulate and amplify them, and then convert them into electromagnetic waves for radiation via antenna 2.

[0099] The electronic device 200 implements display functions through a GPU, a display screen 294, and an application processor. In some embodiments, the electronic device 200 may include one or N display screens 294, where N is a positive integer greater than 1.

[0100] Electronic device 200 can perform shooting functions through ISP, camera 293, video codec, GPU, display 294, and application processor. External memory interface 220 can be used to connect external memory cards, such as Micro SD cards, to expand the storage capacity of electronic device 200.

[0101] Internal memory 221 can be used to store computer executable program code, which includes instructions. Processor 210 executes various functional applications and data processing of electronic device 200 by running the instructions stored in internal memory 221. Internal memory 221 may include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function, etc. The data storage area may store data created during the use of electronic device 200, etc. Furthermore, internal memory 221 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0102] Electronic device 200 can implement audio functions such as music playback and recording through audio module 270, speaker 270A, receiver 270B, microphone 270C, headphone jack 270D, and application processor.

[0103] Buttons 290 include a power button, volume buttons, etc. Indicators 292 can be used to indicate charging status or to indicate messages, etc. Sensor module 280 may include pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, proximity sensors, proximity sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, bone conduction sensors, etc.

[0104] This application proposes an image processing method. For example... Figure 4As shown, firstly, the electronic device can input the image to be processed into a global convolutional neural network model to determine the global features corresponding to the image. The height of the image to be processed is H, and the width is W. Next, the electronic device can input the global features corresponding to the image to be processed into a target policy network model, so that the target policy network model determines the target cropping parameters and the target connected component area threshold that match the image to be processed. The target cropping parameters can be used as pre-processing parameters, and the target connected component area threshold can be used as post-processing parameters. Then, the electronic device can use the target cropping parameters to crop the image to be processed, obtaining a screenshot 1 corresponding to the image to be processed. The proportion of the object to be segmented in screenshot 1 is appropriate, thereby achieving accurate pre-processing of the image to be processed and ensuring the accuracy of subsequent semantic segmentation using screenshot 1. The height of screenshot 1 is h, and the width is w. Then, the electronic device can input screenshot 1 into a target segmentation network model, so that the target segmentation network model performs semantic segmentation on screenshot 1, generating and outputting a mask 1 corresponding to screenshot 1. In this diagram, mask 1 is a binarized image of screenshot 1. The pixel values ​​in the region to be segmented on mask 1 differ from the pixel values ​​in the non-segmented regions. The electronic device can then perform connected component analysis on mask 1, comparing the area of ​​the region to be segmented in mask 1 with a connected component area threshold. This corrects false positive regions (i.e., false positive areas) to non-segmented regions, resulting in mask 2. This removes false positive regions from mask 1, achieving accurate post-processing of the image and ensuring the semantic segmentation accuracy of the image. Both mask 1 and mask 2 have a height of h and a width of w.

[0105] Afterwards, the electronic device can restore the mask 2 to the size of the image to be processed, and obtain the target mask corresponding to the image to be processed. The electronic device can then perform relevant processing on the image to be processed based on the target mask to obtain an image that meets the requirements.

[0106] The target segmentation network model and the target policy network model mentioned above are obtained through training. The electronic device can first train the initial policy network model and the initial segmentation network model based on the training image and its corresponding ground truth to obtain the target policy network model that can accurately determine the cropping parameters and connected component area thresholds that match the image, as well as the target segmentation network model that can achieve accurate semantic segmentation.

[0107] The image processing method provided in this application will be described in two parts, taking the example of a mobile phone as the electronic device and a person as the object to be segmented. One part describes the training process of the initial policy network model and the initial segmentation network model, that is, the training process based on a general adaptive segmentation training framework. The other part describes the process of applying the trained target policy network model and target segmentation network to perform semantic segmentation, that is, the semantic segmentation process based on a general adaptive segmentation inference framework.

[0108] The following will first introduce Figure 5 This section introduces the training process based on an adaptive segmentation training framework. For example... Figure 5 As shown, the image processing method provided in this application embodiment may include S301-S315.

[0109] S301. The mobile phone acquires a training image sample set. The training image sample set includes at least one training image and the ground truth value corresponding to each training image in the at least one training image.

[0110] Each of the training images mentioned above includes an object to be segmented, which can be a person. The ground truth image corresponding to the training image represents the standard binarized image of the training image, which can be obtained by relevant personnel through accurate segmentation of the person in the training image. For example, ... Figure 6 The image shown is a training image and its corresponding ground truth value.

[0111] The training image has a height of H and a width of W. The ground truth image corresponding to the training image also has a height of H and a width of W. Furthermore, the dimensions (i.e., height and width) of the training images can be the same or different; this application does not impose any restrictions on this.

[0112] Optionally, the size of the training image can be the same as or different from the size of the image to be processed.

[0113] S302. For each training image, the mobile phone inputs the training image into the global neural network model to obtain the global features corresponding to the training image.

[0114] In this embodiment of the application, in order to improve image processing efficiency and thus enhance the training efficiency of the initial network model (such as the initial policy network model or the initial segmentation network model), for each training image, the mobile phone can input the training image into the global convolutional neural network model. This allows the global convolutional neural network model to perform coarse processing on the training image, namely, dimensionality reduction and compression, to obtain the global feature e corresponding to the training image. G In other words, e G =f G (i). Where i represents the training image. fG () represents a global convolutional neural network model.

[0115] Optionally, the global features corresponding to the training images can be one-dimensional features.

[0116] In some embodiments, the global convolutional neural network model described above may be a lightweight convolutional network (such as MobileNet-V2) model or a shallow neural network model to reduce computational complexity and thereby improve the training efficiency of the initial network model.

[0117] In some embodiments, S302 above is an optional step, that is, the mobile phone may not know the global features corresponding to the training image, but directly use the training image to train the initial network model. In simple terms, the operation on the global features corresponding to the training image described below can be replaced by describing the operation on the training image.

[0118] S303: The mobile phone trains the initial policy network model based on the global features corresponding to each training image and the ground truth value corresponding to each training image.

[0119] S304. After this training is completed, for each training image, the mobile phone obtains the initial cropping parameters and initial connected component area thresholds corresponding to the training image output by the initial policy network model.

[0120] Here, the initial cropping parameters and initial connected component area thresholds corresponding to the training image represent the initial cropping parameters and initial connected component area thresholds predicted by the initial policy network model to match the training image.

[0121] S305. The mobile phone crops the training image based on the initial cropping parameters to obtain screenshot 2 corresponding to the training image.

[0122] In this embodiment, the initial policy network model updates its parameters by learning the global features corresponding to the training images and the ground truth values ​​corresponding to the training images. This enables the initial policy network model to output pre- and post-processing parameters suitable for the training images, thereby achieving the training of the initial policy network model. Furthermore, this training can include multiple rounds of training (i.e., multiple training iterations). After one round of training, for each training image, the mobile phone can obtain the initial cropping parameters and initial connected component area threshold predicted by the initial policy network model during (or described as after) this round of training, corresponding to the training image. These initial cropping parameters and initial connected component area thresholds are used to determine the training effect of the initial policy network model, thereby determining whether further training of the initial policy network model is necessary.

[0123] The initial cropping parameters mentioned above can be determined by the initial policy network model based on the constraints corresponding to the preprocessing parameters. For example, the initial cropping parameters corresponding to the training image include w1, w2, h1, and h2 corresponding to the training image. In one case, such as... Figure 7A As shown, w1 represents the distance between the left boundary of the character boundary region in screenshot 2 (here described as the training image, ground truth) and the left boundary of screenshot 2. w2 represents the distance between the right boundary of the character boundary region in screenshot 2 and the right boundary of screenshot 2. h1 represents the distance between the upper boundary of the character boundary region in screenshot 2 and the upper boundary of screenshot 2. h2 represents the distance between the lower boundary of the character boundary region in screenshot 2 and the lower boundary of screenshot 2. Accordingly, the above constraints can be as follows:

[0124]

[0125] Where x1 can be the x-coordinate of the top-left vertex of the person boundary region in the training image (or alternatively, the ground truth), and y1 can be the y-coordinate of the top-left vertex of the person boundary region in the training image. Similarly, x2 can be the x-coordinate of the bottom-right vertex of the person boundary region in the training image, and y2 can be the y-coordinate of the bottom-right vertex of the person boundary region in the training image. Figure 7A or Figure 7B As shown, the top-left vertex of the training image (here or alternatively described as the ground truth) can be the origin (0,0). Accordingly, x1 < x2, y1 < y2. Of course, using the top-left vertex as the origin is just one example; other vertices can also be used as the origin, and this application does not limit this.

[0126] In some embodiments, the positional information of the character boundary region in the aforementioned initial policy network model (such as the coordinates of the top-left and bottom-right vertices of the character boundary region) can be determined by the initial policy network model. On one hand, this initial policy network model has the ability to detect the positional information of the character. Specifically, the initial policy network model can obtain image region information related to the task (such as the positional information of the character boundary region). Simply put, it's equivalent to adding a detection module to the initial policy network model, using the global features corresponding to the image (such as the training image) as input to determine the image region information related to the task, reducing image information redundancy, and minimizing interference from irrelevant image information. On the other hand, the training image or the ground truth image corresponding to the training image is labeled with the character boundary region, enabling the initial policy network model to directly obtain the positional information of the character boundary region from the training image or the ground truth image.

[0127] Of course, the method described above for the initial policy network model to determine the location information of the boundary region of a person is only one example. The mobile phone (such as the initial policy network model) can also determine it based on other methods. For example, the initial policy network model can determine it using the global features corresponding to the training image, or the initial policy network model can determine it using the value of the pixel point on the ground truth (i.e., 255 or 0).

[0128] It should be noted that the operations performed by the network model (such as the initial policy network model described above) in this application embodiment are actually performed by the mobile phone. Furthermore, in this case, the mobile phone can use the positional information of the person boundary region on the training image. For each training image, the mobile phone can crop the training image based on the positional information of the person boundary region on that training image, combined with the initial cropping parameters, to obtain a screenshot 2 including the person boundary region (or a cropped image after training), reducing the probability of cropping out the person boundary region on the training image and achieving accurate cropping of the training image. The height of screenshot 2 can be h, and the width can be w, where h = h1 + h2 + y2 - y1, and w = w1 + w2 + x2 - x1. For example, combined with... Figure 7A First, the phone can determine the position information of the left boundary of screenshot 2 based on the position information of the boundary region of the person and w1. Then, based on the boundary information of the left boundary of screenshot 2, the phone can crop the portion of the image between the left boundary of screenshot 2 and the left boundary of the training image. Similarly, the phone can crop the portion of the image between the right boundary of screenshot 2 and the right boundary of the training image. The phone can also crop the portion of the image between the top boundary of screenshot 2 and the top boundary of the training image. Finally, the phone can crop the portion of the image between the bottom boundary of screenshot 2 and the bottom boundary of the training image.

[0129] In another case, such as Figure 7B As shown, w1 represents the distance between the left boundary of the training image and the left boundary of screenshot 2. w2 represents the distance between the right boundary of the training image and the right boundary of screenshot 2. h1 represents the distance between the top boundary of the training image and the top boundary of screenshot 2. h2 represents the distance between the bottom boundary of the training image and the bottom boundary of screenshot 2. Accordingly, the above constraints can be as follows:

[0130]

[0131] In another scenario, for each training image, the phone can crop it based on initial cropping parameters to obtain a screenshot 2 that includes the boundary region of the person, reducing the probability of cropping out the boundary region of the person in the training image and achieving accurate cropping of the training image. For example, combined with Figure 7BFirst, the phone can determine the position of the left boundary of screenshot 2 based on the location information of the person's boundary region and w1. Then, based on the boundary information of the left boundary of screenshot 2, the phone can crop the portion of the image between the left boundary of screenshot 2 and the left boundary of the training image. Similarly, the phone can crop the portion of the image between the right boundary of screenshot 2 and the right boundary of the training image. The phone can also crop the portion of the image between the top boundary of screenshot 2 and the top boundary of the training image. Finally, the phone can crop the portion of the image between the bottom boundary of screenshot 2 and the bottom boundary of the training image.

[0132] Understandably, in this other case, the phone does not need to use the location information of the person's boundary region to crop the training image. For example, when the person in the training image is located on the left side of the training image, the left boundary of the person's boundary region coincides with the left boundary of the training image. Therefore, w1 in the initial cropping parameters determined by the initial policy network model can be 0.

[0133] In some embodiments, the initial policy network model can determine a constraint range based on the above constraints, so as to select pruning parameters from the constraint range and determine them as initial pruning parameters.

[0134] Optionally, the initial policy network model can randomly select cropping parameters from within the constraints. Alternatively, the cropping parameters can start from the boundary values, i.e., from the maximum value, and gradually decrease in size (this can be simply understood as the size of screenshot 2 being the same as the size of the training image at the beginning, and then the size of screenshot 2 gradually decreasing). Alternatively, the values ​​can start from half of the range, i.e., w1 can start from x1 / 2, w2 from (W-x2) / 2, h1 from y1 / 2, and h2 from (H-y2) / 2, thereby accelerating the convergence speed of the initial policy network model, and consequently accelerating the convergence speed of the overall adaptive segmentation training framework, improving training efficiency.

[0135] It should be noted that in practical applications, the distribution of objects to be segmented in the images to be semantically segmented (i.e., the images to be processed mentioned above) may vary. For example, the objects to be segmented in some images may be located on the left side of the image, while the objects to be segmented in others may be located on the right side. Therefore, in order to better train the initial policy network model so that the trained initial policy network model can accurately determine the target cropping parameters and target connected component area threshold that match the distribution (or distribution information) of the objects to be segmented in the input images to be processed, the training image sample set mentioned above may include training images with different distributions.

[0136] The training process for the initial policy network model has been described above. The training process for the initial segmentation network will be described below.

[0137] S306. The mobile phone trains the initial segmentation network model based on the screenshots 2 corresponding to each training image and the ground truth values ​​corresponding to each training image.

[0138] S307. After this training is completed, for each screenshot 2, the mobile phone obtains the training mask 1 corresponding to the screenshot 2 output by the initial segmentation network model.

[0139] The initial segmentation network model is used to perform semantic segmentation on the image.

[0140] The training mask 1 (or first training mask) corresponding to the above screenshot 2 (or the cropped training image) refers to the binarized image corresponding to screenshot 2 obtained by the initial segmentation network model. It can be understood as the binarized image predicted by the initial segmentation network model. This training mask 1 includes at least one person region (i.e., the region to be segmented) and at least one non-person region (i.e., the region not to be segmented). Each person region (or alternatively described as the second target object region) includes a person. Each non-person region (or alternatively described as the second non-target object region) does not include a person.

[0141] The pixel value in the aforementioned person area can be 255, and correspondingly, the color of the person area can be white. The pixel value in the aforementioned non-person area can be 0, and the color of the non-person area can be black. Of course, the pixel value in the person area can also be 0, and the pixel value in the non-person area can be 255. That is, the pixel value in the person area can be 255 or 0, and the pixel value in the non-person area can be 0 or 255, as long as the pixel values ​​in the person area and the non-person area are different.

[0142] In some embodiments, the training mask 1 described above can be determined based on a predicted probability map. The predicted probability map represents the probability corresponding to each pixel in the screenshot, where the probability indicates the probability that the pixel belongs to a person. The higher the probability of a pixel, the higher the probability that the pixel belongs to a person. This application can set a binarization threshold. When the probability of a pixel is greater than the binarization threshold, the probability that the pixel belongs to a person is very high, and the phone can treat the pixel as foreground. When the probability of a pixel is less than the binarization threshold, the probability that the pixel does not belong to a person is very low, and the phone can treat the pixel as background.

[0143] The probability of each pixel is between 0 and 1. The binarization threshold can be set according to actual needs, such as 0.5, 0.4, or 0.6, as long as it is a value greater than 0 and less than or equal to 1.

[0144] In some embodiments, the initial segmentation network model can perform semantic segmentation on the screenshot 2 corresponding to the training image and output the predicted probability map 1 corresponding to the training image. Then, the mobile phone can use the pixels with a probability greater than the binarization threshold on the predicted probability map 1 as pixels in the person region, and the pixels with a probability less than or equal to the binarization threshold as pixels in the non-target region, thereby obtaining the corresponding training mask 1.

[0145] In other embodiments, the initial segmentation network model first performs semantic segmentation on the screenshot 2 corresponding to the training image to obtain the prediction probability map 1 corresponding to the training image. Then, the initial segmentation network model identifies pixels with probabilities greater than a binarization threshold on the prediction probability map as pixels in the person region, and pixels with probabilities less than or equal to the binarization threshold as pixels in the non-target region, thereby obtaining and outputting the corresponding training mask 1. In other words, the initial segmentation network model can directly output the training mask 1, i.e., m. crop =S(i crop ), where m crop Indicates training mask 1, i crop 2 represents the screenshot corresponding to the training image. S represents the initial segmentation network model.

[0146] Optionally, in this embodiment, the initial segmentation network model can output not only the training mask 1 corresponding to the training image, but also the predicted probability map 1.

[0147] It should be noted that when the probability corresponding to a pixel on the predicted probability map is equal to the binarization threshold, the mobile phone can treat the pixel as a pixel in the human area or as a pixel in the non-human area. This application does not impose any restrictions on this.

[0148] In this embodiment, the initial segmentation network model updates its parameters based on the screenshot 2 corresponding to the training image and the ground truth value of the corresponding training image. This enables the initial segmentation model to accurately predict the human and non-human regions on screenshot 2, achieving semantic segmentation of screenshot 2 and thus training the initial segmentation network model. Furthermore, this training can include multiple rounds of training (i.e., multiple training iterations). After one round of training, for each training image, the mobile phone can obtain the training mask 1 corresponding to screenshot 2 predicted by the initial segmentation network model during (or described as after) this round of training. This training mask 1 can be used to determine the training effect of the initial segmentation network model, thereby determining whether further training of the initial segmentation network model is necessary.

[0149] The training process of the initial segmentation network model has been introduced above. The following section will introduce how to determine whether to stop training the initial segmentation network model and the initial policy network model.

[0150] S308. For each character region in training mask 1, the mobile phone determines whether the area of ​​the character region is greater than or equal to the initial connected component area threshold.

[0151] In this embodiment of the application, generally speaking, the proportion of people in the image is relatively large, and correspondingly, the area of ​​the people region is also relatively large. Therefore, for each person region on the training mask 1 corresponding to screenshot 2, the mobile phone determines whether the area of ​​the person region is greater than or equal to the initial connected component area threshold in order to determine whether the person region is a false positive region.

[0152] If the area of ​​the person region is greater than or equal to the initial connected region area threshold, it indicates that the area of ​​the person region is large and that the person region is not a false positive region. Then the mobile phone can execute S309.

[0153] If the area of ​​the person region is less than the initial connected region area threshold, it indicates that the area of ​​the person region is small, and the person region may not actually be a person region. The person region is a false positive region, and the phone can then execute S310.

[0154] S309, The mobile phone does not correct the training mask 1 for the area of ​​the person.

[0155] In cases where the person region on training mask 1 is not a false positive region, the phone does not process that person region.

[0156] S310, the mobile phone corrects the area of ​​the person to a non-person area, and obtains training mask 2.

[0157] Correcting a person area to a non-person area can mean updating the pixel values ​​of the person area to the values ​​of pixels in the non-person area, such as changing them from 255 to 0. For example, the phone corrects all person areas on training mask 1 with areas smaller than the initial connected component area threshold to non-person areas. That is, after connected component analysis, training mask 2 (i.e., m) corresponding to screenshot 2 is obtained. thr This is used to remove false positive regions on training mask 1, thereby further improving semantic segmentation accuracy.

[0158] S311. Based on the size of the training image, the mobile phone restores the training mask 2 to obtain the target training mask corresponding to the training image.

[0159] The size of the target training mask corresponding to the training image is the same as the size of the training image.

[0160] In this embodiment of the application, for each training image, the mobile phone can restore (or extend) the training mask 2 (or second training mask) corresponding to the training image to obtain a target training mask with the same size as the training image, thereby restoring the original image size. This facilitates the determination of the training efficiency of the initial network model by comparing the target training mask corresponding to the training image with the ground truth corresponding to the training image.

[0161] The target training mask corresponding to the above training image can be understood as the predicted ground truth value corresponding to the training image, that is, the ground truth value obtained by semantic segmentation after the initial network model.

[0162] In some embodiments, the above restoration process may include: the mobile phone may use the training mask 2 corresponding to the training image as a reference, and extend the background pixels in the top, bottom, left, and right sides of the training mask 2 until a target training mask is obtained with a height and width equal to the height and width of the training image. The pixel values ​​in the extended portion are the same as the pixel values ​​in the non-human area.

[0163] The aforementioned extended portion may include a left extended portion, which represents the portion between the left boundary of the training mask 2 and the left boundary of the training image. Taking the initial cropping parameters w1, w2, h1, and h2 in the above initial cropping parameters as an example, where w1, w2, h1, and h2 are the same as in the above case, the extended distance corresponding to the left extended portion can be x1-w1.

[0164] The aforementioned extension may include a right-side extension, which represents the portion between the right boundary of training mask 2 and the right boundary of the training image. The extension distance corresponding to the right-side extension may be W-x2-w2.

[0165] The aforementioned extension may include an upper extension, which represents the portion between the upper boundary of training mask 2 and the upper boundary of the training image. The extension distance corresponding to the upper extension may be y1-h1.

[0166] The aforementioned extension may include a lower extension, which represents the portion between the lower boundary of training mask 2 and the lower boundary of the training image. The extension distance corresponding to the lower extension may be H-y2-h2.

[0167] In this embodiment, the mobile phone extends the training mask 2 to obtain the target training mask, so that the extended part of the target training mask relative to the training mask 2 does not have false positive regions, reducing false positives in segmentation and avoiding segmentation of the entire image. When there are objects in the extended part of the image that are easily segmented into human objects, the object in the image is mistakenly segmented into a human object, resulting in semantic segmentation errors. This can effectively improve the semantic segmentation accuracy.

[0168] In some embodiments, after a round of training, the mobile phone can determine the semantic segmentation accuracy by comparing the similarity between the predicted ground truth and the actual ground truth corresponding to the training images. This semantic segmentation accuracy allows the mobile phone to determine whether the accuracy of the initial network model after this round of training meets the requirements, thereby determining whether to stop training the initial network model. The following will use the similarity as an overlap value as an example, combined with steps S312-S315, to describe the process of determining whether to stop training the initial network model.

[0169] S312. The mobile phone calculates the overlap value between the target training mask corresponding to the training image and the ground truth value corresponding to the training image.

[0170] The higher the overlap value, the higher the accuracy of the initial network model. For example, the overlap value can be the IoU value.

[0171] In some embodiments, S312 describes only one possible way to calculate the overlap value. The mobile phone can also determine the overlap value in other ways, such as calculating the overlap value based on the above training mask 1 and the ground truth. Specifically, the overlap value between the character area in the training mask 1 and the character area in the ground truth can be calculated.

[0172] S313. The mobile phone determines whether the overlap value is greater than or equal to the preset overlap threshold.

[0173] In this embodiment, when the overlap value is greater than or equal to a preset overlap threshold, it indicates that the overlap between the training mask 1 corresponding to the training image predicted by the initial segmentation network model and the ground truth value corresponding to the training image is high. This means that the accuracy of the training mask 1 obtained based on the initial segmentation network model is high, which also means that the semantic segmentation accuracy of the initial segmentation network model is high. In other words, the screenshot 2 obtained by cropping based on the initial cropping parameters output by the initial policy network model can enable the semantic segmentation of the initial segmentation network model to achieve high accuracy. Therefore, the accuracy of the initial policy network model is also high, and it can output appropriate cropping parameters and connected component area thresholds. The mobile phone can execute S314.

[0174] If the overlap value is less than the preset overlap threshold, it indicates that the overlap between the training mask 1 corresponding to the training image predicted by the initial segmentation network model and the ground truth value corresponding to the training image is low. This means that the accuracy of the training mask 1 obtained based on the initial segmentation network model is low, which in turn means that the semantic segmentation accuracy of the initial segmentation network model is low. In other words, the screenshot 2 obtained by cropping based on the initial cropping parameters output by the initial policy network model cannot achieve a high accuracy in the semantic segmentation of the initial segmentation network model. Therefore, the accuracy of the initial policy network model is also low, and it cannot output suitable cropping parameters and connected component area thresholds. The mobile phone can execute S315.

[0175] In some embodiments, the aforementioned preset overlap threshold can be pre-set or adjusted by the phone as the loss function of the segmentation network model changes. As the number of training epochs increases, the segmentation accuracy of the initial segmentation network model becomes higher, and correspondingly, the loss function value of the initial segmentation network model decreases. As the loss function value decreases, the preset overlap threshold can be increased.

[0176] Understandably, the loss function of the initial segmentation network model does not decrease continuously. As the number of training epochs increases, the initial segmentation network model gradually converges, and its loss function typically decreases and then stabilizes. Therefore, the aforementioned preset overlap threshold can be gradually increased as the loss function decreases. Then, when the loss function stabilizes, the preset overlap threshold also stabilizes, ensuring that the convergence speeds of the policy network model and the segmentation network model are consistent. This allows the adaptive segmentation training pipeline to achieve optimal results with the fastest speed and lowest computational cost.

[0177] In some embodiments, the loss function value of the initial segmentation network model can be the overlap value. Of course, the loss function value can also be determined in other ways, such as calculating the overlap value between the screenshot 2 corresponding to the training image and the ground truth value corresponding to the training image, and using it as the loss function value.

[0178] S314. The mobile phone uses the initial policy network model as the target policy network model and the initial segmentation network model as the target segmentation network model.

[0179] In this embodiment, the mobile phone stops training the initial policy network model and the initial segmentation network model, and uses the initial policy network model obtained in the latest round of training as the target policy network model, and uses the initial segmentation network model obtained in the latest round of training as the target segmentation network model.

[0180] S315, the phone returns to the above-mentioned S303.

[0181] For example, when the above overlap value is less than the preset overlap threshold, it indicates that the accuracy of the initial network model obtained in this round of training is still low and cannot accurately achieve semantic segmentation. Therefore, the mobile phone needs to continue to train the initial network model in the next round.

[0182] For example, such as Figure 8 As shown, the mobile phone inputs the training image into the global convolutional neural network model to obtain the global features corresponding to the training image. This allows the mobile phone to begin multiple rounds of training on the initial training network model based on the training image, the corresponding global features, and the corresponding ground truth values. In the first round of training, the mobile phone inputs the global features and ground truth values ​​corresponding to the training image into the initial policy network model. This allows the initial policy network model to determine and output the initial cropping parameters and the initial connected component area threshold for the training image based on policy p1. Afterward, the mobile phone can crop the training image based on the initial cropping parameters to obtain screenshot 2. Then, the mobile phone can input screenshot 2 into the initial segmentation network model to perform semantic segmentation on screenshot 2, obtaining training mask 1.

[0183] Next, based on the initial connected component area threshold, the phone performs connected component analysis and reconstruction on training mask 1 to obtain the target training mask corresponding to the training image. Then, the phone can calculate the IoU value between training mask 2 and the target training mask corresponding to the training image.

[0184] Afterwards, the phone can calculate whether the IoU value is greater than or equal to the preset overlap threshold 'a'. If so, it indicates that training is complete, and the phone can stop training, using the initial policy network model after this round of training as the target policy network model, and the initial segmentation network model after this round of training as the target segmentation network model. Optionally, the phone can also output the policy p1 so that the target policy network model can subsequently use the policy p1 to determine the pruning parameters.

[0185] If not, the phone can undergo a second round of training, which is similar to the first round and will not be described in detail here.

[0186] Optionally, the initial policy network model described above is a recurrent network model, which can cyclically output different pruning parameters.

[0187] In some embodiments, the training images may further include training images with different levels of segmentation difficulty. The segmentation difficulty refers to the ease or difficulty of semantic segmentation of the objects to be segmented in the image. For example, if a training image contains objects that are easily missegmented as the objects to be segmented, then the segmentation difficulty of that training image is higher. Based on this, for training images with higher segmentation difficulty, the mobile phone may need to train the initial network model for multiple rounds to meet the requirements, while for training images with lower segmentation difficulty, the mobile phone may only need to train for a few rounds to meet the requirements, achieving adaptive training, dynamically allocating computational resources, and thus improving overall training efficiency. Furthermore, this enables the trained target policy network model to adaptively determine the target cropping parameters corresponding to the images to be processed with different segmentation difficulty levels, and enables the trained target segmentation network model to perform accurate semantic segmentation of the screenshots corresponding to the images to be processed with different segmentation difficulty levels, ensuring the segmentation accuracy of different images to be processed.

[0188] In some embodiments, during training, the overall computational cost of training the initial network model can be adjusted by changing the threshold (such as the preset overlap threshold and the target connected component area threshold mentioned above). For example, for cases with different segmentation accuracy requirements, the mobile phone does not need to retrain the initial network model. It can dynamically achieve the required segmentation accuracy with minimal computational cost by adjusting the threshold, or maximize the use of all available computational resources in real time to improve segmentation accuracy and ensure semantic segmentation performance.

[0189] In some embodiments, the initial policy network model described above can be modeled as an agent, trained using a reinforcement learning algorithm. The agent's actions are the initial cropping parameters (such as w1, w2, h1, and h2 mentioned above) and the connected component area threshold. The state can be the proportion of the person region in screenshot 2 corresponding to the training image. The discount reward function can be the IoU value obtained based on the current action. A positive reward is given when the IoU shows an upward trend, and the initial policy network model improves over time, meaning the output initial cropping parameters match the training image more and more closely, thus finding the most suitable cropping parameters. The optimization objective is to maximize the discount reward function. Furthermore, an IoU value greater than or equal to 'a' can be a stopping condition for the iterative training of the initial policy network. Optionally, the positive reward can be understood as increasing the IoU value based on the actually calculated IoU value.

[0190] In some embodiments, for a pre-trained target segmentation network model, when the distribution of people in the image to be processed (i.e., target domain data) differs from that in the training image (i.e., source domain data) (e.g., the location and proportion of people), the semantic segmentation accuracy may decrease. When the semantic segmentation accuracy decreases, the mobile phone can train the target policy network model using only the target domain data, without retraining the target segmentation network model. This allows the semantic segmentation accuracy to meet the requirements without changing the target segmentation network model, thereby achieving fast semantic segmentation of the target domain data.

[0191] The above describes the process of training an initial network model to obtain a high-precision target network model (such as an object segmentation network model or an object policy network model). After obtaining the target network model, the mobile phone can apply this target network model for semantic segmentation. The following will combine... Figure 9 We will continue to introduce the process of semantic segmentation using the target network model.

[0192] S401, The mobile phone receives the user's command to launch the camera application.

[0193] S402. In response to the above-mentioned startup operation, the mobile phone displays the shooting interface. The shooting interface includes a preview area, which displays the image of the person captured by the mobile phone's camera.

[0194] The shooting interface is the interface displayed when the phone is recording video or taking a photo. This interface may include a preview area and a shutter button. The preview area displays the image captured by the camera. The following section will use the phone in photo-taking mode as an example to illustrate the semantic segmentation process. For example, when a user clicks... Figure 10A The camera app shown is activated by the phone in response to a tap on it, launching the camera app and entering photo-taking mode (e.g., ...). Figure 10B As shown in the image, the phone displays an image of a person captured by its camera. Here, the image of the person refers to the image to be processed.

[0195] The aforementioned shooting interface may also include, for example: Figure 10B The portrait control shown is used to trigger the phone to enter portrait mode.

[0196] S403, The mobile phone receives the user's click operation on the portrait control on the shooting interface.

[0197] S404. In response to the user's click on the portrait control, the mobile phone inputs the portrait image into the global convolutional neural network model to obtain the global features corresponding to the portrait image.

[0198] For example, when a user wants to enhance the people in the preview image displayed on their phone, they can click as described above. Figure 10B The portrait control shown. In response to the user's click on the portrait control, the phone enters portrait mode (e.g., ...). Figure 10C As shown), the process of enhancing the portrait begins, as described in steps S405-S417 below.

[0199] S405. The mobile phone inputs the global features corresponding to the person image into the target policy network model to obtain the target cropping parameters and the target connected region area threshold corresponding to the person image.

[0200] S406. The mobile phone crops the image of the person based on the target cropping parameters to obtain the corresponding screenshot 1.

[0201] The target policy network model outputs target cropping parameters and target connected component area thresholds that match the person image. The target cropping parameters are used to crop the person image so that the proportion of the person in the cropped image is appropriate, thus ensuring high accuracy of the mask obtained from semantic segmentation based on the cropped image, and consequently, good semantic segmentation results.

[0202] It's important to note that when a mobile phone crops a person image based on target cropping parameters, it doesn't crop out the person from the image. For example, the target policy network model mentioned above can determine the location information of the person's boundary region in the image, and then determine appropriate target cropping parameters and target connected component area thresholds based on this location information. The target policy network model can then output the target cropping parameters, target connected component area thresholds, and the location information of the person's boundary region, allowing the mobile phone to use this boundary information and the target cropping parameters for cropping. Alternatively, the mobile phone can first input the person image into a human detection model to identify the person's boundary region and determine its location. Then, the mobile phone can input the person image and the location information of the person's boundary region into the target policy network model, allowing the model to determine and output appropriate target cropping parameters and target connected component area thresholds. Finally, the mobile phone can use this boundary information and the target cropping parameters for cropping.

[0203] Optionally, the aforementioned human detection model can be a small human detection model, such as a lightweight human detection model, to enable rapid detection of people.

[0204] In this embodiment, the mobile phone enters portrait mode in response to a user's click on the portrait control on the shooting interface to perform portrait enhancement processing on the person in the image. To achieve portrait enhancement, the mobile phone first needs to perform semantic segmentation to determine the person region in the image. The mobile phone can then input the image into a global convolutional neural network model to obtain the global features corresponding to the image. These global features are used to determine appropriate pre- and post-processing parameters, thereby achieving accurate semantic segmentation of the image using target cropping parameters and target connected component area thresholds. Furthermore, since the pre- and post-processing parameters are determined using the global features corresponding to the image, the computational load can be reduced, and the efficiency of the pre- and post-processing parameters can be improved.

[0205] In some embodiments, similar to S302 above, S404 is also an optional step. Accordingly, if S404 is not performed, S405 can be described as the mobile phone inputting a person image into a target policy network model to obtain the target cropping parameters and target connected component area threshold corresponding to the person image.

[0206] In some embodiments, the mobile phone's use of the target policy network model to determine the target connected component area threshold and target cropping parameters matching the person image is only one possible implementation of determining the target connected component area threshold and target cropping parameters matching the person image. The mobile phone can also use other methods to determine the target connected component area threshold and target cropping parameters matching the person image, such as determining the distribution of people in the person image and determining the matching target connected component area threshold and target cropping parameters. The specific determination process is similar to the process described above where the target policy network model determines the matching target connected component area threshold and target cropping parameters based on the distribution of the person image.

[0207] In some embodiments, the user's click on the portrait control described above is only one possible triggering operation for the phone to perform semantic segmentation tasks; this triggering operation can also be of other types. For example, in response to the user's click on, such as... Figure 10D When clicking on the gallery app, the phone displays something like this: Figure 10E The gallery interface shown is 20. Afterwards, the mobile phone receives user input... Figure 10E Clicking on the displayed image 21 will show the following: Figure 10FThe interface including the image is shown. Among them, the interface 21 also includes controls for image processing, such as edit, more, and other controls. In response to the user's click operation on the edit control, the mobile phone can display more edit controls, such as blur, filter, and other controls. After that, when the user wants to add a filter to the image 21, the user can click on the filter. Here, the image 21 can be used as the above-mentioned portrait image 1, and the click on the filter can be the above-mentioned trigger operation that can trigger the mobile phone to perform a semantic segmentation task. For another example, the trigger operation can be the user's operation to enable the AI mode on the shooting interface, etc.

[0208] S407. The mobile phone inputs the screenshot 1 corresponding to the portrait image into the target segmentation network model, so that the target segmentation network model performs semantic segmentation on the screenshot 1 and outputs the predicted probability map corresponding to the screenshot 1.

[0209] In the embodiment of the present application, the portrait image is cropped by using the target cropping parameters that match the screenshot 1 (or the cropped image) for the target segmentation network model to perform semantic segmentation on the cropped portrait image, which can effectively reduce the computational overhead of the target segmentation network model and improve the segmentation accuracy when the proportion of the person in the portrait image is small, ensuring the semantic segmentation effect.

[0210] S408. The mobile phone counts the number of pixel points on the predicted probability map whose probabilities are equal to the binary threshold.

[0211] S409. The mobile phone determines whether the number is greater than or equal to a preset number threshold.

[0212] In the embodiment of the present application, when the probability corresponding to a pixel point is equal to 0.5, the pixel point may belong to a person or may not belong to a person. Based on this, when the number of pixel points on the predicted probability map whose probabilities are equal to the binary threshold is large, that is, when the number (or the first number) is greater than or equal to the preset number threshold (or the first number threshold), it indicates that the number of pixel points whose belonging to a person is uncertain is large, that is, it indicates that the semantic segmentation accuracy of the target segmentation network model for the screenshot 1 is low, and the proportion of the person in the screenshot 1 affects the semantic segmentation accuracy. Therefore, the above target cropping parameters may not be suitable for the portrait image, so the mobile phone needs to re-determine the cropping parameters to crop the portrait image, and then the mobile phone can execute S410.

[0213] When the number of pixel points on the predicted probability image whose probabilities are equal to the binary threshold is small, that is, when the number is less than the preset number threshold, it indicates that the number of pixel points whose belonging to a person is uncertain is small, that is, the semantic segmentation accuracy is high. In other words, the above target cropping parameters are suitable for the portrait image and are matched with the portrait image. Therefore, the mobile phone does not need to re-crop the portrait image, and then the mobile phone can execute S411.

[0214] S410, the phone returns to the above S405.

[0215] In this embodiment, the mobile phone re-inputs the image of the person into the target policy network model to determine new target cropping parameters and a new target connected component area threshold corresponding to the image. The new target cropping parameters differ from the previously determined target cropping parameters. However, the new target connected component threshold may be the same as or different from the previously determined target connected component threshold; this application does not limit it.

[0216] S411. The mobile phone determines the mask 1 corresponding to the image of the person based on the predicted probability map.

[0217] Mask 1 (or the first mask) is similar to the training mask 1 mentioned above, and may include at least one human figure region (or may be described as the first target object region) and at least one non-human figure region (or the first non-target object region).

[0218] In some embodiments, the probability corresponding to a pixel in the aforementioned person region is greater than a binarization threshold. The probability corresponding to a pixel in a non-person region is less than or equal to the binarization threshold. Furthermore, when the probability corresponding to a pixel on the predicted probability map is equal to the binarization threshold, the mobile phone can determine that the pixel belongs to the person region or not; this application does not impose any limitations on this.

[0219] In some embodiments, the mask 1 can be determined by the mobile phone based on the predicted probability image, or it can be directly output by the target segmentation network model. That is, the mobile phone outputs screenshot 1 to the target segmentation network model, which outputs the predicted probability image and mask 1.

[0220] In addition, S408-S410 above are optional steps. The mobile phone can also directly execute S411 above to determine the mask 1 without judging whether the number of pixels with a probability equal to the binarization threshold on the predicted probability map is greater than the preset number threshold to determine whether the semantic segmentation accuracy is high.

[0221] In some embodiments, the mask 1 can be the output of the segmentation network model. Accordingly, if the above steps S408-S410 are optional, then it is acceptable whether the target segmentation network model outputs a predicted probability map or not.

[0222] Although using a target segmentation network model to perform semantic segmentation on screenshot 1 obtained by cropping based on target clipping parameters can achieve good semantic segmentation results, the resulting mask 1 may still contain false positive regions. Therefore, the phone can use the target connected component area threshold to perform connected component analysis on mask 1 to remove false positive regions on mask 1, thereby further improving the semantic segmentation accuracy. The connected component analysis process will be described in detail below with reference to steps S412-S414.

[0223] In some embodiments, the mobile phone's use of the target segmentation network model to determine the mask 1 corresponding to the person image is merely an example. The mobile phone can also directly determine the mask 1 corresponding to the person image based on a semantic segmentation algorithm. The determination process is similar to that of the target segmentation network model in determining the mask 1, and this application does not limit it. Similarly, during the training phase, the mobile phone can train only the target policy network model without training the target segmentation network model. Correspondingly, the mobile phone can also perform semantic segmentation on the cropped training images without using the initial segmentation network model.

[0224] S412. For each character region on mask 1, the mobile phone determines whether the area of ​​the character region is greater than or equal to the target connected component area threshold.

[0225] In this embodiment of the application, for each person region on the mask 1, if the area of ​​the person region is greater than or equal to the target connected region area threshold, it indicates that the area of ​​the person region is large and the person region is not a false positive region, then the mobile phone can execute S413.

[0226] If the area of ​​the person region is less than the target connected region area threshold, it indicates that the area of ​​the person region is small and the person region is a false positive region. In this case, the mobile phone can execute S414.

[0227] S413, The mobile phone does not correct the area of ​​the person in the mask 1.

[0228] S414. The mobile phone corrects the area of ​​the person in mask 1 to obtain mask 2.

[0229] The process of connected component analysis described in S412-S414 above can be referred to the relevant content in S308-S310 above, and will not be repeated here.

[0230] The above describes the process by which the mobile phone obtains mask 2, which does not include false positive regions. After obtaining mask 2, in order to better process the image of the person, the mobile phone can restore mask 2 to obtain a target mask with the same size as the image of the person. For details, please refer to the relevant content of S415 below.

[0231] S415. The mobile phone restores the mask 2 based on the size of the person image to obtain the target mask corresponding to the person image.

[0232] The target mask corresponding to the person image has the same size as the person image, that is, the height and width are the same.

[0233] In this embodiment, the mobile phone can use the mask obtained by performing connected component analysis on mask 1 as mask 2. Then, the mobile phone can restore the size of mask 2 (or the second mask) to the size of the person image to obtain a target mask with the same size as the person image. This facilitates subsequent editing operations on the person image using the target mask, i.e., image processing. Alternatively, the mobile phone can directly use mask 2 as the target mask corresponding to the aforementioned person image, achieving rapid determination of the target mask.

[0234] The process of restoring mask 2 by the mobile phone can be referred to as the process of restoring training mask 2 by the mobile phone, and will not be repeated here.

[0235] S416. The mobile phone performs portrait enhancement processing on the image of a person based on the target mask, and obtains the enhanced image of the person.

[0236] S417, the phone displays an enhanced image of a person.

[0237] For example, a mobile phone can perform portrait enhancement processing on a person image based on a target mask to improve the quality of the image. For instance, in response to a user's click on the aforementioned portrait control, the mobile phone displays an enhanced person image (or target person image) in which the person is clearer.

[0238] It should be noted that the above-mentioned portrait enhancement processing is only one possible processing method for human images. Mobile phones can also perform other types of processing on portrait enhancement, such as image cutout, filter processing (or smart filters), and background blurring. In other words, a mobile phone can perform a first processing on a human image based on a target mask, which can include one or more of the following: portrait enhancement, image cutout, filter processing, and background blurring.

[0239] In some embodiments, during application, the overall computational cost of the target network model can be adjusted by changing the threshold (such as the preset quantity threshold mentioned above). For example, for cases where the segmentation accuracy requirement is low, the mobile phone will set a smaller threshold, thereby achieving the required segmentation accuracy with less computational cost.

[0240] It should be noted that the embodiments of this application illustrate the solution by using the same device (such as a mobile phone) to train the initial network model and the same device to apply the trained target network model. Of course, the device for training the initial network model and the device for applying the trained target network model do not have to be the same device. For example, the initial network model can be trained on a server, and the target network model can be installed on the device that needs to perform semantic segmentation. Furthermore, the number of objects to be segmented on the image to be processed or the training image can be one or more. The number of objects to be segmented shown in Figure 1 is merely one example, and this application does not limit it.

[0241] In some embodiments, the object to be segmented described above as a person is merely an example. The image processing method provided in this application can be applied to any single-class semantic segmentation task (such as segmenting target objects into categories like sky, pets, and plants), or multi-class semantic segmentation tasks. However, if applied to a multi-class semantic segmentation task, the phone can first segment the target objects of each category in the multi-class segmentation process.

[0242] In some embodiments, this application provides a chip system that can be applied to the above-described electronic device, which performs the image processing method described above.

[0243] In some embodiments, this application provides a computer storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the image processing method described above.

[0244] In some embodiments, this application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the image processing method described above.

[0245] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0246] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0247] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0248] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0249] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0250] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized by, Applied to electronic devices, the method includes: Based on the image to be processed, target cropping parameters and target connected component area thresholds that match the image to be processed are determined, wherein the target cropping parameters and the target connected component area thresholds are determined by a target policy network model; Based on the target cropping parameters, the image to be processed is cropped to obtain the cropped image; The cropped image is subjected to semantic segmentation to obtain a first mask; wherein, the first mask is a binarized image of the cropped image, and the first mask includes at least one first target object region and a first non-target object region; the first target object region refers to the region that includes the target object, and the first non-target object region refers to the region that does not include the target object; Based on the target connected component area threshold, connected component analysis is performed on the first mask to determine the target mask corresponding to the image to be processed; wherein, the connected component analysis indicates that the region that was erroneously semantically segmented into the first target object region is updated into the first non-target object region.

2. The method of claim 1, wherein, The semantic segmentation process performed on the cropped image to obtain the first mask includes: The cropped image is input into a target segmentation network model to obtain a predicted probability map; wherein, the predicted probability map includes a first probability corresponding to each pixel in the cropped image; the first probability corresponding to each pixel represents the probability that the pixel belongs to the target object; Based on the predicted probability map, the first mask is determined; wherein, the first probability corresponding to the pixel in the first target object region of the first mask is greater than the first preset probability value, and the first probability corresponding to the pixel in the first non-target object region of the first mask is less than the first preset probability value.

3. The method of claim 2, wherein, Before determining the first mask, the method further includes: Obtain the first number of pixels in the predicted probability map whose first probability is equal to the first preset probability value; It is determined that the first quantity is less than the first quantity threshold.

4. The method of claim 3, wherein, The method further includes: If the first quantity is determined to be greater than or equal to the first quantity threshold, the process returns to the step of determining the target cropping parameters and the target connected component area threshold that match the image to be processed based on the image to be processed.

5. The method according to any one of claims 1 to 4, characterized in that, The step of determining the target cropping parameters and target connected component area threshold that match the image to be processed, based on the image to be processed, includes: The image to be processed is input into a global convolutional neural network model to obtain the global features corresponding to the image to be processed; Based on the global features corresponding to the image to be processed, the target cropping parameters and target connected component area thresholds that match the image to be processed are determined through the target policy network model.

6. The method of claim 1, wherein, The step of performing connected component analysis on the first mask based on the target connected component area threshold to determine the target mask corresponding to the image to be processed includes: The first target object region in the first mask whose area is smaller than the target connected component area threshold is updated to the first non-target object region to obtain the second mask; The target mask is determined based on the second mask.

7. The method of claim 6, wherein, Determining the target mask based on the second mask includes: Based on the size of the image to be processed, the second mask is extended to obtain the target mask; wherein, the size of the target mask is the same as the size of the image to be processed, and the pixel values ​​of the extended region in the target mask are the same as the pixel values ​​of the first non-target object region.

8. The method according to any one of claims 1 to 4, characterized in that, The step of determining the target cropping parameters and target connected component area threshold that match the image to be processed, based on the image to be processed, includes: The image to be processed is input into the target policy network model to obtain the target cropping parameters and the target connected component area threshold.

9. The method of claim 8, wherein, The method further includes: Obtain a training image sample set; wherein the training image sample set includes at least one training image and a ground truth value corresponding to each training image; each training image is an image including the target object; the ground truth value corresponding to the training image represents the actual mask corresponding to the training image; Based on the training image sample set, the initial policy network model is trained to obtain the target policy network model.

10. The method of claim 9, wherein, The step of training the initial policy network model based on the training image sample set to obtain the target policy network model includes: For each training image, the training image is input into the initial policy network model to obtain initial cropping parameters that match the training image; Based on the initial cropping parameters, the training image is cropped to obtain the cropped training image; The cropped training image is subjected to semantic segmentation to obtain a first training mask; wherein the first training mask includes at least one second target object region and a second non-target object region; Based on the ground truth value corresponding to the first training mask and the training image, an overlap value is determined; wherein, the larger the overlap value, the higher the degree of overlap between the second target object region in the first training mask and the second target object region in the ground truth value; If the overlap value is less than a preset overlap threshold, return to the step of inputting the training image into the initial policy network model to obtain initial cropping parameters that match the training image; If the overlap value is greater than or equal to the preset overlap threshold, the initial policy network model is determined to be the target policy network model.

11. The method of claim 10, wherein, The initial policy network model is also used to output an initial connected component area threshold; the determination of the overlap value based on the first training mask and the ground truth value corresponding to the training image includes: Based on the initial connected component area threshold, connected component analysis is performed on the first training mask to determine the target training mask corresponding to the training image. Determine the degree of overlap between the target training mask corresponding to the training image and the ground truth value corresponding to the training image.

12. The method of claim 10, wherein, The step of determining the overlap value based on the first training mask and the ground truth value corresponding to the training image includes: Determine the degree of overlap between the second target object region in the first training mask and the second target object region in the ground truth.

13. The method of claim 1, wherein, The image to be processed includes an image of a person; before determining the target cropping parameters and target connected component area threshold that match the image to be processed based on the image to be processed, the method further includes: The shooting interface is displayed; wherein, the shooting interface includes the image of the person; After determining the target mask corresponding to the image to be processed, the method further includes: Displaying a target person image; wherein the target person image is obtained by performing a first processing on the person image based on the target mask corresponding to the image to be processed; wherein the first processing includes at least one or more of portrait enhancement processing, image cutout processing, filter processing, and background blur processing.

14. A chip system, characterized by The chip system is applied to an electronic device that performs the method as described in any one of claims 1 to 13.

15. An electronic device, comprising: The electronic device includes a memory and one or more processors; the memory and the processors are coupled; the memory is used to store computer program code, the computer program code including computer instructions; when the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 1 to 13.

16. A computer readable storage medium characterized by: Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 13.

17. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 13.

Images