Image processing method, intelligent terminal and storage medium

Abstract

The present application proposes an image processing method, an intelligent terminal and a storage medium. The processing method is applied to a processing device and includes the following steps: obtaining a picture to be processed. The picture to be processed includes a skin area. The skin area may include a facial skin area and / or a body skin area. The intelligent terminal can input the picture to be processed into a recognition model. The recognition model will process the picture to be processed according to its preset parameters and algorithm steps, thereby obtaining first information of the skin area. The intelligent terminal can determine the target area based on the first information, thereby cropping the target image according to the target area. The present application improves the accuracy of skin segmentation and the processing efficiency of the recognition model.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method, an intelligent terminal and a storage medium. Background Art

[0002] With the development of science and technology, face detection, gesture recognition, posture recognition and other functions are increasingly being applied to various terminal devices. In the process of implementing these functions, using skin segmentation methods to pre-process images can effectively reduce external interference such as background in the process of face detection and gesture recognition.

[0003] In some implementations, skin segmentation can be achieved by region-based segmentation methods such as region growing method and watershed method. Alternatively, thresholds can be set by statistical methods, machine learning methods, etc. to achieve skin segmentation by pixels.

[0004] In the process of conceiving and implementing the present application, the inventors discovered that there are at least the following problems: the above-mentioned skin segmentation method has the problem of low segmentation accuracy for users with dark skin.

[0005] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the invention

[0006] In response to the above technical problems, the present application provides an image processing method, an intelligent terminal and a storage medium, which can improve the skin segmentation accuracy for users with dark skin.

[0007] In order to solve the above technical problems, the present application provides an image processing method which can be applied to a smart terminal, comprising the following steps:

[0008] S1: Acquire a picture to be processed, optionally, the picture to be processed includes a skin area;

[0009] S2: Inputting the image to be processed into a recognition model for processing to obtain first information of the skin area;

[0010] S3: Process the image to be processed (such as cropping) according to the first information to obtain a target image.

[0011] Optionally, the step S3 includes:

[0012] Determine a target area contour of the target object according to the first information;

[0013] According to the target area contour, the image to be processed is processed to obtain a target image of the target object.

[0014] Optionally, the step S2 includes:

[0015] Inputting the image to be processed into a feature extraction module of the recognition model to obtain at least one primary feature map;

[0016] Inputting at least one of the primary feature maps into the attention module of the recognition model to obtain a high-level feature map;

[0017] First information of the skin area is determined or generated based on the high-level feature map.

[0018] Optionally, the method further comprises:

[0019] Inputting at least one of the primary feature maps into an edge extraction module of the recognition model to obtain an edge feature map;

[0020] The high-level feature map, the edge feature map and at least one of the primary feature maps are input into an output module of the recognition model to obtain the first information.

[0021] Optionally, inputting at least one of the primary feature maps into the attention module of the recognition model to obtain a high-level feature map comprises:

[0022] Inputting the primary feature map into the feature dimension reduction module of the attention module for dimension reduction processing to obtain a first dimension reduction feature map and a second dimension reduction feature map;

[0023] Inputting the first dimension reduction feature map into the position perception attention module of the attention module to perform up and down position information perception processing to obtain a first output feature map;

[0024] Inputting the second dimensionality reduction feature map into the channel attention module of the attention module for upper and lower channel perception processing to obtain a second output feature map;

[0025] According to the first output feature map and the second output feature map, a high-level feature map is obtained by fusing.

[0026] Optionally, the step of inputting the first dimension reduction feature map into the position perception attention module of the attention module to perform up and down position information perception processing to obtain a first output feature map includes:

[0027] Using the feature transformation submodule in the position-aware attention module, transforming the first reduced-dimensionality feature map to obtain a third reduced-dimensionality feature map;

[0028] Using the multi-scale mixed distance relationship submodule in the position-aware attention module, calculating the distance features of each scale of each feature point in the third dimensionality reduction feature map;

[0029] Using the feature interaction submodule in the position-aware attention module, the distance features of each scale of each feature point in the third dimensionality reduction feature map are fused to obtain a first attention feature map;

[0030] Determine the first output feature map based on the first attention feature map and the first dimensionality reduction feature map.

[0031] Optionally, determining the first output feature map according to the first attention feature map and the first dimensionality reduction feature map includes:

[0032] Determine, using a feature activation module in the position-aware attention module, a second attention feature map based on the first dimensionality reduction feature map and the first attention feature map;

[0033] Using the feature transformation submodule in the position-aware attention module, transforming the second attention feature map to obtain a third attention feature map;

[0034] Determine a first output feature map based on the third attention feature map and the first dimensionality reduction feature map.

[0035] Optionally, the using the multi-scale mixed distance relationship submodule in the position-aware attention module to calculate the distance features of each scale of each feature point in the third dimensionality reduction feature map includes:

[0036] Performing a pooling operation on the third dimensionality reduction feature map to obtain pooling features of each scale for each feature point in the third dimensionality reduction feature map;

[0037] Calculate the distance feature between each feature point in the third dimensionality reduction feature map and each pooling feature corresponding to the feature point.

[0038] Optionally, inputting the second dimension reduction feature map into the channel attention module of the attention module for upper and lower channel perception processing to obtain a second output feature map, including:

[0039] Obtaining a fourth attention feature map according to the reconstructed feature map and the transposed feature map of the second dimensionality reduction feature map;

[0040] Determine a second output feature map based on the reconstructed feature map of the fourth attention feature map and the second dimensionality reduction feature map.

[0041] Optionally, obtaining a fourth attention feature map according to the reconstructed feature map and the transposed feature map of the second dimensionality reduction feature map includes:

[0042] According to the reconstructed feature map and the transposed feature map of the second dimensionality reduction feature map, the product thereof is calculated as a product feature map;

[0043] Normalizing the product feature map using a normalization function to obtain a normalized feature map;

[0044] According to the reconstructed feature map of the normalized feature map and the second dimensionality reduction feature map, the product thereof is calculated as a fourth attention feature map.

[0045] Optionally, inputting at least one of the primary feature maps into an edge extraction module of the recognition model to obtain an edge feature map comprises:

[0046] Performing convolution calculation on the at least one primary feature map to obtain at least one intermediate feature map;

[0047] Performing convolution calculation on at least one intermediate feature map to obtain at least one edge sub-feature map;

[0048] The at least one edge sub-feature map is fused to obtain the edge feature map.

[0049] Optionally, the method further comprises:

[0050] Inputting the training image into the recognition model for processing to obtain at least one of an edge loss value, a segmentation loss value, and an auxiliary loss value;

[0051] Use at least one of the segmentation loss value, the edge loss value, and the auxiliary loss value to adjust the parameters of the recognition model to obtain a trained recognition model.

[0052] Optionally, the step of inputting the training image into the recognition model for processing to obtain at least one of an edge loss value, a segmentation loss value, and an auxiliary loss value comprises:

[0053] Inputting the training image into the feature extraction module of the recognition model for prediction to obtain first information;

[0054] At least one of an edge loss value, a segmentation loss value, and an auxiliary loss value is obtained according to the first information and the edge label of the training image.

[0055] Optionally, the method further comprises:

[0056] Input the training image into the recognition model for processing to obtain an edge loss value and a segmentation loss value;

[0057] The segmentation loss value and the edge loss value are used to adjust the parameters of the recognition model to obtain a trained recognition model.

[0058] Optionally, the method further comprises:

[0059] Input the training image into the recognition model for processing to obtain a segmentation loss value;

[0060] The segmentation loss value is used to adjust the parameters of the recognition model to obtain a trained recognition model.

[0061] Optionally, inputting the training image into the recognition model for processing to obtain a segmentation loss value includes:

[0062] Inputting the training image into the feature extraction module of the recognition model for prediction to obtain first information;

[0063] A segmentation loss value is obtained according to the first information and the edge label of the training image.

[0064] Optionally, the method further comprises:

[0065] Get the original image;

[0066] Preprocessing the original image to obtain the image to be processed;

[0067] Use the brightness enhancement module to optimize the brightness of the image to be processed.

[0068] Optionally, the first information includes facial skin contour and / or body skin contour.

[0069] The present application also provides an intelligent terminal, comprising: a memory and a processor, wherein an image processing method program is stored in the memory, and when the image processing method program is executed by the processor, the steps of any of the above-mentioned image processing methods are implemented.

[0070] The present application also provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned image processing methods are implemented.

[0071] As described above, the image processing method of the present application can be applied to a smart terminal, and includes the following steps: the smart terminal can obtain a picture to be processed. The picture to be processed includes a skin area. The skin area may include a facial skin area and / or a body skin area. The smart terminal can input the picture to be processed into a recognition model. The recognition model will process the picture to be processed according to its preset parameters and algorithm steps, thereby obtaining first information of the skin area. The smart terminal can determine the target area based on the first information, and thus obtain a target picture by cropping according to the target area. Through the above technical solution, it is possible to achieve an effect of improving the accuracy of skin segmentation and improving the processing efficiency of the recognition model, thereby solving the problem of low segmentation accuracy of the skin segmentation method in the prior art for users with dark skin, thereby improving the user experience of users with dark skin. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application. In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor.

[0073] Figure 1 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present application;

[0074] Figure 2 A communication network system architecture diagram provided for an embodiment of the present application;

[0075] Figure 3 is a flowchart of an image processing method according to the first embodiment;

[0076] Figure 4 is a schematic diagram of a picture to be processed according to the first embodiment;

[0077] Figure 5 is a schematic diagram of an application scenario of an image processing method according to the first embodiment;

[0078] Figure 6 is a schematic diagram of a training picture according to the second embodiment;

[0079] Figure 7 is a schematic diagram of a recognition model structure according to the second embodiment;

[0080] Figure 8 is a schematic diagram of a flow chart of a feature extraction module according to the third embodiment;

[0081] Fig. 9 is a flow chart of an attention module according to the third embodiment;

[0082] Fig.10 is a schematic diagram of the structure of a position-aware attention module according to the third embodiment;

[0083] Fig.11 is a schematic structural diagram of a channel attention module according to the third embodiment;

[0084] Fig.12 is a schematic diagram of a flow chart of an edge extraction module according to the third embodiment;

[0085] Fig.13 is a schematic structural diagram of an edge extraction module according to the third embodiment;

[0086] Fig.14 is a flow chart of an output module according to the third embodiment;

[0087] Fig.15 is a schematic structural diagram of a module according to a third embodiment;

[0088] Fig.16 is a schematic structural diagram of a decoder according to the third embodiment;

[0089] Fig.17 is a schematic diagram of a process flow of a deep supervision module according to the third embodiment;

[0090] Fig.18 is a schematic diagram of a label according to the third embodiment.

[0091] The realization of the purpose, functional features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. The above-mentioned drawings have shown clear embodiments of this application, which will be described in more detail later. These drawings and textual descriptions are not intended to limit the scope of the concept of this application in any way, but to illustrate the concept of this application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0092] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0093] It should be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element. In addition, components, features, and elements with the same name in different embodiments of the present application may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.

[0094] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in this article, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this article, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "at the time of..." or "when..." or "in response to determination". Furthermore, as used in this article, the singular forms "one", "one" and "the" are intended to also include plural forms, unless there is an opposite indication in the context. It should be further understood that the terms "comprising", "including" indicate the existence of features, steps, operations, elements, components, projects, kinds, and / or groups, but do not exclude the existence, occurrence or addition of one or more other features, steps, operations, elements, components, projects, kinds, and / or groups. The terms "or", "and / or", "including at least one of the following" used in this application, etc. can be interpreted as inclusive, or mean any one or any combination. For example, “comprising at least one of the following: A, B, C” means “any of the following: A; B; C; A and B; A and C; B and C; A and B and C”, and for another example, “A, B or C” or “A, B and / or C” means “any of the following: A; B; C; A and B; A and C; B and C; A and B and C”. An exception to this definition will only occur when a combination of elements, functions, steps or operations are inherently mutually exclusive in some manner.

[0095] It should be understood that, although the various steps in the flowchart in the embodiment of the present application are displayed in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and it can be performed in other orders. Moreover, at least a portion of the steps in the figure may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and their execution order is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0096] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0097] It should be noted that, in this article, step codes such as S101, S102, etc. are used for the purpose of expressing the corresponding content more clearly and concisely, and do not constitute a substantial limitation on the sequence. When implementing the step, those skilled in the art may execute S102 first and then S101, etc., but these should all be within the scope of protection of this application.

[0098] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0099] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0100] The smart terminal may be implemented in various forms. For example, the smart terminal described in this application may include electronic devices such as mobile phones, tablet computers, laptop computers, PDAs (Personal Digital Assistants), PMPs (Portable Media Players), navigation devices, wearable devices, smart bracelets, pedometers, and fixed terminals such as digital TVs and desktop computers.

[0101] The subsequent description will be explained by taking an electronic device as an example, and those skilled in the art will understand that, in addition to components specifically used for mobile purposes, the construction according to the embodiments of the present application can also be applied to fixed type terminals.

[0102] See also Figure 1 , which is a schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present application, the electronic device 100 may include: RF (Radio Frequency) unit 101, WiFi module 102, audio output unit 103, A / V (audio / video) input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, processor 110, and power supply 111 and other components. Those skilled in the art can understand that Figure 1 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0103] Combine the following Figure 1 A detailed introduction to the various components of electronic equipment:

[0104] The radio frequency unit 101 can be used for receiving and sending signals during information transmission or communication. Specifically, after receiving the downlink information of the base station, it is sent to the processor 110 for processing; in addition, the uplink data is sent to the base station. Generally, the radio frequency unit 101 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc. In addition, the radio frequency unit 101 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communications can use any communication standard or protocol, including but not limited to GSM (Global System of Mobile communication), GPRS (General Packet Radio Service), CDMA2000 (Code Division Multiple Access 2000), WCDMA (Wideband Code Division Multiple Access), TD-SCDMA (Time Division-Synchronous Code Division Multiple Access), FDD-LTE (Frequency Division Duplexing-Long Term Evolution), TDD-LTE (Time Division Duplexing-Long Term Evolution) and 5G, etc.

[0105] WiFi is a short-range wireless transmission technology. Electronic devices can help users send and receive emails, browse web pages, and access streaming media through WiFi module 102. It provides users with wireless broadband Internet access. Figure 1 The WiFi module 102 is shown, but it is understandable that it is not an essential component of the electronic device and can be omitted as needed without changing the essence of the invention.

[0106] The audio output unit 103 can convert the audio data received by the RF unit 101 or the WiFi module 102 or stored in the memory 109 into an audio signal and output it as sound when the electronic device 100 is in a call signal receiving mode, a talk mode, a recording mode, a voice recognition mode, a broadcast receiving mode, etc. Moreover, the audio output unit 103 can also provide audio output related to a specific function performed by the electronic device 100 (for example, a call signal receiving sound, a message receiving sound, etc.). The audio output unit 103 may include a speaker, a buzzer, etc.

[0107] The A / V input unit 104 is used to receive audio or video signals. The A / V input unit 104 may include a GPU (Graphics Processing Unit) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The processed image frame can be displayed on the display unit 106. The image frame processed by the graphics processor 1041 can be stored in the memory 109 (or other storage medium) or sent via the radio frequency unit 101 or the WiFi module 102. The microphone 1042 can receive sound (audio data) via the microphone 1042 in a telephone call mode, a recording mode, a voice recognition mode, and other operating modes, and can process such sound into audio data. The processed audio (voice) data can be converted into a format output that can be sent to a mobile communication base station via the radio frequency unit 101 in the case of a telephone call mode. The microphone 1042 can implement various types of noise elimination (or suppression) algorithms to eliminate (or suppress) noise or interference generated in the process of receiving and sending audio signals.

[0108] The electronic device 100 also includes at least one sensor 105, such as a light sensor, a motion sensor, and other sensors. Optionally, the light sensor includes an ambient light sensor and a proximity sensor. Optionally, the ambient light sensor can adjust the brightness of the display panel 1061 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 1061 and / or the backlight when the electronic device 100 is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can also be configured on the mobile phone, such as fingerprint sensors, pressure sensors, iris sensors, molecular sensors, gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0109] The display unit 106 is used to display information input by the user or information provided to the user. The display unit 106 may include a display panel 1061, which may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc.

[0110] The user input unit 107 can be used to receive input digital or character information, and to generate key signal input related to user settings and function control of the electronic device. Optionally, the user input unit 107 may include a touch panel 1071 and other input devices 1072. The touch panel 1071, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1071 or near the touch panel 1071 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Optionally, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 110, and can receive and execute commands sent by the processor 110. In addition, the touch panel 1071 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1071, the user input unit 107 may further include other input devices 1072. Optionally, the other input devices 1072 may include, but are not limited to, one or more of a physical keyboard, a function key (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, etc., which are not specifically limited here.

[0111] Optionally, the touch panel 1071 may cover the display panel 1061. When the touch panel 1071 detects a touch operation on or near it, it transmits the information to the processor 110 to determine the type of touch event. Then, the processor 110 provides corresponding visual output on the display panel 1061 according to the type of touch event. Figure 1 In the figure, the touch panel 1071 and the display panel 1061 are two independent components to realize the input and output functions of the electronic device. However, in some embodiments, the touch panel 1071 and the display panel 1061 can be integrated to realize the input and output functions of the electronic device, which is not limited here.

[0112] The interface unit 108 serves as an interface through which at least one external device can be connected to the electronic device 100. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, etc. The interface unit 108 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or at least one element within the electronic device 100 or may be used to transmit data between the electronic device 100 and an external device.

[0113] The memory 109 can be used to store software programs and various data. The memory 109 can mainly include a program storage area and a data storage area. Optionally, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 109 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0114] The processor 110 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 109, and calling data stored in the memory 109, so as to monitor the electronic device as a whole. The processor 110 may include one or at least one processing unit; preferably, the processor 110 may integrate an application processor and a modem processor. Optionally, the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 110.

[0115] The electronic device 100 may also include a power supply 111 (such as a battery) for supplying power to various components. Preferably, the power supply 111 may be logically connected to the processor 110 via a power management system, thereby managing functions such as charging, discharging, and power consumption through the power management system.

[0116] although Figure 1 Not shown, the electronic device 100 may further include a Bluetooth module, etc., which will not be described in detail here.

[0117] To facilitate understanding of the embodiments of the present application, the communication network system on which the electronic device of the present application is based is described below.

[0118] See also Figure 2 , Figure 2 A communication network system architecture diagram is provided for an embodiment of the present application, wherein the communication network system is an LTE system of universal mobile communication technology, and the LTE system includes a UE (User Equipment) 201, an E-UTRAN (Evolved UMTS Terrestrial Radio Access Network) 202, an EPC (Evolved Packet Core) 203 and an operator's IP service 204, which are connected in sequence for communication.

[0119] Optionally, UE201 may be the above-mentioned terminal 100, which will not be described in detail here.

[0120] E-UTRAN 202 includes eNodeB 2021 and other eNodeBs 2022 , etc. Optionally, eNodeB 2021 may be connected to other eNodeBs 2022 via a backhaul (eg, an X2 interface), and eNodeB 2021 is connected to EPC 203 , and eNodeB 2021 may provide UE 201 with access to EPC 203 .

[0121] EPC203 may include MME (Mobility Management Entity) 2031, HSS (Home Subscriber Server) 2032, other MME 2033, SGW (Serving GateWay) 2034, PGW (PDN Gate Way) 2035 and PCRF (Policy and Charging Rules Function) 2036, etc. Optionally, MME 2031 is a control node that processes signaling between UE 201 and EPC 203, and provides bearer and connection management. HSS 2032 is used to provide some registers to manage functions such as home location register (not shown in the figure), and save some user-specific information such as service features and data rates. All user data can be sent through SGW2034, PGW2035 can provide IP address allocation and other functions for UE 201, PCRF2036 is the policy and charging control policy decision point for service data flow and IP bearer resources, which selects and provides available policy and charging control decisions for the policy and charging execution function unit (not shown in the figure).

[0122] The IP service 204 may include the Internet, an intranet, an IMS (IP Multimedia Subsystem) or other IP services.

[0123] Although the above introduction takes the LTE system as an example, those skilled in the art should know that the present application is not only applicable to the LTE system, but also to other wireless communication systems, such as GSM, CDMA2000, WCDMA, TD-SCDMA and future new network systems (such as 5G), etc., without limitation here.

[0124] Based on the above-mentioned electronic device hardware structure and communication network system, various embodiments of the present application are proposed.

[0125] First embodiment

[0126] See also Figure 3 , Figure 3 A flowchart of an image processing method provided in an embodiment of the present application. Figure 1 and Figure 2 Based on the embodiment shown, Figure 3 As shown, with the intelligent terminal as the execution subject, the method of this embodiment may include the following steps:

[0127] S101. Obtain a picture to be processed. Optionally, the picture to be processed includes a skin area.

[0128] Optionally, in the process of implementing functions such as face detection, gesture recognition, and posture recognition, the use of skin segmentation methods to pre-process images can effectively improve processing efficiency. In application scenarios such as face detection, gesture recognition, and posture recognition, functions such as face detection, gesture recognition, and posture recognition can be executed in a smart terminal. The smart terminal can send the original image or original video to the smart terminal after acquiring the original image or original video. The smart terminal can obtain the image to be processed from the original image or original video. Optionally, when the smart terminal acquires at least one original image sent by the smart terminal, the smart terminal can select an original image from the at least one original image as the image to be processed. Optionally, when the smart terminal acquires the original video sent by the smart terminal, the smart terminal can convert the original video into at least one original image according to a preset sampling frequency, and select one from the at least one original image as the image to be processed. Optionally, the image to be processed can be as follows Figure 4 (a). The image to be processed includes a skin area. The skin area may include a facial skin area and / or a body skin area.

[0129] Optionally, the intelligent terminal may also pre-process the selected original picture after selecting an original picture from at least one original picture to obtain a picture to be processed. The pre-processing may include scaling and cropping the original picture. Optionally, the pre-processing may also include using a brightness enhancement module to optimize the brightness of the picture to be processed. Since the pictures to be processed in the present application include skin area recognition for dark skin. People with dark skin are easily affected by the brightness of light, forming light spots, etc., which will cause discontinuity in the segmentation effect. Therefore, in order to reduce the impact of brightness on recognition, a brightness enhancement module may also be used to optimize the picture to be processed to improve the image processing efficiency and segmentation accuracy.

[0130] S102: Input the image to be processed into a recognition model for processing to obtain first information of the skin area.

[0131] Optionally, the intelligent terminal can input the image to be processed into a recognition model. The recognition model processes the image to be processed according to its preset parameters and algorithm steps, thereby obtaining first information of the skin area. Optionally, the first information includes facial skin contour and / or body skin contour. The first information can be as follows: Figure 4 Optionally, the recognition model includes an attention module based on context information.

[0132] Optionally, in some implementations, the recognition model is usually a skin region segmentation model based on a convolutional neural network. Among them, the neural network is mainly implemented using a pyramid model and an attention mechanism. However, these methods either fail to capture long-term dependencies, leading to feature isolation problems, or have excessive computational complexity, which easily leads to the loss of feature location information or the expansion of the error range. For this reason, the recognition model is a skin segmentation model based on a dual attention mechanism.

[0133] Optionally, in order to further improve the skin area segmentation effect of dark-skinned people, an encoder-decoder neural network structure is also used in the network model of the recognition model. In addition, in the encoder-decoder neural network structure, Mobilenetv3 is also introduced to achieve the effect of reducing parameters and calculation amount, thereby improving the calculation efficiency of the model, improving the recognition efficiency of the model, and meeting the real-time requirements.

[0134] Optionally, the recognition model also adopts a dual attention mechanism to solve the problem of feature isolation, so that the skin area continuity in the first information is stronger. Optionally, the recognition model also includes an edge supervision module. The use of the edge supervision module can enhance edge information guidance and edge label supervision. The use of the edge supervision module makes the detected skin area edge more refined. At the same time, the use of the edge supervision module can also effectively solve the problem that the continuous effect of dual attention may expand the error range and damage the segmentation result, and propose deep supervision to improve and accelerate network convergence.

[0135] S103: Process the image to be processed (such as cropping) according to the first information to obtain a target image.

[0136] Optionally, after acquiring the first information, the smart terminal may determine the target area according to the first information, thereby obtaining a target image by processing (such as cropping) the target area. For example, when the first information includes a facial skin contour, the smart terminal may determine the facial area according to the facial area contour. When the facial area is the target area, the smart terminal may obtain a target image by processing (such as cropping) the facial area. For another example, when the target area is a hand and the first information includes a biological skin contour, the smart terminal may determine the hand area according to the biological skin contour. The smart terminal may obtain the target image by processing (such as cropping) the hand area.

[0137] Optionally, the specific steps of processing (such as cropping) to obtain the target image may include:

[0138] Step 1: The intelligent terminal obtains the target area contour of the target object according to the first information. For example, the target area contour may be a facial contour, a hand contour, a body contour, etc. Optionally, in order to facilitate subsequent cropping, the target area contour may be a rectangle. Optionally, in order to improve feature extraction efficiency, the target area contour may be a polygon close to the target object.

[0139] Step 2: The intelligent terminal processes (e.g., crops) the image to be processed according to the target area contour to obtain a target image of the target object. For example, when the target area contour is a rectangle, the intelligent terminal can crop the image according to the rectangular frame to obtain the target image containing the target object. For another example, when the target area contour is a polygon, the intelligent terminal can crop the polygon according to the polygon to obtain the minimum circumscribed rectangle of the polygon. In the minimum circumscribed rectangle of the polygon, the area outside the polygon can be covered with a fixed color such as 0 or 255.

[0140] In the image processing method provided in the embodiment of the present application, the intelligent terminal can obtain a picture to be processed. The picture to be processed includes a skin area. The skin area may include a facial skin area and / or a body skin area. The intelligent terminal can input the picture to be processed into a recognition model. The recognition model will process the picture to be processed according to its preset parameters and algorithm steps, thereby obtaining first information of the skin area. The intelligent terminal can determine the target area based on the first information, and thus obtain the target picture by cropping according to the target area. The present application, by using the recognition model, realizes the segmentation of dark skin, improves the effect of skin segmentation, and improves the processing efficiency of the recognition model.

[0141] Second embodiment

[0142] See also Figure 5 , Figure 5 A schematic diagram of an application scenario of an image processing method provided in an embodiment of the present application. Figures 1 to 4Based on the embodiment shown, Figure 5 As shown, with the smart terminal as the execution subject, in this embodiment, the smart terminal can complete the training of the recognition model through the following steps:

[0143] S201. Obtain a training data set.

[0144] Optionally, during the training phase of the network, the intelligent terminal can add a large amount of dark skin training data on the basis of the existing training data set to enhance the recognition effect of the recognition model for dark skin data. Optionally, in order to increase the number of samples of the training data, the original image can also be randomly horizontally flipped, scaled, cropped, brightness enhanced, etc. to obtain a training picture. Optionally, in order to increase the recognition accuracy and avoid the influence of brightness on dark skin, the intelligent terminal can also pre-process the training picture using brightness enhancement and other methods. Optionally, the training pictures in the training data set are divided into at least one data block. One data block is a batch. The training pictures can be sent to the network in the form of a batch for training. Optionally, the number of training pictures included in a batch can be set as needed. For example, when batch_size=8, a batch includes 8 training pictures.

[0145] For example, the original image can be Figure 6 (a). The intelligent terminal can obtain a batch of training image inputs by randomly flipping, scaling, cropping, and enhancing the brightness of the original image. The training image can be Figure 6 (b) or Figure 6 Optionally, the dimension of the training images of a batch can be (8, 3, 513, 513).

[0146] Optionally, existing skin detection datasets generally have the problem of uneven skin color distribution. There are more light skin pictures and fewer dark skin pictures in the existing skin detection datasets. The recognition model trained using the skin detection dataset as training data is prone to the problem of poor robustness of the recognition model due to the domain differences in the dataset itself. In order to solve this problem and make full use of the existing public skin detection dataset, the present application has produced a semantically richer dark skin dataset as a training dataset. The present application has additionally collected a large number of dark skin portrait pictures and annotated them to form a dark skin image skin area segmentation dataset. In the process of annotating dark skin portrait pictures, in order to make the most of the semantic information in the dark skin image skin area dataset, provide semantic guidance for the skin segmentation task or promote potential downstream tasks, the present application has formulated more fine-grained annotation rules. The present application has made a fine-grained division of the annotation rules for the training pictures in the dataset. This includes a more detailed division of the skin area into facial skin area and body skin area. In addition, it also includes dividing the non-skin area into facial feature non-skin area (including eyes, nostrils, lips, etc.), hair non-skin area (including hair, beard, etc.) and other non-skin area according to its facial features, hair and other characteristics.

[0147] Optionally, the present application realizes joint training of multiple data sets. In the joint training process, training is first performed on a public skin segmentation data set to obtain a pre-trained model. Afterwards, the present application uses the pre-trained model and uses the dark skin area segmentation data to perform fine-tuning training again, changing the prediction category from binary classification to multi-classification. Based on this training strategy, the present application can complement the knowledge obtained from different data sets, solve the domain differences in the data sets, and improve the robustness of the model and the accuracy of dark skin area segmentation.

[0148] S202: Input the training image into the recognition model for processing to obtain at least one of an edge loss value, a segmentation loss value, and a segmentation auxiliary loss value.

[0149] Optionally, after the intelligent terminal inputs the training picture into the recognition model, the intelligent terminal can use Figure 7 The recognition model shown in the figure processes the training image. Figure 7 As shown, the recognition model may include five modules: feature extraction module (feature extraction backbone network, encoder), edge extraction module (edge ​​extraction branch), attention module (high-level feature fusion module), output module (decoder) and deep supervision module. The specific processing process of the training image in the recognition model may include the following steps:

[0150] Step 1: The intelligent terminal inputs the training image into the feature extraction module of the recognition model to obtain at least one primary feature map.

[0151] Optionally, after the intelligent terminal obtains the image to be processed, it can input the image to be processed into a feature extraction module for feature extraction. The specific steps performed in the encoder can be as follows: Figure 8 As shown. The intelligent terminal can input the image to be processed into the encoder. The image to be processed can be represented by input. The encoder can perform data preprocessing on the image to be processed. The encoder can input the preprocessed data into the feature extraction backbone network to complete the extraction of primary features and obtain at least one primary feature map. The at least one primary feature map can be represented as c0, c1, c2, c3, c4 respectively. Optionally, the feature extraction backbone network can be a MobileNetv3_small_075 network pre-trained with ImageNet. Optionally, the dimensions of the primary feature maps c0, c1, c2, c3, c4 can be (8, 16, 257, 257), (8, 16, 129, 129), (8, 24, 65, 65), (8, 40, 33, 33), (8, 432, 17, 17) respectively.

[0152] Step 2: The intelligent terminal inputs at least one of the at least one primary feature map into the attention module of the recognition model to obtain a high-level feature map. Optionally, the high-level feature map is used to determine or generate the first information of the skin area.

[0153] Optionally, the intelligent terminal can input the primary feature map obtained by the feature extraction module into the attention module to obtain a high-level feature map. The calculation process of the attention module can be as follows: Fig. 9 As shown. The attention module can perform dimensionality reduction processing on the primary feature map after obtaining the primary feature map. The dimensionality reduction processing can be completed by the feature dimensionality reduction module of the attention module. The attention module specifically includes two parts: a location-aware attention module (LAA) and a channel attention module (CAM). On the basis of the two attention modules, the present application also adjusts the number of channels so that the data in the present application can better meet the real-time requirements and match the channels of the encoder. The intelligent terminal can input the reduced primary feature map into the location-aware attention module and the channel attention module respectively to obtain a first output feature map and a second output feature map. The attention module can calculate a high-level feature map based on the first output feature map and the second output feature map. Optionally, the process can be specifically divided into the following steps:

[0154] Step 21, the intelligent terminal inputs the primary feature map into the feature dimension reduction module of the attention module for dimension reduction processing to obtain a first dimension reduction feature map and a second dimension reduction feature map. Optionally, the feature dimension reduction module can use two 3x3 convolution kernels to perform feature dimension reduction on the primary feature map c4 to obtain the first dimension reduction feature map m1 and the second dimension reduction feature map m2 after dimension reduction. Optionally, the dimensions of the first dimension reduction feature map m1 and the second dimension reduction feature map m2 can be (8, 108, 17, 17).

[0155] Step 22. The intelligent terminal inputs the first dimensionality reduction feature map into the position-aware attention module of the attention module for up and down position information perception processing to obtain a first output feature map. Optionally, the position-aware attention module LAA proposes a relationship model based on pixels and context areas to address the problem of loss of position information caused by feature map discreteness in existing algorithms. The position-aware attention module LAA enhances the continuity of feature maps from the object level. The position-aware attention module LAA can specifically include four parts: a feature transformation submodule, a multi-scale hybrid distance relationship submodule (Hybrid Distance Relation, HDR), a feature interaction submodule, and a feature activation submodule. The model structure of the position-aware attention module LAA can be as follows Fig.10 Optionally, the calculation process of the LAA module specifically includes the following steps:

[0156] Step 221, the intelligent terminal uses the feature transformation submodule in the location-aware attention module to transform the first reduced-dimensionality feature map to obtain a third reduced-dimensionality feature map. Optionally, there may be an input transformation parameter θ in the feature transformation submodule. The feature transformation submodule may use a 1x1 convolution kernel to further reduce the dimension of the first reduced-dimensionality feature map m1 to obtain a third reduced-dimensionality feature map m3. Optionally, the dimension of the third reduced-dimensionality feature map m3 may be (8, 27, 17, 17).

[0157] Step 222: The intelligent terminal uses the multi-scale hybrid distance relationship submodule in the location-aware attention module to calculate the distance features of each scale of each feature point in the third dimensionality reduction feature map. Optionally, the distance feature calculation process may specifically include the following steps:

[0158] Step 2221: The intelligent terminal performs pooling operation on the third dimension-reduced feature map to obtain the pooling features of each scale for each feature point in the third dimension-reduced feature map. Optionally, the multi-scale hybrid distance relationship sub-module HDR can perform pooling operation on the third dimension-reduced feature map m3 after selecting n maximum pooling operators and average pooling operators, and obtain the pooling features of each scale for each feature point in the third dimension-reduced feature map. Optionally, the sizes of the n pooling operators can be the maximum pooling operator and average pooling operator with sizes of [3, 5, 7, 9, 11, 15, 19] respectively. The pooling feature can be represented using the aggregation function A k (x i,j ) for representation. Where x i,j is the center point aggregated by the aggregation function, k is the range of the context area from the center point, and 0 ≤ k < n. The subscripts i, j in x i,j are used to represent the coordinates of the center point in the third dimension-reduced feature map m3. Its dimension can be (8, 27, 17, 17).

[0159] Step 2222: The intelligent terminal calculates the distance feature between each feature point in the third dimension-reduced feature map and each corresponding pooling feature of the feature point. Optionally, the intelligent terminal can calculate the third dimension-reduced feature map m3 and the distance feature A k (x i,j ) according to the following formula to obtain the distance feature R k (i, j) for measuring the distance between two high-dimensional feature vectors. Its dimension can be (8, 1, 17, 17). The formula can be:

[0160] R k (i, j) = F(x i,j , A k (x i,j ))

[0161] Where F(x i,j , A k (x i,j )) is the distance metric function, and the formula of the distance metric function can be specifically expressed as:

[0162]

[0163] Step 223: The intelligent terminal uses the feature interaction sub-module in the position-aware attention module to fuse the distance features of each scale for each feature point in the third dimension-reduced feature map to obtain the first attention feature map. Optionally, the intelligent terminal can input the distance feature into the feature interaction sub-module of the position-aware attention module LAA. The intelligent terminal can fuse the obtained n R k (i, j) according to the following formula based on the different relationships of each pixel point. The formula is:

[0164]

[0165] The fusion function ω(·) can be simplified to a 3x3 convolution operator. The smart terminal can calculate the first attention feature map attention through this formula. The first attention feature map is a weight matrix. Where i, j represents the coordinates in the first attention feature map attention. Each u i,j is a point in the first attention feature map attention. Optionally, the dimension of the first attention feature map attention can be (8, 1, 17, 17).

[0166] Step 224: The intelligent terminal determines a first output feature map according to the first attention feature map and the first dimensionality reduction feature map. Optionally, the process may include the use of an activation function and feature transformation again, and the specific process includes:

[0167] Step 2241, the intelligent terminal uses the feature activation module in the location-aware attention module to determine the second attention feature map according to the first dimensionality reduction feature map and the first attention feature map. Optionally, the intelligent terminal can use the feature activation submodule of the location-aware attention module LAA to activate the weight matrix attention and the first dimensionality reduction feature map m1 according to the following formula to obtain the second attention feature map attention1. Optionally, the dimension of the second attention feature map attention1 can be (8,108,17,17). The formula is:

[0168] v i,j =u i,j × i,j

[0169] Among them, i, j represent the coordinates in the second attention feature map attention1. Each v i,j is a point in the second attention feature map attention1.

[0170] Step 2242, the intelligent terminal uses the feature transformation submodule in the position-aware attention module to transform the second attention feature map to obtain a third attention feature map. Optionally, the feature transformation submodule also includes an output transformation parameter φ. The intelligent terminal can use the output transformation parameter to transform the second attention feature map attention1. The intelligent terminal can also use a 1x1 convolution kernel to perform feature transformation on the second attention feature map attention1 to obtain a third attention feature map attention2. Optionally, the dimension of the third attention feature map attention2 can be (8,108,17,17).

[0171] Step 2243, the intelligent terminal determines the first output feature map based on the third attention feature map and the first dimensionality reduction feature map. Optionally, the intelligent terminal can also use the following formula to calculate the third attention feature map attention2 and the first dimensionality reduction feature map to obtain the first output feature map output1. The first output feature map output1 is the final output of the LAA module. Optionally, the dimension of the first output feature map output1 can be (8,108,17,17). The formula is specifically:

[0172] y i,j =x i,j +φ(v i,j )

[0173] Step 23: The intelligent terminal inputs the second dimensionality reduction feature map into the channel attention module of the attention module for upper and lower channel perception processing to obtain a second output feature map. Optionally, the channel attention module CAM is used to selectively emphasize interdependent channel mappings by integrating the associated features between all channel mappings. The specific calculation process can be as follows: Fig.11 Optionally, the calculation process may specifically include the following steps:

[0174] Step 231: The intelligent terminal obtains a fourth attention feature map based on the reconstructed feature map and the transposed feature map of the second dimensionality reduction feature map. Optionally, in the channel attention module, the intelligent terminal may perform dimensionality transformation and transposition on the second dimensionality reduction feature map m2 to obtain a reconstructed feature map A and a transposed feature map A. T Optionally, the reconstructed feature map A and the transposed feature map A T The dimensions of can be (8,108,17×17) and (8,17×17,108) respectively. The intelligent terminal can use matrix multiplication and softmax function to calculate A and A. T Calculate and get the characteristics Figure X Optionally, the feature Figure X The dimension can be (8,108,108). This feature Figure X Each feature point x in ji The calculation formula can be shown as follows:

[0175]

[0176] Smart terminals can use matrix multiplication to calculate X Tand A, and calculate to obtain the fourth attention feature map attention3. Optionally, the dimension of the fourth attention feature map attention3 can be (8, 108, 17, 17). Optionally, the calculation process of the fourth attention feature map can specifically include the following steps:

[0177] Step 2311: The intelligent terminal calculates the product of the reconstructed feature map and the transposed feature map of the second dimensionality reduction feature map as a product feature map.

[0178] Step 2312: The intelligent terminal uses a normalization function to normalize the product feature map to obtain a normalized feature map.

[0179] Step 2313: The intelligent terminal calculates the product of the normalized feature map and the reconstructed feature map of the second dimensionality reduction feature map as a fourth attention feature map.

[0180] Step 232, the intelligent terminal determines the second output feature map based on the reconstructed feature map of the fourth attention feature map and the second reduced dimension feature map. Optionally, the intelligent terminal can add the fourth attention feature map attention3 after dimension transformation and the second reduced dimension feature map m2 to obtain the second output feature map output2. Optionally, the dimension of the second output feature map output2 can be (8, 108, 17, 17). The calculation formula of the second output feature map output2 can be expressed as:

[0181]

[0182] Among them, E j is the row vector in the second output feature map output2.

[0183] Step 24, according to the first output feature map and the second output feature map, a high-level feature map is obtained by fusing. Optionally, the intelligent terminal can also combine the first output feature map output1 and the second output feature map output2 and perform feature dimensionality upgrade to obtain a high-level feature map. The combination process specifically includes adding the first output feature map output1 and the second output feature map output2. Feature dimensionality upgrade includes using a Dropout operation and using a 1x1 convolution kernel to upgrade the sum of the first output feature map output1 and the second output feature map output1 to obtain the high-level feature map output that is finally output by the dual attention module. Optionally, the dimension of the high-level feature map output can be (8,40,17,17).

[0184] Step 3: The intelligent terminal inputs at least one primary feature map into the edge extraction module of the recognition model to obtain an edge feature map.

[0185] Optionally, the intelligent terminal can input the primary feature map into the edge extraction module to obtain the edge feature map. The calculation process of the edge extraction module can be as follows: Fig.12 As shown. After acquiring at least one primary feature map extracted by the feature extraction module, the intelligent terminal can complete edge feature extraction and edge prediction based on these primary feature maps to obtain edge feature maps and edge prediction maps. In the calculation process of the edge extraction module, in order to capture more edge details and other information, the edge extraction module can process the primary feature maps c1, c2, and c3. Optionally, the dimensions of the three primary feature maps can be (8, 16, 129, 129), (8, 24, 65, 65), and (8, 40, 33, 33), respectively. The data processing process of the edge extraction module can be as follows Fig.13 Optionally, the specific steps may include:

[0186] Step 31, the intelligent terminal performs convolution calculation on at least one primary feature map to obtain at least one intermediate feature map. Optionally, the edge extraction module first uses three different 3x3 convolutions to perform dimensionality reduction processing on the three primary feature maps c1, c2, and c3 respectively, obtains three unified channel numbers and obtains intermediate feature maps edge1_fea, edge2_fea, and edge3_fea. Optionally, the dimensions of the three intermediate feature maps after convolution calculation can be (8, 16, 129, 129), (8, 16, 65, 65), and (8, 16, 33, 33) respectively.

[0187] Step 32, the intelligent terminal performs convolution calculation on at least one intermediate feature map to obtain at least one edge sub-feature map. Optionally, the edge extraction module uses another three 3x3 convolutions to perform edge prediction on the intermediate feature maps edge1_fea, edge2_fea, and edge3_fea. The edge prediction process also realizes feature extraction and dimensionality reduction of the intermediate feature map. The edge extraction module can obtain three edge sub-feature maps edge1, edge2, and edge3 with the same number of channels through the edge prediction. Optionally, the dimensions of the three edge sub-feature maps after the convolution calculation can be (8, 2, 129, 129), (8, 2, 65, 65), and (8, 2, 33, 33), respectively.

[0188] Step 33, the intelligent terminal fuses at least one edge sub-feature map to obtain an edge feature map. Optionally, since the three primary feature maps are obtained at different stages of the feature extraction module, the dimensions of the three primary feature maps are not the same. In the edge extraction module, in order to fuse features of different scales, the intelligent terminal can also use bilinear interpolation to process the intermediate feature maps edge2_fea and edge3_fea, so that the dimensions of the intermediate feature maps edge2_fea and edge3_fea are the same as the dimensions of the intermediate feature map edge1_fea. Optionally, the dimensions after bilinear interpolation can be (8, 16, 129, 129). In order to perform feature fusion, the intelligent terminal superimposes edge1_fea, edge2_fea, and edge3_fea according to the channel to obtain the feature map edge_fea. Optionally, the dimension of the feature map edge_fea can be (8, 48, 129, 129). Then, the intelligent terminal can use convolution to reduce the dimension of the feature map edge_fea to obtain the final result edge feature map edge_fea of ​​the edge extraction module. The edge feature map edge_fea is used to provide more edge detail information for the decoder module. Optionally, the dimension of the edge feature map edge_fea can be (8, 26, 129, 129).

[0189] Optionally, the intelligent terminal can also perform bilinear interpolation on the edge sub-feature graphs edge2 and edge3 so that the dimensions of the two edge sub-feature graphs are the same as the dimension of edge1. Optionally, the dimension after bilinear interpolation can be (8, 2, 129, 129). The intelligent terminal can also superimpose edge1, edge2, and edge3 according to the channel to obtain the feature graph edge. Optionally, the dimension of the superimposed feature graph edge can be (8, 6, 129, 129). Then, the intelligent terminal can use convolution to reduce the dimension of the feature graph edge to obtain the final result edge prediction graph edge of the edge extraction module. The edge prediction graph edge is used to perform the final edge loss calculation, thereby supervising the network to learn edge information. Optionally, the dimension of the edge prediction graph edge can be (8, 1, 129, 129).

[0190] Step 4: The intelligent terminal inputs the high-level feature map, the edge feature map and at least one primary feature map into an output module of the recognition model to obtain first information.

[0191] Optionally, the intelligent terminal may also decode the high-level feature map by using an output module to obtain the first information. The execution steps of the output module (decoder) may be as follows: Fig.14After obtaining the high-level feature map, the primary feature map and the edge feature map, the intelligent terminal can calculate the first information based on these data. The first information may include a prediction map and an intermediate prediction map. Optionally, the first information may include a final prediction map and an intermediate prediction map. The structure diagram of the output module can be as shown in Fig.15 As shown. The feature extraction module and the attention module input the primary feature map and the high-level sign map into the decoder of the output module. The output module may include at least one decoder. Optionally, Fig.15 The specific structure of each decoder in can be as follows Fig.16 As shown. Fig.15 The specific decoding process of the five decoders in may include:

[0192] Decoder 1 is used to concatenate the high-level feature map output with the primary feature map c3 to obtain the first sampling data upsample1. Optionally, the dimension of the first sampling data upsample1 can be (8, 80, 33, 33). Thereafter, the decoder 1 can reduce the dimension of the first sampling data upsample1 through a 1x1 convolution, and perform feature extraction on the reduced-dimensional first sampling data upsample1 through a 3x3 convolution kernel to obtain a first decoding feature map decoder1. Optionally, the dimension of the first decoding feature map decoder1 can be (8, 40, 33, 33). The processing of the first sampling data upsample1 by the 3x3 convolution kernel does not change the dimension of the first sampling data upsample1.

[0193] Decoder 2 is used to concatenate the first decoding feature map decoder1 with the primary feature map c2 to obtain the second sampling data upsample2. Optionally, the dimension of the second sampling data upsample2 can be (8, 64, 65, 65). Thereafter, the decoder 2 can reduce the dimension of the second sampling data upsample2 through a 1x1 convolution, and perform feature extraction on the reduced-dimensional second sampling data upsample2 through a 3x3 convolution kernel to obtain a second decoding feature map decoder2. Optionally, the dimension of the second decoding feature map decoder2 can be (8, 24, 65, 65). The processing of the second sampling data upsample2 by the 3x3 convolution kernel does not change the dimension of the second sampling data upsample2.

[0194] In order to fuse the edge information extracted by the edge extraction module, the decoder 3 may superimpose the primary feature map c1 and the edge feature map edge_fea obtained by the edge extraction module by channel to obtain the feature map edge_c1. Optionally, the dimension of the feature map edge_c1 may be (8, 48, 129, 129). Then, the decoder 3 may use a convolution layer to reduce the dimension of the feature map edge_c1 to obtain the reduced-dimensional feature map edge_c1. Optionally, the dimension of the reduced-dimensional feature map edge_c1 may be (8, 24, 129, 129).

[0195] Decoder 3 may upsample the second decoding feature map decoder2. Decoder 3 may concatenate the upsampled second decoding feature map decoder2 with the dimension-reduced feature map edge_c1 to obtain third sampling data upsample3. Optionally, the dimension of the third sampling data upsample3 may be (8, 48, 129, 129). The decoder 3 may reduce the dimension of the third sampling data upsample3 by 1x1 convolution. The decoder 3 may also perform feature extraction on the dimension-reduced third sampling data upsample3 by a 3x3 convolution kernel, and keep the dimension unchanged to obtain a third decoding feature map decoder3. Optionally, the dimension of the third decoding feature map decoder3 may be (8, 16, 129, 129).

[0196] Decoder 4 is used to upsample the third decoding feature map decoder3. Decoder 4 can concatenate the upsampled third decoding feature map decoder3 with the primary feature map c0 to obtain fourth sampling data upsample4. Optionally, the dimension of the fourth sampling data upsample4 can be (8, 32, 257, 257). Decoder 4 can reduce the dimension of the fourth sampling data upsample4 through 1x1 convolution. The decoder 4 can also perform feature extraction on the fourth sampling data upsample4 through a 3x3 convolution kernel, and keep the dimension unchanged to obtain a fourth decoding feature map decoder4. Optionally, the dimension of the fourth decoding feature map decoder4 can be (8, 16, 259, 259).

[0197] Decoder 5 is used to upsample the fourth decoding feature map decoder4. Decoder 5 can splice the upsampled fourth decoding feature map decoder4 with the image to be processed input to obtain fifth sampling data upsample5. Optionally, the dimension of the fifth sampling data upsample5 can be (8,19,513,513). Decoder 5 can reduce the dimension of the fifth sampling data upsample5 through 1x1 convolution. Decoder 5 can also perform feature extraction on the fifth sampling data upsample5 through a 3x3 convolution kernel, and keep the dimension unchanged to obtain the first information seg. The first information seg can be a skin edge contour. Optionally, the dimension of the first information seg can be (8,4,513,513).

[0198] Optionally, the recognition model also includes a deep supervision module. The execution steps of the deep supervision module can be as follows: Fig.17 As shown. Optionally, the first information may include a final prediction map and an intermediate prediction map. The deep supervision module may calculate the segmentation loss value using the true segmentation label and the first information. Optionally, as Figure 6 (b) and Figure 6 The labels corresponding to the training images shown in (c) can be as follows Fig.18 (a) and Fig.18 As shown in (b), the intelligent terminal can obtain the true segmentation label and the true edge label based on the label.

[0199] Optionally, the smart terminal can obtain the edge prediction graph edge through the previous edge extraction module. In order to match the size of the original image, the smart terminal can also use bilinear interpolation to upsample the edge prediction graph edge so that its size is the same as the training image input. The smart terminal can use the sigmoid function to activate the edge prediction graph edge. The value range of each point in the activated edge prediction graph edge will be constrained to the range of (0,1). Each point in the activated edge prediction graph edge will be used as the probability value of the edge. The formula of the sigmoid function is as follows:

[0200]

[0201] The smart terminal can obtain the true segmentation label of the edge of the input image from the training data set. The smart terminal can use the binary cross entropy loss function to supervise the predicted first information and obtain the edge loss value loss edge . The formula of the binary cross entropy loss function is as follows:

[0202] loss edge =-w n [y n·logx n +(1-y n )·log(1-x n )]

[0203] Among them, w n is the weight coefficient, y n is the label value in the true segmentation label, x n is the predicted value in the first information, wherein n represents the nth pixel of the training image.

[0204] Optionally, the deep supervision module may also include auxiliary loss supervision. In order to speed up model convergence and alleviate the problem of incomplete segmentation in the dual attention module, the intelligent terminal can use at least one decoded feature map generated by the output module (decoder) to predict the segmentation result. The intelligent terminal can use the convolution layer as an auxiliary segmentation head to predict the intermediate results of the decoder respectively to obtain at least one segmentation result. For example, the intelligent terminal can predict the second decoding feature map decoder2, the third decoding feature map decoder3 and the fourth decoding feature map decoder4 to obtain the first segmentation result out_aux1, the second segmentation result out_aux2 and the third segmentation result out_aux3. Optionally, the dimensions of the three segmentation results can be (8, 4, 65, 65), (8, 4, 129, 129) and (8, 4, 257, 257), respectively. The intelligent terminal can use the cross entropy loss function to calculate the loss of the above at least one segmentation result to obtain at least one segmentation auxiliary loss value. For example, the intelligent terminal may calculate a first segmentation auxiliary loss value loss_aux1, a second segmentation auxiliary loss value loss_aux2, and a third segmentation auxiliary loss value loss_aux3.

[0205] Optionally, the intelligent terminal can also apply cross entropy loss to calculate the final input first information seg to obtain the final segmentation loss value loss out The calculation process of the segmentation loss value specifically includes the following steps:

[0206] Step 1: The intelligent terminal inputs the training image into the feature extraction module of the recognition model for prediction to obtain the first information.

[0207] Step 2: The intelligent terminal obtains a segmentation loss value according to the first information and the edge label of the training image, wherein the edge label includes a true segmentation label.

[0208] S203. Use at least one of the edge loss value, the segmentation loss value, and the segmentation auxiliary loss value to adjust the parameters of the recognition model to obtain a trained recognition model.

[0209] Optionally, the intelligent terminal can calculate the edge loss value based on the edge loss value. edge , segmentation loss value loss out And the segmentation auxiliary loss values ​​loss_aux1, loss_aux2, loss_aux3. The final model loss is calculated. The calculation formula for the model loss can be:

[0210] L loss =loss edge +loss_aux1+loss_aux2+loss_aux3+loss out

[0211] Optionally, the intelligent terminal may optimize the parameters of the recognition model according to the model loss through a gradient descent method and a back propagation algorithm.

[0212] In the image processing method provided by the embodiment of the present application, during the training phase of the network, the intelligent terminal can obtain a training data set. After the intelligent terminal inputs the training picture into the recognition model, the recognition model can calculate the edge prediction map, the first information and the decoding feature map of each training picture. The intelligent terminal can calculate the edge loss value based on the edge prediction map and the true edge label. The intelligent terminal can calculate the segmentation loss value based on the first information and the true segmentation label. The intelligent terminal can calculate the segmentation auxiliary loss value based on at least one decoding feature map. The intelligent terminal can calculate the model loss using at least one of the edge loss value, the segmentation loss value and the segmentation auxiliary loss value. The intelligent terminal can use the model loss to optimize the parameters of the recognition model through the gradient descent method and the back propagation algorithm. In the present application, by training the recognition model, the segmentation of dark skin is realized, the effect of skin segmentation is improved, and the processing efficiency of the recognition model is improved. In addition, in the present application, by adding an edge extraction module and a deep supervision module to the recognition model, the convergence speed and context relevance of the attention module are improved, and the training efficiency and model accuracy are improved. In addition, this application has also made a training data set for dark skin with richer semantics through improvements at the data set level, combined multiple data sets to realize the training of the recognition model, solved the problem of domain differences in the skin color domain in the data set, and improved the segmentation ability of dark skin areas. At the same time, this application has also introduced Mobi lenetv3 to meet the real-time requirements of the model through improvements at the model level, adopted a dual attention mechanism to obtain skin areas with strong continuity, proposed an edge supervision module to make the detected skin area edges finer, and proposed deep-level supervision to solve the defects of the dual attention module and accelerate network convergence. Therefore, this application, through the above method, ultimately makes the model significantly improved in real-time and accuracy, the skin segmentation results are more accurate, the regional continuity is stronger, the skin edges are finer, the model effect is more robust, and the prediction speed is faster and the model parameters are smaller.

[0213] An embodiment of the present application further provides an intelligent terminal, which includes a memory and a processor. An image processing program is stored in the memory. When the image processing program is executed by the processor, the steps of the image processing method in any of the above embodiments are implemented.

[0214] An embodiment of the present application further provides a computer-readable storage medium, on which an image processing program is stored. When the image processing program is executed by a processor, the steps of the image processing method in any of the above embodiments are implemented.

[0215] In the embodiments of the smart terminal and computer-readable storage medium provided in the present application, all technical features of any of the above-mentioned image processing method embodiments may be included, and the expanded and explained contents of the specification are basically the same as those of the embodiments of the above-mentioned methods, and will not be repeated here.

[0216] The embodiment of the present application further provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer executes the methods in the above various possible implementation modes.

[0217] An embodiment of the present application also provides a chip, including a memory and a processor, the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that a device equipped with the chip executes the methods in various possible implementation modes as described above.

[0218] It is understood that the above scenarios are only examples and do not constitute a limitation on the application scenarios of the technical solutions provided in the embodiments of the present application. The technical solutions of the present application can also be applied to other scenarios. For example, it is known to those skilled in the art that with the evolution of the system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0219] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0220] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0221] The units in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0222] In the present application, the same or similar terminology concepts, technical solutions and / or application scenario descriptions are generally described in detail only the first time they appear. When they appear again later, they are generally not repeated for the sake of brevity. When understanding the technical solutions and other contents of the present application, for the same or similar terminology concepts, technical solutions and / or application scenario descriptions that are not described in detail later, reference can be made to the previous related detailed descriptions.

[0223] In the present application, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0224] The various technical features of the technical solution of the present application can be arbitrarily combined. In order to make the description concise, not all possible combinations of the various technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present application.

[0225] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, an intelligent terminal, a controlled terminal, or a network device, etc.) to execute the method of each embodiment of the present application.

[0226] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes at least one computer instruction. When the computer program instruction is loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instruction can be transmitted from a website site, a computer, an intelligent terminal or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, intelligent terminal or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as an intelligent terminal, a data center, etc. that contains at least one available medium integrated. The available medium can be a magnetic medium, (e.g., a floppy disk, a storage disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state storage disk Solid State Disk (SSD)), etc.

[0227] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image processing method, characterized in that: The following steps are involved: S1: Acquire a picture to be processed, wherein the picture to be processed includes a skin area; S2: Inputting the image to be processed into a recognition model for processing to obtain first information of the skin area; S3: Process the image to be processed according to the first information to obtain a target image; The step S2 comprises: Inputting the image to be processed into a feature extraction module of the recognition model to obtain at least one primary feature map; Inputting the primary feature map into the feature dimension reduction module of the attention module for dimension reduction processing to obtain a first dimension reduction feature map and a second dimension reduction feature map; Inputting the first dimension reduction feature map into the position perception attention module of the attention module to perform up and down position information perception processing to obtain a first output feature map; Obtaining a fourth attention feature map according to the reconstructed feature map and the transposed feature map of the second dimensionality reduction feature map; Determining a second output feature map according to the reconstructed feature map of the fourth attention feature map and the second dimensionality reduction feature map; According to the first output feature map and the second output feature map, a high-level feature map is obtained by fusing; First information of the skin area is determined or generated based on the high-level feature map.

2. The method according to claim 1, characterized in that The S3 step comprises: Determine a target area contour of the target object according to the first information; According to the target area contour, the image to be processed is processed to obtain a target image of the target object.

3. The method according to claim 1, characterized in that The inputting the first dimension reduction feature map into the position perception attention module of the attention module to perform up and down position information perception processing to obtain a first output feature map includes: Using the feature transformation submodule in the position-aware attention module, transforming the first reduced-dimensionality feature map to obtain a third reduced-dimensionality feature map; Using the multi-scale mixed distance relationship submodule in the position-aware attention module, calculating the distance features of each scale of each feature point in the third dimensionality reduction feature map; Using the feature interaction submodule in the position-aware attention module, the distance features of each scale of each feature point in the third dimensionality reduction feature map are fused to obtain a first attention feature map; Determine the first output feature map based on the first attention feature map and the first dimensionality reduction feature map.

4. The method according to claim 1, characterized in that: The step of obtaining a fourth attention feature map according to the reconstructed feature map and the transposed feature map of the second dimensionality reduction feature map comprises: According to the reconstructed feature map and the transposed feature map of the second dimensionality reduction feature map, the product thereof is calculated as a product feature map; Normalizing the product feature map using a normalization function to obtain a normalized feature map; According to the reconstructed feature map of the normalized feature map and the second dimensionality reduction feature map, the product thereof is calculated as a fourth attention feature map.

5. The method according to any one of claims 1 to 4, characterized in that Also includes: Inputting the training image into the recognition model for processing to obtain at least one of an edge loss value, a segmentation loss value, and an auxiliary loss value; Use at least one of the segmentation loss value, the edge loss value, and the auxiliary loss value to adjust the parameters of the recognition model to obtain a trained recognition model.

6. An intelligent terminal, characterized in that: The intelligent terminal comprises: a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the image processing method according to any one of claims 1 to 5 are implemented.

7. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the image processing method according to any one of claims 1 to 5 are implemented.

Images