Image processing method, device, equipment and storage medium

Through the target model, feature extraction and segmentation prediction of face images is solved, and the problem of difficulty in positioning different types of skin symptoms in face images is achieved, and the precise positioning and recognition efficiency of target object position information is improved.

CN113378696BInactive Publication Date: 2024-11-05BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110636583.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-08
Publication Date
2024-11-05
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The prior art is difficult to accurately locate different types of skin symptoms in facial images, and it is impossible to accurately identify and locate multiple types of skin symptoms overlapping areas.

Method used

By obtaining the face area image from the target image, input the target model to obtain the segmentation result of the target area, determine the sub-image, and determine the position information of the target object in the sub-image, extract the feature map using the first network of the target model, The second network performs segmentation prediction of target areas to avoid the difficulty of dividing multiple types of overlapping areas.

Benefits of technology

The precise positioning of target object position information in different types of target areas in the face image is achieved, the problem of recognition ambiguity in multiple types of overlapping areas is avoided, and the recognition accuracy and speed are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113378696B_ABST
    Figure CN113378696B_ABST
Patent Text Reader

Abstract

The present disclosure provides an image processing method, apparatus, device and storage medium, which relate to the field of artificial intelligence technology, specifically computer vision and deep learning technology, and can be applied to image processing scenarios. The specific implementation scheme is: obtaining a face area image from a target image; inputting the face area image into a target model to obtain segmentation results corresponding to at least one type of target area output by the target model; based on the segmentation results corresponding to the at least one type of target area, determining the sub-images corresponding to the at least one type of target area in the face area image; in the sub-images corresponding to the at least one type of target area, determining the position information corresponding to the target objects respectively contained in the at least one type of target area. The embodiments of the present disclosure can ensure the accuracy of detecting the position information of the target objects contained in at least one type of target area.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically computer vision and deep learning technology. Background Art

[0002] Object recognition is an important means of image processing. One type of object recognition processing is to identify the skin condition of a person's face. In the process of identifying the skin condition of a person's face, it is often only possible to determine whether there are one or more types of skin symptoms, but it is impossible to accurately locate different types of skin symptoms. Therefore, how to accurately locate different types of skin symptoms in a face image becomes a problem that needs to be solved. Summary of the invention

[0003] The present disclosure provides an image processing method, apparatus, device and storage medium.

[0004] According to one aspect of the present disclosure, there is provided an image processing method, comprising:

[0005] Obtain a face region image from a target image;

[0006] Inputting the face region image into a target model to obtain segmentation results corresponding to at least one type of target region output by the target model;

[0007] Determine sub-images corresponding to the at least one type of target area in the face area image based on the segmentation results corresponding to the at least one type of target area;

[0008] In the sub-images respectively corresponding to the at least one type of target regions, position information corresponding to the target objects respectively included in the at least one type of target regions is determined.

[0009] According to another aspect of the present disclosure, there is provided an image processing apparatus, comprising:

[0010] An extraction module, used for obtaining a face region image from a target image;

[0011] A recognition module, used for inputting the face region image into a target model to obtain segmentation results corresponding to at least one type of target region output by the target model;

[0012] A region acquisition module, used for determining sub-images corresponding to the at least one type of target region in the face region image based on the segmentation results corresponding to the at least one type of target region;

[0013] The position determination module is used to determine the position information corresponding to the target objects respectively contained in the at least one type of target area in the sub-images respectively corresponding to the at least one type of target area.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method in any embodiment of the present disclosure.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided. The computer instructions are used to enable a computer to execute the method in any embodiment of the present disclosure.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method in any embodiment of the present disclosure is implemented.

[0020] According to the technology of the present disclosure, the accuracy of detecting the position information of the target object contained in at least one type of target area can be ensured.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a schematic diagram of an image processing method according to an embodiment of the present disclosure;

[0024] Figure 2 is a schematic diagram of an image processing method according to another embodiment of the present disclosure;

[0025] Figure 3 is a schematic diagram of a processing flow of model training according to an embodiment of the present disclosure;

[0026] Figure 4 is a schematic diagram of an image processing device according to another embodiment of the present disclosure;

[0027] Figure 5is a schematic diagram of an image processing device according to another embodiment of the present disclosure;

[0028] Figure 6 It is a block diagram of an electronic device used to implement the image processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0030] The present disclosure first provides an image processing method, such as Figure 1 As shown, including:

[0031] Step S11: obtaining a face region image from a target image;

[0032] Step S12: inputting the face region image into a target model to obtain segmentation results corresponding to at least one type of target region output by the target model;

[0033] Step S13: determining sub-images corresponding to the at least one type of target area in the face area image based on the segmentation results corresponding to the at least one type of target area;

[0034] Step S14: determining, in the sub-images corresponding to the at least one type of target regions, location information corresponding to the target objects respectively contained in the at least one type of target regions.

[0035] The method provided in this embodiment can be applied to an image processing device, which may include a server, a terminal device or a chip; the terminal device may include a mobile phone, a notebook, a tablet computer, a desktop computer or other devices, etc. This embodiment does not limit the specific form of the image processing device.

[0036] In one implementation, the target image may be an image containing a human face. Preferably, the target image may be an RGB (Red Green Blue) image containing a human face.

[0037] In one implementation, obtaining a face region image from a target image may include: detecting the target image to obtain a face region; and obtaining an image of the face region from the target image as the face region image when the detected face region meets a preset condition. The preset condition may include: the face is not obstructed, such as not wearing sunglasses or a mask, etc.

[0038] In one implementation, the segmentation results corresponding to the at least one type of target region may specifically be: the segmentation results corresponding to at least one target region of each type of the at least one type. That is, the number of each type of target region of the at least one type may be one or more, and this embodiment does not limit the number of each type of target region.

[0039] In one implementation, the determining of the sub-images corresponding to the at least one type of target areas in the face area image based on the segmentation results corresponding to the at least one type of target areas can be: processing based on the masks corresponding to at least one target area of ​​each type of the at least one type, to obtain the sub-images corresponding to the at least one target area of ​​each type of the at least one type contained in the face area image.

[0040] The position information corresponding to the target objects respectively contained in the at least one type of target area may specifically be: the position information corresponding to the target object contained in at least one target area of ​​each type of the at least one type. Each target area of ​​the at least one target area of ​​each type may contain a corresponding target object. For example, assuming that a certain type of target area is a blackhead area, correspondingly, the target object contained in any blackhead area is a blackhead.

[0041] The position information corresponding to the target object may refer to the position coordinates of the target object in the face area image.

[0042] In this embodiment, the face region image in the target image is first obtained, and then the target model is used to predict the segmentation results corresponding to at least one type of target region contained in the face region image, and then the position information corresponding to the target objects respectively contained in the at least one type of target region is determined according to the segmentation results corresponding to the at least one type of target region. Since the target model can determine the corresponding segmentation results for at least one type of target region, in the subsequent processing of positioning the target objects contained in at least one type of target region, the target objects contained in different types of target regions can be positioned respectively, thereby ensuring the accuracy of positioning the target objects contained in each type of target region, and also avoiding the problem that the target objects contained in different types of regions cannot be positioned respectively or the positioning is ambiguous in regions where multiple types overlap.

[0043] In one implementation, inputting the face region image into a target model to obtain segmentation results corresponding to at least one type of target region output by the target model includes:

[0044] Inputting the face region image into the first network of the target model to obtain a feature map corresponding to the face region image;

[0045] The feature map is input into the second network of the target model to obtain segmentation results corresponding to the at least one type of target area.

[0046] The first network may be a Resnet (residual network).

[0047] It should be noted that, among the at least one type of target regions, the number of each type of target regions may be one or more. The segmentation results corresponding to the at least one type of target regions may specifically be masks corresponding to at least one target region of each type of the at least one type.

[0048] For example, the at least one type of target area may include at least one of the following: a blackhead area, a pore area, and a color spot area. Accordingly, the segmentation results corresponding to the at least one type of target area may include at least one of the following: a mask corresponding to at least one pore area, a mask corresponding to at least one blackhead area, and a mask corresponding to at least one color spot area.

[0049] In this embodiment, the first network in the target model first extracts image features to obtain a feature map of the face area image, and then the second network in the target model predicts the segmentation result corresponding to at least one type of target area, and respectively obtains the segmentation results corresponding to the at least one type of target area. This avoids the problem of not being able to accurately divide different types of target areas when multiple types of target areas overlap in the face area image, thereby ensuring the accuracy of ultimately locating the target object contained in at least one type of target area.

[0050] In this embodiment, the step of inputting the feature map into the second network of the target model to obtain the segmentation results corresponding to the at least one type of target area includes:

[0051] The feature map is input into the detection network in the second network of the target model to obtain at least one candidate box; each of the at least one candidate box is input into at least one prediction branch in the second network of the target model for prediction to obtain at least one type of target area contained in each candidate box and its corresponding segmentation result; wherein different prediction branches in the at least one prediction branch are used to predict different types of target areas and their corresponding segmentation results.

[0052] The detection network in the second network of the target model may be an RPN (Region Proposal Network). It should be noted that the detection network in the second network of the target model does not output the type of the target region respectively contained in the at least one candidate box when outputting the at least one candidate box.

[0053] The second network of the target model includes at least one prediction branch; wherein different prediction branches in the at least one prediction branch are used to predict different types of target regions contained in the candidate frame and their corresponding segmentation results. That is, different prediction branches correspond to different types of target regions, and accordingly, the result output by any one of the at least one prediction branch can be the target region of the type corresponding to the prediction branch contained in the candidate frame and its corresponding segmentation result, or the prediction result of the target region of the type corresponding to the prediction branch not contained in the candidate frame.

[0054] Furthermore, inputting each candidate box in the at least one candidate box into at least one prediction branch in the second network of the target model for prediction to obtain at least one type of target region contained in each candidate box and its corresponding segmentation result may include:

[0055] The kth candidate box among the at least one candidate box is input into the yth prediction branch among the at least one prediction branch of the second network for prediction, and the kth candidate box output by the yth prediction branch contains a target area of ​​the type corresponding to the yth prediction branch and its corresponding segmentation result, or the prediction result output by the yth prediction branch does not contain the target area of ​​the type corresponding to the yth prediction branch; wherein k and y are both integers greater than or equal to 1.

[0056] The k-th candidate box mentioned above is any one of at least one candidate box. In addition, in the above example, only the processing of the k-th candidate box inputting one of the at least one prediction branch is illustrated. In actual processing, the k-th candidate box can be input in parallel to each of the at least one prediction branch, but the processing of each prediction branch is the same as the processing of the y-th prediction branch, so it is not repeated.

[0057] It should be understood that after any candidate box is input into each prediction branch of the at least one prediction branch, the result output by each prediction branch can be obtained. Therefore, after the at least one prediction branch of the second network of the target model outputs all predicted results, a candidate box may contain only one type of target area or multiple types of target areas. For example, the second network of the target model may include two prediction branches, the first prediction branch of the two prediction branches is used to predict whether the candidate box contains a blackhead area, and if so, outputs the segmentation result corresponding to the blackhead area; the second prediction branch is used to predict whether the candidate box contains a pore area, and if so, outputs the segmentation result corresponding to the pore area. In the case where the detection network in the second network of the target model outputs at least one candidate frame, any one of the at least one candidate frame (such as the k-th candidate frame) is input into the two prediction branches, respectively, wherein the first prediction branch predicts the k-th candidate frame to include a blackhead region, and outputs a segmentation result (i.e., a mask) corresponding to the blackhead region; wherein the second prediction branch predicts the k-th candidate frame to include a pore region, and outputs a segmentation result (i.e., a mask) corresponding to the pore region. In other words, the k-th candidate frame includes two types of target regions, the blackhead region and the pore region. In this way, when a candidate frame is an overlapping region of multiple types of target regions, different types of target regions contained in the candidate frame can be accurately predicted.

[0058] Here, the candidate box output by the detection network in the second network of the target model can be a rectangular box; the shape of the segmentation result (i.e., mask) of the target area output by any one of the prediction branches in the at least one prediction branch in the second network of the target model can be an irregular shape, for example, a circle or a polygon.

[0059] In this embodiment, image features are extracted by the first network in the target model to obtain a feature map of the face area image, and then the feature map is processed by the detection network of the second network in the target model to obtain at least one candidate frame, and then the type and segmentation result of the target area contained in each candidate frame are predicted respectively by at least one prediction branch to obtain at least one type of target area contained in each candidate frame and its corresponding segmentation result. This can avoid the problem of inability to perform accurate segmentation when different types of target areas overlap, thereby ensuring the accuracy of the final recognition result.

[0060] In one embodiment, determining, in the sub-images respectively corresponding to the at least one type of target regions, the position information corresponding to the target objects respectively contained in the at least one type of target regions includes:

[0061] In the sub-image corresponding to the i-th type of target area in the sub-images corresponding to the at least one type of target area, the target pixel point of the sub-image corresponding to the i-th type of target area is determined based on the color and / or grayscale value of the pixel point, and the position coordinates of the target pixel point are used as the position information corresponding to the target object contained in the i-th type of target area; wherein i is an integer greater than or equal to 1.

[0062] The sub-images corresponding to the at least one type of target area may specifically include: sub-images corresponding to at least one target area of ​​each type of the at least one type. The sub-image corresponding to the i-th type of target area may be a sub-image corresponding to any one type of target area of ​​the at least one type. It should be understood that the number of sub-images corresponding to the i-th type of target area may be one or more. This embodiment describes the processing of any one of the sub-images. The processing of each sub-image included in the i-th type of target area is the same as the aforementioned processing and will not be repeated.

[0063] Among them, the method of determining the target pixel point of the sub-image corresponding to the i-th type of target area based on the color and / or grayscale value of the pixel point, and using the position coordinates of the target pixel point as the position information corresponding to the target object contained in the i-th type of target area can be: determining, from the sub-image corresponding to the i-th type of target area, a pixel point that has the largest color and / or grayscale difference with the surrounding area as the target pixel point of the sub-image corresponding to the i-th type of target area, and using the position coordinates of the target pixel point as the position information corresponding to the target object contained in the i-th type of target area.

[0064] Taking the at least one type of target area as a blackhead area and a pore area as an example, since blackheads and pores have differences in color and / or grayscale value from the surrounding skin, a pixel point with the largest difference in color and / or grayscale value from the surrounding area can be found from the sub-image corresponding to the i-th type of target area as the blackhead or pore (i.e., the target pixel point), and then the specific position coordinates of the blackhead or pore are obtained.

[0065] Specifically, the aforementioned processing method of determining a pixel point having the largest color and / or grayscale difference from the surrounding area from the sub-image corresponding to the i-th type of target area may include:

[0066] The sub-image corresponding to the i-th type of target area is convolved with at least two Gaussian function difference operators to obtain at least two blurred images of the sub-image corresponding to the i-th type of target area; a local maximum is determined based on the at least two blurred images, and a pixel point at a position of the local maximum value is taken as a pixel point in the sub-image corresponding to the i-th type of target area that has the largest color and / or grayscale difference with the surrounding area.

[0067] Wherein, the at least two blurred images may be at least two low-pass filtering results. The standard deviations between the at least two Gaussian function difference operators are different, that is, at least two Gaussian function difference operators may be obtained by adjusting the standard deviations.

[0068] The determining of the local maximum based on the at least two blurred images may be to form a cube with the at least two blurred images and to use a spot in the cube as the local maximum. The spot in the cube may be generated by stacking the difference between any two consecutive blurred images in the at least two blurred images in the cube.

[0069] In this embodiment, the segmentation results of multiple types of target areas are identified by the target model, and then the position coordinates of the target objects contained therein are identified in the sub-image corresponding to each target area corresponding to each type. Since the recognition process of the target model can avoid the overlap problem between multiple types of target areas, the accuracy of the position information of the target objects contained in each type of target area is guaranteed in the final recognition; in addition, since at least one type of target area can be segmented only by the target model, the position information of the target objects contained in each type of target area can be located, so the speed of identifying the position information of the target objects contained in each type of target area can be guaranteed.

[0070] In one implementation, the detecting and obtaining a face region sub-image from a target image includes:

[0071] Detecting the target image to obtain a rectangular frame corresponding to the face area;

[0072] Determine an initial image of the face region based on the rectangular frame corresponding to the face region;

[0073] The initial face region image is normalized to obtain the face region image.

[0074] The detecting of the target image to obtain the rectangular frame corresponding to the face area may specifically be: inputting the target image into a face detection model to obtain the rectangular frame corresponding to the face area output by the face detection model.

[0075] Determining the initial image of the face area based on the rectangular frame corresponding to the face area may include: determining the face area based on the upper left corner coordinates of the rectangular frame corresponding to the face area and the width and height corresponding to the face area; expanding the face area based on a preset ratio to obtain an enlarged face area; extracting the image contained in the enlarged face area from the target image and interpolating it to a preset scale to obtain the initial image of the face area.

[0076] The preset scale may be preset according to actual needs, for example, the size may be 512x512.

[0077] The preset ratio can be set according to actual conditions, such as 5%. Accordingly, the enlarging the face area based on the preset ratio to obtain the enlarged face area can refer to: moving the four frames of the face area outwards based on the preset ratio to obtain four new frames, and using the areas contained in the four new frames as the enlarged face area.

[0078] The four frames of the face area may be respectively: an upper frame of the face area, a lower frame of the face area, a left frame of the face area, and a right frame of the face area. The outward movement refers to a movement in the opposite direction of a perpendicular line between any frame and the center point of the face area; for example, the outward movement of the upper frame of the face area refers to an upward movement, the outward movement of the lower frame of the face area refers to a downward movement, the outward movement of the left frame of the face area refers to a leftward movement, and the outward movement of the right frame of the face area refers to a rightward movement.

[0079] The normalizing the initial image of the face region to obtain the face region image may include: adjusting the pixel value of each pixel in the initial image of the face region to within a preset pixel value interval to obtain the face region image. The adjusting the pixel value of each pixel in the initial image of the face region to within a preset pixel value interval may be: subtracting 128 from the pixel value of each pixel in the initial image of the face region and then dividing by 256, so that the pixel value of each pixel is between [-0.5, 0.5].

[0080] Furthermore, the step of adjusting the pixel value of each pixel in the initial image of the face region to a preset pixel value interval to obtain the face region image may also include: adjusting the pixel value of each pixel in the initial image of the face region to a preset pixel value interval to obtain a normalized image, and performing data enhancement processing on the normalized image to obtain the face region image. The data enhancement processing may include at least one of the following: data scaling, translation, and rotation.

[0081] Taking the at least one type of target area as a pore area and a blackhead area, and the target objects contained in the at least one type of target area as an example, combined with Figure 2 , the solution provided in this embodiment is described, including:

[0082] S21: input a target image; illustratively, the target image may be a human image including a face;

[0083] S22: Performing face detection on the target image;

[0084] S23: Acquire a face region image from the target image;

[0085] S24: inputting the face region image into a target model to obtain a segmentation result of a pore region and a segmentation result of a blackhead region;

[0086] S25: Determine a sub-image corresponding to the pore area in the face area image based on the segmentation result of the pore area, and determine a sub-image corresponding to the blackhead area in the face area image based on the segmentation result of the blackhead area; determine position information of the pores contained in the pore area from the sub-image corresponding to the pore area, and determine position information of the blackheads contained in the blackhead area from the sub-image corresponding to the blackhead area.

[0087] In this embodiment, an initial face area image can be extracted from the target image in advance, and then the initial face area image can be normalized to obtain a face area image. In this way, in the subsequent recognition processing of at least one type of target area segmentation results, recognition can be performed based on a more accurate face area image, thereby ensuring the accuracy of subsequent recognition.

[0088] In one embodiment, the method of training to obtain the aforementioned target model may include:

[0089] The preset model is trained based on the training samples to obtain the trained preset model; the trained preset model is used as the target model; wherein the training samples contain the annotation results of at least one type of target object.

[0090] Specifically, the training sample may be a picture; the number of the training samples may be one or more; and each training sample includes a labeling result of at least one type of target object.

[0091] The labeling result of the at least one type of target object may specifically refer to the position coordinates of the at least one type of target object. The at least one type of target object may specifically refer to at least one target object in each of the at least one type; for example, the at least one type of target object may include blackheads and pores, and any training sample may be pre-labeled with the position coordinates of all blackheads and all pores.

[0092] In this embodiment, by training the preset model with the annotation results pre-annotated with at least one type of target object, the trained target model can more quickly and accurately identify the segmentation result of at least one type of target area.

[0093] The training of the preset model based on the training sample includes:

[0094] Inputting the training sample into a preset first network of the preset model to obtain a feature map output by the preset first network of the preset model;

[0095] Inputting the feature map into a preset second network in the preset model to obtain a segmentation result of at least one type of target area output by the preset second network in the preset model;

[0096] Based on the segmentation result of the at least one type of target region and the labeling result of the at least one type of target object of the training sample, determine a first type of loss function and a second type of loss function corresponding to the segmentation result of the at least one type of target region respectively;

[0097] Backward conduction is performed according to the first type of loss function and the second type of loss function to update the preset model.

[0098] Wherein, determining the first type of loss function and the second type of loss function corresponding to the segmentation results of the at least one type of target region respectively based on the segmentation results of the at least one type of target region and the labeling results of the at least one type of target object of the training sample may specifically include:

[0099] Based on the segmentation result of the j-th type of target area among the at least one type and the position coordinates of the j-th type of target object in the training sample, determine the first type of loss function and the second type of loss function corresponding to the segmentation result of the j-th type of target area; j is an integer greater than or equal to 1.

[0100] Here, the j-th type of target area may be any one of the at least one type of target area. Since the processing for each type of target area is the same as the above description, they will not be described one by one.

[0101] Among them, the first type of loss function can be a Dice loss function, and the second type of loss function can be a BCE (binary cross entropy) loss function.

[0102] The method for determining the convergence of the preset model may include at least one of the following: the number of iterations of the preset model reaches a preset number; the first type of loss function and the second type of loss function of the preset model are lower than a preset threshold value.

[0103] Exemplarily, assuming that the at least one type of target area is a blackhead area and a pore area, the at least one type of target object is a blackhead and a pore, the first loss function is a Dice loss function, the second loss function is a BCE loss function, and the training sample includes the labeling results of blackheads (i.e., the position coordinates of blackheads) and the labeling results of pores (i.e., the position coordinates of pores); combined Figure 3The training of the preset model is described as follows: inputting a training sample into a preset first network of the preset model; wherein the first network may be ResNet;

[0104] Inputting the feature map output by the first network into a preset second network of a preset model to obtain a blackhead area segmentation result and a pore area segmentation result output by the preset second network; wherein the second network may include an RPN;

[0105] Based on the blackhead region segmentation result and the blackhead labeling result in the training sample, the Dice loss and BCE loss corresponding to the blackhead region segmentation result are calculated, and based on the pore region segmentation result and the pore labeling result in the training sample, the Dice loss and BCE loss corresponding to the pore region segmentation result are calculated; then, according to the Dice loss and BCE loss corresponding to the blackhead region segmentation result and the Dice loss and BCE loss corresponding to the pore region segmentation result, the preset model is updated by reverse conduction until the preset model converges to obtain the target model. Wherein, the reverse conduction updating of the preset model may specifically refer to the reverse conduction updating of the parameters of the preset model.

[0106] In this embodiment, a training sample containing annotation results of at least one type of target object is used to determine the first type of loss function and the second type of loss function corresponding to the segmentation results of each type of target area output by the preset model, and then the preset model is reversely conducted according to the first type of loss function and the second type of loss function corresponding to the segmentation results of each type of target area to update the preset model, so that the trained target model can more accurately identify the segmentation results of at least one type of target area.

[0107] The present disclosure also provides an image processing device, such as Figure 4 As shown, including:

[0108] An extraction module 41 is used to obtain a face region image from a target image;

[0109] A recognition module 42, configured to input the face region image into a target model, and obtain segmentation results corresponding to at least one type of target region output by the target model;

[0110] A region acquisition module 43, configured to determine sub-images corresponding to the at least one type of target region in the face region image based on the segmentation results corresponding to the at least one type of target region;

[0111] The position determination module 44 is used to determine, in the sub-images respectively corresponding to the at least one type of target area, the position information corresponding to the target objects respectively contained in the at least one type of target area.

[0112] The recognition module 42 is used to input the face area image into the first network of the target model to obtain a feature map corresponding to the face area image; input the feature map into the second network of the target model to obtain segmentation results corresponding to at least one type of target area.

[0113] The recognition module 42 is used to input the feature map into the detection network in the second network of the target model to obtain at least one candidate box; input each candidate box in the at least one candidate box into at least one prediction branch in the second network of the target model for prediction to obtain at least one type of target area contained in each candidate box and its corresponding segmentation result; wherein different prediction branches in the at least one prediction branch are used to predict different types of target areas and their corresponding segmentation results.

[0114] The position determination module 44 is used to determine the target pixel point of the sub-image corresponding to the i-th type of target area in the sub-image corresponding to the i-th type of target area in the sub-images corresponding to the at least one type of target area based on the color and / or grayscale value of the pixel point, and use the position coordinates of the target pixel point as the position information corresponding to the target object contained in the i-th type of target area; wherein i is an integer greater than or equal to 1.

[0115] The extraction module 41 is used to detect the target image to obtain a rectangular frame corresponding to the face area; determine an initial image of the face area based on the rectangular frame corresponding to the face area; and normalize the initial image of the face area to obtain the face area image.

[0116] like Figure 5 As shown, the device also includes:

[0117] The model training module 45 is used to train the preset model based on the training samples to obtain the trained preset model; and use the trained preset model as the target model; wherein the training samples contain the annotation results of at least one type of target object.

[0118] The model training module 45 is used to input the training sample into the preset first network of the preset model to obtain a feature map output by the preset first network of the preset model; input the feature map into the preset second network in the preset model to obtain a segmentation result of at least one type of target area output by the preset second network in the preset model; determine the first type of loss function and the second type of loss function corresponding to the segmentation result of the at least one type of target area based on the segmentation result of the at least one type of target area and the labeling result of the at least one type of target object of the training sample; and perform back conduction to update the preset model according to the first type of loss function and the second type of loss function.

[0119] The solution provided by the present disclosure uses a face detection model to obtain a face region image, first obtains a feature map of the face region image through a target model, then detects the feature map to obtain at least one candidate frame, then predicts at least one type of target region contained in the candidate frame and its corresponding segmentation result, and finally determines the location information of the target object contained in each target region of each type. In this way, at least one type of target region and its corresponding segmentation result can be efficiently predicted for the face region image through the target model, thereby improving processing efficiency and speed; in addition, since different types of target regions are predicted for the candidate frames respectively, the problem of being unable to be identified or located due to overlap of different types of target regions is avoided, thereby ensuring the accuracy of the location information of the target object finally identified.

[0120] The functions of each unit, module or sub-module in each device of the embodiments of the present disclosure can be found in the corresponding description in the above method embodiments, and will not be repeated here.

[0121] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0122] Figure 6 A schematic block diagram of an example electronic device 60 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0123] like Figure 6As shown, the electronic device 60 includes a computing unit 61, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 62 or a computer program loaded from a storage unit 68 into a random access memory (RAM) 63. In the RAM 63, various programs and data required for the operation of the electronic device 60 can also be stored. The computing unit 61, the ROM 62, and the RAM 63 are connected to each other via a bus 64. An input / output (I / O) interface 65 is also connected to the bus 64.

[0124] A number of components in the electronic device 60 are connected to the I / O interface 65, including: an input unit 66, such as a keyboard, a mouse, etc.; an output unit 67, such as various types of displays, speakers, etc.; a storage unit 68, such as a disk, an optical disk, etc.; and a communication unit 69, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 69 allows the electronic device 60 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0125] The computing unit 61 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 61 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 61 performs the various methods and processes described above, such as image processing methods. For example, in some embodiments, the image processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 68. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 60 via the ROM 62 and / or the communication unit 69. When the computer program is loaded into the RAM 63 and executed by the computing unit 61, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 61 may be configured to perform the image processing method in any other appropriate manner (e.g., by means of firmware).

[0126] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0128] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0129] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0130] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0131] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0132] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0133] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image processing method, comprising: Obtain a face region image from a target image; Inputting the face region image into a first network of a target model to obtain a feature map corresponding to the face region image; Input the feature map into a detection network in a second network of the target model of the target model to obtain at least one candidate box; input each candidate box in the at least one candidate box into at least one prediction branch in the second network for prediction to obtain at least one type of target region contained in each candidate box and its corresponding segmentation result; wherein different prediction branches in the at least one prediction branch are used to predict different types of target regions and their corresponding segmentation results; Determine, based on the segmentation results respectively corresponding to the at least one type of target area, sub-images respectively corresponding to the at least one type of target area in the face area image; In the sub-images respectively corresponding to the at least one type of target regions, position information corresponding to the target objects respectively included in the at least one type of target regions is determined.

2. The method according to claim 1, wherein: The determining, in the sub-images respectively corresponding to the at least one type of target regions, position information corresponding to the target objects respectively contained in the at least one type of target regions comprises: In the sub-image corresponding to the i-th type of target area in the sub-images corresponding to the at least one type of target area, the target pixel point of the sub-image corresponding to the i-th type of target area is determined based on the color and / or grayscale value of the pixel point, and the position coordinates of the target pixel point are used as the position information corresponding to the target object contained in the i-th type of target area; wherein i is an integer greater than or equal to 1.

3. The method according to claim 1, wherein: Determining a face region image from a target image includes: Detecting the target image to obtain a rectangular frame corresponding to the face area; Determine an initial image of the face region based on the rectangular frame corresponding to the face region; The initial face region image is normalized to obtain the face region image.

4. The method according to any one of claims 1 to 3, wherein: The method further comprises: The preset model is trained based on the training samples to obtain the trained preset model; the trained preset model is used as the target model; wherein the training samples contain the annotation results of at least one type of target object.

5. The method according to claim 4, wherein: The training of the preset model based on the training sample includes: Inputting the training sample into a preset first network of the preset model to obtain a feature map output by the preset first network of the preset model; Inputting the feature map into a preset second network in the preset model to obtain a segmentation result of at least one type of target area output by the preset second network in the preset model; Based on the segmentation result of the at least one type of target region and the labeling result of the at least one type of target object of the training sample, determine a first type of loss function and a second type of loss function corresponding to the segmentation result of the at least one type of target region respectively; Backward conduction is performed according to the first type of loss function and the second type of loss function to update the preset model.

6. An image processing device, comprising: An extraction module, used for obtaining a face region image from a target image; A recognition module, used for inputting the face area image into the first network of the target model to obtain a feature map corresponding to the face area image; inputting the feature map into the detection network in the second network of the target model of the target model to obtain at least one candidate box; inputting each candidate box in the at least one candidate box into at least one prediction branch in the second network for prediction to obtain at least one type of target area contained in each candidate box and its corresponding segmentation result; wherein different prediction branches in the at least one prediction branch are used to predict different types of target areas and their corresponding segmentation results; A region acquisition module, used for determining sub-images corresponding to the at least one type of target region in the face region image based on the segmentation results corresponding to the at least one type of target region; The position determination module is used to determine the position information corresponding to the target objects respectively contained in the at least one type of target area in the sub-images respectively corresponding to the at least one type of target area.

7. The device according to claim 6, wherein: The position determination module is used to determine the target pixel point of the sub-image corresponding to the i-th type of target area in the sub-image corresponding to the i-th type of target area in the sub-images corresponding to the at least one type of target area based on the color and / or grayscale value of the pixel point, and use the position coordinates of the target pixel point as the position information corresponding to the target object contained in the i-th type of target area; wherein i is an integer greater than or equal to 1.

8. The device according to claim 6, wherein: The extraction module is used to detect the target image to obtain a rectangular frame corresponding to the face area; determine an initial image of the face area based on the rectangular frame corresponding to the face area; and normalize the initial image of the face area to obtain the face area image.

9. The device according to any one of claims 6 to 8, wherein: The device also includes: A model training module is used to train a preset model based on training samples to obtain a trained preset model; the trained preset model is used as the target model; wherein the training samples contain annotation results of at least one type of target object.

10. The device according to claim 9, wherein: The model training module is used to input the training sample into the preset first network of the preset model to obtain a feature map output by the preset first network of the preset model; Inputting the feature map into a preset second network in the preset model to obtain a segmentation result of at least one type of target area output by the preset second network in the preset model; Based on the segmentation results of at least one type of target area and the labeling results of at least one type of target object of the training sample, determine the first type of loss function and the second type of loss function corresponding to the segmentation results of at least one type of target area respectively; and perform back conduction to update the preset model according to the first type of loss function and the second type of loss function.

11. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 5.