Image processing method and device, electronic equipment and readable storage medium

Abstract

The invention provides an image processing method and device, electronic equipment and a readable storage medium. The method comprises the following steps: acquiring a to-be-classified first picture set; detecting key points of the pictures in the first picture set, and filtering the pictures which do not meet a preset key point condition to obtain a second picture set which meets the key point condition; performing target object region segmentation on the pictures in the second picture set, and filtering the pictures which do not meet a preset target object condition to obtain a third picture set which meets the preset target object condition; target object features are extracted from target object areas segmented from all the pictures in the third picture set; and aggregating the pictures with the same target object features into a group of picture sets.

Description

Technical Field

[0001] The present application relates to the technical field of image classification, and in particular to an image processing method, device, electronic device and readable storage medium. Background Art

[0002] With the widespread popularity of electronic devices and the increase in the frequency of daily use, the number of images stored in terminals has shown an exponential growth trend, and managing these images has become a challenge. Traditional management methods often rely on manual classification, which is not only time-consuming and labor-intensive, but also difficult to ensure the consistency and accuracy of classification. Therefore, a more intelligent, accurate and efficient image processing method is urgently needed. Summary of the invention

[0003] The present application provides an image processing method, the method comprising:

[0004] Obtain a first set of pictures to be classified;

[0005] Detect key points of the pictures in the first picture set, and filter out pictures that do not meet the preset key point conditions to obtain a second picture set that meets the key point conditions;

[0006] Segmenting the target object regions of the pictures in the second picture set, and filtering the pictures that do not meet the preset target object conditions, to obtain a third picture set that meets the preset target object conditions;

[0007] Extracting target object features from target object regions segmented from each picture of the third picture set;

[0008] Aggregate the pictures with the same target object features into a set of pictures.

[0009] Optionally, the detecting key points of the pictures in the first picture set includes:

[0010] Inputting each picture in the first picture set into a pre-trained target object detection model, so that the target object detection model identifies a target object region in each picture, and determines a target object key point contained in the picture in the target object region;

[0011] According to the target object key points, determine whether the image meets the preset key point conditions.

[0012] Optionally, determining whether the image meets a preset key point condition according to the key point of the target object includes:

[0013] Obtaining the area of ​​the target object region corresponding to each picture in the first picture set output by the target object detection model, and / or the number of key points of the target object;

[0014] Based on the area or the number of key points of the target object, determining whether the image meets a preset key point condition;

[0015] or,

[0016] Based on the target object shape formed by the target object key points, determine whether the image meets the preset key point conditions.

[0017] Optionally, before inputting each picture in the first picture set into a pre-trained target object detection model, the method further includes:

[0018] Inputting each picture in the first picture set into a pre-trained scene classification model, so that the scene classification model identifies the picture scene to which each picture belongs;

[0019] Obtaining the picture scenes output by the scene classification model and to which each picture in the first picture set belongs, and filtering pictures that do not conform to the target picture scene; wherein the target picture scene is a picture scene including a target object;

[0020] The step of inputting each picture in the first picture set into a pre-trained human body detection model includes:

[0021] The pictures in the first picture set that match the target picture scene are input into a pre-trained target object detection model.

[0022] Optionally, the method is applied to a mobile device; the scene classification model is a classification model constructed based on a lightweight neural network that can run on a mobile device.

[0023] Optionally, the lightweight neural network includes a Ghostnetv2 network.

[0024] Optionally, segmenting the pictures in the second picture set into target object areas and filtering pictures that do not meet the preset target object conditions to obtain a third picture set that meets the preset target object conditions includes:

[0025] Inputting each picture in the second picture set into a pre-trained target object segmentation model, so as to determine, by the target object segmentation model, an unobstructed area in the target object area corresponding to each picture in the second picture set;

[0026] Obtaining an unobstructed area in the target object region corresponding to each picture in the second picture set output by the target object segmentation model, and determining a picture whose ratio of the area of ​​the unobstructed area to the area of ​​the target object region is less than a preset ratio threshold as a picture that does not meet the preset target object condition;

[0027] The pictures that do not meet the preset target object condition are filtered out from the second picture set to obtain a third picture set.

[0028] Optionally, obtaining an unobstructed area in the target object region corresponding to each image in the second image set output by the target object segmentation model, and determining an image whose ratio of the area of ​​the unobstructed area to the area of ​​the target object region is less than a preset ratio threshold as an image that does not meet the preset target object condition, includes:

[0029] Obtaining an unobstructed area in the target object region corresponding to each picture in the second picture set output by the target object segmentation model, and determining a first number of target object key points in the unobstructed area and a second number of target object key points in the target object region;

[0030] A ratio of the first number to the second number is calculated, and pictures whose ratio is less than a preset ratio threshold are determined as pictures that do not meet preset target object conditions.

[0031] Optionally, the pictures with the same target object features are aggregated into a set of pictures, including:

[0032] Create a target album for storing pictures with the same characteristics as the target object;

[0033] Add pictures with the same characteristics as the target object to the target album.

[0034] The present application also provides an image processing device, the device comprising:

[0035] A picture acquisition unit, used to acquire a first picture set to be classified;

[0036] A first filtering unit, configured to detect key points of the pictures in the first picture set, and filter the pictures that do not meet the preset key point conditions, to obtain a second picture set that meets the key point conditions;

[0037] A second filtering unit, configured to segment the pictures in the second picture set into target object regions, and filter the pictures that do not meet the preset target object conditions, to obtain a third picture set that meets the preset target object conditions;

[0038] a feature extraction unit, configured to extract target object features from target object regions segmented from each picture of the third picture set;

[0039] The picture aggregation unit is used to aggregate the pictures with the same target object features into a group of picture sets.

[0040] The present application also provides an electronic device, comprising a communication interface, a processor, a memory and a bus, wherein the communication interface, the processor and the memory are interconnected via the bus;

[0041] The memory stores machine-readable instructions, and the processor executes the above method by calling the machine-readable instructions.

[0042] The present application also provides a computer-readable storage medium, which stores machine-readable instructions. When the machine-readable instructions are called and executed by a processor, the above method is implemented.

[0043] In the scheme recorded in the above embodiment, pictures that do not meet the requirements can be filtered out by detecting key points and segmenting target object areas of the pictures. For the filtered pictures, target object features are further extracted from the segmented target object areas, so that pictures with the same target object features are aggregated into a group of pictures. This can effectively and automatically classify the pictures to be classified, thereby improving the efficiency of picture management and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flowchart of an image processing method provided by an exemplary embodiment.

[0045] Figure 2 The figure is a schematic diagram showing a face detection model identifying face regions and face key points according to an exemplary embodiment.

[0046] Figure 3 It is a schematic diagram of a human body detection model for identifying human body regions and key points of a human body provided by an exemplary embodiment.

[0047] Figure 4 It is a schematic diagram showing the working principle of a decoupled fully connected CRF layer according to an exemplary embodiment.

[0048] Figure 5 It is a flowchart of a practical application of an image processing method provided by an exemplary embodiment.

[0049] Figure 6 It is a hardware structure diagram of an electronic device in which an image processing device is provided according to an exemplary embodiment.

[0050] Figure 7It is a block diagram of an image processing device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0051] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0052] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0053] See also Figure 1 , Figure 1 FIG. 1 is a flowchart of an image processing method provided by an exemplary embodiment. The method may include the following steps:

[0054] Step 100: Obtain a first set of pictures to be classified.

[0055] In this specification, the first picture set may include, for example, a picture set in an album, or a picture set in other specified locations (such as a specified folder), or a picture set in an unspecified location automatically recognized by an application (such as a folder storing pictures recognized when scanning a terminal), etc.

[0056] Taking the photo album as an example, the application can traverse the photo album directory in the device storage through the API provided by the system, and obtain the first set of pictures to be classified in the photo album.

[0057] It should be noted that before obtaining the first set of pictures to be classified in the album, it should be ensured that the application has the permission to access the user's album.

[0058] Typically, a user may grant an application permission to read the photo album in the device storage when using the application for the first time. A user may also grant an application permission to read one or more specific pictures in the photo album in the device storage when using the application for the first time, which is not specifically limited in this manual.

[0059] In one embodiment, the application can filter out pictures to be classified from the album according to preset conditions or user-specified parameters. The preset conditions can be set based on timestamps, file attributes, and picture content. For example, pictures with file sizes greater than a threshold can be used as pictures to be classified, or pictures with specified file names can be used as pictures to be classified. These filtered pictures can form a first picture set.

[0060] Step 102: Detect key points of the pictures in the first picture set, and filter out pictures that do not meet the preset key point conditions to obtain a second picture set that meets the key point conditions.

[0061] After determining the first picture set to be classified, the second picture set containing the target object can be identified from each picture in the first picture set. The specific method of identifying the second picture set containing the target object from each picture in the first picture set is not specifically limited in this specification.

[0062] The target object may include any type of object, for example, it may include items (such as mobile phones, women's bags, etc.), and the image processing method provided in this specification may classify pictures of the same items together. For another example, the target object may include animals (such as pet cats, pet dogs, etc.), and the image processing method provided in this specification may classify pictures of the same animals together. For another example, the target object may include plants (such as potted plants, flowers, etc.), and the image processing method provided in this specification may classify pictures of the same plants together. For another example, the target object may include human faces, and the image processing method provided in this specification may classify pictures of the same human faces together.

[0063] In this specification, a pre-trained target object detection model may be used to identify a second set of pictures containing a target object from a first set of pictures to be classified.

[0064] Exemplarily, each picture in the first picture set may be input into a pre-trained target object detection model, so that the target object detection model identifies a target object region in each picture, and determines the key points of the human face contained in the picture in the target object region;

[0065] Then, according to the target object key point, it is determined whether the picture meets the preset key point condition; then, the pictures that do not meet the preset key point condition are filtered out from the first picture set to obtain the second picture set.

[0066] Furthermore, the aforementioned determination of whether the image meets the preset key point conditions based on the key points of the target object can be determined in the following ways:

[0067] [The first method of determining key point conditions]:

[0068] Obtain the area of ​​the target object region corresponding to the picture in the first picture set output by the target object detection model, and / or the number of key points of the target object; based on the area or the number of key points of the target object, determine whether the picture meets the preset key point condition.

[0069] [The second method of determining key point conditions]:

[0070] Based on the target object shape formed by the target object key points, determine whether the image meets the preset key point conditions.

[0071] The following describes how to determine these two key point conditions:

[0072] Taking the target object of face as an example, the above target object detection model can use a pre-trained face detection model and a pre-trained human body detection model to identify a second set of pictures containing faces from a first set of pictures to be classified.

[0073] [The first method of determining key point conditions]

[0074] For example, a face detection method based on traditional computer vision algorithms can be used. This method can detect faces based on features such as color, texture, and shape of the image. For another example, face detection based on deep learning can be used. Using deep learning algorithms such as convolutional neural networks (CNN), high-precision face detection can be performed. For another example, a key point detection method can be used to determine the specific location and feature points of the face, such as eyes, nose, and mouth, through key point detection. This can help locate and analyze faces more precisely.

[0075] Exemplarily, each picture in the first picture set may be input into a pre-trained face detection model, so that the face detection model can identify a face region in each picture, and determine the face key points contained in the picture in the face region;

[0076] Next, the aforementioned obtaining the area of ​​the target object region corresponding to the picture in the first picture set output by the target object detection model and / or the number of key points of the target object; based on the area or the number of key points of the target object, determining whether the picture meets the preset key point condition may further include:

[0077] Obtaining the ratio of the area of ​​the face region corresponding to each picture in the first picture set output by the face detection model to the total area of ​​the picture, and / or the number of the face key points;

[0078] The picture whose ratio is not greater than a preset ratio threshold or whose number of facial key points is not greater than a preset number threshold is determined as a picture that does not meet the preset key point conditions.

[0079] It should be noted that, for other target objects, the same method can be used to obtain the ratio of the area of ​​the target object region corresponding to each picture in the first picture set output by the target object detection model to the total area of ​​the picture, and / or the number of target object key points; the picture whose ratio is not greater than the preset ratio threshold or the number of target object key points is not greater than the preset number threshold is determined as a picture that does not meet the preset key point conditions.

[0080] In this example, each picture in the first picture set can be input in batches into a pre-trained face detection model, and the face detection model can identify the face areas in the pictures and output the positions of the detected face areas in the pictures, for example, a face detection box composed of a series of coordinates.

[0081] Further, the facial key point positioning model can be applied in the determined face area to accurately find the positions of key feature points such as eyes, nose, mouth, etc. Among them, the facial key point positioning model can be an additional pre-trained model, or as a part of the face detection model, which is not specifically limited in this specification. For example, the facial key point detector in the Dlib library can be used to detect the eyes, nose, eyebrows and mouth, etc. in the face area.

[0082] For each face region detected in the image, the ratio of the area of ​​the face region to the total area of ​​the image may be calculated, and / or the total number of facial key points recognized in each face region may be counted.

[0083] The pictures whose ratio is not greater than the preset ratio threshold or whose number of key points of the face is not greater than the preset number threshold are determined as pictures that do not meet the preset key point conditions, thereby filtering pictures whose ratio of the area of ​​the face region to the total area of ​​the picture is lower than the preset ratio threshold, or filtering pictures whose number of key points of the face is lower than the preset number threshold. The remaining pictures in the first picture set after filtering are the second picture set that meets the key point conditions.

[0084] By calculating the ratio of the area of ​​the face region to the total area of ​​the image, or counting the number of facial key points, and comparing them with a preset threshold, low-quality images containing faces can be directly filtered out.

[0085] It should be noted that before the first set of pictures to be classified is input into the target object detection model, the pictures can also be preprocessed. Since not all pictures need to be classified, some unnecessary pictures can be screened out, such as screenshots, pictures in the application cache, and so on. Users can set filtering conditions, such as file size, format (for example, only processing common picture formats such as .jpg or .png), and excluding specific directories, etc. In addition, for pictures that take up a lot of storage space, they can also be compressed appropriately to reduce memory usage. In order to meet the input requirements of the model, it may also be necessary to adjust the size of the picture, or normalize the pixel value, etc.

[0086] See also Figure 2 , Figure 2 It is a schematic diagram of a face detection model identifying face areas and face key points provided by an exemplary embodiment.

[0087] like Figure 2 As shown, Figure 2 On the left is a picture to be classified. After the picture is input into the face detection model, the face detection model can identify the face area of ​​the picture. Figure 2 The dotted rectangular box in FIG. 1 shows the area corresponding to the picture.

[0088] Furthermore, the face detection model can determine the locations of key feature points such as eyes, nose, eyebrows, and mouth in the face area. Figure 2 The key points corresponding to two eyes, a nose, two eyebrows and a mouth are shown in FIG.

[0089] [The second method of determining key point conditions]

[0090] In this specification, in addition to using the face detection model to filter out images that do not meet the preset key point conditions, the body detection model can also be used to filter out images that do not meet the preset key point conditions.

[0091] Exemplarily, each picture in the first picture set can be input into a pre-trained human body detection model so that the human body detection model can identify the human body region corresponding to each picture and determine the human body key points contained in the picture in the human body region.

[0092] Next, the above-mentioned determining whether the image meets the preset key point conditions based on the target object form formed by the key points of the target object further includes:

[0093] Acquire human key points corresponding to each picture in the first picture set output by the human detection model, and construct a human skeleton structure based on the human key points;

[0094] The similarity between the human skeleton structure and the human skeleton structure corresponding to the illegal posture is calculated, and the picture with a similarity greater than a threshold is determined as a picture that does not meet the preset key point conditions.

[0095] It should be noted that for other target objects, their key points can also be constructed into corresponding target object forms according to the type of specific target objects. For example, a dog skeleton structure is constructed according to the key points of a pet dog, and then the similarity between the dog skeleton structure and the skeleton structure corresponding to the illegal posture is calculated to determine the pet dog images that do not meet the preset key point conditions.

[0096] In this example, the pre-trained human detection model can identify and select the positions of all human-shaped objects in the image, that is, determine the approximate area of ​​each human body. At the same time, the position of the human body area is output, for example, a human body detection frame composed of a series of coordinates, each detection frame marks the position of a human body.

[0097] After determining the human body area, human key point detection can be performed to accurately locate key points in the previously defined human body area, such as neck, shoulders, elbows, wrists, hips, knees, ankles, etc. These key points help understand human posture, limb position, and clothing wearing effect.

[0098] After determining the key points of the human body, the detected key points can be connected to construct a human skeleton structure that reflects the human posture in the image.

[0099] The constructed human skeleton structure can be further compared with the human skeleton structure corresponding to the preset illegal posture, for example, by calculating the similarity between the two to determine whether the human posture in the current picture is normal.

[0100] In addition, a machine learning model can also be used. The model can recognize and classify different postures or actions by learning a large number of labeled posture samples. The human skeleton structure is input into the machine learning model, and the machine learning model determines whether the human posture is normal.

[0101] After determining the pictures with abnormal human postures, the pictures with abnormal human postures can be further filtered, thereby ensuring that the remaining pictures in the first picture set are all high-quality and exquisite pictures, thereby improving the user experience when viewing the classification results.

[0102] See also Figure 3 , Figure 3 It is a schematic diagram of a human body detection model for identifying human body regions and key points of a human body provided by an exemplary embodiment.

[0103] like Figure 3 As shown, Figure 3On the left is a picture containing a portrait or human body. After the picture is input into the human body detection model, the human body detection model can identify the human body area in the picture. Figure 3 The dotted rectangular box in FIG. 1 shows the area corresponding to the picture.

[0104] Furthermore, the human body detection model can determine the locations of key feature points such as the neck, shoulders, elbows, wrists, hips, knees, ankles, etc. in the human body area. Figure 3 These key feature points are shown in . Further, these key feature points can be connected to determine the corresponding human skeleton structure.

[0105] In this specification, in addition to pictures of people or human bodies, each picture in the first picture set to be classified may also include pictures with various scenes. For example, natural scenery, urban street scenes, scenes of people's activities, etc. Through the pre-trained scene classification model, the picture scene to which each picture in the pictures to be classified belongs can be quickly identified, and the scene label corresponding to each picture scene can be output. Therefore, pictures containing people or human bodies can be screened out based on the scene classification model.

[0106] Specifically, before inputting each picture in the first picture set into the pre-trained target object detection model, each picture in the first picture set may be input into the pre-trained scene classification model so that the scene classification model identifies the picture scene to which each picture belongs;

[0107] Then, the picture scenes output by the scene classification model and to which each picture in the first picture set belongs are obtained, and pictures that do not conform to the target picture scene are filtered; wherein the target picture scene is a picture scene including a target object;

[0108] Next, the remaining pictures in the first picture set that are not in accordance with the target picture scene may be filtered out and input into a pre-trained target object detection model.

[0109] In this specification, the scene classification model can identify the picture scenes described by each input picture, and simultaneously output the scene label corresponding to the picture scene to which each picture belongs.

[0110] The scene tags may specifically include tags corresponding to natural scenery, such as the sea, mountains, sunset, etc.; they may also include tags corresponding to city street scenes, such as offices, cafes, etc.; in addition, they may also include tags corresponding to portraits or human bodies.

[0111] Based on the scene labels, it can be determined which pictures in the first picture set contain the target object. These pictures are pictures that meet the target picture scene, while those that do not contain the target object are pictures that do not meet the target picture scene.

[0112] After determining the pictures that do not conform to the target picture scene, the pictures that do not conform to the target picture scene can be further filtered, thereby ensuring that the pictures input into the target object detection model are all pictures containing the target object, thereby avoiding the waste of computing resources of the target object detection model (target object detection on pictures without target objects cannot detect results, thereby wasting resources).

[0113] In one embodiment, since the pictures in the user's photo album are private to the user, they cannot be uploaded to the cloud for classification, and thus a lightweight scene classification model that can run on a mobile device is required for classification.

[0114] The lightweight neural network may include but is not limited to Ghostnetv2 network, MobileNet network, SqueezeNet network, RegNet network, MnasNet network, etc.

[0115] Take the Ghostnetv2 network as an example. Ghostnetv2 is an efficient convolutional neural network architecture, which is an upgraded version of the Ghostnet network. Ghostnet reduces the computational complexity and parameter amount of the model by generating low-cost "Ghost" filters while maintaining high accuracy. Ghostnetv2 is further optimized on this basis, aiming to improve the performance of the model, especially for applications on mobile devices and resource-constrained environments. The Ghostnetv2 network is suitable for tasks that require fast and real-time processing of images, such as image classification, object recognition, etc.

[0116] When the image is input into the Ghostnetv2 network, the Ghostnetv2 network will perform multi-level feature extraction on the image through a series of convolutional layers, activation functions and other components. This can be understood as the network understanding and analyzing the content of the image at different scales.

[0117] After multiple levels of feature extraction, the high-level features of the image are passed to the fully connected layer or other classification layers of the network. These layers are responsible for mapping the learned features to predefined categories, such as scene labels such as sea, mountains, sunset, etc. Finally, the Ghostnetv2 network will output one or more scene labels with the highest probability, indicating the scene category that the model believes the image is most likely to belong to.

[0118] Specifically, after the image to be classified is input into the Ghostnetv2 network, the image will pass through a series of convolutional layers in the Ghostnetv2 network.

[0119] Compared with the convolutional layers of traditional convolutional neural networks, the convolutional layers in the Ghostnetv2 network can effectively generate more feature maps using fewer operations, especially on mobile devices and in resource-constrained environments, which can reduce computational complexity and the amount of parameters.

[0120] A convolutional layer in a traditional convolutional neural network (CNN). The convolutional layer slides over the input image through a set of fixed-size, learnable weight filters (kernels), performs weighted summation on local areas and applies nonlinear activation functions to extract features. Each filter is responsible for identifying different features, such as edges, lines, or more complex patterns.

[0121] In the Ghostnetv2 network, the image first passes through a basic convolutional layer to output a small number of feature maps. Further, through operations on the internal feature maps, such as linear transformation and nonlinear activation, the feature maps are "decomposed" or "expanded" into more feature maps. These additional feature maps are called "virtual feature maps". These virtual feature maps are generated based on the actual calculated feature maps through low-cost operations, so they do not introduce a lot of additional computing overhead.

[0122] In this way, the Ghostnetv2 network can generate more feature maps without significantly increasing the demand for computing resources, thereby improving the model's ability and diversity to capture different features and enhancing the model's expressiveness.

[0123] After the image passes through a series of convolutional layers and activation functions, the Ghostnetv2 network outputs a preliminary classification result for each pixel in the image.

[0124] Furthermore, the preliminary classification results can be input into the DFC (Decoupled Fully Connected Conditional Random Fields) CRF layer.

[0125] Decoupled fully connected CRF is also a conditional random field, which is used to post-process pixel-level annotations in image classification tasks to obtain more accurate results. CRF takes into account the dependencies between adjacent pixels and optimizes the segmentation results by maximizing the joint probability distribution. However, the traditional fully connected CRF has a large computational overhead, so a decoupled fully connected CRF is proposed to process the adjacency relationships in the horizontal and vertical directions separately, thereby reducing the computational complexity.

[0126] See also Figure 4 , Figure 4 It is a schematic diagram of the working principle of a decoupled fully connected CRF layer provided by an exemplary embodiment.

[0127] like Figure 4 As shown in Figure 1, the input of the decoupled fully connected CRF layer is a preliminarily classified image, in which each pixel has an initial category label.

[0128] The preliminarily classified images are further input into the vertical fully connected layer. The vertical fully connected layer only considers the adjacency relationship in the vertical direction and updates the label of each pixel through a vertical fully connected layer.

[0129] The preliminarily classified images are further input into the horizontal fully connected layer, which only considers the adjacency relationship in the horizontal direction. The label of each pixel is updated again through a horizontal fully connected layer.

[0130] Finally, the updates in both directions are combined to form an attention map that reflects the final category label of each pixel.

[0131] Since the computational overhead of the decoupled fully connected CRF is relatively small, it can reduce the computational complexity of the model without affecting the overall running time, making it more suitable for mobile devices and resource-constrained environments.

[0132] After the attention map is generated, the attention map can be further input into one or more fully connected layers to extract classification information from the attention map and finally output the classification result. The function of these fully connected layers is to map the features in the attention map to specific category labels, such as converting them into probability distributions of each category through a softmax function. Based on the probability distribution output by the fully connected layer, the category label with the highest probability can be selected as the final classification result.

[0133] By combining the convolutional layer of the lightweight Ghostnetv2 network with the decoupled fully connected CRF layer, the computational complexity of the scene classification model can be reduced as much as possible, making the scene classification model suitable for mobile devices. In this way, images do not need to be uploaded to the cloud, but only classified on mobile devices, thereby better protecting user privacy.

[0134] Step 104 , segmenting the target object regions of the pictures in the second picture set, and filtering the pictures that do not meet the preset target object conditions, to obtain a third picture set that meets the preset target object conditions.

[0135] After obtaining the second picture set, the pictures in the second picture set may be segmented into target object areas, and pictures that do not meet the preset target object conditions may be filtered to obtain a third picture set that meets the preset target object conditions.

[0136] Exemplarily, each picture in the second picture set may be input into a pre-trained target object segmentation model, so that the target object segmentation model determines an unobstructed area in the target object area corresponding to each picture in the second picture set;

[0137] Obtaining an unobstructed area in the target object region corresponding to each picture in the second picture set output by the target object segmentation model, and determining a picture whose ratio of the area of ​​the unobstructed area to the area of ​​the target object region is less than a preset ratio threshold as a picture that does not meet the preset target object condition;

[0138] The pictures that do not meet the preset target object condition are filtered out from the second picture set to obtain a third picture set.

[0139] In this specification, the target object segmentation model can output a pixel-level segmentation map for the target object. In this segmentation map, each pixel is assigned a label indicating which part of the target object the pixel belongs to.

[0140] Taking the target object of face as an example, the target object segmentation model can be a face segmentation model, which performs face segmentation on the input image and outputs various components of the face such as skin, eyes, mouth, hair or background.

[0141] In the segmentation results output by the face output model, in addition to identifying the various components of the face, it also clearly identifies which areas are actually exposed parts of the face and which are blocked parts, such as glasses, masks, hair blocking the face, etc. By analyzing the segmentation map, the unblocked areas can be further identified and marked.

[0142] The unobstructed areas in the face region corresponding to each image in the character image set output by the face segmentation model are obtained, and the ratio of the area of ​​the unobstructed area to the area of ​​the face region can be further calculated. If the ratio of the area of ​​the unobstructed area to the area of ​​the face region is less than a preset ratio threshold, it means that there are more obstructed parts. Therefore, it can be determined that the pictures have poor actual effects and these pictures can be directly filtered.

[0143] By filtering out images in which faces are obscured, it can be ensured that the images in the generated third image set all contain large face areas. This can improve the efficiency of subsequently aggregating images with the same facial features into a group of image sets, and can also improve the user experience when viewing classification results.

[0144] In one embodiment, after completing the target object segmentation and determining the unobstructed area, a key point detection model can be used to identify and count the target object key points in the unobstructed area to count the first number of target object key points in the unobstructed area.

[0145] Furthermore, the key point detection model can also be used, but this time, the target object region is not subjected to occlusion judgment, and all key points are directly searched and located in the entire initially detected target object region. The second number of all target object key points in the entire target object region is counted.

[0146] The first number reflects the amount of effective target object information that can be used for analysis or recognition, and the second number provides the total number of key points in the target object area in theory. If the ratio of the first number to the second number is less than the preset ratio threshold, it means that there are many blocked parts, so it can be determined as a picture with poor actual effect, and these pictures can be directly filtered.

[0147] Step 106: extract target object features from target object regions segmented from each picture of the third picture set.

[0148] In this specification, for each picture in the third picture set, a target object detection technology can be used, such as a Haar feature cascade classifier and a HOG+SVM method of Dlib. These model methods can identify the position of the target object in the picture and output a target object detection frame. Further, the target object area can be cropped from the picture according to the target object detection frame output by the target object detection model to prepare for subsequent feature extraction.

[0149] The specific method of extracting the target object features in the target object area is not specifically limited in this specification.

[0150] Taking the face as an example, we can use pre-trained face feature extraction models, such as FaceNet, ArcFace, VGGFace2, etc. These models can convert face images in the face area into feature vectors of fixed length. These vectors can better represent the information of the face.

[0151] Among them, the FaceNet model is trained through the triplet loss function to make the distance between image features of the same person as small as possible and the distance between different people as large as possible, thereby learning the embedding vector in the high-dimensional space.

[0152] The ArcFace model optimizes the separability of features and enhances the discriminative ability of features by introducing an additional loss function, so that the inter-class distance in the feature space increases and the intra-class distance decreases.

[0153] The VGGFace2 model is modified based on the image classification network and learns facial feature representation by fine-tuning training based on a large-scale face dataset.

[0154] Step 108: Aggregate the pictures with the same target object features into a set of pictures.

[0155] In this specification, after extracting the target object features from each picture in the third picture set, a unique feature vector may be generated for the target object in each picture, and the vector represents the identity information of the target object.

[0156] Further, an appropriate similarity calculation method may be used to compare the similarities between feature vectors. Common similarity calculation methods include Euclidean distance, cosine similarity or other complex calculation methods.

[0157] Then a similarity threshold can be set. When the similarity between two feature vectors exceeds this threshold, it is considered that the target object features they represent are the same.

[0158] Pictures with the same target object features may be further aggregated into a group of picture sets. For example, pictures with the same target object features may be aggregated into a subfolder or marked with the same label in a database, thereby completing the classification of the first picture set to be classified.

[0159] In one implementation, for each group of pictures with the same target object characteristics, a target album for storing pictures with the same target object characteristics may be newly created, and the pictures with the same target object characteristics may be added to the target album.

[0160] Therefore, after completing the image classification, users can quickly find all photos of a specific target object, which is more convenient whether it is used for personal memory organization, album management or professional portrait photography archiving.

[0161] See also Figure 5 , Figure 5 It is a flowchart of a practical application of an image processing method for human faces provided by an exemplary embodiment.

[0162] In step 502, the application may obtain a first set of pictures to be classified.

[0163] Taking the photo album as an example, the application can traverse the photo album directory in the device storage through the API provided by the system and obtain the pictures to be classified in the photo album. It should be noted that these pictures can be accessed after the user's authorization.

[0164] Step 504, input the acquired pictures in the first picture set into a pre-trained face detection model. Through the face detection model, the face area corresponding to each picture can be identified, and the number of facial key points contained in the picture can be determined in the face area.

[0165] Step 506: The face detection model can filter out pictures with smaller face areas and fewer face key points, output selfie and group photo pictures containing faces, and generate a second picture set.

[0166] Step 508, input the pictures in the first picture set obtained into the pre-trained scene classification model. Through the scene classification model, the picture scene to which each picture belongs can be identified, and the picture can be marked with corresponding scene labels, such as "mountains", "sea", "human body", "portrait", etc.

[0167] Step 510: The scene classification model outputs a picture containing a target scene label, wherein the target scene label includes a picture scene label of a portrait or a human body.

[0168] Step 512: The scene classification model can also output pictures with other scene labels.

[0169] By filtering out pictures with other scene labels, the remaining pictures in the first picture set are all pictures including portraits or human bodies.

[0170] Step 514, the remaining pictures in the first picture set after filtering out the pictures with other scene labels are input into the pre-trained human body detection model. The human body detection model can identify the human body area and determine the human body key points contained in the object picture in the human body area. Further, the human body skeleton structure can be constructed based on the human body key points, and whether the human body posture in the picture is not good can be determined based on the human body skeleton structure, and pictures with poor human body posture can be filtered.

[0171] Step 516, input the second image set consisting of the images output by the face detection model and the images output by the body detection model into the face segmentation model. The face segmentation model can determine the unobstructed face area, thereby filtering out images in which the face is obstructed too much, thereby obtaining a third image set.

[0172] Step 518: extracting facial features from the facial regions segmented from each picture in the third picture set, and aggregating pictures with the same facial features into a group of picture sets.

[0173] Step 520, create a new target album, and add the pictures aggregated into a set of pictures to the target album for user management.

[0174] In the scheme recorded in the above embodiment, pictures that do not meet the requirements can be filtered out by detecting key points and segmenting face areas of pictures. For the filtered pictures, facial features are further extracted from the segmented face areas, so that pictures with the same facial features are aggregated into a group of pictures. In this way, the pictures to be classified can be automatically classified efficiently, thereby improving the efficiency of picture management and user experience.

[0175] The above needs to be explained Figure 5 This is only a schematic diagram of the image processing flow for the target object of human face. In fact, when the target object is other types such as objects, animals, plants, etc., it is possible to classify other types of pictures such as objects, animals, plants, etc. by simply adjusting the targets of the target object detection model and the segmentation model.

[0176] See also Figure 6 , Figure 6 It is a hardware structure diagram of an electronic device in which an image processing device is located, which is shown as an exemplary embodiment. At the hardware level, the device includes a processor 602, an internal bus 604, a network interface 606, a memory 608, and a non-volatile memory 610, and of course may also include hardware required for other services. One or more embodiments of this specification can be implemented based on software, such as the processor 602 reading the corresponding computer program from the non-volatile memory 610 into the memory 608 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0177] See also Figure 7 , Figure 7 FIG. 1 is a block diagram of an image processing device according to an exemplary embodiment. The image processing device can be applied to Figure 6 In the electronic device shown in the figure, the technical solution of this specification is implemented. Among them, the above-mentioned image processing device may include:

[0178] The picture acquisition unit 700 is used to acquire a first picture set to be classified;

[0179] A first filtering unit 702 is used to detect key points of the pictures in the first picture set, and filter the pictures that do not meet the preset key point conditions to obtain a second picture set that meets the key point conditions;

[0180] A second filtering unit 704 is used to segment the pictures in the second picture set into target object areas, and filter the pictures that do not meet the preset target object conditions to obtain a third picture set that meets the preset target object conditions;

[0181] A feature extraction unit 706 is used to extract target object features from the target object regions segmented from each picture of the third picture set;

[0182] The picture aggregation unit 708 is used to aggregate the pictures with the same target object features into a group of pictures.

[0183] In an exemplary embodiment, the first filtering unit 702 includes:

[0184] an identification subunit, configured to input each picture in the first picture set into a pre-trained target object detection model, so that the target object detection model identifies a target object region in each picture, and determines a target object key point contained in the picture in the target object region;

[0185] A determination subunit, used to determine whether the image meets a preset key point condition based on the key point of the target object;

[0186] The first filtering subunit is used to filter out pictures that do not meet the preset key point conditions from the first picture set to obtain a second picture set.

[0187] In an exemplary embodiment, the determining subunit includes:

[0188] A first determination subunit is configured to obtain the area of ​​the target object region corresponding to each image in the first image set output by the target object detection model, and / or the number of key points of the target object; and determine whether the image meets a preset key point condition based on the area or the number of key points of the target object;

[0189] The second determination subunit is used to determine whether the image meets the preset key point conditions based on the target object form formed by the target object key points.

[0190] In an exemplary embodiment, before the identification subunit, the device further includes:

[0191] A scene classification subunit, configured to input each picture in the first picture set into a pre-trained scene classification model, so that the scene classification model can identify the picture scene to which each picture belongs;

[0192] A first acquisition subunit is used to acquire the picture scene output by the scene classification model and to which each picture in the first picture set belongs, and filter pictures that do not conform to a target picture scene; wherein the target picture scene is a picture scene including a target object;

[0193] The human body detection subunit is further used to input the remaining pictures in the first picture set that do not conform to the target picture scene into a pre-trained target object detection model.

[0194] In an exemplary embodiment, the apparatus is applied to a mobile device; the scene classification model is a classification model constructed based on a lightweight neural network that can run on the mobile device.

[0195] In an exemplary embodiment, the lightweight neural network includes a Ghostnetv2 network.

[0196] In an exemplary embodiment, the second filtering unit 704 includes:

[0197] a segmentation subunit, configured to input each picture in the second picture set into a pre-trained target object segmentation model, so as to determine, by the target object segmentation model, an unobstructed area in the target object area corresponding to each picture in the second picture set;

[0198] a second acquisition subunit, configured to acquire an unobstructed area in the target object region corresponding to each image in the second image set output by the target object segmentation model, and determine an image whose ratio of the area of ​​the unobstructed area to the area of ​​the target object region is less than a preset ratio threshold as an image that does not meet the preset target object condition;

[0199] The second filtering subunit is used to filter out the pictures that do not meet the preset target object condition from the second picture set to obtain a third picture set.

[0200] In an exemplary embodiment, the second acquisition subunit is further used to obtain unobstructed areas in the target object area corresponding to each picture in the second picture set output by the target object segmentation model, and determine a first number of target object key points in the unobstructed area, and a second number of target object key points in the target object area; and, calculate a ratio of the first number to the second number, and determine a picture whose ratio is less than a preset ratio threshold as a picture that does not meet the preset target object condition.

[0201] In an exemplary embodiment, the picture aggregation unit 708 includes:

[0202] Create a new subunit, and create a new target album for storing pictures with the same characteristics of the target object;

[0203] An adding subunit is used to add pictures with the same characteristics as the target object to the target album.

[0204] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0205] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. Ordinary technicians in this field can understand and implement it without paying creative work.

[0206] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0207] This specification also provides an embodiment of a computer-readable storage medium. The computer-readable storage medium stores machine-readable instructions, and when the machine-readable instructions are called and executed by a processor, the image processing method provided in any embodiment of this specification can be implemented.

[0208] The computer-readable storage medium provided in the embodiments of this specification may specifically include, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical fiber card. That is, the readable storage medium includes a readable medium that can store or transmit information.

[0209] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0210] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0211] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0212] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0213] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0214] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0215] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "one", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0216] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0217] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. An image processing method, the method comprising: Obtain a first set of pictures to be classified; Detect key points of the pictures in the first picture set, and filter out pictures that do not meet the preset key point conditions to obtain a second picture set that meets the key point conditions; Segmenting the target object regions of the pictures in the second picture set, and filtering the pictures that do not meet the preset target object conditions, to obtain a third picture set that meets the preset target object conditions; Extracting target object features from target object regions segmented from each picture of the third picture set; Aggregate the pictures with the same or similar target object features into a set of pictures.

2. According to the method of claim 1, the detecting of key points of the pictures in the first picture set comprises: Inputting pictures in the first picture set into a pre-trained target object detection model, so that the target object detection model identifies target object regions in each picture, and determines target object key points contained in the picture in the target object region; According to the target object key points, determine whether the image meets the preset key point conditions.

3. The method according to claim 2, wherein determining whether the image meets a preset key point condition based on the target object key point comprises: Obtaining the area of ​​the target object region corresponding to the picture in the first picture set output by the target object detection model, and / or the number of key points of the target object; Based on the area or the number of key points of the target object, determining whether the image meets a preset key point condition; or, Based on the target object shape formed by the target object key points, determine whether the image meets the preset key point conditions.

4. The method according to claim 2, before inputting each picture in the first picture set into a pre-trained target object detection model, the method further comprises: Inputting each picture in the first picture set into a pre-trained scene classification model, so that the scene classification model identifies the picture scene to which each picture belongs; Obtain the picture scenes output by the scene classification model and to which each picture in the first picture set belongs, and obtain pictures that meet the target picture scene; wherein the target picture scene is a picture scene including a target object; The step of inputting each picture in the first picture set into a pre-trained human body detection model includes: The pictures in the first picture set that match the target picture scene are input into a pre-trained target object detection model.

5. The method according to claim 4 is applied to a mobile device; the scene classification model is a classification model constructed based on a lightweight neural network that can run on a mobile device.

6. According to the method of claim 5, the lightweight neural network includes a Ghostnetv2 network.

7. The method according to claim 1, wherein the segmenting of the target object regions of the pictures in the second picture set and filtering the pictures that do not meet the preset target object conditions to obtain the third picture set that meets the preset target object conditions comprises: Inputting each picture in the second picture set into a pre-trained target object segmentation model, so as to determine, by the target object segmentation model, an unobstructed area in the target object area corresponding to each picture in the second picture set; Obtaining an unobstructed area in the target object region corresponding to each picture in the second picture set output by the target object segmentation model, and determining a picture whose ratio of the area of ​​the unobstructed area to the area of ​​the target object region is less than a preset ratio threshold as a picture that does not meet the preset target object condition; The pictures that do not meet the preset target object condition are filtered out from the second picture set to obtain a third picture set.

8. The method according to claim 7, obtaining an unobstructed area in the target object area corresponding to each image in the second image set output by the target object segmentation model, and determining an image whose ratio of the area of ​​the unobstructed area to the area of ​​the target object area is less than a preset ratio threshold as an image that does not meet the preset target object condition, comprising: Obtaining an unobstructed area in the target object region corresponding to each picture in the second picture set output by the target object segmentation model, and determining a first number of target object key points in the unobstructed area and a second number of target object key points in the target object region; A ratio of the first number to the second number is calculated, and pictures whose ratio is less than a preset ratio threshold are determined as pictures that do not meet preset target object conditions.

9. The method according to claim 1, aggregating the pictures with the same target object features into a set of pictures, comprising: Create a target album for storing pictures with the same characteristics as the target object; Add pictures with the same characteristics as the target object to the target album.

10. An image processing device, comprising: A picture acquisition unit, used to acquire a first picture set to be classified; A first filtering unit, configured to detect key points of the pictures in the first picture set, and filter the pictures that do not meet the preset key point conditions, to obtain a second picture set that meets the key point conditions; A second filtering unit, configured to segment the pictures in the second picture set into target object regions, and filter the pictures that do not meet the preset target object conditions, to obtain a third picture set that meets the preset target object conditions; a feature extraction unit, configured to extract target object features from target object regions segmented from each picture of the third picture set; The picture aggregation unit is used to aggregate the pictures with the same target object features into a group of picture sets.

11. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 9 by running the executable instructions.

12. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images