Image object recognition processing method, device and equipment, storage medium and product

Abstract

Embodiments of the present application disclose a kind of image object identification processing method, device and equipment, storage medium, product, the method comprises: the target image obtained is carried out object area identification processing, obtain in the predicted area in the target image;Call image object identification model to the sub-image area in the predicted area carries out image object identification, determine whether each sub-image area belongs to the image object to be identified;According to the identification result of image object identification, determine the image object area in the target image, wherein the identification result includes: the result information for recording the sub-image area belonging to the image object to be identified.Using the present application, the missed detection and false detection of image object area can be reduced, the efficiency of identifying image object from image is improved, the overall scheme is feasible, low in cost and high in precision.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a method, apparatus, intelligent device, storage medium, and computer program product for recognizing and processing image objects. Background Technology

[0002] With the rapid popularization of deep learning technology and the improvement of computing power of smart devices, the performance of semantic segmentation technology has been greatly improved, and its application is becoming increasingly widespread. In the fields of beauty and makeup, photo editing, and changing hairstyles or dyeing hair, the demand is increasing day by day. In order to find the image object to be processed in post-processing, using deep learning technology for semantic segmentation of the entire image would undoubtedly bring huge computational load and false detection of irrelevant regions. Summary of the Invention

[0003] This application provides a method, apparatus, intelligent device, storage medium, and computer program product for recognizing and processing image objects, which can improve the efficiency of recognizing image objects from images.

[0004] On the one hand, this application provides a method for recognizing and processing image objects, including:

[0005] The acquired target image is subjected to object region recognition processing to obtain the predicted region in the target image;

[0006] The image object recognition model is invoked to perform image object recognition on the sub-image regions in the prediction region to determine whether each sub-image region belongs to the image object to be recognized.

[0007] The image object region in the target image is determined based on the recognition result of the image object recognition, wherein the recognition result includes result information for recording sub-image regions belonging to the image object to be recognized.

[0008] On the other hand, this application also provides an image object recognition processing apparatus, comprising:

[0009] The prediction module is used to perform object region recognition processing on the acquired target image to obtain the predicted region in the target image;

[0010] The recognition module is used to call the image object recognition model to perform image object recognition on the sub-image regions in the prediction region, and determine whether each sub-image region belongs to the image object to be recognized;

[0011] The determination module is used to determine the image object region in the target image based on the recognition result of image object recognition, wherein the recognition result includes result information for recording sub-image regions belonging to the image object to be recognized.

[0012] In another aspect, this application also provides an intelligent device, including a storage device and a processor, wherein the storage device stores program instructions, and the processor invokes the program instructions to implement an image object recognition processing method.

[0013] Accordingly, this application also provides a computer-readable storage medium storing program instructions, which, when executed, implement a method for recognizing and processing image objects.

[0014] This application first detects the approximate area of ​​image objects such as hair, and then performs fine detection of that area. On the one hand, it can effectively reduce false detections of image object areas and reduce the detection workload. On the other hand, fine detection of local areas can also improve the accuracy of identifying image objects from the image, thereby greatly improving the efficiency of image object detection. Moreover, the overall solution is highly feasible, low in cost, and high in accuracy. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating an image object recognition processing method according to an embodiment of the present invention;

[0017] Figure 2a This is a schematic diagram showing the dimensions of the target image according to an embodiment of the present invention;

[0018] Figure 2b This is a schematic diagram of relevant image region parameters according to an embodiment of the present invention;

[0019] Figure 3 This is a schematic diagram of the masking layer for the hair object according to an embodiment of the present invention;

[0020] Figure 4 This is a schematic diagram of an adjustment method for adjusting the position of an initial region in a target image according to an embodiment of the present invention;

[0021] Figure 5 This is a schematic diagram of the initial region and the predicted region after the height value is expanded according to an embodiment of the present invention;

[0022] Figure 6 This is a schematic diagram illustrating the adjustment of the initial region according to an embodiment of the present invention;

[0023] Figure 7This is a schematic diagram of the structure of the image object recognition and processing device according to an embodiment of the present invention;

[0024] Figure 8 This is a schematic diagram of the structure of a smart device according to an embodiment of the present invention. Detailed Implementation

[0025] For the acquired target images, users can perform various processing as needed. For example, they can identify facial features such as eyes, nose, and mouth, apply beauty filters or add decorative elements. Or, they can identify features like hair in the target image and perform image processing such as coloring. For instance, for hair in the target image, the hair region can be segmented and extracted separately, and then specially processed, such as dyeing, changing hairstyles, or adding hair accessories. Specifically, the target image can first be segmented into regions, and then the region containing the user's hair can be dyed red based on the segmentation results. In some embodiments, after processing, these target images can be uploaded to social media platforms and shared with other users via a smart terminal. If these target images are captured during video communication between users, after processing, the processed video footage can be displayed on the video communication interface of both parties or the other party, showing the video screen with the image object processing described above.

[0026] The target image can be acquired by the user through a smart terminal such as a smartphone or tablet, or it can be extracted from storage or downloaded from the network. Processing of these target images can be performed on a smart terminal with a corresponding application app installed. For example, when a user takes a photo with their smartphone, they can process the captured target image, or process the target image used for preview during the shooting preview stage so that the processed image can be directly captured when the camera clicks to shoot. Alternatively, the image can be uploaded to a server via the smart terminal, where the server performs the corresponding image processing. For example, during a video call, the video communication server automatically processes the transmitted video image frames (target images) based on user actions to display the processed video image on the other party's communication interface, such as displaying a video image of dyed hair.

[0027] The processing of the target image in this embodiment of the invention includes multiple stages, specifically including object region recognition processing of the target image, region prediction, image object recognition processing within the predicted region, and finally determining the region where the image object is located based on the recognition processing result. In one implementation, more specifically, after acquiring the target image, a rough estimation of the target image region is first performed to obtain the image location region of the image object to be detected, i.e., the initial region. After the initial region is roughly estimated and determined, it is expanded based on the initial region by stretching and enlarging the rectangular region corresponding to the initial region to obtain the final prediction region. Then, the image object to be detected is identified and determined in the prediction region, and the region where the image object is located is determined. For example, the region of hair in the target image mentioned above. In this embodiment, by processing the target object's location region twice, a more accurate image region to be identified (corresponding to the prediction region mentioned above) can be obtained, which can better reduce the missed detection and false detection of image objects. Furthermore, identifying and determining the region of the image object to be detected in the final determined prediction region can improve the efficiency of detecting image objects such as hair, and also greatly facilitate subsequent image processing in the target region. For example, after determining the location region of hair in the target image, the pixels in this location region can be assigned values ​​to achieve the purpose of "hair dyeing" processing, or a mask can be displayed over the location region of hair to conveniently complete the masking processing.

[0028] Please see Figure 1 The diagram shown is a flowchart illustrating an image object recognition and processing method according to an embodiment of the present invention. This method can be applied to various image processing scenarios, such as social applications that allow image sharing, video call applications, video conferencing systems, and some graphic design and image editing applications. The method can be executed by a smart terminal with the corresponding application installed, or by a powerful image processing server. The method of this embodiment includes the following steps.

[0029] Users with image editing needs (such as beauty and makeup enhancement) can obtain target images containing a specific object they want to detect and adjust through various means, such as taking pictures of people with a smartphone, retrieving task images from memory, or downloading task images from the network. After loading and obtaining the target image, object region recognition processing is performed on the obtained target image in S101 to obtain the predicted region in the target image. There are various methods for performing object region recognition processing. In one embodiment, an initial region can be roughly identified based on the user's clicked location or a selected area.

[0030] In one embodiment, an image object prediction model can be invoked to perform a coarse full-image detection in the target image to determine a predicted region. The image object prediction model can directly obtain a predicted region as needed, or obtain regional geometric parameters of the predicted region, based on which a predicted region can be directly determined. Alternatively, S101 may also include: invoking the image object prediction model to perform region prediction on the target image to determine an initial region; adjusting the position of the initial region in the target image according to a region expansion rule to obtain the predicted region. That is, a coarse initial region is first obtained, and then a fine-tuning is performed based on this initial region to obtain a predicted region. The expansion rule is a supplementary processing to prevent the initial region from failing to encompass all the hair. In some cases, such as when hair is sparse or short, the initial region determined by the image object prediction model may cover all the hair. In this case, the initial region can be directly used as the prediction region. That is, after calling the image object prediction model to predict the region of the target image and determining the initial region, the position of the initial region can be displayed on the user interface, along with a region editing button and a confirmation button. If the confirmation button is selected, the initial region can be directly used as the prediction region. If the region editing button is selected, the position of the initial region in the target image will be adjusted according to the region expansion rules to obtain the prediction region.

[0031] In one implementation, the step of calling the image object prediction model to predict the region of the target image and determining the initial region may specifically include: calling the image object prediction model to predict the region of the target image and obtaining the region geometric feature parameters output by the image object prediction model after prediction; and determining the initial region based on the region geometric feature parameters.

[0032] In another embodiment, the face location region in the target image can be identified first. Relatively mature face recognition technology can be used to accurately determine the face's location, and this face region can be used as the initial region. Then, starting from the face location region, the predicted regions for image objects such as hair, eyes, or nose can be roughly estimated. For example, after confirming the initial region corresponding to the face location, the initial region can be edited according to the position of the eyes on the face to obtain the predicted region.

[0033] In another embodiment, face detection can be performed first. If a face is detected, the predicted region is estimated and determined based on the face location region determined by the face detection. If no face is detected, the image object prediction model mentioned above is called to predict the region of the target image so as to output the region geometric feature parameters of the predicted initial region to finally obtain the predicted region.

[0034] The image object prediction model can be built based on a neural network. It predicts a rectangular region corresponding to an initial region, which includes at least some or all of the image objects to be detected and identified. The model predicts and outputs geometric feature parameters of the region, which may include offset information and rectangular region-related parameters (height and width information). In one embodiment, the offset information includes the offset between the center of the rectangular region and the center of the target image; the height information includes the ratio of the height of the rectangular region to the height of the target image; and the width information includes the ratio of the width of the rectangular region to the width of the target image. In other embodiments, the geometric feature parameters may also be the image coordinates of the center position of the rectangular region, the height of the rectangular region, and the width of the rectangular region.

[0035] In one embodiment, to quickly predict regional geometric feature parameters such as offset, a neural network with an overall computational cost of less than 2M, consisting of convolutional layers, pooling layers, nonlinear layers, and fully connected layers, can be designed. An image object prediction model can then be trained based on this neural network. The predicted values ​​output by the neural network, or the image object prediction model, are the offset of the center point of the rectangular region corresponding to the image object (e.g., hair) relative to the image center (i.e., offset information), and the relative proportions of the rectangular region's length and width to the image's length and width (i.e., the proportion of the rectangular region's height to the target image's height, and the proportion of the rectangular region's width to the target image's width), which can be expressed as (Δsx, Δsy), Δsh, and Δsw, respectively, all of which are normalized values. Assuming the target image's center point and width / height are (cx, cy), w, and h, the calculation method for the rectangular center, width, and height of the initial region where the image object (e.g., hair) is located is as follows:

[0036]

[0037] Specifically, the offset information in the geometric feature parameters of the region is (Δsx, Δsy), the height information is Δsh, the width information is Δsw, the rectangular center of the initial region, or the rectangular image center, is (rx, ry), the height of the initial region is rh, and the width of the initial region is rw. The width of the target image is w, the height of the target image is h, and the image center coordinates of the target image are (cx, cy). For a detailed illustration of the target image, the initial region, and the geometric feature parameters of the initial region, please refer to [the relevant documentation / reference]. Figure 2a , Figure 2b As shown. The original input image is... Figure 2a As shown, the rectangular region and related parameters determined after inputting it into the neural network, i.e., the image object prediction model, are as follows: Figure 2b As shown. Figure 2bThe dashed rectangle in the image represents the initial region of the target image.

[0038] The image object prediction model described above can be trained on a set of positive sample images containing objects such as hair and a set of negative sample images without objects such as hair. The trained image object prediction model can then be used to make a rough prediction of the region and output the initial region's geometric feature parameters.

[0039] In one embodiment, after determining the initial region, the aforementioned region expansion rule is used to adjust the position of the initial region in the target image to obtain the predicted region. The region expansion rule is mainly used to adjust the size of the initial region to determine a more suitable region containing the image object, i.e., the predicted region. In one embodiment, the size of the initial region can be adjusted based on its regional geometric feature parameters to obtain the predicted region. For example, the predicted region is obtained by increasing the height by 20% or other values, and the width by 10% or other values, based on the initial region. A detailed description of the adjustment process from the initial region to the predicted region can be found in the following description. Figure 2b The solid-line rectangle in the image represents the prediction region.

[0040] After determining the prediction region, in S102, an image object recognition model is invoked to perform image object recognition on the sub-image regions within the prediction region, determining whether each sub-image region belongs to the image object to be recognized. The image object recognition model can be a dedicated model specifically designed to recognize a particular image object, such as a model specifically designed to recognize hair objects in an image, or a model specifically designed to recognize eye objects in an image. In one embodiment, the sub-image region is the region containing a pixel within the prediction region; invoking the image object recognition model to perform image object recognition on the sub-image regions within the prediction region, determining whether each sub-image region belongs to the image object to be recognized, may include: invoking the image object recognition model to perform image object recognition on each pixel within the prediction region, determining the probability that each pixel in the prediction region belongs to the image object to be recognized; wherein pixels with a probability value greater than or equal to a preset threshold belong to the image object to be recognized; and the recognition result of the image object recognition is a probability map corresponding to the prediction region determined based on the obtained probabilities. Obtaining the probability map facilitates subsequent determination of the image object region within the prediction region based on the probability map.

[0041] Image object recognition models can also be built based on neural networks. These models segment the image objects, such as hair, into regions within the prediction area. The neural network for this model can consist of dilated convolutional layers, pooling layers, nonlinear layers, and upsampling layers. Inputting a rectangular region from the target image (the prediction region) into the model yields a probability map of the same size as the prediction region. The probability map ranges from 0 to 1, with each point on the map corresponding to a pixel (i.e., a sub-image region). The probability value on the map indicates the probability that the corresponding pixel belongs to the image object to be identified (e.g., hair). When the probability is greater than or equal to a preset probability threshold, the sub-image region, i.e., the corresponding pixel, is considered to belong to the image object to be identified (e.g., the pixel is one of the pixels of the hair image object). When the probability is less than the preset probability threshold, the sub-image region, i.e., the corresponding pixel, is considered not to belong to the image object to be identified. Training the image object recognition model can also be done using a training set of images containing images of hair or other image objects. A loss function can be established using cross-entropy during training, as shown in the following formula:

[0042] L = -logP i Formula 2

[0043] Where P i Let be the probability that the i-th pixel is predicted to be an image object such as hair.

[0044] In S103, the image object region in the target image is determined based on the recognition result of the image object recognition. The recognition result includes result information for recording sub-image regions belonging to the image object to be recognized. After obtaining the probability map, probability values ​​less than a preset probability threshold are set to 0, and probability values ​​greater than or equal to the preset probability threshold are set to 1, resulting in a binary image of the same size as the predicted region. Based on the predicted region, it is mapped to the target image. The image object region in the target image is determined based on the points with a value of 1 in the binary image, forming the final mask layer for image object regions such as hair. Figure 3 It corresponds to Figure 2a A schematic diagram of the hair mask in the target image, such as... Figure 3 As shown, by adjusting parameters such as the color of the mask layer, one can achieve the purpose of coloring image objects such as hair.

[0045] Please see again Figure 4 This is a schematic diagram of an adjustment method for adjusting the position of an initial region in a target image according to an embodiment of the present invention. This adjustment method corresponds to S101 described above. The method can still be implemented by a smart terminal or a server.

[0046] S400: Determine whether the area of ​​the initial region determined based on the region's geometric feature parameters satisfies the area condition. The area of ​​the initial region can be calculated based on the length and width of the rectangular region of the initial region, and the length and width of the rectangular region can be calculated based on the region's geometric feature parameters.

[0047] S401: When the initial region area determined according to the region geometric feature parameters meets the area condition, the second updated geometric parameters are determined according to the geometric parameters of the target image; wherein, the initial region area meeting the area condition includes: the ratio between the initial region area and the area of ​​the target image is greater than a preset area ratio threshold. Alternatively, the initial region area meeting the area condition includes: the initial region area is greater than a times the area of ​​the target image, where a can be, for example, 0.8 or 0.9. When the area condition is met, the target image captured this time can be considered mainly a head image, and the size of the target image can be directly used as the size included in the second updated geometric parameters, and the entire image region of the target image can be used as the prediction region. Alternatively, b times the size of the target image can be used as the size included in the second updated geometric parameters, for example, b can be 0.9 or other values, the width of the target image * 0.9 can be used as the width included in the second updated geometric parameters, and the height of the target image * 0.9 can be used as the height included in the second updated geometric parameters.

[0048] S402: The position of the initial region in the target image is adjusted according to the second updated geometric parameters to obtain the predicted region. As described in S401, the predicted region obtained after adjustment can be the entire target image region, or 0.9 times the size of the entire target image region, etc.

[0049] S403: When the initial region area determined based on the region geometric feature parameters does not meet the area condition, select the corresponding region expansion rule based on the region geometric feature parameters of the initial region.

[0050] S404: Determine the first updated geometric parameters according to the region expansion rules. When the area condition is not met, different region expansion rules can be selected based on the region's geometric characteristic parameters to determine the first updated geometric parameters according to different region expansion rules.

[0051] S405: Adjust the position of the initial region in the target image according to the first updated geometric parameters to obtain the predicted region. In this embodiment of the invention, the predicted region obtained after adjusting the initial region according to the first updated geometric parameters is different from the predicted region obtained after adjusting the initial region according to the second updated geometric parameters. The area or size of the predicted region obtained according to the second updated geometric parameters is larger than the area or size of the predicted region obtained according to the first updated geometric parameters.

[0052] In one embodiment, in step S403, selecting a corresponding region expansion rule based on the region geometric feature parameters of the initial region includes: calculating the width value of the initial region based on the region geometric feature parameters of the initial region, and calculating the height value of the initial region based on the region geometric feature parameters of the initial region. The calculation method can refer to Formula 1 above. If the width value is greater than N times the height value, then the first region expansion rule is selected, where N is a positive number greater than 1. In one embodiment, N can be equal to 1.5 or other values, that is, when the width of the initial region is 1.5 times the height, the first region expansion rule is selected.

[0053] In step S404, determining the first updated geometric parameter according to the region expansion rule includes: using the width value of the initial region as the width value of the first updated geometric parameter according to the first region expansion rule; and using the updated height value obtained by expanding the height value of the initial region by a first preset ratio as the height value of the first updated geometric parameter according to the first region expansion rule. The updated height value is greater than a first height threshold and less than a second height threshold. The second height threshold can be, for example, the height value of the target image or P times the height value of the target image, where P is, for example, 0.8 or 0.9. Under the first region expansion rule, the width value of the initial region is the same as the width value of the predicted region, but it is expanded vertically by a first preset ratio. This first preset ratio can be, for example, 20%. For instance, when the height value of the initial region is 100 pixels, the height value of the first updated geometric parameter corresponding to the predicted region is 140 pixels, meaning it is expanded upwards by 20 pixels and downwards by 20 pixels. The pixel values ​​in this example are merely illustrative.

[0054] In one embodiment, in step S403, selecting a corresponding region expansion rule based on the region geometric feature parameters of the initial region includes: calculating the width value of the initial region based on the region geometric feature parameters of the initial region, and calculating the height value of the initial region based on the region geometric feature parameters of the initial region. The calculation method can refer to Formula 1 above. If the width value is greater than the height value, but the width value is less than or equal to N times the height value, then the second region expansion rule is selected, where N is a positive number greater than 1. That is, if the width value is greater than the height value, but the width value is less than 1.5 times the height value (or N is another value), then the second region expansion rule is selected.

[0055] In step S404, determining the first updated geometric parameter according to the region expansion rule includes: according to the second region expansion rule, expanding the height value of the initial region by a second preset ratio to obtain an updated height value as the height value of the first updated geometric parameter, wherein the updated height value is greater than a first height threshold and less than a second height threshold, the second height threshold being, for example, the height value of the target image or P times the height value of the target image, where P is, for example, a value such as 0.8 or 0.9; and according to the second region expansion rule, expanding the width value of the initial region by a third preset ratio to obtain an updated width value as the width value of the first updated geometric parameter, wherein the updated width value is greater than a first width threshold and less than a second width threshold, the second width threshold being, for example, the width value of the target image or P times the width value of the target image, where P is, for example, a value such as 0.8 or 0.9. Under the second region expansion rule, the height value of the initial region is expanded vertically by a second preset ratio, such as 10%, while updating the height value to be greater than a first height threshold (e.g., 0) and less than a second height threshold. The second height threshold can be, for example, the height value of the target image or P times the height value of the target image, where P is, for example, a value of 0.8 or 0.9. The width value w of the initial region is expanded horizontally by a third preset ratio, such as 5%, while updating the width value to be greater than a first width threshold (e.g., 0) and less than a second width threshold. The second width threshold can be, for example, the width value of the target image or P times the width value of the target image, where P is, for example, a value of 0.8 or 0.9. For example, when the initial region has a height of 100 pixels, the height of the first updated geometric parameter of the predicted region is 120 pixels, which means it expands upwards by 10 pixels and downwards by 10 pixels in the height direction; when the initial region has a width of 100 pixels, the width of the first updated geometric parameter of the predicted region is 110 pixels, which means it expands to the left by 5 pixels and to the right by 5 pixels in the width direction. The pixel values ​​in this example are just for illustration.

[0056] In one embodiment, in step S403, selecting a corresponding region expansion rule based on the region geometric feature parameters of the initial region includes: calculating the width value of the initial region based on the region geometric feature parameters of the initial region, and calculating the height value of the initial region based on the region geometric feature parameters of the initial region, the calculation method can refer to Formula 1 above; if the height value is greater than M times the width value, then a third region expansion rule is selected, where M is a positive number greater than 1; in one embodiment, M can be equal to 1.5 or other values, that is, when the height of the initial region is 1.5 times the width, the third region expansion rule is selected.

[0057] In step S404, determining the first updated geometric parameter according to the region expansion rule includes: according to the third region expansion rule, using the height value of the initial region as the height value of the first updated geometric parameter, and according to the third region expansion rule, expanding the width value of the initial region by a fourth preset ratio to obtain the updated width value as the width value of the first updated geometric parameter. The updated width value is greater than a first width threshold and less than a second width threshold. The second width threshold can be, for example, the width value of the target image or P times the width value of the target image, where P is, for example, 0.8 or 0.9. Under the third expansion rule, the height value of the predicted region is the same as the height value of the initial region, but the width value of the initial region is expanded by a fourth preset ratio. The fourth preset ratio can be, for example, 20%. For example, when the width value of the initial region is 100 pixels, the width value of the first updated geometric parameter corresponding to the predicted region is 140 pixels, that is, it expands 20 pixels to the left and 20 pixels to the right in the width direction. The pixel values ​​in this example are only illustrative.

[0058] In one embodiment, in step S403, selecting a corresponding region expansion rule based on the region geometric feature parameters of the initial region includes: calculating the width value of the initial region based on the region geometric feature parameters of the initial region, and calculating the height value of the initial region based on the region geometric feature parameters of the initial region. The calculation method can refer to Formula 1 above. If the height value is greater than the width value and the height value is less than or equal to M times the width value, then the fourth region expansion rule is selected, where M is a positive number greater than 1. M can be, for example, 1.5 or other values. That is, when the height value is greater than the width value, but the height value is less than or equal to 1.5 times the width value, the fourth region expansion rule can be selected.

[0059] In step S404, determining the first updated geometric parameter according to the region expansion rule includes: according to the fourth region expansion rule, expanding the height value of the initial region by a fifth preset ratio to obtain an updated height value as the height value of the first updated geometric parameter, wherein the updated height value is greater than a first height threshold and less than a second height threshold, and the second height threshold may be, for example, the height value of the target image or P times the height value of the target image, where P is, for example, a value such as 0.8 or 0.9; and according to the fourth region expansion rule, expanding the width value of the initial region by a sixth preset ratio to obtain an updated width value as the width value of the first updated geometric parameter, wherein the updated width value is greater than a first width threshold and less than a second width threshold, and the second width threshold may be, for example, the width value of the target image or P times the width value of the target image, where P is, for example, a value such as 0.8 or 0.9. Based on the fourth region expansion rule, the height value h of the initial region is expanded vertically by a fifth preset ratio, such as 5%, while updating the height value to be greater than a first height threshold (e.g., 0) and less than a second height threshold. The second height threshold can be, for example, the height value of the target image or P times the height value of the target image, where P is, for example, a value of 0.8 or 0.9. The width value w of the initial region is expanded horizontally by a sixth preset ratio, such as 10%, while updating the width value to be greater than a first width threshold (e.g., 0) and less than a second width threshold. The second width threshold can be, for example, the width value of the target image or P times the width value of the target image, where P is, for example, a value of 0.8 or 0.9. For example, when the initial region has a height of 100 pixels, the height of the first update set parameter of the predicted region is 110 pixels, which means it expands upwards by 5 pixels and downwards by 5 pixels in the height direction; when the initial region has a width of 100 pixels, the width of the first update geometry parameter of the predicted region is 120 pixels, which means it expands to the left by 10 pixels and to the right by 10 pixels in the width direction. The pixel values ​​in this example are just for illustration.

[0060] like Figure 5 As shown, the dashed rectangle represents the initial region, while the solid rectangle represents the predicted region obtained by expanding vertically and horizontally. The predicted region determined by the empirical values ​​in the example mentioned above can basically ensure that all image objects are within this region. Although there may be some redundant regions where no hair objects exist, compared to performing hair recognition on the entire target image, it can greatly reduce false detections, reduce the amount of recognition computation, save hardware and software resources, and relatively ensure the accuracy of recognition.

[0061] Please see again Figure 6 ,exist Figure 6 The text intuitively presents the methods for expanding and adjusting the initial region. Figure 6The diagram above Figure 4 The aforementioned adjustment process corresponds to the following: When the initial region area is greater than 0.8 times the area of ​​the target image, the size of the target image is directly used as the size of the predicted region, and the target image is used as the predicted region for image object recognition. When the initial region area is less than or equal to 0.8 times the area of ​​the target image, it is determined whether the width value of the initial region is greater than 1.5 times the height value of the initial region. If so, the height value of the initial region is expanded vertically by 20%. If the width value of the initial region is less than or equal to 1.5 times the height value of the initial region, it is further determined whether the width value of the initial region is greater than the height value. If so, the height value of the initial region is expanded vertically by 10%, but the expanded height value is greater than 0 and less than the height value of the target image. The width value of the initial region is expanded horizontally by 5%, but the expanded width value is greater than 0 and less than the width value of the target image. If the width value of the initial region is less than or equal to the height value, it is further determined whether the height value of the initial region is greater than 1.5 times the width value. If so, the width value of the initial region is expanded horizontally by 20%. If the height of the initial region is less than or equal to 1.5 times the width, the height of the initial region is expanded vertically by 5%, but the expanded height is greater than 0 and less than the height of the target image. The width of the initial region is expanded horizontally by 10%, but the expanded width is greater than 0 and less than the width of the target image.

[0062] This application employs a strategy of first detecting the approximate initial region where image objects such as hair are located, and then performing a fine expansion of the initial region. This strategy can effectively reduce the missed detections and false detections of image object regions, improve the efficiency of identifying image objects from images, and the overall solution is highly feasible, low in cost, and high in accuracy.

[0063] Please see again Figure 7 This is a schematic diagram of the structure of an image object recognition and processing device according to an embodiment of the present invention. The device can be installed in a smart device, such as a smartphone, tablet computer, or personal computer. The smart device can also be a server, which can be a dedicated image object processing server, or a social platform server or communication application server. In this embodiment, the device includes a prediction module 701, a recognition module 702, and a determination module 703.

[0064] The prediction module 701 is used to perform object region recognition processing on the acquired target image to obtain a predicted region in the target image. The predicted region is obtained by adjusting the size of the initial region based on the regional geometric feature parameters of the initial region. The recognition module 702 is used to call an image object recognition model to perform image object recognition on the sub-image regions in the predicted region and determine whether each sub-image region belongs to the image object to be recognized. The determination module 703 is used to determine the image object region in the target image based on the recognition result of the image object recognition. The recognition result includes result information for recording the sub-image regions belonging to the image object to be recognized. The device also includes a module for setting a mask layer on the image object region and adjusting the parameters of the mask layer to adjust the image objects in the image object region.

[0065] In one possible implementation, the prediction module 701 is used to call an image object prediction model to perform region prediction on the target image to determine an initial region; and adjust the position of the initial region in the target image according to a region expansion rule to obtain a predicted region.

[0066] In one possible implementation, the prediction module 701 is used to call an image object prediction model to perform region prediction on the target image and obtain the region geometric feature parameters output by the image object prediction model after prediction; determine an initial region based on the region geometric feature parameters; the initial region includes a rectangular region, and the region geometric feature parameters include: offset information, height information, and width information; the offset information includes: the offset between the center of the rectangular region and the center of the target image; the height information includes: the ratio of the height of the rectangular region to the height of the target image; and the width information includes: the ratio of the width of the rectangular region to the width of the target image.

[0067] In one possible implementation, the prediction module 701 is configured to select a corresponding region expansion rule based on the regional geometric feature parameters of the initial region; determine a first updated geometric parameter according to the region expansion rule; and adjust the position of the initial region in the target image according to the first updated geometric parameter to obtain an adjusted predicted region.

[0068] In one possible implementation, the prediction module 701 is configured to: determine a second updated geometric parameter based on the geometric parameter of the target image when the initial region area determined according to the region geometric feature parameter satisfies the area condition; adjust the position of the initial region in the target image according to the second updated geometric parameter to obtain an adjusted predicted region; and select a corresponding region expansion rule based on the region geometric feature parameter of the initial region when the initial region area determined according to the region geometric feature parameter does not satisfy the area condition.

[0069] In one possible implementation, the initial region area satisfies the area condition by: the ratio between the initial region area and the area of ​​the target image is greater than a preset area ratio threshold.

[0070] In one possible implementation, the device further includes a module for identifying face location regions in the target image and using the face location regions as initial regions; or a module for performing face detection on the target image, estimating and determining a predicted region based on the face location regions determined by face detection if a face location region is detected, and calling an image object prediction model to predict regions in the target image in order to obtain the initial region if no face location region is detected.

[0071] In one possible implementation, the prediction module 701 is configured to calculate the width value of the initial region based on the regional geometric feature parameters of the initial region, and calculate the height value of the initial region based on the regional geometric feature parameters of the initial region; if the width value is greater than N times the height value, then a first region expansion rule is selected, where N is a positive number greater than 1; according to the first region expansion rule, the width value of the initial region is used as the width value of the first updated geometric parameter; according to the first region expansion rule, the updated height value obtained by expanding the height value of the initial region by a first preset ratio is used as the height value of the first updated geometric parameter, wherein the updated height value is greater than a first height threshold and less than a second height threshold.

[0072] In one possible implementation, the prediction module 701 is configured to calculate the width value of the initial region based on the regional geometric feature parameters of the initial region, and calculate the height value of the initial region based on the regional geometric feature parameters of the initial region; if the width value is greater than the height value, but the width value is less than or equal to N times the height value, then a second region expansion rule is selected, where N is a positive number greater than 1; according to the second region expansion rule, the updated height value obtained by expanding the height value of the initial region by a second preset ratio is used as the height value of the first updated geometric parameter, wherein the updated height value is greater than a first height threshold and less than a second height threshold; according to the second region expansion rule, the updated width value obtained by expanding the width value of the initial region by a third preset ratio is used as the width value of the first updated geometric parameter, wherein the updated width value is greater than a first width threshold and less than a second width threshold.

[0073] In one possible implementation, the prediction module 701 is configured to calculate the width value of the initial region based on the regional geometric feature parameters of the initial region, and calculate the height value of the initial region based on the regional geometric feature parameters of the initial region; if the height value is greater than M times the width value, then a third region expansion rule is selected, where M is a positive number greater than 1; according to the third region expansion rule, the height value of the initial region is used as the height value of the first updated geometric parameter; according to the third region expansion rule, the updated width value obtained by expanding the width value of the initial region by a fourth preset ratio is used as the width value of the first updated geometric parameter, wherein the updated width value is greater than a first width threshold and less than a second width threshold.

[0074] In one possible implementation, the prediction module 701 is configured to calculate the width value of the initial region based on the regional geometric feature parameters of the initial region, and calculate the height value of the initial region based on the regional geometric feature parameters of the initial region; if the height value is greater than the width value and the height value is less than or equal to M times the width value, then a fourth region expansion rule is selected, where M is a positive number greater than 1; according to the fourth region expansion rule, the updated height value obtained by expanding the height value of the initial region by a fifth preset ratio is used as the height value of the first updated geometric parameter, wherein the updated height value is greater than a first height threshold and less than a second height threshold; according to the fourth region expansion rule, the updated width value obtained by expanding the width value of the initial region by a sixth preset ratio is used as the width value of the first updated geometric parameter, wherein the updated width value is greater than a first width threshold and less than a second width threshold.

[0075] In one possible implementation, the sub-image region is the region where a pixel in the initial region is located; the recognition module 702 is used to call an image object recognition model to perform image object recognition on each pixel in the prediction region, and determine the probability that each pixel in the prediction region belongs to the image object to be recognized; wherein, pixels with a probability value greater than or equal to a preset threshold belong to the image object to be recognized; the recognition result of the image object recognition is a probability map corresponding to the prediction region determined according to the obtained probability, so that the determining module 703 can determine the image object region within the prediction region according to the probability map.

[0076] The specific implementation of each module in the embodiments of the present invention can be referred to the description of the relevant content in the foregoing method embodiments, and will not be repeated here. The strategy of first detecting the approximate initial region where image objects such as hair are located, and then performing fine expansion of the initial region, can effectively reduce the missed detection and false detection of image object regions, improve the efficiency of identifying image objects from images, and the overall solution is highly feasible, low in cost, and high in accuracy.

[0077] Please see again Figure 8 This is a schematic diagram of the structure of a smart device according to an embodiment of the present invention. The smart device in this embodiment can be a smart terminal such as a smartphone, tablet computer, or personal computer, or it can be a server. The smart device includes a power module and various housing structures. In this embodiment, the smart device includes a processor 801 and a storage device 802. It may also include a communication interface 803, a user interface 804, etc., as needed, to realize various data transmission and communication functions and meet the interaction needs with users.

[0078] The storage device 802 may include volatile memory, such as random-access memory (RAM); the storage device 802 may also include non-volatile memory, such as flash memory, solid-state drive (SSD), etc.; the storage device 802 may also include a combination of the above types of memory.

[0079] The processor 801 may be a central processing unit (CPU). The processor 801 may further include hardware chips. These hardware chips may be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), etc. PLDs may be field-programmable gate arrays (FPGAs), generic array logic (GALs), etc.

[0080] The storage device 802 is also used to store program instructions. The processor 801 can call the program instructions to implement the relevant steps of the image object recognition and processing method mentioned in the foregoing embodiments.

[0081] In one embodiment, the processor 801 calls program instructions stored in the storage device 802 to perform object region recognition processing on the acquired target image to obtain a predicted region in the target image. The predicted region is obtained by adjusting the size of the initial region based on the regional geometric feature parameters of the initial region. The processor then calls an image object recognition model to perform image object recognition on the sub-image regions in the predicted region to determine whether each sub-image region belongs to the image object to be recognized. Based on the recognition results of the image object recognition, the processor determines the image object region in the target image, sets a mask layer on the image object region, and adjusts the image object in the image object region by adjusting the parameters of the mask layer. The recognition results include result information for recording the sub-image regions belonging to the image object to be recognized.

[0082] In one embodiment, the processor 801 is configured to invoke an image object prediction model to perform region prediction on the target image, determine an initial region, and adjust the position of the initial region in the target image according to a region expansion rule to obtain a predicted region.

[0083] In one embodiment, the processor 801 is configured to invoke an image object prediction model to perform region prediction on the target image, and obtain the region geometric feature parameters output by the image object prediction model after prediction; determine an initial region based on the region geometric feature parameters; the initial region includes a rectangular region, and the region geometric feature parameters include: offset information, height information, and width information; the offset information includes: the offset between the center of the rectangular region and the center of the target image; the height information includes: the ratio of the height of the rectangular region to the height of the target image; and the width information includes: the ratio of the width of the rectangular region to the width of the target image.

[0084] In one embodiment, the processor 801 is configured to select a corresponding region expansion rule based on the region geometric feature parameters of the initial region; determine a first update geometric parameter according to the region expansion rule; and adjust the position of the initial region in the target image according to the first update geometric parameter to obtain a predicted region.

[0085] In one embodiment, the processor 801 is configured to: determine a second updated geometric parameter based on the geometric parameter of the target image when the initial region area determined according to the region geometric feature parameter satisfies the area condition; adjust the position of the initial region in the target image according to the second updated geometric parameter to obtain a predicted region; and when the initial region area determined according to the region geometric feature parameter does not satisfy the area condition, execute the step of selecting a corresponding region expansion rule based on the region geometric feature parameter of the initial region.

[0086] In one embodiment, the initial region area satisfies the area condition by: the ratio between the initial region area and the area of ​​the target image is greater than a preset area ratio threshold.

[0087] In one embodiment, the processor 801 is further configured to identify face location regions in the target image and use the face location regions as initial regions; or to perform face detection on the target image, and if a face location region is detected, to estimate and determine a predicted region based on the face location region determined by the face detection; if no face location region is detected, to call an image object prediction model to perform region prediction on the target image in order to obtain the initial region.

[0088] In one embodiment, the processor 801 is configured to calculate the width value of the initial region and the height value of the initial region based on the regional geometric feature parameters of the initial region; if the width value is greater than N times the height value, a first region expansion rule is selected, where N is a positive number greater than 1; according to the first region expansion rule, the width value of the initial region is used as the width value of the first updated geometric parameter; according to the first region expansion rule, the updated height value obtained by expanding the height value of the initial region by a first preset ratio is used as the height value of the first updated geometric parameter, wherein the updated height value is greater than a first height threshold and less than a second height threshold.

[0089] In one embodiment, the processor 801 is configured to calculate the width value of the initial region and the height value of the initial region based on the regional geometric feature parameters of the initial region; if the width value is greater than the height value but less than or equal to N times the height value, then a second region expansion rule is selected, where N is a positive number greater than 1; according to the second region expansion rule, the updated height value obtained by expanding the height value of the initial region by a second preset ratio is used as the height value of the first updated geometric parameter, wherein the updated height value is greater than a first height threshold and less than a second height threshold; according to the second region expansion rule, the updated width value obtained by expanding the width value of the initial region by a third preset ratio is used as the width value of the first updated geometric parameter, wherein the updated width value is greater than a first width threshold and less than a second width threshold.

[0090] In one embodiment, the processor 801 is configured to calculate the width value of the initial region based on the regional geometric feature parameters of the initial region, and calculate the height value of the initial region based on the regional geometric feature parameters of the initial region; if the height value is greater than M times the width value, then a third region expansion rule is selected, where M is a positive number greater than 1; according to the third region expansion rule, the height value of the initial region is used as the height value of the first updated geometric parameter; according to the third region expansion rule, the updated width value obtained by expanding the width value of the initial region by a fourth preset ratio is used as the width value of the first updated geometric parameter, wherein the updated width value is greater than a first width threshold and less than a second width threshold.

[0091] In one embodiment, the processor 801 is configured to calculate the width value of the initial region and the height value of the initial region based on the regional geometric feature parameters of the initial region; if the height value is greater than the width value and the height value is less than or equal to M times the width value, then a fourth region expansion rule is selected, where M is a positive number greater than 1; according to the fourth region expansion rule, the updated height value obtained by expanding the height value of the initial region by a fifth preset ratio is used as the height value of the first updated geometric parameter, wherein the updated height value is greater than a first height threshold and less than a second height threshold; according to the fourth region expansion rule, the updated width value obtained by expanding the width value of the initial region by a sixth preset ratio is used as the width value of the first updated geometric parameter, wherein the updated width value is greater than a first width threshold and less than a second width threshold.

[0092] In one embodiment, the sub-image region is the region where a pixel is located in the prediction region; the processor 801 is used to call an image object recognition model to perform image object recognition on each pixel in the prediction region, and determine the probability that each pixel in the prediction region belongs to the image object to be recognized; wherein, pixels with a probability value greater than or equal to a preset threshold belong to the image object to be recognized; the recognition result of the image object recognition is a probability map corresponding to the prediction region determined according to the obtained probability.

[0093] The specific implementation of the processor 801 in this embodiment can be found in the description of the relevant content in the foregoing method embodiments, and will not be repeated here. This application employs a strategy of first detecting the approximate initial region where image objects such as hair are located, and then performing a fine-grained expansion of the initial region. This strategy can effectively reduce missed and false detections of image object regions, improve the efficiency of identifying image objects from images, and the overall solution is highly feasible, low-cost, and highly accurate.

[0094] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0095] The above-disclosed embodiments are merely some examples of the present invention and should not be construed as limiting the scope of the invention. Those skilled in the art will understand that implementing all or part of the above-described embodiments and making equivalent changes in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A method for recognizing and processing image objects, characterized in that, include: The image object prediction model is invoked to predict the region of the acquired target image, and the region geometric feature parameters output by the image object prediction model after prediction are obtained. The initial region is determined based on the region geometric feature parameters. The initial region includes a rectangular region, and the geometric feature parameters of the region include: offset information, height information, and width information; the offset information includes: the offset between the center of the rectangular region and the center of the target image; the height information includes: the ratio of the height of the rectangular region to the height of the target image; and the width information includes: the ratio of the width of the rectangular region to the width of the target image. The width and height of the initial region are calculated based on the region's geometric feature parameters, and a corresponding region expansion rule is selected based on the width and height of the initial region. If the width value is greater than N times the height value, then the first region expansion rule is selected, where N is a positive number greater than 1; The first updated geometric parameter is determined according to the first region expansion rule, which includes: using the width value of the initial region as the width value of the first updated geometric parameter; and using the updated height value obtained by expanding the height value of the initial region by a first preset ratio as the height value of the first updated geometric parameter, wherein the updated height value is greater than a first height threshold and less than a second height threshold. The position of the initial region in the target image is adjusted according to the first updated geometric parameters to obtain the predicted region; An image object recognition model is invoked to perform image object recognition on each pixel in the prediction region, and the probability that each pixel in the prediction region belongs to the image object to be recognized is determined; wherein, pixels with a probability value greater than or equal to a preset threshold belong to the image object to be recognized; the recognition result of the image object recognition is a probability map corresponding to the prediction region determined based on the obtained probabilities; The image object region in the target image is determined based on the probability map corresponding to the predicted region. A mask layer is set for the image object region, and the image object in the image object region is adjusted by adjusting the parameters of the mask layer.

2. The method as described in claim 1, characterized in that, Before selecting the corresponding region expansion rule based on the region geometric feature parameters of the initial region, the method further includes: When the initial region area determined based on the region geometric feature parameters satisfies the area condition, the second updated geometric parameters are determined based on the geometric parameters of the target image. The predicted region is obtained by adjusting the position of the initial region in the target image according to the second updated geometric parameters; When the initial region area determined based on the region geometric feature parameters does not meet the area condition, the corresponding region expansion rule is selected based on the region geometric feature parameters of the initial region.

3. The method as described in claim 2, characterized in that, The initial region area satisfies the area condition as follows: the ratio between the initial region area and the area of ​​the target image is greater than a preset area ratio threshold.

4. The method according to any one of claims 1-3, characterized in that, Also includes: Face detection is performed on the target image. If a face location region is detected, the predicted region is estimated and determined based on the face location region determined by the face detection. If no face location region is detected, the image object prediction model is called to predict the region of the target image in order to obtain the initial region.

5. The method as described in claim 1, characterized in that, Also includes: If the width value is greater than the height value, but the width value is less than or equal to N times the height value, then the second region expansion rule is selected, where N is a positive number greater than 1; According to the second region expansion rule, the updated height value obtained by expanding the initial region height value by a second preset ratio will be used as the height value of the first updated geometric parameter. The updated height value is greater than the first height threshold and less than the second height threshold. According to the second region expansion rule, the updated width value obtained by expanding the initial region width value by a third preset ratio will be used as the width value of the first updated geometric parameter. The updated width value is greater than the first width threshold and less than the second width threshold.

6. The method as described in claim 1, characterized in that, Also includes: If the height value is greater than M times the width value, then the third region expansion rule is selected, where M is a positive number greater than 1; According to the third region expansion rule, the height value of the initial region is used as the height value of the first updated geometric parameter; According to the third region expansion rule, the updated width value obtained by expanding the width value of the initial region by a fourth preset ratio will be used as the width value of the first updated geometric parameter. The updated width value is greater than the first width threshold and less than the second width threshold.

7. The method as described in claim 1, characterized in that, Also includes: If the height value is greater than the width value and the height value is less than or equal to M times the width value, then the fourth region expansion rule is selected, where M is a positive number greater than 1; According to the fourth region expansion rule, the updated height value obtained by expanding the initial region height value by a fifth preset ratio will be used as the height value of the first updated geometric parameter. The updated height value is greater than the first height threshold and less than the second height threshold. According to the fourth region expansion rule, the updated width value obtained by expanding the initial region width value by a sixth preset ratio will be used as the width value of the first updated geometric parameter. The updated width value is greater than the first width threshold and less than the second width threshold.

8. An image object recognition processing device, characterized in that, include: The prediction module is used to call the image object prediction model to predict the region of the acquired target image, and to obtain the region geometric feature parameters output by the image object prediction model after prediction, and to determine the initial region based on the region geometric feature parameters. The initial region includes a rectangular region, and the region's geometric feature parameters include offset information, height information, and width information. The offset information includes the offset between the center of the rectangular region and the center of the target image. The height information includes the ratio of the height of the rectangular region to the height of the target image. The width information includes the ratio of the width of the rectangular region to the width of the target image. The width and height values ​​of the initial region are calculated based on the region's geometric feature parameters. A corresponding region expansion rule is selected based on the width and height values. If the width value is greater than N times the height value, a first region expansion rule is selected, where N is a positive number greater than 1. A first update geometric parameter is determined according to the first region expansion rule. The position of the initial region in the target image is adjusted according to the first update geometric parameter to obtain the predicted region. The prediction module, when determining the first updated geometric parameter according to the first region expansion rule, uses the width value of the initial region as the width value of the first updated geometric parameter; and uses the updated height value obtained by expanding the height value of the initial region by a first preset ratio as the height value of the first updated geometric parameter, wherein the updated height value is greater than a first height threshold and less than a second height threshold. The recognition module is used to call an image object recognition model to perform image object recognition on each pixel in the prediction region, and determine the probability that each pixel in the prediction region belongs to the image object to be recognized; wherein, pixels with a probability value greater than or equal to a preset threshold belong to the image object to be recognized; the recognition result of the image object recognition is a probability map corresponding to the prediction region determined based on the obtained probability; The determination module is used to determine the image object region in the target image based on the probability map corresponding to the predicted region; The device further includes a module for setting a mask layer on the image object region and adjusting the parameters of the mask layer to adjust the image object in the image object region.

9. A smart device, characterized in that, The method includes a storage device and a processor, wherein the storage device stores program instructions, and the processor invokes the program instructions to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that, when executed, will implement the method as described in any one of claims 1-7.

11. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1-7.

Images