Image acquisition method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses an image acquisition method and device, electronic equipment and a storage medium. The method comprises the steps that under the condition that it is determined that distinguishing processing operation needs to be carried out on a target in an image, a target instance of the target is recognized, the category of at least part of objects in the image is recognized, and a recognition result of the target is obtained; and according to the identification result, performing the distinguishing processing operation on an identified target and / or an image area except the target to obtain a target image. According to the embodiment of the invention, a good image processing effect can be ensured, and the target image after accurate processing can be obtained.

Description

Technical Field

[0001] The present invention relates to the field of image processing but is not limited to the field of image processing, and in particular to an image acquisition method, device, electronic device and storage medium. Background Art

[0002] There are more and more application scenarios of image processing in related technologies. In some application scenarios of acquiring images or videos, various image processing needs to be performed on image frames in the images or videos. For example, in portrait mode, relevant image processing can be performed on the characters in the image, or in portrait mode, the background area other than the portrait can be blurred.

[0003] However, in the related art, when performing various image processing on image frames in images or videos, there may be large processing errors, resulting in the final processing effect not meeting the requirements and failing to obtain accurately processed images. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the embodiments of the present disclosure disclose an image acquisition method, device, electronic device and storage medium to overcome the problem that the image cannot be accurately processed, resulting in difficulty in obtaining an accurately processed image.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image acquisition method, the method comprising:

[0006] In the case where it is determined that a distinguishing processing operation needs to be performed on a target in the image, a target instance of the target is identified, and the category of at least part of the objects in the image is identified to obtain an identification result of the target;

[0007] According to the recognition result, the distinguishing processing operation is performed on the recognized target and / or the image area other than the target to obtain a target image.

[0008] In some embodiments, identifying the target instance of the target and identifying the category of at least part of the objects in the image to obtain the recognition result of the target includes:

[0009] Recognizing a target instance of the target, identifying the target object from the image and determining an edge of the target object;

[0010] Identify the category of at least part of the object in the image to determine the category to which each part of the target object belongs;

[0011] Determining whether the target object includes content other than the target according to the categories to which each part of the target object belongs;

[0012] In response to the target object including content other than the target, the target object and the edge of the target object are adjusted according to the categories to which the various parts belong, so as to obtain a recognition result of the target.

[0013] In some embodiments, there are multiple target objects; identifying the target object from the image and determining the edge of the target object includes:

[0014] In response to the existence of an overlapping area between at least two of the target objects, identifying each of the target objects from the image and determining an edge of each of the target objects;

[0015] and / or,

[0016] In response to the distance between at least two of the target objects being less than a distance threshold, each of the target objects is identified from the image and an edge of each of the target objects is determined.

[0017] In some embodiments, identifying the target object from the image and determining the edge of the target object includes:

[0018] Using an instance segmentation algorithm to segment the image features of the image, obtaining instance features and weight features of each of the target objects; wherein the instance features are used to characterize the region of interest corresponding to the target object in the image; and the weight features include a weight value corresponding to the instance features, a category, and a confidence level that the region of interest is located in the foreground region of the image;

[0019] Based on the instance features and the weight features, edges of the target object are determined.

[0020] In some embodiments, determining the category to which each part of the target object belongs includes:

[0021] The image features of the image are segmented using a semantic segmentation algorithm to obtain semantic features and edge features of the image; wherein the semantic features are used to characterize the category to which at least part of the pixels in the image belong; and the edge features include features of the contour edge of the entire target in the image;

[0022] Based on the semantic features and the edge features, the categories to which the various parts of the target object belong are determined.

[0023] In some embodiments, the method further comprises:

[0024] Identify at least some of the pixels in the image, and determine instance attributes and / or category attributes of the at least some of the pixels;

[0025] The identifying of the target instance of the target and identifying the category of at least part of the objects in the image to obtain the recognition result of the target includes:

[0026] Based on the instance attribute and / or the category attribute of at least part of the pixels, the target instance of the target is identified, and the category of at least part of the objects in the image is identified to obtain a recognition result of the target.

[0027] In some embodiments, the identifying the target instance of the target based on the instance attribute and / or the category attribute of the at least part of the pixels, and identifying the category of at least part of the objects in the image to obtain the recognition result of the target includes:

[0028] Based on the instance attribute and the category attribute, determining a target pixel whose instance attribute and the category attribute meet target requirements; wherein the target is composed of the target pixels;

[0029] Based on the target pixels, a target instance of the target is identified, and categories of at least part of the objects in the image are identified to obtain a recognition result of the target.

[0030] In some embodiments, performing the differentiation processing operation on the identified target and / or the image area other than the target includes:

[0031] performing a first differentiation processing operation on the target; and / or,

[0032] performing a second distinguishing processing operation on the image area other than the target;

[0033] The processing effect corresponding to the first distinguishing processing operation is different from the processing effect corresponding to the second distinguishing processing operation.

[0034] In some embodiments, there are multiple targets to be identified; performing the differentiation processing operation on the identified targets includes:

[0035] performing the first distinguishing processing operation on some of the multiple targets, and performing the second distinguishing processing operation on other targets except the some of the targets; or

[0036] Performing different first distinguishing processing operations on each of the targets respectively; or

[0037] The second distinguishing processing operation is performed differently on each of the targets.

[0038] In some embodiments, the second distinguishing processing operation is a virtualization operation; and performing different second distinguishing processing operations on each of the targets respectively includes:

[0039] Determining a blur degree corresponding to each target based on depth information of each target in the image;

[0040] According to the blurring degree, corresponding blurring operations are performed on each of the targets respectively.

[0041] In some embodiments, the method further comprises:

[0042] In the case where the current scene is a target usage scene, determining that the distinguishing processing operation needs to be performed on the target in the image;

[0043] The target usage scenario includes at least one of the following:

[0044] Portrait recognition scene, vehicle recognition scene and scene recognition scene.

[0045] According to a second aspect of an embodiment of the present disclosure, there is provided an image acquisition device, the device comprising:

[0046] A recognition module, configured to, when it is determined that a distinguishing processing operation needs to be performed on a target in an image, recognize a target instance of the target and recognize the category of at least part of the objects in the image to obtain a recognition result of the target;

[0047] The processing module is used to perform the differentiation processing operation on the identified target and / or the image area other than the target according to the recognition result to obtain the target image.

[0048] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:

[0049] processor;

[0050] a memory for storing instructions executable by the processor;

[0051] The processor is configured to implement any method described in any one of the embodiments of the present disclosure when running the executable instructions.

[0052] According to a fourth aspect of an embodiment of the present disclosure, a computer storage medium is provided, wherein the computer storage medium stores a computer executable program, and when the executable program is executed by a processor, the method described in any embodiment of the present disclosure is implemented.

[0053] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0054] In the embodiment of the present disclosure, when it is determined that a distinguishing processing operation needs to be performed on a target in an image, the target instance of the target is identified, and the category of at least part of the objects in the image is identified to obtain a recognition result of the target; according to the recognition result, the distinguishing processing operation is performed on the identified target and / or the image area other than the target to obtain a target image. Here, since when it is determined that a distinguishing processing operation needs to be performed on the target in the image, the recognition result of the target is obtained based on the recognition result of identifying the target instance of the target and the recognition result of identifying the category of at least part of the objects in the image, it is fully adaptable to the recognition result of identifying the target instance of the target and the recognition result of identifying the category of at least part of the objects in the image, and the target is accurately identified from the image. Compared with the method in the related art that cannot accurately process the image, in the embodiment of the present disclosure, in the process of performing the distinguishing processing operation on the accurately identified target and / or the image area other than the target to obtain the target image, the distinguishing processing operation can be accurately performed on the accurately identified target and / or the image area other than the target to obtain the target image that meets the image processing requirements. In this way, it can be ensured that the image processing effect is good and the target image can be obtained after accurate processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0056] Figure 1 is a flow chart of an image acquisition method according to an exemplary embodiment;

[0057] Figure 2 is a schematic diagram of an application scenario corresponding to an image acquisition method according to an exemplary embodiment;

[0058] Figure 3 is a schematic diagram of a framework of an image segmentation model according to an exemplary embodiment;

[0059] Figure 4 is a schematic diagram of a framework of an image segmentation model according to an exemplary embodiment;

[0060] Figure 5 is a schematic diagram showing a comparison of image recognition results according to an exemplary embodiment;

[0061] Figure 6 is a schematic diagram of a multi-task framework according to an exemplary embodiment;

[0062] Figure 7is a schematic structural diagram of an image acquisition device according to an exemplary embodiment;

[0063] Figure 8 The figure is a schematic diagram showing the structure of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present invention. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.

[0065] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0066] In the following description, the terms "first / second / third" are used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first / second / third" can be interchanged with the specific order or sequence where permitted, so that the embodiments of the present invention described herein can be implemented in an order other than that illustrated or described herein.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0068] like Figure 1 As shown, the embodiment of the present disclosure provides an image acquisition method, the method comprising:

[0069] Step S110, when it is determined that a distinguishing processing operation needs to be performed on a target in the image, a target instance of the target is identified, and categories of at least part of the objects in the image are identified to obtain a recognition result of the target;

[0070] Step S120: performing the differentiation processing operation on the identified target and / or the image region other than the target according to the recognition result to obtain a target image.

[0071] It can be understood that the target image is an image obtained after performing the differentiation processing operation on the target identified in the image and / or the image area other than the target.

[0072] Any of the methods described in the embodiments of the present disclosure may be performed by an intelligent device having computing power resources. For example, the intelligent device may be an electronic device, and the electronic device may be a vehicle-mounted terminal, a monitoring device, a mobile phone, and a computer. Alternatively, the device may also be a physical server or a cloud server. Here, the type of the intelligent device having computing power resources may not be limited.

[0073] In some embodiments, the target instance of the target may be identified first, and then the category of at least part of the objects in the image may be identified, so as to obtain the recognition result of the target. Alternatively, the category of at least part of the objects in the image may be identified first, and then the target instance of the target may be identified, so as to obtain the recognition result of the target. Alternatively, the target instance of the target may be identified, and the category of at least part of the objects in the image may be identified at the same time. Here, the recognition order of identifying the target instance and identifying the category of at least part of the objects may not be limited.

[0074] In some embodiments, when it is determined that a distinguishing operation needs to be performed on a target in an image, a target instance of the target is identified to obtain an instance recognition result; the category of at least part of the objects in the image is identified to obtain a semantic recognition result; based on the instance recognition result and the semantic recognition result, the target is identified from the image to obtain a target recognition result; the distinguishing operation is performed on the identified target and / or the image area other than the target to obtain a target image. In some embodiments, when it is determined that a distinguishing operation needs to be performed on a target in an image, a target instance of the target can be identified to obtain an instance recognition result; the category of at least part of the objects in the image is identified to obtain a semantic recognition result. The instance recognition result can be corrected based on the semantic recognition result to obtain a corrected instance recognition result; the target is identified from the image based on the corrected instance recognition result; and / or the semantic recognition result can be corrected based on the instance recognition result to obtain a corrected semantic recognition result; the target is identified from the image based on the corrected semantic recognition result. Here, the target can be identified from the image based on a more accurate instance recognition result and / or a more accurate semantic recognition result.

[0075] It should be noted that, for targets of the same category, a target instance is used to characterize each target individual in the same category. That is, for each target, a target instance corresponding to the target can be identified.

[0076] In some embodiments, there may be only one target in the image, or there may be at least two targets in the image. The target may include but is not limited to at least one of the following: a person, an animal, a plant, and a scene. Here, the specific type of the target may not be limited.

[0077] In some embodiments, identifying the category of at least some objects in the image includes:

[0078] Identify at least some of the objects as being of a target category or a non-target category.

[0079] Here, when the category of the object is the target category, it is determined that the object belongs to the target, or when the category of the object is the non-target category, it is determined that the object does not belong to the target.

[0080] For example, the image may include people, animals and plants, and the target may be a person. When the category of the object is a target category, the object is determined to be a person. Alternatively, when the category of the object is a non-target category, the object is determined not to be a person, for example, the object is determined to be an animal and / or a plant.

[0081] In some embodiments, when the category of the object is a target category, it is determined that the object is in a foreground area of ​​the image; the object is determined to be a candidate object, and the target is identified from the candidate objects.

[0082] In some embodiments, when the category of the object is a non-target category, it is determined that the object is in a background area in the image; it is determined that the object is not a candidate object, and the target is identified from objects other than the candidate objects.

[0083] In the embodiment of the present disclosure, when it is determined that a distinguishing processing operation needs to be performed on a target in an image, the target instance of the target is identified, and the category of at least part of the objects in the image is identified to obtain a recognition result of the target; according to the recognition result, the distinguishing processing operation is performed on the identified target and / or the image area other than the target to obtain a target image. Here, since when it is determined that a distinguishing processing operation needs to be performed on the target in the image, the recognition result of the target is obtained based on the recognition result of identifying the target instance of the target and the recognition result of identifying the category of at least part of the objects in the image, it is fully adaptable to the recognition result of identifying the target instance of the target and the recognition result of identifying the category of at least part of the objects in the image, and the target is accurately identified from the image. Compared with the method in the related art that cannot accurately process the image, in the embodiment of the present disclosure, in the process of performing the distinguishing processing operation on the accurately identified target and / or the image area other than the target to obtain the target image, the distinguishing processing operation can be accurately performed on the accurately identified target and / or the image area other than the target to obtain the target image that meets the image processing requirements. In this way, it can be ensured that the image processing effect is good and the target image can be obtained after accurate processing.

[0084] In some embodiments, identifying the target instance of the target and identifying the category of at least part of the objects in the image to obtain the recognition result of the target includes:

[0085] Recognizing a target instance of the target, identifying the target object from the image and determining an edge of the target object;

[0086] Identify the category of at least part of the object in the image to determine the category to which each part of the target object belongs;

[0087] Determining whether the target object includes content other than the target according to the categories to which each part of the target object belongs;

[0088] In response to the target object including content other than the target, the target object and the edge of the target object are adjusted according to the categories to which the various parts belong, so as to obtain a recognition result of the target.

[0089] Here, after the target object is identified, it is determined whether the target object includes content other than the target according to the categories to which the various parts of the target object belong. If it is determined that the target object includes content other than the target, the target object and the edge of the target object will be adjusted. Therefore, if the identified target object includes content other than the target, that is, if the identified target object cannot accurately represent the target, the target object can be accurately adjusted to exclude the content other than the target in the target object, so that the target object can accurately represent the target. In this way, in the process of identifying the target from the image based on the adjusted target object, the target can be accurately identified from the image to ensure the accuracy of the identified target.

[0090] For example, when the target is a person, the initial target (i.e., the target object described in any of the embodiments of the present disclosure) can be determined first; then, based on the categories to which each part of each target object belongs, the categories of each part of the target object are determined to be human or non-human, so as to determine whether the target object includes non-human parts; when the target object includes non-human parts, the initial target and its edges can be adjusted based on the categories to which each part belongs, so as to correct the initial target and obtain a corrected target. At this time, the corrected target can accurately characterize the person and the edge of the person in the image.

[0091] Exemplarily, when the target is a person, there may be a person and other objects in the image. For example, the other objects may be trees, and the trees may cover part of the human body in the image. At this time, the target objects identified from the image may include trees other than people. In the disclosed embodiment, it is possible to determine whether the target object includes trees by the categories to which the various parts of the target object belong; in response to the target object including trees, the target object and the edge of the target object are adjusted according to the categories to which the various parts belong so that the target object does not include trees; based on the adjusted target object and the edge of the target object, a person is identified from the image.

[0092] In some embodiments, in response to the target object including content other than the target, the target object and the edge of the target object are adjusted according to the categories to which the various parts belong, including: in response to the target object including content other than the target, according to the categories to which the various parts belong, deleting the content other than the target in the target object.

[0093] In some embodiments, there are multiple target objects; identifying the target object from the image and determining the edge of the target object includes:

[0094] In response to the existence of an overlapping area between at least two of the target objects, identifying each of the target objects from the image and determining an edge of each of the target objects;

[0095] and / or,

[0096] In response to the distance between at least two of the target objects being less than a distance threshold, each of the target objects is identified from the image and an edge of each of the target objects is determined.

[0097] Here, even if there is an overlapping area between at least two target objects and / or they are very close to each other, each target object can still be identified from the image and the edge of each target object can be determined. In this way, at least two overlapping or adjacent target objects will not be mistakenly regarded as a whole, and each target object can be accurately identified in the embodiment of the present disclosure.

[0098] In some embodiments, the distance between at least two target objects is less than the distance threshold, which may mean that the distance between the edges of at least two target objects is less than the distance threshold. For example, the distance between the edges of at least two target objects may be zero.

[0099] In some embodiments, identifying the target object from the image and determining the edge of the target object includes:

[0100] Using an instance segmentation algorithm to segment the image features of the image, obtaining instance features and weight features of each of the target objects; wherein the instance features are used to characterize the region of interest corresponding to the target object in the image; and the weight features include a weight value corresponding to the instance features, a category, and a confidence level that the region of interest is located in the foreground region of the image;

[0101] Based on the instance features and the weight features, edges of the target object are determined.

[0102] Here, since it will be based on instance features and weight features (for example, it can be as follows Figure 3 As shown), the edge of the target object is determined, and the instance feature is used to characterize the region of interest corresponding to the target in the image, and the weight feature includes the weight value corresponding to the instance feature, the category and the confidence that the region of interest is located in the foreground area. Therefore, in the process of determining the edge of the target object, it is possible to adapt to the region of interest corresponding to each target and the weight value corresponding to each region of interest (exemplarily, as Figure 4 The weight matrix kernel shown), categories (exemplarily, such as Figure 4 The category class shown) and confidence (exemplarily, as shown Figure 4 The confidence score (score) shown in the figure accurately segments the edge of the target object from the corresponding region of interest in the image.

[0103] In some embodiments, determining the category to which each part of the target object belongs includes:

[0104] The image features of the image are segmented using a semantic segmentation algorithm to obtain semantic features and edge features of the image; wherein the semantic features are used to characterize the category to which at least part of the pixels in the image belong; and the edge features include features of the contour edge of the entire target in the image;

[0105] Based on the semantic features and the edge features, the categories to which the various parts of the target object belong are determined.

[0106] Here, since the categories to which the various parts of the target object belong are determined based on the semantic features and the various edge features, and the semantic features are used to characterize the categories to which at least some of the pixels in the image belong; the edge features include the features of the contour edge of the entire target in the image, therefore, in the process of segmenting the image features of the image using the semantic segmentation algorithm, the results of classifying the various pixels of the target object in the image and the detailed information of the contour edge corresponding to the entire target can be combined simultaneously to accurately perform semantic segmentation on the image to determine the categories to which the various parts of the target object belong. In this way, it is possible to ensure that the accuracy of the semantic recognition results obtained is high, the segmentation effect is good, and it is possible to accurately determine the categories to which the various parts of the target object belong.

[0107] In order to better understand the embodiments of the present disclosure, some application scenarios of image processing are described below:

[0108] In the field of computer vision, images can be segmented by semantic segmentation or instance segmentation. Semantic segmentation is to assign each pixel in the image to each category of objects, while instance segmentation is to divide each object in the image into separate parts. Semantic segmentation and instance segmentation are two different types of segmentation tasks, which are widely used in fields such as autonomous driving, robot vision, medical imaging, intelligent monitoring, and mobile phone imaging.

[0109] In some embodiments, the image can be processed by semantic segmentation to output a high-precision semantic recognition result, or the image can be processed by instance segmentation to output a relatively rough instance recognition result. However, for mobile phone images, the edge effect of individual segmentation is extremely high, especially for the problem of blurred portraits in multi-person scenes. If semantic segmentation or instance segmentation is used, it often leads to problems such as blurred background people or poor edge effects. The current mainstream instance segmentation methods mainly include detection-based instance segmentation and instance segmentation based on dense or sparse feature maps. These methods can improve the instance segmentation effect of some scenes, but lack high-precision semantic information, resulting in unsatisfactory overall and edge segmentation effects, especially in the face of some complex scenes with small targets, occlusions and multiple people. The segmentation effect will be relatively poor.

[0110] In some embodiments, in the related art, the image can be processed based on the technology related to computer vision to segment the object to be identified from the image. However, in the related art, when there are multiple objects to be identified in an image, it is often impossible to accurately and quickly segment each object to be identified from the image. For example, in the process of blurring some objects to be identified in an image taken by a mobile phone, it is difficult to accurately and quickly segment each image to be identified from the image in the related art, resulting in the omission of the object to be identified that needs to be blurred in the process of blurring the object to be identified, the blurring effect is poor, and the blurring speed is slow. In this way, when there are multiple objects to be identified in an image, the related art has the problem of poor segmentation effect and low segmentation efficiency of segmenting the objects to be identified.

[0111] Based on this, an embodiment of the present disclosure may provide an image segmentation model for identifying various targets from an image.

[0112] In some embodiments, an instance segmentation module (such as Figure 3 ) to perform any of the instance segmentation algorithms described in the embodiments of the present disclosure. The semantic segmentation module (such as Figure 3 As shown) executes the semantic segmentation algorithm described in any of the embodiments of the present disclosure.

[0113] In some embodiments, the instance segmentation module may be used to segment each object in the image into separate parts. Each object may correspond to an instance recognition result.

[0114] In some embodiments, the semantic segmentation module is used to assign each pixel in the image to an object of a corresponding category. For example, there are a target and a background area in the image. The semantic segmentation module can be used to assign the pixels constituting the target to the target, and the pixels constituting the background area to the background area. In this way, the target can be accurately identified from the image by accurately classifying the pixels constituting the background area and the pixels constituting the target. Here, it can be understood that regardless of whether the number of targets is one or more, the number of semantic recognition results for targets of the same category can be one.

[0115] In some embodiments, a feature extraction module in an image segmentation model is used to extract features from an image with multiple targets to obtain image features; an instance segmentation module in the image segmentation model is used to segment the image features to obtain instance recognition results corresponding to each of the targets; a semantic segmentation module in the image segmentation model is used to segment the image features to obtain semantic recognition results; based on the instance recognition results and the semantic recognition results, targets are identified from the image; and the differentiation processing operation is performed on the identified targets or image areas other than the targets to obtain a target image.

[0116] In some embodiments, before using the feature extraction module in the image segmentation model to extract features from the image, the image may be preprocessed to obtain an image after data preprocessing. The data preprocessing may be used to achieve data enhancement of the image. At this time, using the feature extraction module in the image segmentation model in the embodiment of the present disclosure to extract features from an image with multiple targets to obtain image features may refer to using the feature extraction module to extract features from the image after data preprocessing to obtain image features. Here, the image data preprocessing may be completed by performing a geometric transformation on the image. The operation of geometric transformation may include but is not limited to at least one of the following: flipping, rotating, cropping, scaling, translating, and shaking. Alternatively, the image data preprocessing may be completed by performing a pixel transformation on the image.

[0117] In some embodiments, a feature extraction module in an image segmentation model is used to extract features from an image having multiple targets to obtain image features, including: inputting the image into the feature extraction module of the image segmentation model to obtain image features.

[0118] In some embodiments, both the semantic segmentation module and the instance segmentation module in the image segmentation model can be connected to the feature extraction module in the image segmentation model. The feature extraction module in the image segmentation model can be the backbone network in the image segmentation model. At this point, it can be understood that the connection between the semantic segmentation module and the feature extraction module can refer to the connection between the input of the semantic segmentation module and the output of the feature extraction module. The connection between the instance segmentation module and the feature extraction module can refer to the connection between the input of the instance segmentation module and the output of the feature extraction module. It can be understood that the image segmentation model in the embodiment of the present disclosure will automatically extract image features using the feature extraction module, that is, the image segmentation model in the embodiment of the present disclosure is an end-to-end network model.

[0119] In some embodiments, there may be one input content for the entire image segmentation model, that is, an image may be input to the image segmentation model, and the image may be a three-channel image. The image segmentation model may have two output contents, one of which is a semantic recognition result, and the other is an instance recognition result corresponding to each target. Here, the semantic recognition result may be a single-channel semantic grayscale image, and each instance recognition result may be a single-channel instance grayscale image.

[0120] In some embodiments, the semantic recognition result may be first mask information corresponding to the image, and the first mask information may be used to mask the background area except all objects in the image; the instance recognition result may be second mask information corresponding to each object. The second mask information may be used to mask all areas except the object in the image for each object. Each object may be segmented from the image based on the first mask information and the second mask information.

[0121] Here, on the one hand, since each target is segmented from the image based on the semantic recognition result after segmenting the image and the instance recognition result corresponding to each target, in the process of segmenting each target from the image, it is possible to fully combine the refined semantic information and the instance features corresponding to the target, and accurately segment each target from the image. In this way, it can be ensured that the segmentation effect of each target segmented from the image is good. On the other hand, since the semantic segmentation module and the instance segmentation module can obtain the semantic recognition result and the instance recognition result respectively using the image features extracted by the same feature extraction module in the same image segmentation model, in the process of obtaining the semantic recognition result and the instance recognition result, the semantic segmentation module and the instance segmentation module can share the image features extracted by the feature extraction module to quickly calculate the segmentation result using the same image features. Compared with the method of obtaining the semantic recognition result and the instance recognition result respectively using different models in the related art, the embodiment of the present disclosure can reduce the number of models and the number of model parameters, thereby simplifying the calculation process, quickly obtaining the semantic recognition result and the instance recognition result, so as to quickly segment each target from the image based on the quickly obtained semantic recognition result and the instance recognition result. In this way, the segmentation efficiency can be ensured. In summary, the embodiments of the present disclosure can improve the segmentation efficiency while ensuring a good segmentation effect.

[0122] In some embodiments, the semantic segmentation module includes: a first semantic segmentation submodule and a second semantic segmentation submodule. Any of the semantic features described in the embodiments of the present disclosure may be features obtained after the first semantic segmentation submodule performs segmentation processing on the image features, and any of the edge features described in the embodiments of the present disclosure may be features obtained after the second semantic segmentation submodule performs segmentation processing on the image features.

[0123] In some embodiments, the semantic feature may be a first result to be fused at a first granularity, and the edge feature may be a second result to be fused at a second granularity. The first result to be fused at the first granularity and the second result to be fused at the second granularity may be fused to obtain any of the semantic recognition results described in the embodiments of the present disclosure. The first granularity may be greater than the second granularity. For example, the semantic feature may be a coarse-grained semantic recognition result extracted from the image features by the first semantic segmentation submodule, and the edge feature may be a fine-grained edge detail result extracted from the image features by the second semantic segmentation submodule.

[0124] In some embodiments, the image segmentation model may further include a fusion module, which is used to fuse semantic features and edge features to obtain any semantic recognition result described in any of the embodiments of the present disclosure. Here, through the fusion module, the semantic features and edge features can be quickly fused without a complex attention mechanism, effectively reducing the computational complexity.

[0125] In some embodiments, the semantic features output by the first semantic segmentation submodule can be a dual-channel image, one of the dual-channel images is used to characterize the background area of ​​the image, and the other dual-channel image is used to characterize the foreground area in the image (i.e., the area where the target is located). The edge features output by the second semantic segmentation submodule can be a single-channel image. The semantic recognition result obtained based on the semantic features and edge features can be a single-channel image. Exemplarily, the semantic recognition result can be a single-channel grayscale image, in which the value of each pixel is not less than 0, and the value of each pixel is not greater than 255.

[0126] In some embodiments, the instance segmentation module includes a first instance segmentation submodule and a second instance segmentation submodule. Any of the instance features described in the embodiments of the present disclosure is a feature obtained after the first instance segmentation submodule performs segmentation processing on the image feature, and any of the weight features described in the embodiments of the present disclosure is a feature obtained after the second instance segmentation submodule performs segmentation processing on the image feature.

[0127] In some embodiments, a semantic fusion feature can be obtained based on the semantic features described in any of the embodiments of the present disclosure and the shallow features extracted from the image using a feature extraction module. The image features and the semantic fusion features can be segmented using a first instance segmentation submodule to obtain the instance features described in any of the embodiments of the present disclosure. The image features and the semantic fusion features can be segmented using a second instance segmentation submodule to obtain the weight features described in any of the embodiments of the present disclosure.

[0128] Here, since the instance segmentation module is used to segment the second semantic features and the image features, the instance features and weight features corresponding to each of the targets are obtained, and the semantic fusion features are determined based on the semantic features extracted from the first semantic segmentation submodule and the shallow features extracted from the feature extraction module. Therefore, compared with the method in the related art that does not consider the shallow features and semantic features in the image segmentation model when performing instance segmentation on the image, the embodiment of the present disclosure will fully consider the influence of semantic features and shallow features on the instance recognition results when performing instance segmentation on the image, so that in the instance segmentation process, the instance features, refined semantic features and shallow features that can better represent the initial image can be combined to accurately perform instance segmentation on each target in the image. In this way, the situation where the segmentation effect is poor due to the lack of detailed information that can represent the initial image during the instance segmentation process can be reduced, and the accuracy of instance segmentation of each target in the image can be ensured.

[0129] In the disclosed embodiment, a relatively light feature extraction module and a segmentation head network (i.e., a semantic segmentation module and an instance segmentation module) can be used to achieve a very high-precision segmentation effect by guiding with semantic features. Thus, compared with the related art which needs to rely on a detection model to detect the target area in order to segment the target, there is no need to rely on a detection model, and poor segmentation accuracy will not be caused by inaccurate detection accuracy. The disclosed implementation can improve the generalization of the image segmentation model, reduce the amount of calculation, and improve the segmentation effect.

[0130] It should be noted that the feature extraction module belongs to the shallow network in the image segmentation model, and any of the shallow features described in the embodiments of the present disclosure actually refers to the features extracted by the shallow network. Here, since the shallow network is often close to the input position in the image segmentation model, the shallow features extracted from the shallow network can contain information about more pixels in the image. For example, the shallow features will include fine-grained image information, and the fine-grained image information includes the color, texture, edge and angular information of the image. In this way, the shallow features can more accurately characterize the detailed information of the initial image, and the shallow features have a good effect in characterizing the image.

[0131] In some embodiments, the first instance segmentation submodule is used to determine the area in the image where the target whose brightness is greater than the brightness threshold is located, so as to obtain the region of interest for the target segmentation. For example, the first instance segmentation submodule can be an instance activation map (IAM) structure, the first instance segmentation submodule can not rely on dense prediction at the pixel level, but can capture key context feature information related to each target through sparse features, the first instance submodule can not rely on post-processing operations such as non-maximum suppression (NMS), so as to improve the overall operation speed of the network in the image segmentation model.

[0132] In some embodiments, see Figure 4 , Figure 4The structure of a feature extraction module and an instance segmentation module in an image segmentation model is exemplarily shown, as well as a process of performing instance segmentation processing on the image features extracted by the feature extraction module. The segmentation processing process includes: using the feature extraction module in the image segmentation model to extract features from an image with multiple targets to obtain image features; wherein the size of the image may be h×w, h is the length of the image, and w is the width of the image. The image features may include features of a first size, a second size, a third size, and a fourth size, respectively, wherein the first size may be (h÷2)×(w÷2), the first size may be (h÷4)×(w÷4); the third size may be (h÷8)×(w÷8); and the fourth size may be (h÷16)×(w÷16). Based on the semantic features output from the semantic segmentation module and the shallow features extracted from the feature extraction module, semantic fusion features are obtained by fusion; wherein the length of the semantic features may be h÷2, and the width of the semantic features may be w÷2. The semantic fusion features are downsampled to obtain downsampled semantic fusion features. The length of the downsampled semantic fusion feature can be h÷4, and the width of the semantic fusion feature can be w÷4. The downsampled semantic fusion feature and the image feature are input into the decoder in the image segmentation model to obtain the target fusion feature. The length of the target fusion feature can be h÷4, and the width of the second semantic feature can be w÷4; the target fusion feature is input into the first instance segmentation submodule to obtain the instance feature corresponding to each of the objects to be identified. The instance feature is used to characterize the region of interest corresponding to the object to be identified in the image. The size of the instance feature can be D×(h÷4)×(w÷4); the length of the instance feature is h÷4, the width of the instance feature is w÷4, and the number of instance features is D. The target fusion feature is input into the second instance segmentation submodule to obtain the weight feature corresponding to each of the instance features; the weight feature includes the weight value corresponding to the instance feature, the category, and the confidence that the region of interest is located in the foreground area; based on the instance feature and the weight feature, the instance recognition result corresponding to each target is obtained.

[0133] In some embodiments, the first instance segmentation submodule may be a fully convolutional network. The second instance segmentation submodule may be composed of convolution, group convolution, and a fully connected network.

[0134] In some embodiments, based on the instance features and the weight features, the instance recognition results corresponding to each of the targets are obtained, including: based on a preset matrix multiplication, multiplying the instance features and the weight features corresponding to the instance features to obtain the instance recognition results of each of the targets.

[0135] In some embodiments, the method further comprises:

[0136] Inputting the training samples into the initial segmentation model, training the first semantic segmentation submodule, the second semantic segmentation submodule and the instance segmentation module in the image segmentation model until a preset convergence condition is met, thereby obtaining the image segmentation model;

[0137] The preset convergence condition includes: the number of training times reaches a first preset number of times, or the convergence function corresponding to the image segmentation model converges.

[0138] In some embodiments, a multi-task training method can be used to train the output of the first semantic segmentation submodule, the output of the second semantic segmentation submodule, and the output of the instance segmentation module in the image segmentation model. The output of the first semantic segmentation submodule is the first semantic feature described in any of the embodiments disclosed herein, and the output of the second semantic segmentation submodule is the edge feature described in any of the embodiments disclosed herein. The first semantic feature and the edge feature are used to improve the semantic and edge detail performance in the semantic segmentation module, respectively. Here, different task branch heads can be replaced through multi-task training to achieve simultaneous training of different tasks, and the features between different tasks can guide and enhance each other, so that after the training of each task is completed, the effect and performance of each task can be improved.

[0139] In some embodiments, the training sample is input into the initial segmentation model, and the first semantic segmentation submodule, the second semantic segmentation submodule and the instance segmentation module in the image segmentation model are trained until a preset convergence condition is met to obtain the image segmentation model, including: constraining the first semantic segmentation submodel based on a first loss function; wherein the first loss function can be a cross entropy loss function CE Loss; constraining the second semantic segmentation submodel based on a second loss function; wherein the second loss function can be an absolute value loss function L1Loss; constraining the semantic recognition result based on the second loss function and / or the third loss function; wherein the semantic recognition result is a segmentation result obtained by fusing the outputs of the first semantic segmentation submodel and the second semantic segmentation submodel; the third loss function can be a Laplacian loss function Laplacian Loss; constraining the instance segmentation module based on a fourth loss function; wherein the third loss function can include the first loss function and / or the dice loss function Dice Loss. Here, Dice Loss can measure the similarity between the predicted results in the instance segmentation module and the true annotations, thereby obtaining the matching coefficient between each predicted object and the true annotation, and then using the Hungarian algorithm to calculate the optimal match, and using the matching relationship to calculate the training loss.

[0140] In some embodiments, the output of the first semantic segmentation submodule can be a dual-channel image, one of the dual-channel images is used to characterize the background area of ​​the image (i.e., the image is a mask image corresponding to the background), and the other image in the dual-channel image is used to characterize the foreground area in the image (i.e., the other image is a mask image of the foreground area). The edge features output by the second semantic segmentation submodule can be a single-channel image. The semantic recognition result obtained based on the semantic features and edge features can be a single-channel image. Exemplarily, the semantic recognition result can be a single-channel grayscale image, wherein the value corresponding to the pixel in the grayscale image is not less than 0, and the value of the pixel is not greater than 255. The value of the pixel can be positively correlated with the confidence that the pixel belongs to the foreground.

[0141] In some embodiments, the preset convergence condition is the convergence of the loss function, the training sample is input into the initial segmentation model, the first semantic segmentation submodule, the second semantic segmentation submodule and the instance segmentation module in the image segmentation model are trained until the preset convergence condition is met, and the image segmentation model is obtained, including: inputting the training sample into the initial segmentation model, training the first semantic segmentation submodule, the second semantic segmentation submodule and the instance segmentation module in the image segmentation model to obtain a predicted segmentation result; the training sample is an image labeled with a segmentation result; based on the difference between the predicted segmentation result and the labeled segmentation result, determining whether the image segmentation model reaches the preset convergence condition; in response to the image segmentation model not reaching the preset convergence condition, updating the model parameters of the image segmentation model based on the difference until the difference is less than a difference threshold.

[0142] In some embodiments, the inputting of the training samples into the initial segmentation model, training the first semantic segmentation submodule, the second semantic segmentation submodule and the instance segmentation module in the image segmentation model until a preset convergence condition is met, and obtaining the image segmentation model includes:

[0143] Inputting the training sample into the semantic segmentation module of the initial segmentation model, training the first semantic segmentation submodule and the second semantic segmentation submodule until the number of training times reaches a second preset number of times, and obtaining the first semantic segmentation submodule to be fine-tuned and the second semantic segmentation submodule to be fine-tuned; wherein the second preset number of times is less than the first preset number of times;

[0144] Determining a pre-training weight based on the first semantic segmentation submodule to be fine-tuned and the second semantic segmentation submodule to be fine-tuned;

[0145] Updating the model parameters of the initial segmentation model based on the pre-trained weights to obtain a fine-tuned segmentation model;

[0146] The training samples are input into the fine-tuning segmentation model, and the first semantic segmentation submodule, the second semantic segmentation submodule and the instance segmentation module of the fine-tuning segmentation model are trained until the preset convergence condition is met.

[0147] Here, before the first semantic segmentation submodule, the second semantic segmentation submodule and the instance segmentation module in the fine-tuning segmentation model are trained as a whole, the first semantic segmentation submodule and the second semantic segmentation submodule to be fine-tuned will be pre-trained, and the initial segmentation model will be updated based on the pre-training weights determined according to the first semantic segmentation submodule and the second semantic segmentation submodule to be fine-tuned to obtain the fine-tuning segmentation model to be trained. Therefore, in the implementation of the present disclosure, the first semantic segmentation submodule, the second semantic segmentation submodule and the instance segmentation module in the fine-tuning segmentation model can be trained as a whole on the premise of giving priority to ensuring the good training effect of the first semantic segmentation submodule and the second semantic segmentation submodule. At this time, it is only necessary to fine-tune the first semantic segmentation submodule and the second semantic segmentation submodule that have been trained, and there is no need to significantly adjust multiple modules in the fine-tuning segmentation model during the training process, and the model parameters of each module will not be inconsistent with the actual needs due to frequent and significant adjustments to multiple modules in the fine-tuning segmentation model. In this way, in the application process of the trained image segmentation model, the segmentation effect of the image segmentation model can be ensured to be good.

[0148] In some embodiments, the second preset number of times can be determined according to a preset duration. The preset duration can be the training duration of training the first semantic segmentation submodule and / or the second semantic segmentation submodule once. The second preset number of times and the preset duration can be inversely correlated. For example, the second preset number of times can be 80,000 times.

[0149] In some embodiments, the model parameters of the initial segmentation model are updated based on the pre-trained weights to obtain a fine-tuned segmentation model, including: updating the model parameters of the first semantic segmentation submodule in the initial segmentation model based on the first pre-trained weights, and updating the model parameters of the second semantic segmentation submodule in the initial segmentation model based on the second pre-trained weights to obtain a fine-tuned segmentation model. The first pre-trained weights can be determined based on the model parameters of the first semantic segmentation submodule to be fine-tuned, and the second pre-trained weights can be determined based on the second semantic segmentation submodule to be fine-tuned.

[0150] In some embodiments, the training of the first semantic segmentation submodule, the second semantic segmentation submodule, and the instance segmentation module of the fine-tuning segmentation model includes:

[0151] In the process of training the first semantic segmentation submodule, the second semantic segmentation submodule and the instance segmentation module of the fine-tuned segmentation model, configuring a first system resource for the first semantic segmentation submodule of the fine-tuned segmentation model;

[0152] configuring second system resources for the second semantic segmentation submodule of the fine-tuned segmentation model; and,

[0153] A third system resource is configured for the instance segmentation module of the fine-tuned segmentation model; wherein the first system resource is smaller than the third system resource, and / or the second system resource is smaller than the third system resource.

[0154] Here, since in the process of training the first semantic segmentation submodule, the second semantic segmentation submodule and the instance segmentation module of the fine-tuning segmentation model, the first system resource corresponding to the first semantic segmentation submodule is less than the third system resource corresponding to the instance segmentation module, and / or the second system resource corresponding to the second semantic segmentation submodule is less than the third system resource corresponding to the instance segmentation module, therefore, in the case where the first semantic segmentation submodule and the second semantic segmentation submodule in the fine-tuning segmentation model are preliminarily trained modules to be fine-tuned, only fewer system resources are needed to train and fine-tune the first semantic segmentation submodule and the second semantic segmentation submodule, and more system resources can be used to focus on training the untrained instance segmentation module. In this way, the proportion of system resources corresponding to the instance segmentation module can be increased, and the segmentation effect of the instance segmentation module can be improved while maintaining the good segmentation effect of the first semantic segmentation submodule and the second semantic segmentation submodule.

[0155] In some embodiments, the ratio of the first system resource, the second system resource, and the third system resource may be a preset ratio. The preset ratio may be pre-set. For example, the preset ratio may be 1:1:2. The preset ratio may be determined according to the structural complexity of the first semantic segmentation submodule, the second semantic segmentation submodule, and the instance segmentation module. In some embodiments, the system resources corresponding to the first semantic segmentation submodule, the second semantic segmentation submodule, and the instance segmentation module are all positively correlated with their corresponding structural complexity.

[0156] In some embodiments, the system resources may be central processing unit (CPU) resources.

[0157] In some embodiments, the feature extraction module in the image segmentation model is used to extract features from an image with multiple objects to obtain image features, including:

[0158] The feature extraction module is used to extract multi-scale features from the image to obtain the image features.

[0159] Here, since the image features are features obtained after multi-scale feature extraction of the image, in the process of segmenting the image features using the semantic segmentation module and the instance segmentation module, the features of objects of different sizes in the image can be accurately extracted based on the refined multi-scale image features, thereby accurately segmenting different objects from the image based on the accurately extracted image features. In this way, it can ensure that the segmentation effect of each object from the image is good.

[0160] In order to better understand the embodiments of the present disclosure, the image recognition scenarios and methods in the embodiments of the present disclosure are described below through some exemplary embodiments:

[0161] In some embodiments, Figure 5 As shown, exemplarily, there are at least two targets in the image. When it is necessary to blur targets other than the target target among the at least two targets, if only semantic segmentation or image segmentation based on monocular depth is performed without instance segmentation, the second target in the image will be omitted, that is, the second target in the image will not be blurred. After instance segmentation, the blurring problem existing in the related art can be solved.

[0162] In some embodiments, the image segmentation model in the embodiments of the present disclosure can be based on an end-to-end neural network, share a feature extraction module, and allow the feature extraction module to connect to two network heads at the same time. At this time, one of the network heads (i.e., the semantic segmentation module described in any of the embodiments of the present disclosure) can be used to extract semantic features from the features extracted by the feature extraction module, and another network head (i.e., the instance segmentation module described in any of the embodiments of the present disclosure) can be used to extract instance features from the features extracted by the feature extraction module. In addition, the performance of each task can be improved by performing loss constraints on each network head and using the features between each character to enhance each other between tasks.

[0163] For example, Figure 6As shown, after inputting an original image (which may be any image described in the embodiments of the present disclosure), the original image can be processed using a data enhancement module to adjust the size of the original image to the image input size corresponding to the feature extraction module, and obtain a resized image; the feature extraction module and the multi-scale neck module are used to extract multi-scale features from the resized image to obtain image features; the image features are respectively put into the head network corresponding to each task, and single or multiple task training is performed to obtain the output corresponding to each task, and the output result is obtained; the output result is post-processed to restore the size of the output result to the size corresponding to the original image, and the task result corresponding to each task is obtained.

[0164] It should be noted that the task here may be a task of outputting semantic recognition results. In this case, the head network corresponding to the task may be a network corresponding to the semantic segmentation module described in any of the embodiments of the present disclosure; the task here may be a task of outputting instance recognition results. In this case, the head network corresponding to the task may be a network corresponding to the instance segmentation module described in any of the embodiments of the present disclosure.

[0165] like Figure 6 The multi-task framework shown can be extended to other technical fields by replacing or adding the head network corresponding to the tasks in the multi-task framework. For example, the multi-task framework can be extended to the technical field of gesture recognition through semantic segmentation. At this time, the tasks here may include the task of outputting semantic recognition results and the task of outputting gesture recognition results. For example, the multi-task framework can be extended to the technical field of pedestrian re-identification through semantic segmentation. At this time, the tasks here may include the task of outputting semantic recognition results and the task of outputting pedestrian re-identification results. The tasks may also include at least one of the following: the task of detecting the area where the human body is located, the task of classifying pixels, or the task of determining the key points of the human body. Moreover, the features extracted in each task can interact with each other, that is, the features extracted in task A can be used in task B. In this way, effective features that are difficult to obtain through single-task training can be added, thereby bringing huge performance and effect benefits to the entire task.

[0166] In some embodiments, the method further comprises:

[0167] Identify at least some of the pixels in the image, and determine instance attributes and / or category attributes of the at least some of the pixels;

[0168] The identifying of the target instance of the target and identifying the category of at least part of the objects in the image to obtain the recognition result of the target includes:

[0169] Based on the instance attribute and / or the category attribute of at least part of the pixels, the target instance of the target is identified, and the category of at least part of the objects in the image is identified to obtain a recognition result of the target.

[0170] Here, the instance attributes and / or category attributes of at least some pixels can be determined at the pixel level, so as to accurately identify the target from the image in combination with finer-grained pixel information to obtain accurate target recognition results.

[0171] In some embodiments, all pixels in the image may be identified to determine instance attributes and / or category attributes of all pixels. Based on the instance attributes and / or category attributes of at least some of the pixels, the target instance of the target is identified, and the category of at least some of the objects in the image is identified to obtain the identification result of the target. Here, the target may be identified from the image based on the instance attributes and / or category attributes corresponding to all pixels of the image.

[0172] In some embodiments, for the target object described in any of the embodiments of the present disclosure, the pixels in the area where the target object is located can be identified to determine the instance attributes and / or category attributes of the pixels. Based on the instance recognition result and the semantic recognition result, and in combination with the instance attributes and / or category attributes corresponding to the pixels in the area where the target object is located, the target is identified from the image to obtain the recognition result of the target. Here, the target object can be determined first, and then the target can be quickly identified from the image based on the instance attributes and / or category attributes corresponding to a small number of pixels in the area where the target object is located.

[0173] In some embodiments, the identifying the target instance of the target and identifying the category of at least part of the objects in the image based on the instance attribute and / or the category attribute of the at least part of the pixels to obtain the recognition result of the target includes: determining the target pixels whose instance attribute and the category attribute meet the target requirements based on the instance attribute and the category attribute; wherein the target is composed of the target pixels;

[0174] Based on the target pixels, a target instance of the target is identified, and categories of at least part of the objects in the image are identified to obtain a recognition result of the target.

[0175] Here, since the target instance of the target is identified based on the target pixels, and the categories of at least some objects in the image are identified to obtain the recognition result of the target, and the target is composed of the target pixels, and the instance attributes and the category attributes of the target pixels meet the target requirements, the target pixels that meet the target requirements can be determined first, and based on the target pixels that meet the target requirements, the target instance of the target can be quickly identified in the area where the target pixels are located, and the categories of at least some objects in the image can be quickly identified to obtain the recognition result of the target.

[0176] In some embodiments, the instance attributes corresponding to different targets may be different, the category attributes corresponding to different targets may be the same, or the category attributes corresponding to different targets may be different.

[0177] In some embodiments, performing the differentiation processing operation on the identified target and / or the image area other than the target includes:

[0178] performing a first differentiation processing operation on the target; and / or,

[0179] performing a second distinguishing processing operation on the image area other than the target;

[0180] The processing effect corresponding to the first distinguishing processing operation is different from the processing effect corresponding to the second distinguishing processing operation.

[0181] Here, the types of the first distinguishing processing operation and the second distinguishing processing operation may not be limited, and it is only necessary to ensure that the processing effect corresponding to the first distinguishing processing operation is different from the processing effect corresponding to the second distinguishing processing operation. Exemplarily, the first distinguishing processing operation may be used to adjust at least one of the exposure, brightness, contrast, and saturation corresponding to the target. The first distinguishing processing operation may be a beauty operation for beautifying the target. Exemplarily, the second distinguishing processing operation may be used to blur the image area outside the target to highlight the area where the target is located.

[0182] In some embodiments, both the first distinguishing processing operation and the second distinguishing processing operation can be used to highlight the target in the image. For example, the processing effects corresponding to the first distinguishing processing operation and the second distinguishing processing operation can both visually enhance the brightness corresponding to the target to highlight the target in the image.

[0183] In some embodiments, there are multiple targets to be identified; performing the differentiation processing operation on the identified targets includes:

[0184] performing the first distinguishing processing operation on some of the multiple targets, and performing the second distinguishing processing operation on other targets except the some of the targets; or,

[0185] Performing different first distinguishing processing operations on each of the targets; or,

[0186] The second distinguishing processing operation is performed differently on each of the targets.

[0187] Here, the corresponding distinguishing processing operations can be flexibly performed on the multiple targets identified. Compared with the method in the related art that can only perform the same distinguishing processing operation on the multiple targets to be identified, the embodiment of the present disclosure can provide a more flexible method for performing distinguishing processing operations to ensure the flexibility of performing distinguishing processing operations on different targets.

[0188] In some embodiments, the first distinguishing and processing operation includes a beautifying operation, and performing different first distinguishing and processing operations on each of the targets includes:

[0189] Determining a beauty level corresponding to each target based on the area range and / or depth information of the area where each target is located;

[0190] According to the beautification degree, a corresponding beautification operation is performed on each of the targets.

[0191] In some embodiments, the area range of the area where the target is located is positively correlated with the beautification degree. That is, the larger the area range of the area where the target is located, the higher the beautification degree. Alternatively, the smaller the area range of the area where the target is located, the lower the beautification degree. The depth information corresponding to the target and the beautification degree may be positively correlated. That is, in the image, the closer the target is visually located to the front, the higher the beautification degree corresponding to the target, or the further back the target is visually located to the back, the lower the beautification degree corresponding to the target.

[0192] In some embodiments, the first distinguishing processing operation includes a brightness enhancement operation, and performing different first distinguishing processing operations on each of the targets includes:

[0193] Determining the brightness enhancement degree corresponding to each target based on the area range and / or depth information of the area where each target is located;

[0194] According to the brightness enhancement degree, a corresponding brightness enhancement operation is performed on each of the targets.

[0195] In some embodiments, the size of the area range of the area where the target is located is positively correlated with the degree of brightness enhancement. That is, the larger the area range of the area where the target is located, the higher the degree of brightness enhancement. Alternatively, the smaller the area range of the area where the target is located, the lower the degree of brightness enhancement. The depth information corresponding to the target and the degree of beauty can be inversely correlated. That is, in the image, the closer the target is visually located to the front, the higher the degree of brightness enhancement corresponding to the target, or, the further back the target is visually located, the lower the degree of brightness enhancement corresponding to the target.

[0196] In some embodiments, the second distinguishing and processing operation may be a virtualization operation, wherein the first distinguishing and processing operation is performed on some of the multiple targets, and the second distinguishing and processing operation is performed on other targets except the some of the targets, including:

[0197] Determining a preset target from a plurality of said targets;

[0198] The first distinguishing operation is performed on the preset target, and the second distinguishing operation is performed on other targets except the preset target. The number of preset targets can be one or more.

[0199] In some embodiments, a first distinguishing processing operation may be performed on the target, and a second distinguishing processing operation may be performed on the image region other than the target. It is understandable that the image region other than the target does not include the region where the target is located.

[0200] In some embodiments, a first distinguishing processing operation may be performed on a preset target determined from the targets, and a second distinguishing processing operation may be performed on an image region other than the preset target. It is understood that, in the case where there are multiple targets, if all of the multiple targets are preset targets, the image region does not include the region where the targets are located; or, if there are non-preset targets among the multiple targets, the image region includes targets other than the preset targets.

[0201] In some embodiments, the second distinguishing processing operation is a virtualization operation; and performing different second distinguishing processing operations on each of the targets respectively includes:

[0202] Determining a blur degree corresponding to each target based on depth information of each target in the image;

[0203] According to the blurring degree, corresponding blurring operations are performed on each of the targets respectively.

[0204] Here, we can adapt to the depth information of each target in the image and accurately perform corresponding blur operations on each target according to different blur degrees to ensure a good blur effect.

[0205] In some embodiments, the depth information corresponding to the target is used to characterize the visual depth of the target in the image, and the visual depth is inversely correlated with the degree of blur. That is, in the image, the closer the target is visually located in the front, the lower the degree of blur corresponding to the target; or, the further back the target is visually located in the back, the higher the degree of blur corresponding to the target.

[0206] In some embodiments, among multiple targets, for a target with the smallest visual depth, the blurring degree may be 0. That is, when there are multiple targets in an image, the blurring operation may not be performed for the frontmost target.

[0207] In some embodiments, the method further comprises:

[0208] In the case where the current scene is a target usage scene, determining that the distinguishing processing operation needs to be performed on the target in the image;

[0209] The target usage scenario includes at least one of the following:

[0210] Portrait recognition scene, vehicle recognition scene and scene recognition scene.

[0211] Here, the portrait recognition scene may be a scene for recognizing a human body in an image, and in this case, in the portrait recognition scene, the target in the image includes the human body. The vehicle recognition scene may be a scene for recognizing a vehicle in an image, and in this case, in the vehicle recognition scene, the target in the image includes the vehicle. The scene recognition scene may be a scene for recognizing a scene in an image, and in this case, in the scene recognition scene, the target in the image includes the scene.

[0212] For example, please refer again to Figure 2 , portrait recognition scenarios may include scenarios for recognizing the human body in intelligent monitoring mode, scenarios for recognizing the human body in automatic driving mode, and scenarios for recognizing the human body in mobile phone camera mode.

[0213] For example, a portrait recognition scenario may be: a scenario where a human body needs to be processed in portrait mode during mobile phone photography. A vehicle recognition scenario may be: a scenario where a vehicle in the surrounding environment needs to be recognized during the process of automatic driving. A scene recognition scenario may be: a scenario where a scene in the environment needs to be monitored to ensure the safety of the monitoring environment during intelligent monitoring.

[0214] It should be noted that the above target usage scenarios are only exemplary usage scenarios, and the embodiments of the present disclosure include but are not limited to the exemplary usage scenarios shown above. Figure 7 As shown, an embodiment of the present disclosure provides an image acquisition device, the device comprising:

[0215] The recognition module 71 is used to recognize the target instance of the target and recognize the category of at least part of the objects in the image to obtain the recognition result of the target when it is determined that a distinguishing processing operation needs to be performed on the target in the image;

[0216] The processing module 72 is used to perform the differentiation processing operation on the identified target and / or the image area other than the target according to the recognition result to obtain the target image.

[0217] Figure 8 6 is a block diagram of an electronic device 600 according to an exemplary embodiment. The disclosed method can be applied to the electronic device.

[0218] Reference Figure 8 , the electronic device 600 may include one or more of the following components: a processing component 602 , a memory 604 , a power component 606 , a multimedia component 608 , an audio component 610 , an input / output (I / O) interface 612 , a sensor component 614 , and a communication component 616 .

[0219] The processing component 602 generally controls the overall operation of the electronic device 600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 602 may include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate the interaction between the multimedia component 608 and the processing component 602.

[0220] The memory 604 is configured to store various types of data to support operations on the device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phone book data, messages, pictures, videos, etc. The memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0221] The power supply component 606 provides power to the various components of the electronic device 600. The power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 600.

[0222] The multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not sense the boundaries of the touch or slide action, but also detects the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the device 600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0223] The audio component 610 is configured to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC), and when the electronic device 600 is in an operation mode, such as a call mode, a recording mode, and a voice determination mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 604 or sent via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting audio signals.

[0224] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0225] The sensor assembly 614 includes one or more sensors for providing various aspects of status assessment for the electronic device 600. For example, the sensor assembly 614 can detect the open / closed state of the device 600, the relative positioning of components, such as the display and keypad of the electronic device 600, and the sensor assembly 614 can also detect the position change of the electronic device 600 or a component of the electronic device 600, the presence or absence of contact between the user and the electronic device 600, the orientation or acceleration / deceleration of the electronic device 600, and the temperature change of the electronic device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 614 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0226] The communication component 616 is configured to facilitate wired or wireless communication between the electronic device 600 and other devices. The electronic device 600 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency determination (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0227] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0228] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, and the instructions can be executed by the processor 820 of the electronic device 600 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0229] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed in this disclosure.

[0230] It will be appreciated that the present invention is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. An image acquisition method, characterized in that: The method comprises: In the case where it is determined that a distinguishing processing operation needs to be performed on a target in the image, a target instance of the target is identified, and categories of at least part of the objects in the image are identified to obtain a recognition result of the target; According to the recognition result, the distinguishing processing operation is performed on the recognized target and / or the image area other than the target to obtain a target image.

2. The method according to claim 1, characterized in that The identifying of the target instance of the target and identifying the category of at least part of the objects in the image to obtain the recognition result of the target includes: Recognizing a target instance of the target, identifying the target object from the image and determining the edge of the target object; Identify the category of at least part of the object in the image to determine the category to which each part of the target object belongs; Determining whether the target object includes content other than the target according to the categories to which each part of the target object belongs; In response to the target object including content other than the target, the target object and the edge of the target object are adjusted according to the category to which each part belongs, so as to obtain a recognition result of the target.

3. The method according to claim 2, characterized in that The number of the target objects is multiple; identifying the target objects from the image and determining the edges of the target objects includes: In response to the existence of an overlapping area between at least two of the target objects, identifying each of the target objects from the image and determining an edge of each of the target objects; and / or, In response to the distance between at least two of the target objects being less than a distance threshold, each of the target objects is identified from the image and an edge of each of the target objects is determined.

4. The method according to claim 2, characterized in that: The step of identifying the target object from the image and determining the edge of the target object comprises: Using an instance segmentation algorithm to segment the image features of the image, obtaining instance features and weight features of each of the target objects; wherein the instance features are used to characterize the region of interest corresponding to the target object in the image; and the weight features include a weight value corresponding to the instance features, a category, and a confidence level that the region of interest is located in the foreground region of the image; Based on the instance features and the weight features, edges of the target object are determined.

5. The method according to claim 2, characterized in that: Determining the category to which each part of the target object belongs includes: The image features of the image are segmented using a semantic segmentation algorithm to obtain semantic features and edge features of the image; wherein the semantic features are used to characterize the category to which at least part of the pixels in the image belong; and the edge features include features of the contour edge of the entire target in the image; Based on the semantic features and the edge features, the categories to which the various parts of the target object belong are determined.

6. The method according to claim 1, characterized in that The method further comprises: Identify at least some of the pixels in the image, and determine instance attributes and / or category attributes of the at least some of the pixels; The identifying of the target instance of the target and identifying the category of at least part of the objects in the image to obtain the recognition result of the target includes: Based on the instance attribute and / or the category attribute of at least part of the pixels, the target instance of the target is identified, and the category of at least part of the objects in the image is identified to obtain a recognition result of the target.

7. The method according to claim 6, characterized in that The identifying the target instance of the target based on the instance attribute and / or the category attribute of at least part of the pixels, and identifying the category of at least part of the objects in the image to obtain the recognition result of the target includes: Based on the instance attribute and the category attribute, determining a target pixel whose instance attribute and the category attribute meet target requirements; wherein the target is composed of the target pixels; Based on the target pixels, a target instance of the target is identified, and categories of at least part of the objects in the image are identified to obtain a recognition result of the target.

8. The method according to claim 1, characterized in that The performing the distinguishing processing operation on the identified target and / or the image area other than the target comprises: performing a first differentiation processing operation on the target; and / or, performing a second distinguishing processing operation on the image area other than the target; The processing effect corresponding to the first distinguishing processing operation is different from the processing effect corresponding to the second distinguishing processing operation.

9. The method according to claim 8, characterized in that The number of the targets to be identified is multiple; the performing the distinguishing processing operation on the identified targets includes: performing the first distinguishing processing operation on some of the multiple targets, and performing the second distinguishing processing operation on other targets except the some of the targets; or Performing different first distinguishing processing operations on each of the targets respectively; or The second distinguishing processing operation is performed differently on each of the targets.

10. The method according to claim 9, characterized in that The second distinguishing and processing operation is a virtualization operation; and performing different second distinguishing and processing operations on each of the targets respectively includes: Determining a blur degree corresponding to each target based on depth information of each target in the image; According to the blurring degree, corresponding blurring operations are performed on each of the targets respectively.

11. The method according to any one of claims 1 to 10, characterized in that: The method further comprises: In the case where the current scene is a target usage scene, determining that the distinguishing processing operation needs to be performed on the target in the image; The target usage scenario includes at least one of the following: Portrait recognition scene, vehicle recognition scene and scene recognition scene.

12. An image acquisition device, characterized in that: The device comprises: A recognition module, configured to, when it is determined that a distinguishing processing operation needs to be performed on a target in an image, recognize a target instance of the target and recognize the category of at least part of the objects in the image to obtain a recognition result of the target; A processing module is used to perform the differentiation processing operation on the identified target and / or the image area other than the target according to the recognition result to obtain a target image.

13. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method according to any one of claims 1 to 11 when running the executable instructions.

14. A computer-readable storage medium, characterized in that: The readable storage medium stores an executable program, wherein the executable program implements the method according to any one of claims 1 to 11 when executed by a processor.

Images