Image processing method and device, electronic equipment and storage medium
By identifying and cropping the target object region in the image and optimizing image operations by combining depth information, the problems of high power consumption and difficulty in target object tracking in high-pixel image processing are solved, achieving more efficient image processing and target object tracking.
Patent Information
- Application Number
- CN202310596451.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-24
- Publication Date
- 2024-11-26
AI Technical Summary
Existing technologies consume a lot of computing resources when processing high-pixel images, resulting in high power consumption of electronic devices and making it difficult to track target objects.
By identifying the region of the target object in the image, cropping it, and performing preset image operations based on depth information, the processing area is reduced, the amount of data is decreased, and the image processing workflow is optimized.
It reduces the power consumption of electronic devices, improves image processing efficiency, and enhances the tracking capability of target objects.
Smart Images

Figure CN119027447A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method and device, an electronic device, and a storage medium. Background Art
[0002] Currently, the image processing function of electronic devices is a commonly used function of users, and users have increasingly higher requirements for the display effects of images. From the process of capturing images to screen display, various algorithms are used to optimize the visual effects of captured images.
[0003] During the optimization process of related technologies, for image acquisition modules with more than one million pixels in current electronic devices, some algorithms calculate large amounts of data, so the calculation process will occupy a large amount of hardware resources, resulting in high power consumption of electronic devices, causing the electronic devices to heat up. Other algorithms have the problem of difficulty in locating the target object during the tracking and shooting process. Summary of the invention
[0004] The present disclosure provides an image processing method and device, an electronic device and a storage medium.
[0005] A first aspect of the present disclosure provides an image processing method, which is applied to an electronic device. The method includes:
[0006] When it is necessary to perform a preset image operation related to the target object, determining a first area in the first image where the target object is located;
[0007] Cropping the first image according to the first area to obtain a second image; wherein the second image at least includes the first area;
[0008] The preset image operation is performed on the second image according to the depth information corresponding to the first image.
[0009] Based on the above solution, when it is necessary to perform a preset image operation related to the target object, determining a first area where the target object is located in the first image includes:
[0010] When it is necessary to perform the preset image operation related to the target object, detecting the target object on the first image to obtain a second area containing the target object;
[0011] According to the depth value of each pixel in the second area obtained from the depth information, pixels in the second area and pixels in an area adjacent to the second area whose depth difference is less than a depth tolerance are assigned to a connected area, thereby obtaining the first area including the connected area.
[0012] Based on the above scheme, the depth value of each pixel in the second area is obtained from the depth information, and the pixels in the second area and the pixels in the adjacent area of the second area whose depth difference is less than the depth tolerance are attributed to a connected area, so as to obtain the first area containing the target object, including:
[0013] Assigning pixels whose depth difference is less than the depth tolerance to a connected area;
[0014] The connected area is expanded according to a preset image size specification to obtain the first area, wherein the first area at least includes the connected area, and the size of the first area meets the preset image size specification.
[0015] Based on the above solution, the method of expanding the connected region according to a preset image size specification to obtain the first region includes:
[0016] The connected region is expanded according to the aspect ratio in the preset image size specification to obtain the first region satisfying the aspect ratio.
[0017] Based on the above solution, the method further includes:
[0018] Determine a first pixel having a maximum depth value and a second pixel having a minimum depth value in the second area;
[0019] The depth tolerance is determined according to a difference between the depth values corresponding to the first pixel and the second pixel.
[0020] Based on the above solution, the step of cropping the first image according to the first area to obtain the second image includes:
[0021] Determining image coordinates of the first area according to an outer contour of the target object in the first image;
[0022] The first image is cropped according to the image coordinates to obtain a second image containing the target object.
[0023] Based on the above solution, performing the preset image operation on the second image according to the depth information includes:
[0024] According to the depth information, obtaining an average depth value of pixels in the area where the target object is located in the second image;
[0025] Determine pixels whose difference between the depth value in the second image and the average depth value is greater than a first threshold as pixels to be processed;
[0026] The preset image operation is performed on the pixel to be processed.
[0027] Based on the above solution, the preset image operation at least includes: background blur operation.
[0028] Based on the above solution, the method further includes:
[0029] Determining a shooting mode of the electronic device;
[0030] When the shooting mode is a portrait mode, it is determined that a preset image operation related to the target object needs to be performed, wherein the target object is a portrait.
[0031] Based on the above solution, determining the first area where the target object is located in the first image includes:
[0032] Determine whether to enable the target object centering display function;
[0033] When the target object centering display function is turned on, a first area where the target object is located is determined, and the target object is placed at the center of the first area.
[0034] A second aspect of the present disclosure provides an image processing device, which is applied to an electronic device, and the device includes:
[0035] A first determining module, configured to determine a first area in a first image where the target object is located when a preset image operation related to the target object needs to be performed;
[0036] A cropping module, configured to crop the first image according to the first area to obtain a second image; wherein the second image at least includes the first area;
[0037] An execution module is used to perform the preset image operation on the second image according to the depth information corresponding to the first image.
[0038] Based on the above solution, the device also includes:
[0039] a detection module, configured to perform target object detection on the first image to obtain a second area containing the target object when the preset image operation related to the target object needs to be performed;
[0040] A first attribution module is used to assign pixels in the second area and pixels in an adjacent area of the second area whose depth difference is less than a depth tolerance to a connected area based on the depth value of each pixel in the second area obtained from the depth information, thereby obtaining the first area including the connected area.
[0041] Based on the above solution, the device also includes:
[0042] A second attribution module, configured to attribute pixels whose depth difference is less than the depth tolerance to a connected area;
[0043] An expansion module is used to expand the connected area according to a preset image size specification to obtain the first area, wherein the first area at least includes the connected area, and the size of the first area meets the preset image size specification.
[0044] Based on the above solution, the expansion module is specifically used to expand the connected area according to the length-to-width ratio in the preset image size specification to obtain the first area that meets the length-to-width ratio.
[0045] Based on the above solution, the device also includes:
[0046] A second determining module, configured to determine a first pixel with a maximum depth value and a second pixel with a minimum depth value in the second area;
[0047] The third determining module is configured to determine the depth tolerance according to a difference between the depth values corresponding to the first pixel and the second pixel.
[0048] Based on the above solution, the cropping module is specifically used to determine the image coordinates of the first area according to the outer contour of the target object in the first image; and perform a cropping operation on the first image according to the image coordinates to obtain a second image containing the target object.
[0049] Based on the above scheme, the execution module is specifically used to obtain the average depth value of pixels in the area where the target object is located in the second image according to the depth information; determine the pixels whose difference between the depth value in the second image and the average depth value is greater than a first threshold as pixels to be processed; and perform the preset image operation on the pixels to be processed.
[0050] Based on the above solution, the preset image operation at least includes: background blur operation.
[0051] Based on the above solution, the device also includes:
[0052] A fourth determining module, used to determine a shooting mode of the electronic device;
[0053] The fifth determining module is used to determine that a preset image operation related to the target object needs to be performed when the shooting mode is the portrait mode, wherein the target object is a portrait.
[0054] Based on the above scheme, the first determination module is specifically used to determine whether to turn on the target object centering display function; when the target object centering display function is turned on, determine the first area where the target object is located, and place the target object at the center of the first area.
[0055] A third aspect of the present disclosure provides an electronic device, including:
[0056] a memory for storing processor-executable instructions;
[0057] A processor connected to the memory;
[0058] Wherein, the processor is configured to execute the image processing method provided in any one of the above items.
[0059] A fourth aspect of an embodiment of the present disclosure provides a non-temporary computer-readable storage medium. When instructions in the storage medium are executed by a processor of a computer, the computer is enabled to execute an image processing method as provided in any one of the above items.
[0060] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:
[0061] The image processing method provided by the embodiment of the present disclosure determines the first area where the target object is located from the first image when it is necessary to perform a preset image operation related to the target object, and cuts the first image according to the first area to obtain the second image, and performs the preset image operation on the second image according to the depth information corresponding to the first image. In this way, on the one hand, by cutting the first image according to the first area to obtain the second image, the preset image operation is performed on the second image containing the target object, and the image area to be processed by performing the preset image operation is reduced, thereby reducing the amount of data to be processed by performing the preset image operation, thereby reducing the power consumption of the electronic device and improving the efficiency of image processing. On the other hand, by determining whether there is a target object in the first image before performing the preset image operation related to the target object, the cutting operation and the preset image operation are performed only when the target object exists, so that the preset image processing operation is not performed when the target object is not included in the first image, so that the amount of picture processing data of the electronic device is further reduced. In addition, by cutting the first image according to the first area to obtain the second image, since the amount of data of the second image is less than that of the first image, the target object can be tracked more easily in the second image. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0063] Figure 1is a flowchart of an image processing method according to an exemplary embodiment;
[0064] Figure 2 is a flowchart of an image processing method according to an exemplary embodiment;
[0065] Figure 3 is a flowchart of an image processing method according to an exemplary embodiment;
[0066] Figure 4 is a flowchart of an image processing method according to an exemplary embodiment;
[0067] Figure 5 is a flowchart of an image processing method according to an exemplary embodiment;
[0068] Figure 6 is a schematic structural diagram of an image processing device according to an exemplary embodiment;
[0069] Figure 7 It is a schematic structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION
[0070] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices consistent with some aspects of the present disclosure as detailed in the appended claims.
[0071] like Figure 1 As shown, an embodiment of the present disclosure provides an image processing method, which is applied to an electronic device, and the method includes:
[0072] S1100: When a preset image operation related to a target object needs to be performed, determining a first area where the target object is located in a first image;
[0073] S1200: Crop the first image according to the first area to obtain a second image; wherein the second image at least includes the first area;
[0074] S1300: Perform the preset image operation on the second image according to the depth information corresponding to the first image.
[0075] The electronic devices include fixed terminals and mobile terminals. For example, the mobile terminals include but are not limited to smart phones, tablet computers, laptop computers or smart wearable devices, etc. For example, the fixed terminals include but are not limited to desktop computers, etc.
[0076] The target object includes but is not limited to a portrait, an animal, an object or a scene, etc. The target object may be determined before capturing the first image, for example, when it is detected that the selected capture mode is a portrait mode, the target object is determined to be a portrait.
[0077] The above-mentioned preset image operations include but are not limited to blurring of the image background and / or various beautification operations of the target object in the image.
[0078] The above-mentioned depth information is the depth information contained in the depth image corresponding to the first image. The depth information here includes at least a depth value, and the depth value can be a relative depth, which can represent the relative distance relationship between pixels. The depth information here can be obtained by a monocular depth estimation or a binocular depth estimation algorithm.
[0079] The first image may be a picture file in RAW (RAW Image Format) image format acquired by an image acquisition module of the electronic device, or a picture file in YUV image format. The RAW picture is an uncompressed picture generated by an image sensor, that is, an original picture acquired by the image acquisition module.
[0080] The above YUV image format can be obtained by converting the data format based on the RAW image. The YUV image format includes three components, Y, U, and V, for each pixel. YUV refers to a pixel format in which the brightness parameter and the chrominance parameter are separately represented, where "Y" represents brightness, that is, the grayscale value, and "U" and "V" represent chrominance.
[0081] The first image may include one or more target objects, and the categories of the multiple target objects may be the same or different. For example, the first image includes a portrait and an animal, or the first image includes two portraits. Then, when the first image includes multiple target objects, the first area includes multiple target objects.
[0082] The image processing method provided by the embodiment of the present disclosure can be applied to image processing of a first image containing a target object captured when taking photos, recording videos, live broadcasting or making video calls using the front camera or rear camera of an electronic device. For example, when it is detected that the front camera captures a portrait A, a first area of the portrait A in the first image is determined. Thus, the first image is cropped according to the first area to obtain a second image, and a preset image operation is performed on the second image using the depth image corresponding to the first image.
[0083] For example, in a video call or video recording, when the front camera is used to shoot the target object, the display area is usually small, and only the target object needs to be displayed in the display area, that is, the display area only needs to display the second image mentioned above. However, the size and data volume of the first image shot by the camera are usually large. If the preset image operation is performed directly on the first image, the amount of image processing data will be large, which will cause the electronic device to consume more power and affect the data processing speed. Therefore, before performing the preset image operation, the first image is cropped according to the first area to obtain the second image, and the preset image operation is performed only on the second image that needs to be displayed in the display area, thereby reducing the amount of data for performing the preset image operation and improving the efficiency of image processing.
[0084] In one embodiment, during the process of shooting an image, a trigger control for a portrait mode is provided in the shooting interface. When it is detected that the trigger control for the portrait mode is selected, it is determined that the target object is a portrait, and the preset operation related to the portrait includes at least a background blur operation. In other words, when it is necessary to perform a background blur operation related to a portrait, the first area where the portrait is located is determined by an image recognition algorithm. The first image is cropped according to the first area to obtain a second image, where the second image includes at least a portrait. Thus, the background blur operation is performed on the second image according to the depth image corresponding to the first image. In this way, the portrait in the second image can be better highlighted, and the preset image operation is performed only on the second image that needs to be displayed in the display area, thereby reducing the amount of data for performing the background blur operation and improving the efficiency of image processing.
[0085] In one embodiment, the target object may be a pre-configured image, and when the first image is captured, the execution of the preset image operation is automatically triggered based on contour sampling detection of multiple first images.
[0086] The first area may be a regular or irregular area, for example, the first area may be a rectangle, a circle, an ellipse, a pentagon or a heart shape. The shape of the first area may be any shape preset by the user, and the user's personalized needs can be met by the user's preset shape of the first area.
[0087] The image processing method provided by the embodiment of the present disclosure determines a first area in a first image where a target object is located, that is, determines the size and boundary coordinates of the first area, and performs a cropping operation on the first image according to the size and boundary coordinates of the first area to obtain a second image, thereby performing a preset image operation on the second image according to the depth information in the depth image corresponding to the first image.
[0088] In one embodiment, the size of the depth image corresponding to the first image is the same as the size of the first image, and the number of pixels is also the same. Specifically, the depth value of the x-th pixel in the first image is the depth value of the x-th pixel in the depth image, where x is less than or equal to Y, and Y is the number of pixels contained in the first image.
[0089] In another embodiment, in order to improve the speed of calculating the depth information, a preset image operation is performed on the second image according to the depth information corresponding to the first image. The depth information here is the depth information of the depth image corresponding to the first image. Specifically, the process of obtaining the depth information corresponding to the first image here is: proportionally reducing the first image according to a preset scaling ratio, and obtaining the depth information of the depth image corresponding to the reduced image. Specifically, during the scaling process, the first image can be split into multiple rectangular pixel areas according to the preset scaling ratio, and the depth information of the pixels in the upper left corner of each rectangular pixel area is obtained respectively, so as to obtain the depth image of the reduced image.
[0090] The depth information of the depth image corresponding to the reduced image is magnified by the reciprocal of the preset magnification ratio (i.e., each rectangular pixel area uses the depth information of the pixel in the upper left corner of the area as the overall depth information of the rectangular pixel area) to obtain the depth information corresponding to the first image. In this way, the amount of data for calculating the depth information of the first image can be reduced, and the efficiency of obtaining the depth information of the first image can be improved.
[0091] Exemplarily, the preset zoom ratio may be 1 / 8, 1 / 6, 1 / 4, 1 / 3 or 1 / 2, etc. The preset zoom ratio may be determined based on a large amount of image processing experience or experiments.
[0092] For example, for a first image of size 1920x1080, when the preset scaling ratio is 1 / 4, the depth image size corresponding to the reduced image is only 480x270 (scaled to 1 / 4). Specifically, the first image is split into multiple 2*2 rectangular pixel areas, and the depth information of the pixel in the upper left corner of each rectangular pixel area is obtained respectively, so as to obtain the depth image of the reduced image. Then, the reciprocal of the preset scaling ratio is 4, and the depth information of each pixel in the depth image corresponding to the reduced image is magnified 4 times (that is, the depth information of the pixel in the upper left corner of the rectangular pixel area in the 2*2 area is used as the depth information of the 2*2 area), and the depth information corresponding to the first image is obtained.
[0093] In S1300, the depth information of the first image is determined. Specifically, the depth information of the first image may include Y pixels on the first image and the depth value of each pixel in the Y pixels, where Y is a positive integer. A depth image of the first image may be generated according to the depth information of the first image. Assuming that the first image includes Y pixels, the depth image of the first image can be generated based on the depth values of the Y pixels. Therefore, the depth image reflects the distance between each pixel in the first image and the camera module.
[0094] In one embodiment, the depth values of Y pixels on the first image may be determined by a preset algorithm to obtain the depth information. Specifically, the preset algorithm may include but is not limited to a depth estimation network, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), etc.
[0095] In one embodiment, when capturing the first image, if a camera module has a depth sensor, depth information of the first image may be acquired through the depth sensor, where the depth information includes at least the depth value of each pixel in the first image.
[0096] In one embodiment, the S1300 includes: cutting the depth image corresponding to the first image according to the first area to obtain the depth image corresponding to the second image. According to the depth image corresponding to the second image, performing the preset image operation on the second image. The size of the depth image corresponding to the second image is the same as the size of the second image, and the number of pixels is also the same. In this way, by using only the depth image of the second image to perform the preset image operation on the second image, the amount of data for performing the preset image operation is reduced, thereby improving the efficiency of performing the preset image operation.
[0097] In one embodiment, when there is no target object in the captured scene, it is determined that there is no need to perform a preset image operation related to the target object, and a first image captured by an image capture module of the electronic device is obtained, and related image processing operations are performed on the first image.
[0098] The image processing method provided by the embodiment of the present disclosure determines the first area where the target object is located from the first image when it is necessary to perform a preset image operation related to the target object, and cuts the first image according to the first area to obtain the second image, and performs the preset image operation on the second image according to the depth image corresponding to the first image. In this way, on the one hand, by cutting the first image according to the first area to obtain the second image, the preset image operation is performed on the second image containing the target object, and the image area to be processed by performing the preset image operation is reduced, thereby reducing the amount of data to be processed by performing the preset image operation, thereby reducing the power consumption of the electronic device and improving the efficiency of image processing. On the other hand, by determining whether there is a target object in the first image before performing the preset image operation related to the target object, the cutting operation and the preset image operation are performed only when the target object exists, so that the preset image processing operation is not performed when the target object is not included in the first image, so that the amount of picture processing data of the electronic device is further reduced. In addition, by cutting the first image according to the first area to obtain the second image, since the amount of data of the second image is less than that of the first image, it is easier to track the target object in the second image.
[0099] In one embodiment, in S1300, a third image is generated after a preset image operation is performed on the second image according to the depth image corresponding to the first image, and the third image is stored in the first directory of the electronic device. By storing the third image, it is convenient for the user to edit the third image again, such as performing beautification, filtering or graffiti operations, and also share the third image.
[0100] In one embodiment, during the image capture process, the first image captured by the image capture module is stored in the second directory of the electronic device. By storing the first image, it is convenient for the user to edit the first image, such as performing beautification, filtering, or graffiti operations, and the first image can also be shared.
[0101] The first directory and the second directory may be the same or different, and the specific positions of the first directory and the second directory may be set according to specific circumstances. When the first directory and the second directory are different, it is convenient for the user to search for the first image and the third image by classification.
[0102] In some embodiments, Figure 2 As shown, the S1100 includes:
[0103] S1110: When the preset image operation related to the target object needs to be performed, detecting the target object on the first image to obtain a second area containing the target object;
[0104] S1120: According to the depth value of each pixel in the second area obtained from the depth information, pixels in the second area and pixels in an area adjacent to the second area whose depth difference is less than a depth tolerance are assigned to a connected area, thereby obtaining the first area including the connected area.
[0105] When determining the first region where the target object is located in the first image, it is necessary to perform target object detection on the first image to obtain a second region, wherein the number of pixels included in the second region is less than the number of pixels in the first region, that is, the second region can be understood as a subregion of the first region, and the second region at least includes the target object. For example, when the target object is a portrait, the second region at least includes the facial region of the person.
[0106] Since the second area obtained by target detection on the first image may have a neck or torso that belongs to the adjacent area of the second area, but is not included in the second area. Therefore, it is necessary to determine whether there are pixels in the adjacent area of the second area with a depth difference less than the depth tolerance through the depth value of each pixel in the second area, and to include all pixels in the second area and the adjacent area of the second area with a depth difference less than the depth tolerance into a connected area, thereby obtaining the first area containing the target object. The above-mentioned connected area can be understood as a sub-area of the first area.
[0107] In S1110, the target object may be detected on the first image by an image recognition algorithm to obtain the second region. The type of the target object may be input into the image recognition algorithm, and the type of the target object may include at least one of a portrait, an animal, a plant, and a scene. For example, when the category of the target object is a portrait, facial features of the portrait may be detected by performing facial recognition on the first image to obtain the second region, that is, the second region at least includes the facial features of the portrait.
[0108] The depth values of the pixels in the connected area are the same, or the depth difference between any two pixels is within the depth tolerance range.
[0109] In an embodiment, the depth tolerance may be a maximum depth difference of pixels in the second area, wherein the maximum depth difference may be understood as a difference between a maximum depth value and a minimum depth value of pixels in the second area.
[0110] In another embodiment, the depth tolerance is determined through experiments and experience of processing a large number of images.
[0111] The disclosed embodiment obtains a first region containing a target object by assigning pixels whose depth difference with pixels in the second region and the adjacent region of the second region is less than the depth tolerance into a connected region. For example, when the second region is the face of a portrait, the highest point of the face is the nose and the lowest point is the ear. The depth tolerance here can be understood as the distance difference between the nose and the ear. In this way, pixels whose depth difference with the adjacent region of the second region is less than the depth tolerance are also assigned to the first region, reducing the possibility of missing pixels in the first region, so that the first region contains a more complete target object. At the same time, the second image obtained by cropping the first image according to the first region also has a relatively complete target object, thereby improving the integrity of the second image information.
[0112] In some embodiments, the S1120 includes:
[0113] Assigning pixels whose depth difference is less than the depth tolerance to a connected area;
[0114] The connected area is expanded according to a preset image size specification to obtain the first area, wherein the first area at least includes the connected area, and the size of the first area meets the preset image size specification.
[0115] The above-mentioned preset image size specifications may include specifications of various shapes, for example, rectangular, circular, elliptical, pentagonal or heart-shaped specifications.
[0116] Usually, a connected region connects multiple different target objects, or connects multiple image elements in one target object, so the connected region may be an irregular arbitrary shape.
[0117] The connected region can be expanded by using the preset image size specification in the embodiment of the present disclosure, thereby obtaining a first region having a shape corresponding to the preset image size specification, and the first region at least includes the connected region. Thus, the first image can be cut according to the first region to obtain a second image, and the size of the second image also meets the preset image size specification.
[0118] In one embodiment, a plurality of image size specifications are stored in the electronic device, and when it is detected that one of the plurality of image size specifications is selected, the selected size specification is used as a preset image size specification.
[0119] It can be understood that before or during shooting, the user can select one of multiple image size specifications as the preset image size specification. By determining the preset image size specification, the size specification of the final second image meets the user's expectations, thereby improving the user experience.
[0120] In some embodiments, the step of expanding the connected region according to a preset image size specification to obtain the first region includes:
[0121] The connected region is expanded according to the aspect ratio in the preset image size specification to obtain the first region satisfying the aspect ratio.
[0122] The aspect ratio in the above-mentioned preset image size specification can be understood as when the preset image size specification is a rectangle, the aspect ratio is the ratio of the length to the width of the rectangle.
[0123] The above aspect ratio may be 1:1, 4:3 or 16:9, etc. The embodiment of the present disclosure does not specifically limit the aspect ratio.
[0124] Determine the aspect ratio in the preset image size specification, determine the first area where the target object is located in the first image according to the aspect ratio and the connected area, and then crop the first image according to the first area, and the obtained second image also satisfies the aspect ratio.
[0125] In one embodiment, the center of the connected area coincides with the center of the first area to ensure that the target object is located at the center of the first area, so that the target object is also located at the center of the second image, thereby improving the display effect of the second image.
[0126] In some embodiments, the method further comprises:
[0127] Determine a first pixel having a maximum depth value and a second pixel having a minimum depth value in the second area;
[0128] The depth tolerance is determined according to a difference between the depth values corresponding to the first pixel and the second pixel.
[0129] It can be understood that the first pixel with the largest depth value and the second pixel with the smallest depth value in the second area are obtained from the depth image, so that the difference in depth value corresponding to the first pixel and the second pixel is used as the depth tolerance. Then, the pixels in the adjacent areas of the second area whose depth difference is less than the depth tolerance are assigned to a connected area, and the connected area is the first area containing the target object.
[0130] In one embodiment, the depth tolerance may be determined by the sum of the difference between the depth values corresponding to the first pixel value and the second pixel value and the first threshold. The first threshold is determined through a large number of image processing experiments or experience. The disclosed embodiment increases the depth tolerance by the first threshold, so that the pixels at the edge position of the target object can also be attributed to the connected area, thereby improving the integrity of the target object in the second image.
[0131] In one embodiment, considering that the depth tolerance is determined based on the difference in depth values corresponding to the first pixel and the second pixel, there may be pixels at the edge of the target object that are not attributed to the first region. Therefore, the embodiment of the present disclosure can also determine the depth tolerance based on the difference in depth values corresponding to the first pixel and the second pixel, the first weight and the offset value, and attribute the pixels in the second region and the pixels in the adjacent region of the second region whose depth difference is less than the adjusted depth tolerance to a connected region, thereby obtaining the first region containing the target object.
[0132] The first weight and bias value are determined through a large number of image processing experiments or experiences, so that it can be ensured that the pixels at the edge of the target object are attributed to the connected area as much as possible, thereby improving the integrity of the target object in the second image.
[0133] Specifically, the depth tolerance can be calculated using the following formula:
[0134] T = w*R + a;
[0135] Wherein, T is the depth tolerance, w is the first weight, R is the difference between the depth values corresponding to the first pixel and the second pixel, a is the offset value, w is greater than 0, and a is greater than or equal to 0.
[0136] The disclosed embodiment uses the difference between the maximum depth value and the minimum depth value of the pixels in the second area as the depth tolerance, so as to more accurately determine the pixels in the adjacent areas of the second area that need to be attributed to a connected area, thereby making the pixels of the first area containing the target object more complete.
[0137] In some embodiments, the S1200 may include:
[0138] Determining image coordinates of the first area according to an outer contour of the target object in the first image;
[0139] The first image is cropped according to the image coordinates to obtain a second image containing the target object.
[0140] It can be understood that when it is necessary to perform a preset image operation related to a target object, the first area in the first image where the target object is located is first determined. The image coordinates of the first area are determined based on the outer contour of the target object in the first image. For example, based on the outer contour of the target object, the image coordinates corresponding to the bounding box of the outer contour of the target object are determined as the image coordinates corresponding to the boundary of the first area. The bounding box may include but is not limited to a rectangular frame.
[0141] As another example, according to the outer contour of the target object, the boundary of the first region substantially matching the outer contour of the target object is connected. Here, determining the image coordinates of the first region may include: determining the image coordinates of the boundary of the first region.
[0142] According to the image coordinates of the first area, the first image is cropped to obtain a second image containing the target object. The cropping operation here is to crop along the outer contour of the target object. After the second image is obtained, a preset image operation can be performed on the second image according to the depth information corresponding to the first image.
[0143] There are many ways to obtain the outer contour of the target object in the first image. Exemplarily, the method of obtaining the outer contour can obtain the outer contour of the target object based on an arbitrary contour model. Exemplarily, edge detection can also be performed on the binary image corresponding to the first image to obtain the outer contour of the target object. Since the binary image is an image in which each pixel is black or white, when the target object is a person, the binary image can only show the edge information of the image, and the texture features inside the image are not obvious. Therefore, by performing edge detection on the binary image corresponding to the first image, the outer contour of the target object can be obtained more quickly and accurately.
[0144] After determining the outer contour of the target object in the first image, the embodiment of the present disclosure determines the image coordinates of the first area according to the outer contour, so that the determined image coordinates of the first area are more accurate. Then, the first image is cropped according to the image coordinates of the first area, and the second image containing the target object can be obtained more accurately.
[0145] In some embodiments, the S1300 may include:
[0146] According to the depth information, obtaining an average depth value of pixels in the area where the target object is located in the second image;
[0147] Determine pixels whose difference between the depth value in the second image and the average depth value is greater than a first threshold as pixels to be processed;
[0148] The preset image operation is performed on the pixel to be processed.
[0149] It can be understood that after obtaining the second image, since the second image is an image cropped from the first image, the depth value of each pixel in the area where the target object is located in the second image can be obtained from the depth information of the first image. The sum of the depth values of each pixel in the area where the target object is located in the second image is divided by the number of pixels in the area where the target object is located, thereby obtaining an average pixel value.
[0150] Since the depth values of the pixels in the target object area and the pixels in the area outside the target object area are greatly different, the second image can be divided into the target object area and the area to be processed by comparing the depth values. The area to be processed here at least includes the area outside the target object area in the second image.
[0151] Specifically, the depth value of each pixel in the second image is obtained, and it is determined whether the difference between the depth value of each pixel and the average depth value is greater than a first threshold. If it is greater than the first threshold, it means that the pixel is a pixel outside the area where the target object is located, and the pixel is determined as a pixel to be processed. The first threshold here is obtained based on a large number of experiments or image processing experience.
[0152] When the preset image operation is a background blur operation, the pixels to be processed are pixels contained in the background area of the second image. By performing the preset image operation on the pixels to be processed, blurring of the background area in the second image is achieved.
[0153] In one embodiment, performing a preset image operation on the pixel to be processed may include: when the target object is a portrait, and the preset image operation is a background blur operation, the depth value of the pixel to be processed may be input into a pre-trained deep convolutional neural network to perform a blur operation on the pixel to be processed.
[0154] In the disclosed embodiment, the average depth value of the pixels in the area where the target object is located in the second image is obtained according to the depth information of the first image; the pixels whose difference between the depth value and the average depth value in the second image is greater than the first threshold value are determined as pixels to be processed; and the preset image operation is performed on the pixels to be processed. In this way, by determining the pixels to be processed from the second image, the preset image operation is only performed on the pixels to be processed, thereby reducing the pixels to be subjected to the preset image operation, that is, reducing the amount of data for performing the preset highlighting operation, and improving the efficiency of performing the preset image operation.
[0155] In some embodiments, the preset image operation includes at least: a background blur operation.
[0156] It can be understood that when it is determined that a background blur operation related to a target object needs to be performed, a first region in the first image where the target object is located is determined, the first region at least includes a background region and a foreground region, and the foreground region at least includes the target object. Then, the first image is cropped according to the first region to obtain a second image, the second image at least includes the first region, that is, the second image includes the background region and the foreground region.
[0157] In one embodiment, the second image and the depth image corresponding to the second image are input into a blurring algorithm, and the background area is smoothed using the blurring algorithm, so that the background area of the second image is blurred, but the foreground area of the second image remains clear, and progressive blurring is achieved at the junction of the background area and the foreground area.
[0158] The above-mentioned background blurring algorithm can be implemented based on a deep convolutional neural network. For example, the depth image corresponding to the first image and the second image are input into a pre-trained deep convolutional neural network, and the second image is segmented into a portrait area and a background area, so that the background blurring image corresponding to the second image is output through the deep convolutional neural network.
[0159] For another example, the depth image corresponding to the first image is cropped according to the first region to obtain the depth image corresponding to the second image. The depth image corresponding to the second image and the second image are input into a pre-trained deep convolutional neural network to segment the second image into a portrait region and a background region, thereby outputting a background blur image corresponding to the second image through the deep convolutional neural network.
[0160] In some embodiments, Figure 3 As shown, the method also includes:
[0161] S1400: Determine a shooting mode of the electronic device;
[0162] S1500: When the shooting mode is a portrait mode, determining that a preset image operation related to the target object needs to be performed, wherein the target object is a portrait.
[0163] The above-mentioned shooting modes may include portrait mode, landscape mode, macro mode, sports mode or panorama mode, etc.
[0164] When it is determined that the shooting mode of the camera in the electronic device is portrait, it is determined that a preset image operation related to the portrait needs to be performed. Therefore, when the shooting mode is portrait mode, a preset image operation related to the target object is performed, and the preset image operation can be a background blur operation, so that the first image captured more prominently highlights the subject portrait.
[0165] When the shooting mode is the portrait mode, the first area includes at least a portrait area and a background area, wherein the first area may include one or more portraits, and the embodiment of the present disclosure does not specifically limit the number of portraits in the first area. Thus, the first image is cropped according to the first area to obtain a second image, and a preset image operation is performed on the second image according to the depth image corresponding to the first image.
[0166] For example, if the shooting mode is a portrait mode, the corresponding preset image operation may be a background blur operation. In the background blur operation, the second image is segmented to obtain a portrait area and a background area to achieve an effect of highlighting the portrait area and blurring the background area.
[0167] In some embodiments, Figure 4 As shown, the S1100 includes:
[0168] S1130: Determine whether to enable the target object centering display function;
[0169] S1140: When the target object centering display function is turned on, a first area where the target object is located is determined, and the target object is placed at a center position of the first area.
[0170] In the interface for taking photos, videos or recording videos, including a first control, determining whether to enable the target object center display function can be understood as, when the first control is selected, it means that the target object center display function is enabled.
[0171] When the target object centering display function is turned on, when determining the first area where the target object is located in the first image, the center position of the target object is determined and the center position of the target object is overlapped with the center position of the first area, thereby achieving the purpose of placing the target object in the center of the first area.
[0172] For example, during the shooting process, the target object may move, so the lens needs to be moved to ensure that the portrait is located in the center of the picture. However, the disclosed embodiment places the target object in the center of the first area before cropping the first image according to the first area, so that as long as the target object is within the viewing angle of the lens, a second image in which the target object is located in the center of the first area can be obtained, thereby reducing the occurrence of moving the lens and improving the shooting effect and user experience.
[0173] The present disclosure provides an image processing method. Figure 5 As shown, the method includes:
[0174] S210: Select a shooting mode and enable the portrait blur tracking function; here, by selecting a shooting mode, when the selected shooting mode is the portrait mode, the portrait blur tracking function is enabled.
[0175] S220: Fast face detection module; here, the face in the picture collected by the image acquisition module can be detected through the image recognition algorithm.
[0176] S230: whether there is a face; according to the image recognition algorithm, determine whether there is a face. When there is no face, proceed to step S231: execute the default preview process; when there is a face, proceed to step S240.
[0177] S240: Calculate the depth map D based on the image S output by the image acquisition module (sensor); the image S here is the above-mentioned first image, and the depth value of each pixel in the first image acquired by the image acquisition module is determined, so as to generate the depth map D according to the depth value of each pixel.
[0178] S250: Obtain the depth value of the area where the face coordinates are located to calculate the depth tolerance T; the area where the face coordinates are located here is equivalent to the second area mentioned above, and the depth tolerance T is determined according to the depth value of each pixel point in the second area.
[0179] S260: Calculate a connected graph G of face coordinates based on the depth map D and the depth tolerance T; here, when the depth values of two adjacent pixel points in the depth map are determined according to the depth tolerance to be within the depth tolerance range, the pixel point belongs to the connected graph G.
[0180] S270: Calculate the minimum rectangular area R that can contain the connected graph G and meets the output aspect ratio requirement; the aspect ratio here is equivalent to the above-mentioned length-to-width ratio, and the minimum rectangular area R is equivalent to the above-mentioned first area.
[0181] S280: Based on the rectangular area R, a sub-picture area S' is cut out from the original image S, and a depth map sub-area D' is cut out from the depth map D; the original image S here is equivalent to the first image mentioned above, the sub-picture area S' is equivalent to the second image mentioned above, and the depth map sub-area D' is equivalent to the depth image of the second image mentioned above.
[0182] S290: passing the sub-picture region S' and the depth map sub-region D' into a blurring algorithm, and performing background blurring processing on the sub-picture region S'.
[0183] After acquiring the image S output by the image acquisition module (Sensor), the portrait blur function of the electronic device in the disclosed embodiment will quickly identify whether a face exists through a face recognition module based on hardware implementation. If a face exists in the picture, the image depth information D will continue to be acquired through a monocular or binocular depth estimation algorithm.
[0184] The above blurring algorithm sets the blur kernel K according to the depth map sub-area D', blur strength, alignment and other parameters, and convolves the sub-image area S' with the blur K to finally obtain the output image after the portrait background is blurred.
[0185] The centering of the person in the electronic device (such as the iPad tablet) mainly uses the portrait recognition technology and image cropping technology based on machine learning. Based on the image S output by the image acquisition module (Sensor), the area A where the portrait is located is directly calibrated by the portrait recognition technology based on machine learning. After converting area A into area A' according to the system aspect ratio requirements, the image content S' of area A' is cropped out from the image S. After blurring the image S' according to the system requirements, a preview image centered on the person is finally obtained.
[0186] This solution uses a connected graph calculation method to cut out the image area where the portrait is located based on the image depth map, which reduces the image area to be processed by the blur algorithm and optimizes the performance and power consumption of the portrait blur algorithm. The depth map area cut out based on the portrait reduces the depth information passed into the blur algorithm, allowing the blur algorithm to focus on blurring the portrait contour. The image area cut out based on the portrait can ensure that the portrait is always located in the center of the image area, which improves the deficiency of the existing algorithm that cannot track the portrait subject and enriches the application scenarios of this function. The connected graph search algorithm executed by this method based on the depth map face area is faster and occupies less hardware resources than the machine learning method used in the existing portrait detection technology.
[0187] like Figure 6 As shown, the embodiment of the present disclosure provides an image processing device 300, which is applied to an electronic device, and the device includes:
[0188] A first determination module 310, configured to determine a first region in a first image where the target object is located when a preset image operation related to the target object needs to be performed;
[0189] A cropping module 320, configured to crop the first image according to the first area to obtain a second image; wherein the second image at least includes the first area;
[0190] The execution module 330 is configured to perform the preset image operation on the second image according to the depth information corresponding to the first image.
[0191] In some embodiments, the apparatus further comprises:
[0192] a detection module, configured to perform target object detection on the first image to obtain a second area containing the target object when the preset image operation related to the target object needs to be performed;
[0193] A first attribution module is used to assign pixels in the second area and pixels in an adjacent area of the second area whose depth difference is less than a depth tolerance to a connected area based on the depth value of each pixel in the second area obtained from the depth information, thereby obtaining the first area including the connected area.
[0194] In some embodiments, the apparatus further comprises:
[0195] A second attribution module, configured to attribute pixels whose depth difference is less than the depth tolerance to a connected area;
[0196] An expansion module is used to expand the connected area according to a preset image size specification to obtain the first area, wherein the first area at least includes the connected area, and the size of the first area meets the preset image size specification.
[0197] In some embodiments, the expansion module is specifically used to expand the connected area according to the aspect ratio in the preset image size specification to obtain the first area that meets the aspect ratio.
[0198] In some embodiments, the apparatus further comprises:
[0199] A second determining module, configured to determine a first pixel with a maximum depth value and a second pixel with a minimum depth value in the second area;
[0200] The third determining module is configured to determine the depth tolerance according to a difference between the depth values corresponding to the first pixel and the second pixel.
[0201] In some embodiments, the cropping module is specifically used to determine the image coordinates of the first area according to the outer contour of the target object in the first image; and to perform a cropping operation on the first image according to the image coordinates to obtain a second image containing the target object.
[0202] In some embodiments, the execution module is specifically used to obtain the average depth value of pixels in the area where the target object is located in the second image based on the depth information; determine the pixels whose difference between the depth value in the second image and the average depth value is greater than a first threshold as pixels to be processed; and perform the preset image operation on the pixels to be processed.
[0203] In some embodiments, the preset image operation includes at least: a background blur operation.
[0204] In some embodiments, the apparatus further comprises:
[0205] A fourth determining module, used to determine a shooting mode of the electronic device;
[0206] The fifth determining module is used to determine that a preset image operation related to the target object needs to be performed when the shooting mode is the portrait mode, wherein the target object is a portrait.
[0207] In some embodiments, the first determination module is specifically used to determine whether to enable the target object centering display function; when the target object centering display function is enabled, determine the first area where the target object is located, and place the target object at the center of the first area.
[0208] An embodiment of the present disclosure provides an electronic device, including:
[0209] a memory for storing processor-executable instructions;
[0210] A processor connected to the memory;
[0211] Among them, the processor is configured to execute the image processing method provided by any of the aforementioned technical solutions.
[0212] The processor may include various types of storage media, which are non-transitory computer storage media that can continue to remember information stored thereon after the electronic device loses power.
[0213] The processor can be connected to the memory through a bus or the like, and is used to read the executable program stored in the memory, for example, to execute the following Figures 1 to 5 At least one of any of the methods shown.
[0214] Figure 7 8 is a block diagram of an electronic device 800 according to an exemplary embodiment. For example, the electronic device 800 may be included in an electronic device such as a mobile phone, a mobile computer, or a server. In short, the data processing electronic device 800 may be included in any electronic device.
[0215] Reference Figure 7 The electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .
[0216] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.
[0217] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0218] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.
[0219] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating state, such as a shooting state or a video state, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0220] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating state, such as a call state, a recording state, and a voice recognition state, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.
[0221] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.
[0222] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0223] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as Wi-Fi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0224] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above methods.
[0225] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0226] The present disclosure provides a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by the processor of the UE, the UE or the base station can execute the image processing method provided by any of the above embodiments, and can execute Figures 1 to 5 At least one of any of the methods shown.
[0227] The image processing method applied in the electronic device may include:
[0228] When it is necessary to perform a preset image operation related to the target object, determining a first area in the first image where the target object is located;
[0229] Cropping the first image according to the first area to obtain a second image; wherein the second image at least includes the first area;
[0230] The preset image operation is performed on the second image according to the depth information corresponding to the first image.
[0231] It can be understood that when it is necessary to perform a preset image operation related to the target object, determining the first area where the target object is located in the first image includes:
[0232] When it is necessary to perform the preset image operation related to the target object, detecting the target object on the first image to obtain a second area containing the target object;
[0233] According to the depth value of each pixel in the second area obtained from the depth information, pixels in the second area and pixels in an area adjacent to the second area whose depth difference is less than a depth tolerance are assigned to a connected area, thereby obtaining the first area including the connected area.
[0234] It can be understood that, according to obtaining the depth value of each pixel in the second area from the depth information, pixels in the second area and pixels in an adjacent area of the second area whose depth difference is less than the depth tolerance are attributed to one connected area, thereby obtaining the first area containing the target object, including:
[0235] Assigning pixels whose depth difference is less than the depth tolerance to a connected area;
[0236] The connected area is expanded according to a preset image size specification to obtain the first area, wherein the first area at least includes the connected area, and the size of the first area meets the preset image size specification.
[0237] It can be understood that the step of expanding the connected region according to a preset image size specification to obtain the first region includes:
[0238] The connected region is expanded according to the aspect ratio in the preset image size specification to obtain the first region satisfying the aspect ratio.
[0239] It is understandable that the method further comprises:
[0240] Determine a first pixel having a maximum depth value and a second pixel having a minimum depth value in the second area;
[0241] The depth tolerance is determined according to a difference between the depth values corresponding to the first pixel and the second pixel.
[0242] It can be understood that the step of cropping the first image according to the first area to obtain the second image includes:
[0243] The image coordinates of the first area are determined according to the outer contour of the target object in the first image; and the first image is cropped according to the image coordinates to obtain a second image containing the target object.
[0244] It can be understood that performing the preset image operation on the second image according to the depth information includes:
[0245] According to the depth information, obtaining an average depth value of pixels in the area where the target object is located in the second image;
[0246] Determine pixels whose difference between the depth value in the second image and the average depth value is greater than a first threshold as pixels to be processed;
[0247] The preset image operation is performed on the pixel to be processed.
[0248] It can be understood that the preset image operation at least includes: a background blur operation.
[0249] It is understandable that the method further comprises:
[0250] Determining a shooting mode of the electronic device;
[0251] When the shooting mode is a portrait mode, it is determined that a preset image operation related to the target object needs to be performed, wherein the target object is a portrait.
[0252] It can be understood that determining the first area where the target object is located in the first image includes:
[0253] Determine whether to enable the target object centering display function;
[0254] When the target object centering display function is turned on, a first area where the target object is located is determined, and the target object is placed at the center of the first area.
[0255] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0256] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. An image processing method, applied to electronic equipment, It is characterized in that The method comprises: When it is necessary to perform a preset image operation related to the target object, determining a first area in the first image where the target object is located; Cropping the first image according to the first area to obtain a second image; wherein the second image at least includes the first area; The preset image operation is performed on the second image according to the depth information corresponding to the first image.
2. The method according to claim 1, It is characterized in that When it is necessary to perform a preset image operation related to the target object, determining a first area in the first image where the target object is located includes: When it is necessary to perform the preset image operation related to the target object, detecting the target object on the first image to obtain a second area containing the target object; According to the depth value of each pixel in the second area obtained from the depth information, pixels in the second area and pixels in an area adjacent to the second area whose depth difference is less than a depth tolerance are assigned to a connected area, thereby obtaining the first area including the connected area.
3. The method according to claim 2, It is characterized in that The method of obtaining the depth value of each pixel in the second area from the depth information and attributing the pixels in the second area and the pixels in the adjacent area of the second area whose depth difference is less than the depth tolerance to one connected area, thereby obtaining the first area containing the target object, includes: Assigning pixels whose depth difference is less than the depth tolerance to a connected area; The connected area is expanded according to a preset image size specification to obtain the first area, wherein the first area at least includes the connected area, and the size of the first area meets the preset image size specification.
4. The method according to claim 3, It is characterized in that The step of expanding the connected region according to a preset image size specification to obtain the first region includes: The connected region is expanded according to the aspect ratio in the preset image size specification to obtain the first region satisfying the aspect ratio.
5. The method according to claim 2, It is characterized in that The method further comprises: Determine a first pixel having a maximum depth value and a second pixel having a minimum depth value in the second area; The depth tolerance is determined according to a difference between the depth values corresponding to the first pixel and the second pixel.
6. The method according to claim 1, It is characterized in that The step of cutting the first image according to the first area to obtain a second image includes: Determining image coordinates of the first area according to an outer contour of the target object in the first image; The first image is cropped according to the image coordinates to obtain a second image containing the target object.
7. The method according to claim 1, It is characterized in that The performing the preset image operation on the second image according to the depth information includes: According to the depth information, obtaining an average depth value of pixels in the area where the target object is located in the second image; Determine pixels whose difference between the depth value in the second image and the average depth value is greater than a first threshold as pixels to be processed; The preset image operation is performed on the pixel to be processed.
8. The method according to claim 1, It is characterized in that The preset image operation at least includes: a background blur operation.
9. The method according to claim 1, It is characterized in that The method further comprises: Determining a shooting mode of the electronic device; When the shooting mode is a portrait mode, it is determined that a preset image operation related to the target object needs to be performed, wherein the target object is a portrait.
10. The method according to claim 1, It is characterized in that The determining a first area where the target object is located in the first image includes: Determine whether to enable the target object centering display function; When the target object centering display function is turned on, a first area where the target object is located is determined, and the target object is placed at the center of the first area.
11. An image processing device, applied to electronic equipment, It is characterized in that The device comprises: A first determining module, configured to determine a first area in a first image where the target object is located when a preset image operation related to the target object needs to be performed; A cropping module, configured to crop the first image according to the first area to obtain a second image; wherein the second image at least includes the first area; An execution module is used to perform the preset image operation on the second image according to the depth information corresponding to the first image.
12. The device according to claim 11, It is characterized in that The device also includes: a detection module, configured to detect the target object on the first image to obtain a second area containing the target object when the preset image operation related to the target object needs to be performed; A first attribution module is used to assign pixels in the second area and pixels in an area adjacent to the second area whose depth difference is less than a depth tolerance to a connected area based on the depth value of each pixel in the second area obtained from the depth information, thereby obtaining the first area including the connected area.
13. The device according to claim 12, It is characterized in that The device also includes: A second attribution module, configured to attribute pixels whose depth difference is less than the depth tolerance to a connected area; An expansion module is used to expand the connected area according to a preset image size specification to obtain the first area, wherein the first area at least includes the connected area, and the size of the first area meets the preset image size specification.
14. The device according to claim 13, It is characterized in that The expansion module is specifically used to expand the connected area according to the length-to-width ratio in the preset image size specification to obtain the first area that meets the length-to-width ratio.
15. The device according to claim 12, It is characterized in that The device also includes: A second determining module, configured to determine a first pixel with a maximum depth value and a second pixel with a minimum depth value in the second area; The third determining module is configured to determine the depth tolerance according to a difference between the depth values corresponding to the first pixel and the second pixel.
16. The device according to claim 11, It is characterized in that The cropping module is specifically used to determine the image coordinates of the first area according to the outer contour of the target object in the first image; and perform a cropping operation on the first image according to the image coordinates to obtain a second image containing the target object.
17. The device according to claim 11, It is characterized in that The execution module is specifically configured to obtain an average depth value of pixels in the area where the target object is located in the second image according to the depth information; and determine pixels whose difference between the depth value in the second image and the average depth value is greater than a first threshold as pixels to be processed; The preset image operation is performed on the pixel to be processed.
18. The device according to claim 11, It is characterized in that The preset image operation at least includes: a background blur operation.
19. The device according to claim 11, It is characterized in that The device also includes: A fourth determining module, used to determine a shooting mode of the electronic device; The fifth determining module is used to determine that a preset image operation related to the target object needs to be performed when the shooting mode is the portrait mode, wherein the target object is a portrait.
20. The device according to claim 11, It is characterized in that The first determination module is specifically used to determine whether to enable the target object centering display function; when the target object centering display function is enabled, determine the first area where the target object is located, and place the target object at the center of the first area.
21. An electronic device, It is characterized in that include: a memory for storing processor-executable instructions; A processor connected to the memory; The processor is configured to execute the image processing method provided in any one of claims 1 to 10.
22. A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a computer, enables the computer to perform the image processing method provided in any one of claims 1 to 10.
Citation Information
Patent Citations
Photographing method, mobile terminal and computer readable storage medium
CN107786812A
Image processing method and device
CN108154466A
Image processing method and device, electronic equipment and storage medium
CN110111281A
Image segmentation method and device, terminal and storage medium
CN110751659A
Image processing method and device, electronic equipment and medium
CN110910304A
