Image generation method and device and storage medium

Abstract

The invention relates to an image generation method and device and a storage medium. The image generation method comprises the steps that a first image is acquired, the first image comprises a plurality of instances, and depth values of at least two instances in the plurality of instances are different; determining a first depth value range of a to-be-displayed instance in the first image; performing blurring processing on a first instance outside the first depth value range; and generating a second image based on the first image after blurring processing. According to the method and the device, the missing virtual phenomenon caused by not blurring all examples can be avoided.

Description

Technical Field

[0001] The present disclosure relates to image processing, and in particular to an image generating method, device and storage medium. Background Art

[0002] Currently, more and more terminals are using computational photography to improve their photography capabilities, so as to achieve a shooting effect close to that of a single-lens reflex camera. Among them, background blur is achieved by using algorithmic simulation to achieve a large aperture photography effect. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides an image generation method, device and storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an image generating method, comprising: acquiring a first image, the first image comprising a plurality of instances, at least two of the plurality of instances having different depth values; determining a first depth value range of instances to be displayed in the first image; blurring the first instances outside the first depth value range; and generating a second image based on the blurred first image.

[0005] In one embodiment, blurring the first instance outside the first depth value range includes: in response to a plurality of first instances having different depth values, respectively determining first blurring intensities corresponding to the plurality of first instances, wherein different first instances correspond to different first blurring intensities;

[0006] The first instance is blurred based on the corresponding first blur strength.

[0007] In one embodiment, before determining the first depth value range of the first image, the method also includes: obtaining the depth value of each pixel in the first image; and respectively determining the first blurring intensities corresponding to the multiple first instances includes: based on the depth value of each pixel in the multiple first instances, and the correspondence between the depth value and the first blurring intensities, respectively determining the first blurring intensities corresponding to the multiple first instances.

[0008] In one embodiment, the first blur intensities corresponding to the multiple first instances are determined based on the depth values ​​of each pixel point in the multiple first instances and the corresponding relationship between the depth value and the first blur intensities, including: calculating a first average value of the depth values ​​of each pixel point in the same first instance; and determining the first blur intensities corresponding to the multiple first instances based on the first average values ​​corresponding to the multiple first instances and the corresponding relationship between the depth value and the first blur intensities.

[0009] In one embodiment, the first blurring intensity corresponding to the first instance is determined based on the depth value of each pixel point in the first instances and the correspondence between the depth value and the first blurring intensity, including: determining a first pixel point and at least one second pixel point in the first instances, the first pixel point represents any pixel point in the first instances, and the second pixel point represents a pixel point adjacent to the first pixel point; in response to the first pixel point and the second pixel point belonging to the same instance, calculating a second average value between the depth value of the first pixel point and the depth value of the second pixel point; based on the second average value and the correspondence between the depth value and the first blurring intensity, determining the first blurring intensity of the first pixel point.

[0010] In one embodiment, determining the first depth value range of the first image includes: determining a third pixel and at least one fourth pixel in the first image, the third pixel representing a focus point, and the fourth pixel representing a pixel adjacent to the third pixel; in response to the third pixel and the fourth pixel belonging to the same instance, calculating a third average value between the depth value of the third pixel and the depth value of the fourth pixel; and determining the first depth value range based on the third average value and a preset offset.

[0011] In one embodiment, multiple instances in the first image are determined in the following manner: the first image and the depth value of each pixel in the first image are input into a first neural network model to obtain an output third image, wherein the third image includes multiple identifiers; based on the identifiers in the third image, multiple instances in the first image are determined, and different identifiers in the third image correspond to different instances in the first image.

[0012] In one embodiment, blurring the first instance based on the corresponding first blur intensity includes: inputting the first image and the depth value of each pixel in the first image into a second neural network model to obtain an output fourth image, wherein the fourth image includes a plurality of reference values ​​of each pixel in the first instance; determining a second blur intensity based on the first blur intensity and the reference value; and blurring the first instance based on the corresponding second blur intensity.

[0013] In one embodiment, determining the second blurring intensity based on the first blurring intensity and the reference value includes: determining the second blurring intensity of the first pixel based on the first blurring intensity of the first pixel and the difference between the reference value of the first pixel and the reference value of at least one fifth pixel; wherein the first pixel represents any pixel within the plurality of first instances, and the fifth pixel represents a pixel adjacent to the first pixel; wherein the greater the first blurring intensity of the first pixel, the greater the second blurring intensity; and the smaller the difference between the reference value of the first pixel and the reference value of at least one fifth pixel, the greater the second blurring intensity.

[0014] In one implementation, the number of the third pixel points is determined based on the first blurring intensity of the first pixel point, and the greater the first blurring intensity of the first pixel point, the greater the number of the fifth pixel points.

[0015] According to a second aspect of an embodiment of the present disclosure, there is provided an image generating device, comprising: an acquiring unit, for acquiring a first image, wherein the first image comprises a plurality of instances, and depth values ​​of at least two of the plurality of instances are different; a processing unit, for determining a first depth value range of instances to be displayed in the first image; blurring the first instances outside the first depth value range; and generating a second image based on the blurred first image.

[0016] In one embodiment, the processing unit blurs the first instance outside the first depth value range in the following manner: in response to a plurality of first instances having different depth values, first blur intensities corresponding to the plurality of first instances are determined respectively, and different first instances correspond to different first blur intensities; and the first instance is blurred based on the corresponding first blur intensities.

[0017] In one embodiment, before the processing unit determines the first depth value range of the first image, the acquisition unit is also used to: obtain the depth value of each pixel in the first image; the processing unit determines the first blurring intensity corresponding to multiple first instances respectively in the following manner: based on the depth value of each pixel in the multiple first instances, and the correspondence between the depth value and the first blurring intensity, determine the first blurring intensity corresponding to multiple first instances respectively.

[0018] In one embodiment, the processing unit determines the first blur intensities corresponding to multiple first instances respectively based on the depth values ​​of each pixel point in the multiple first instances and the correspondence between the depth value and the first blur intensities in the following manner: calculate the first average value of the depth values ​​of each pixel point in the same first instance; determine the first blur intensities corresponding to multiple first instances respectively based on the first average values ​​corresponding to the multiple first instances and the correspondence between the depth value and the first blur intensities.

[0019] In one embodiment, the processing unit determines the first blur intensity corresponding to the first instance based on the depth value of each pixel in the first instances and the correspondence between the depth value and the first blur intensity in the following manner: determine a first pixel and at least one second pixel in the first instances, wherein the first pixel represents any pixel in the first instances and the second pixel represents a pixel adjacent to the first pixel; in response to the first pixel and the second pixel belonging to the same instance, calculate a second average value between the depth value of the first pixel and the depth value of the second pixel; and determine the first blur intensity of the first pixel based on the second average value and the correspondence between the depth value and the first blur intensity.

[0020] In one embodiment, the processing unit determines the first depth value range of the first image in the following manner: determine a third pixel and at least one fourth pixel in the first image, the third pixel representing the focus point, and the fourth pixel representing the pixel adjacent to the third pixel; in response to the third pixel and the fourth pixel belonging to the same instance, calculate a third average value between the depth value of the third pixel and the depth value of the fourth pixel; based on the third average value and a preset offset, determine the first depth value range.

[0021] In one embodiment, the processing unit determines multiple instances in the first image in the following manner: input the first image and the depth value of each pixel in the first image into a first neural network model to obtain an output third image, wherein the third image includes multiple identifiers; based on the identifiers in the third image, multiple instances in the first image are determined, and different identifiers in the third image correspond to different instances in the first image.

[0022] In one embodiment, the processing unit blurs the first instance based on the corresponding first blur intensity in the following manner: input the first image and the depth value of each pixel in the first image into a second neural network model to obtain an output fourth image, wherein the fourth image includes multiple reference values ​​of each pixel in the first instance; determine the second blur intensity based on the first blur intensity and the reference value; and blur the first instance based on the corresponding second blur intensity.

[0023] In one embodiment, the processing unit determines the second blurring intensity based on the first blurring intensity and the reference value in the following manner: based on the first blurring intensity of the first pixel and the difference between the reference value of the first pixel and the reference value of at least one fifth pixel, determine the second blurring intensity of the first pixel; wherein the first pixel represents any pixel within the plurality of first instances, and the fifth pixel represents a pixel adjacent to the first pixel; wherein the greater the first blurring intensity of the first pixel, the greater the second blurring intensity; and the smaller the difference between the reference value of the first pixel and the reference value of at least one fifth pixel, the greater the second blurring intensity.

[0024] In one implementation, the number of the third pixel points is determined based on the first blurring intensity of the first pixel point, and the greater the first blurring intensity of the first pixel point, the greater the number of the fifth pixel points.

[0025] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: by acquiring an image, determining the depth range corresponding to the image, blurring the instances outside the depth range, and not blurring the instances within the depth range, thereby avoiding the blur leakage phenomenon caused by not blurring all instances.

[0026] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0028] Figure 1 The figure is a schematic diagram showing an image blurring effect according to an exemplary embodiment.

[0029] Figure 2 The figure is a flowchart of an image generating method according to an exemplary embodiment.

[0030] Figure 3The figure is a flow chart of a blurring processing method according to an exemplary embodiment.

[0031] Figure 4 The figure is a flow chart of a first blurring intensity determination method according to an exemplary embodiment.

[0032] Figure 5 is a first image shown according to an exemplary embodiment.

[0033] Figure 6 is a diagram showing a relationship between a depth value and a first blurring strength according to an exemplary embodiment.

[0034] Figure 7 The figure is a flow chart of a first blurring intensity determination method according to an exemplary embodiment.

[0035] Figure 8 The figure is a flow chart of a first blurring intensity determination method according to an exemplary embodiment.

[0036] Fig. 9 The figure is a flowchart of a method for determining a first depth value range according to an exemplary embodiment.

[0037] Fig.10 The figure is a flowchart showing an example determination method according to an exemplary embodiment.

[0038] Fig.11 is a third image shown according to an exemplary embodiment.

[0039] Fig.12 The figure is a flow chart of an image blurring method according to an exemplary embodiment.

[0040] Fig.13 is a second disparity value image shown according to an exemplary embodiment.

[0041] Fig.14 is a fourth image shown according to an exemplary embodiment.

[0042] Fig.15 The figure is a block diagram of an image generating device according to an exemplary embodiment.

[0043] Fig.16 The figure is a block diagram of an image generating device according to an exemplary embodiment. DETAILED DESCRIPTION

[0044] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.

[0045] Currently, more and more terminals are using computational photography to improve their photography capabilities, so as to achieve a shooting effect close to that of a single-lens reflex camera. Among them, background blur is achieved by algorithmic simulation to achieve a large aperture photography effect.

[0046] In some embodiments, the terminal is equipped with multiple cameras, and the background blur can be achieved through portrait recognition technology and binocular depth estimation technology to achieve a large aperture shooting effect. Among them, portrait recognition uses artificial intelligence (AI) or other target recognition methods to identify the portrait area in the image, thereby blurring the background outside the portrait. Binocular depth estimation is to estimate the depth of the object in the image by calculating the parallax between the images taken by the two cameras, and blurring the area outside the depth of field to achieve the background blur effect.

[0047] In some embodiments, the components of the portrait and the connecting parts between the components can be identified through a neural network, based on which the portrait area is determined, and the areas outside the area are blurred to obtain the final result.

[0048] In some embodiments, the red (R), green (G), and blue (B) images may be segmented to obtain a portrait area and a background area, and the background area may be blurred. An RGB image is an image in which red, green, and blue are mixed in different proportions and intensities and displayed on a display.

[0049] In some embodiments, a portrait area may be identified, and if the focus is in the portrait area, a consistent blurring process is performed on the portrait area.

[0050] However, in the above embodiment, only the portrait area and the non-portrait area are distinguished, and when there are multiple people in the scene, the people in the background are often blurred. Figure 1 As shown, Figure 1 The middle person in the image should be blurred, but it is not blurred. In addition, the blurring of the image is consistent, that is, the degree of blurring of all parts is the same, which often results in an unnatural effect and a sense of clipping.

[0051] Therefore, the present disclosure provides an image generation method, which obtains an image, determines the depth range corresponding to the image, blurs the instances outside the depth range, and does not blur the instances within the depth range, thereby avoiding the blur leakage phenomenon caused by not blurring all instances.

[0052] Among them, the embodiments of the present disclosure can be applied to scenarios of image generation, as well as scenarios of video recording, scenarios of using applications to edit images and videos, video conferencing scenarios, etc.

[0053] The image generation method disclosed in the present invention can be applied to robots, conference televisions, terminals, etc. Among them, the terminal can also be called a terminal device, a mobile station (MS), a mobile terminal (MT), etc., which is a device that provides voice and / or data connectivity to users. For example, the terminal can be a handheld device with wireless connection function, a vehicle-mounted device, etc. At present, some examples of terminals are: smart phones (mobile phones), pocket personal computers (pocket personal computers, PPCs), handheld computers, personal digital assistants (personal digital assistants, PDAs), laptops, tablet computers, wearable devices, or vehicle-mounted devices, etc. In addition, when it is a vehicle to everything (V2X) communication system, the terminal device can also be a vehicle-mounted device. It should be understood that the embodiments of the present disclosure do not limit the specific technology and specific device form adopted by the terminal.

[0054] Figure 2 is a flowchart of an image generating method according to an exemplary embodiment. Figure 2 As shown, the image generating method includes the following steps.

[0055] In step S11, a first image is acquired, wherein the first image includes a plurality of instances, and depth values ​​of at least two of the plurality of instances are different.

[0056] In one embodiment, a first image may be acquired. For example, the first image may be acquired by shooting with a camera, or an image frame during shooting. For another example, in a scenario where an image is edited, the first image may be read from a memory. The first image may also be acquired from the cloud, or the first image transmitted by other devices may be acquired, etc., which is not limited in the present disclosure. The first image includes multiple instances. The instances may be, for example, people, animals, buildings, etc.

[0057] In step S12, a first depth value range of the first image is determined.

[0058] In one embodiment, a first depth value range of the first image may be determined. The depth value may be understood as the distance of the photographed object from the photographing device. For example, when there is a distant mountain and a nearby person in the image, the depth value of the mountain is greater than the depth value of the person. The first depth value range may also be referred to as a depth of field (DOF) range.

[0059] In one embodiment, the reference depth value can be determined according to the focus point coordinates of the camera parameters, wherein the focus point coordinates correspond to the instance to be displayed selected by the user. For example, the user can click on instance A to focus, and the instance to be displayed corresponding to the focus point coordinates is instance A. After determining the reference depth value, the first depth value range of the instance to be displayed can be determined according to the reference depth and the preset offset. For example, the reference depth value minus the preset offset obtains the minimum value A of the first depth value range, and the reference depth value plus the preset offset obtains the maximum value B of the first depth value range, and the first depth value range is (A, B). Among them, the focus point coordinates are camera parameters, which are used for the shooting device to focus at the focus point, and the position corresponding to the focus point is the clearest.

[0060] In step S13, blurring is performed on the first instance outside the first depth value range.

[0061] In one embodiment, the first instance outside the first depth value range may be blurred. On the contrary, the instance within the first depth value range is not blurred. The present disclosure refers to the instance outside the first depth value range as the first instance.

[0062] In step S14, a second image is generated based on the blurred first image.

[0063] In one implementation, the second image may be generated based on the blurred first image, and some instances of the generated second image may be blurred while others may not be blurred, so that the blurring effect is more reasonable.

[0064] Of course, in some cases, the part that the terminal user chooses to blur may be the instance, and the part not to be blurred may be the background outside the instance, that is, multiple instances may be outside the first depth value range, and then all of them may be blurred. In other cases, multiple instances may be within the first depth value range, and then all of them may not be blurred.

[0065] The present disclosure acquires an image, determines the depth range corresponding to the image, blurs the instances outside the depth range, and does not blur the instances within the depth range, thereby avoiding the blur leakage phenomenon caused by not blurring all instances.

[0066] In some embodiments, Figure 3is a flowchart of a blurring processing method according to an exemplary embodiment. Figure 3 As shown, the following steps are included.

[0067] In step S21 , in response to the number of first instances with different depth values ​​being multiple, first blurring intensities corresponding to the multiple first instances are determined respectively, and different first instances correspond to different first blurring intensities.

[0068] In one embodiment, if there are multiple instances outside the first depth value range, and they have different depth values. That is, when the number of first instances with different depth values ​​is multiple, the first blurring strengths corresponding to different first instances can be determined respectively. The first blurring strengths corresponding to different first instances are different. Among them, the first instances with different depth values ​​can also be understood as first instances with different depth value ranges. For example, for each pixel in the same instance, its depth value may be different, then the depth value range corresponding to the instance is between the minimum depth value and the maximum depth value of each pixel in the instance. Exemplarily, when the instance is a portrait, the depth value of the tip of the nose of the person may be smaller than the depth value of the eyes. When the instance is a portrait, and the person takes a step or swings his arms, the depth value of the front leg or arm is smaller than the depth value of the back leg or arm. For example, when the instance is an animal such as a cat or dog, and faces the camera, the depth value of its face is smaller than the depth value of its tail. This disclosure does not give examples one by one.

[0069] In some embodiments, the first blur intensity corresponding to the first instance can be matched according to different depth values. For example, the smaller the depth value of the first instance, the smaller the corresponding first blur intensity. Conversely, the larger the depth value of the first instance, the larger the corresponding first blur intensity.

[0070] In step S22, blurring is performed on the first instance based on the corresponding first blurring strength.

[0071] In one implementation, blurring may be performed on multiple first instances according to the first blurring intensity corresponding to the first instance, wherein the first blurring intensity corresponding to the first instance may be understood as the first blurring intensity corresponding to each pixel in the first instance, and the first blurring intensity corresponding to each pixel may be the same or different.

[0072] The present disclosure performs different degrees of blurring on different first instances to achieve a more gradual and natural blurring effect, without requiring the user to manually adjust blurring parameters, thereby improving the user experience.

[0073] In some embodiments, Figure 4 is a flow chart of a first blur intensity determination method according to an exemplary embodiment. Figure 4 As shown, the following steps are included.

[0074] In step S31, the depth value of each pixel in the first image is obtained.

[0075] In one embodiment, the depth value of each pixel in the first image can be obtained. For example, the depth value of each pixel in the first image can be calculated by AI. For another example, a fifth image and camera parameters can be obtained. The fifth image represents an image taken simultaneously with another shooting device different from the first image. For example, the terminal has a main camera and a sub-camera, and the images taken simultaneously by the main camera and the sub-camera can be obtained, namely, the first image and the fifth image. Figure 5 As shown, Figure 5 The image on the left is the image taken by the main camera, which can be the first image, and the image on the right is the image taken by the secondary camera, which can be the fifth image. The camera parameters are divided into the camera parameters of the main camera and the camera parameters of the secondary camera. The camera parameters include camera intrinsic parameters, such as the coordinates of the principal point, focal length, distortion parameters, etc. Among them, the coordinates of the intersection of the camera imaging plane and the camera optical axis are the principal point coordinates. The camera parameters include camera extrinsic parameters, such as the spatial coordinates and orientation of the camera. The camera intrinsic parameters and camera extrinsic parameters can be calibrated at the factory and read from the configuration file. They can also be calculated in real time through an online calibration algorithm. The camera intrinsic parameters and camera extrinsic parameters are used for image alignment and correction. The parallax calculation can be performed on the first image and the fifth image. After the first image and the fifth image are corrected, each pixel in the first image and the corresponding pixel in the fifth image are offset only in the horizontal direction. For example, the pixel A in the first image corresponds to the pixel a in the fifth image, and the coordinates of A are (X1, Y1) and the coordinates of a are (X2, Y2). Then, (X1-X2) represents the disparity value between pixel A and pixel a. The depth value of each pixel can be obtained by calculating the difference between the disparity value of each pixel and the maximum disparity value. For example, the disparity value can be divided into 256 levels, that is, the range of the disparity value is (0, 255), and the maximum disparity value is 255. Then, when the disparity value of a certain pixel is X, the depth value is (255-X). Of course, the level of the disparity value can also be other values, such as 64 levels, 128 levels, etc., which is not limited in the present disclosure.

[0076] In one embodiment, the correction of the first image and the fifth image may be stereoscopic correction. Due to factors such as the different distortion parameters and field of view (FOV) of the primary and secondary cameras, the disparity value is calculated after stereoscopic correction. For example, the stereoscopic correction may refer to the following formula 1.

[0077]

[0078] Where, I1 represents the first image, I3 represents the corrected first image, I2 represents the fifth image, and I4 represents the corrected fifth image. H1 and H2 represent homography matrices. The homography matrix can be used for mapping between two planes. It belongs to the projective group transformation.

[0079] In one embodiment, a semi-global block matching (SGBM) algorithm may be used to calculate the disparity value between the corrected first image and the corrected fifth image. The disparity value between the first image and the fifth image is obtained by inverse transforming the first image to the corrected first image. The inverse transformation may refer to Formula 2.

[0080] D2=H1 -1 D1

[0081] Formula 2

[0082] Wherein, D2 represents the disparity value between the first image and the fifth image, which can be called the second disparity value. D1 represents the disparity value between the corrected first image and the corrected fifth image, which can be called the first disparity value. H1, as in Formula 1, represents the homography matrix used to correct the first image to obtain the fifth image. -1 represents the inversion of the matrix, H -1 Represents the inverse matrix of the H1 matrix.

[0083] It can be understood that the disparity value between images in the present disclosure represents the disparity value between each pixel point corresponding to the image.

[0084] In step S32, based on the depth value of each pixel in the plurality of first instances and the corresponding relationship between the depth value and the first blurring intensity, the first blurring intensity corresponding to the plurality of first instances is determined respectively.

[0085] In one embodiment, the first blurring strengths of the plurality of first instances can be determined respectively based on the depth values ​​of the pixels in the first instance and the corresponding relationship between the depth values ​​and the first blurring strengths. For example, the first instance being outside the first depth value range includes two possible situations: being before the first depth value range and being after the first depth value range. Among them, being before the first depth value range can be understood as the depth value of the first instance being less than the minimum value of the first depth value range, that is, if the first depth value range is (A, B), the depth value of the first instance is less than A. At this time, the distance between the first instance and the shooting device is relatively close. Being after the first depth value range can be understood as the depth value of the first instance being greater than the maximum value of the first depth value range, for example, the depth value of the first instance is greater than B. At this time, the distance between the first instance and the shooting device is relatively far. For the first instance of the two possible situations of being before the first depth value range and being after the first depth value range, different first blurring strengths can be matched. For another example, the depth value size order of the first instance can be distinguished, and the first blurring strength can be matched in a graded manner according to the depth value size order. For example, if there are N first instances, the blur intensity can be divided into N levels, and the blur intensity of the smallest depth value is the first level, and the blur intensity of the largest depth value is the Nth level. For another example, a functional relationship between the depth value and the first blur intensity is preset, and the first blur intensity of the first instance with different depth values ​​is determined according to the functional relationship.

[0086] The present disclosure obtains the depth value of each pixel in the first image, and when there are multiple first instances with different depth values, the first blur strength corresponding to the first instance can be determined based on the depth value. The first blur strength determined by the depth value is more reasonable, and different first instances correspond to different first blur strengths, so that the blur effect is more gradual and more natural.

[0087] In some embodiments, the first blur intensity of the first instance as a whole may be roughly determined, or the first blur intensity of each pixel in the first instance may be determined. For example, the first blur intensity of the pixel may be determined based on the functional relationship between the depth value and the blur intensity, as shown in Formula 3 below.

[0088]

[0089] Wherein, B(p) represents the first blur intensity of pixel point p. d Indicates the depth value, D d (p) represents the depth value of pixel p. F Indicates the minimum value of the first depth value range, d B represents the maximum value of the first depth value range. That is, the first depth value range is (d F ,d B ). F(d) represents a first blur parameter, which is used to calculate the difference between the depth value of the pixel point p and the minimum value of the first depth value range when the depth value of the pixel point p is less than the minimum value of the first depth value range, and multiply the first blur parameter by the difference to obtain the first blur strength of the pixel point p. K B (d) represents a second blur parameter, which is used to calculate the difference between the depth value of the pixel point p and the maximum value of the first depth value range when the depth value of the pixel point p is greater than the maximum value of the first depth value range, and multiply the second blur parameter by the difference to obtain the first blur strength of the pixel point p. Figure 6 As shown in FIG. 1 , it is a graph showing the relationship between the depth value and the first blur strength. The horizontal axis in the graph represents the depth value, d F Indicates the minimum value of the first depth value range, d B Indicates the maximum value of the first depth value range. F and d B When it is between , that is, it is within the first depth value range, no blurring is performed, that is, the first blurring intensity is 0. F represents the first blur parameter, k B k represents the second blur parameter. F The indicated straight line represents the functional relationship between the depth value and the first blur strength when the depth value of the pixel point is less than the minimum value of the first depth value range. B The indicated straight line represents a functional relationship between the depth value and the first blurring intensity when the depth value of the pixel point is greater than the maximum value of the first depth value range.

[0090] In one embodiment, the first blur parameter and the second blur parameter can be calibrated in advance and recorded in two fixed-length tables. The first blur parameter can also be called a foreground blur parameter, and the second blur parameter can also be called a background blur parameter. The first blur parameter and the second blur parameter can be manually calibrated to achieve a SLR camera effect close to the same equivalent focal length.

[0091] The present disclosure provides a corresponding relationship between a depth value and a first blurring intensity, and can determine the first blurring intensity of each pixel point according to the depth value, thereby achieving a more natural blurring effect.

[0092] In some embodiments, Figure 7 is a flow chart of a first blur intensity determination method according to an exemplary embodiment. Figure 7 As shown, the following steps are included.

[0093] In step S41, a first average value of the depth values ​​of each pixel in the same first instance is calculated.

[0094] In one implementation, the average value of the depth values ​​of each pixel in the same first instance, i.e., the first average value, may be calculated. That is, the average value of the depth values ​​of each pixel in each first instance in different first instances is calculated respectively to obtain the first average value corresponding to each first instance.

[0095] In step S42, based on the first average values ​​corresponding to the plurality of first instances and the corresponding relationship between the depth value and the first blurring intensity, the first blurring intensity corresponding to the plurality of first instances is determined respectively.

[0096] In one implementation, the first average value is used as the depth value corresponding to the first instance, and the first blur intensity corresponding to the first instance can be obtained according to the corresponding relationship between the depth value and the first blur intensity.

[0097] The present disclosure calculates the average value of the depth value of each pixel point, and then determines the corresponding first blurring intensity based on the average value as the depth value of the first instance. This can determine the blurring intensity of the first instance more quickly, simplify calculations, and improve efficiency.

[0098] In some embodiments, Figure 8 is a flow chart of a first blur intensity determination method according to an exemplary embodiment. Figure 8 As shown, the following steps are included.

[0099] In step S51, a plurality of first pixel points and at least one second pixel point in first instances are determined.

[0100] In one embodiment, any pixel point within multiple first instances can be determined, and at least one pixel point around the arbitrary pixel point can be determined. The present disclosure refers to any pixel point within multiple instances as a first pixel point. The pixel point adjacent to the first pixel point is called a second pixel point. For example, a window of a fixed size can be determined with each pixel point as the center, such as a 5X5 window, that is, 24 second pixel points are selected around the first pixel point. Of course, the 5X5 window is just a specific example given in the present disclosure. The present disclosure does not limit the size of the window, and the number of second pixel points can also be adjusted according to different first pixel points.

[0101] In step S52 , in response to the first pixel and the second pixel belonging to the same instance, a second average value between the depth value of the first pixel and the depth value of the second pixel is calculated.

[0102] In one embodiment, an average value between the depth value of the first pixel and the depth value of the second pixel, i.e., a second average value, may be calculated. A first blur intensity of the first pixel is determined based on the second average value, so that the first blur intensity of each pixel is not a discrete value, but a relatively continuous value, and the blur effect is smoother.

[0103] In one embodiment, only the second pixel points that belong to the same instance as the first pixel point are selected, and the average value of the depth value is calculated. That is, in response to the first pixel point and the second pixel point belonging to the same instance, the second average value between the depth value of the first pixel point and the depth value of the second pixel point is calculated. If there are connected first instances, then for the first pixel point at the position where the first instance is connected to the first instance, when selecting the second pixel point around it, the pixel point of another first instance may be selected, resulting in the first blurring intensity of the first pixel point being greatly different from the first blurring intensity of other surrounding pixel points, and being discontinuous, so that noise appears in the final generated image. For example, assuming that there are A second pixel points corresponding to the first pixel point, among which there are B second pixel points that belong to the same instance as the first pixel point, the depth values ​​of the B second pixel points and the depth value of the first pixel point are averaged.

[0104] In step S53, based on the second average value and the corresponding relationship between the depth value and the first blurring intensity, the first blurring intensity of the first pixel is determined.

[0105] In one embodiment, the first blurring intensity of the first pixel can be determined based on the second average value, the correspondence between the depth value and the first blurring intensity. That is, the second average value is used as the depth value of the first pixel, and the correspondence between the depth value and the first blurring intensity is substituted into the correspondence between the depth value and the first blurring intensity to obtain the first blurring intensity corresponding to the first pixel. The first blurring intensity corresponding to each first pixel in the first instance is the first blurring intensity corresponding to the first instance. It can be understood that the first blurring intensity corresponding to the first instance in the present disclosure represents the first blurring intensity corresponding to each pixel in the first instance. When the first blurring intensities corresponding to each pixel are the same, the first instance corresponds to one first blurring intensity; when the first blurring intensities corresponding to each pixel are different, the first instance corresponds to multiple first blurring intensities.

[0106] The present disclosure achieves a smoother blur effect in the first instance by taking the average value of the depth value of a pixel point and the depth value of at least one adjacent pixel point, and determining the first blur intensity according to the average value. At the same time, when determining the adjacent pixels of the first pixel point, they are limited to the same instance, so as to avoid the first blur intensity of the first pixel point being greatly different from the first blur intensity of other surrounding pixels, and being discontinuous, and avoiding the appearance of noise in the generated image.

[0107] In some embodiments, the depth value of each pixel in the first instance can be determined based on the disparity value of each pixel. That is, the depth value of each pixel can be obtained by calculating the difference between the disparity value of each pixel and the maximum disparity value. The specific method can refer to the other embodiments mentioned above, and the present disclosure will not be repeated here. In response to the first pixel and the second pixel belonging to the same instance, the average value between the disparity value of the first pixel and the disparity value of the second pixel can be calculated to obtain the disparity average value. The difference between the disparity average value and the maximum disparity value is then calculated to obtain the depth average value, that is, the second average value. Based on the depth average value, and the corresponding relationship between the depth value and the first blurring intensity, the first blurring intensity of the first pixel is determined. By way of example, the following formula 4 can be referred to.

[0108]

[0109] Wherein, D4(p) represents the average disparity of pixel p. ∑ represents the summation symbol. q∈N1(p) represents taking window N1 with pixel p as the center, and q represents the pixel in window N1. Wherein, q also includes p. That is, p represents the first pixel, and q represents the first pixel and the second pixel. D2 represents the second disparity value of pixel q, that is, the disparity value of pixel q in the first image and the fifth image. σ(p,q) represents the first weighting coefficient. L(p)=L(q) represents that the first pixel and the second pixel belong to the same instance, and L(p)≠L(q) represents that the first pixel and the second pixel do not belong to the same instance. Wherein, when q=p, L(p) must be equal to L(q). That is, when the second pixel and the first pixel belong to the same instance, the second disparity value of the second pixel is not 0, and when the second pixel and the first pixel do not belong to the same instance, the second disparity value of the second pixel is 0. The second disparity value of the first pixel is not 0. The average disparity value of the pixel point p is obtained by adding up all the second disparity values ​​that are not 0 and dividing by the number of the pixels whose second disparity values ​​are not 0.

[0110] The present disclosure calculates the average disparity of the pixels first, then the average depth, and obtains the first blur strength based on the average depth, thereby reducing the calculation of converting the disparity value into the depth value as much as possible. The calculation process is simpler and more efficient.

[0111] In some embodiments, Fig. 9 is a flow chart of a method for determining a first depth value range according to an exemplary embodiment. Fig. 9 As shown, the following steps are included.

[0112] In step S61, a third pixel point and at least one fourth pixel point in the first image are determined, the third pixel point represents a focus point, and the fourth pixel point represents a pixel point adjacent to the third pixel point.

[0113] In one embodiment, a third pixel point in the first image may be determined. The third pixel point represents a focus point. The focus point coordinates may be obtained based on obtaining camera parameters. For example, the point that a user clicks to focus during shooting is the focus point. Adjacent pixels may be selected around the third pixel point, which are the fourth pixel point.

[0114] It is understandable that the focus point may be within the instance to be displayed or not. That is, when selecting the focus point, the user can choose to display the instance and blur the part outside the instance, or can choose to display the non-instance part. In some cases, if the pixel point is within the instance, it can be determined that the instance is the instance to be displayed, and the first depth value range is determined based on the following steps. If the pixel point is not within the instance, the depth value of the third pixel point can be used as the reference depth value, and the first depth value range is obtained based on the reference depth value.

[0115] In step S62 , in response to the third pixel and the fourth pixel belonging to the same instance, a third average value between the depth value of the third pixel and the depth value of the fourth pixel is calculated.

[0116] In one implementation, when the third pixel and the fourth pixel belong to the same instance, the average value between the depth value of the third pixel and the depth value of the fourth pixel can be calculated, which is the third average value. The specific implementation method can refer to the above-mentioned method of calculating the average value between the depth value of the first pixel and the depth value of the second pixel to obtain the second average value, which is not repeated in this disclosure.

[0117] In one embodiment, the number of fourth pixel points can be set by itself, for example, it can be 9X9, but this disclosure is not limited. A fourth pixel point that meets the conditions can be determined within the window. The conditions may include at least one of the following: belonging to the same instance as the third pixel point; the fourth pixel point whose depth value difference with the third pixel point is less than or equal to the preset offset, and the average value is calculated. For example, the following formula 5 can be referred to.

[0118]

[0119] Where d represents the reference depth value. ∑ represents the summation symbol. Tp represents the focus point, that is, the third pixel point, and Tq represents the third pixel point and the fourth pixel point. That is, Tq includes Tp and the pixels adjacent to Tp. q∈N2(Tp) represents taking window N2 with Tp as the center. Tq is the pixel point in window N2. D d Indicates the depth value, D d (Tq) represents the depth value of Tq. Δ(Tp, Tq) represents the second first weighting coefficient, and Δ(Tp, Tq) can be 0 or 1. dr represents the preset offset, |D d(Tp)-D(Tq)|≤dr indicates that the difference between the depth values ​​of the third pixel and the fourth pixel is less than the preset offset. L(Tp)=L(Tq) indicates that the third pixel and the fourth pixel belong to the same instance. That is, when the difference between the depth values ​​of the third pixel and the fourth pixel is less than the preset offset, and the third pixel and the fourth pixel belong to the same instance, the second first weighting coefficient can be determined to be 1. |D d (Tp)-D d (Tq)|>dr means that the difference between the depth values ​​of the third pixel and the fourth pixel is greater than the preset offset. L(Tp)≠L(Tq) means that the third pixel and the fourth pixel do not belong to the same instance. That is, when the difference between the depth values ​​of the third pixel and the fourth pixel is greater than the preset offset, or the third pixel and the fourth pixel do not belong to the same instance, it can be determined that the second first weighting coefficient is 0. When the second first weighting coefficient is 1, the depth value of the fourth pixel is not 0, and when the second first weighting coefficient is 0, the depth value of the fourth pixel is 0, and the depth value of the third pixel is not 0. Add all non-zero depth values ​​and divide by the number of pixels whose depth values ​​are not 0 to obtain the reference depth value d. That is, It means to add all non-zero depth values. Indicates the number of pixels whose depth value is not 0.

[0120] In step S63, a first depth value range is determined based on the third average value and the preset offset.

[0121] In one implementation, the third average value can be used as a reference depth value, and the third average value minus the preset offset can be used to obtain the minimum value of the first depth value range, and the third average value plus the preset offset can be used to obtain the maximum value of the first depth value range. The preset offset can be obtained through multiple experiments or can be set according to actual conditions.

[0122] In one implementation, reference may be made to the following formula 6.

[0123]

[0124] Among them, d F Indicates the minimum value of the first depth value range, d B represents the maximum value of the first depth value range. D represents the reference depth. dr represents the preset offset.

[0125] The present disclosure obtains a reference depth value by determining and calculating the average value of the depth values ​​of the focus point and the pixels near the focus point, wherein the difference between the depth value of the pixels near the focus point and the focus point is required to be less than a preset offset, so as to achieve that the reference depth value is as unaffected as possible by the pixels outside the first depth value range to be selected, so that the reference depth is more accurate. In addition, the pixels near the focus point are limited to belong to the same instance as the focus point, so that the reference depth value is closer to the center position of the instance to be displayed, thereby making the first depth value range more accurate.

[0126] In some embodiments, Fig.10 is a flowchart of an example determination method according to an exemplary embodiment. Fig.10 As shown, the following steps are included.

[0127] In step S71, the first image and the depth value of each pixel in the first image are input into the first neural network model to obtain an output third image, which includes multiple logos.

[0128] In one embodiment, the first image and the depth value of each pixel in the first image can be input into the first neural network model to obtain a third image output by the network model. The depth value of each pixel in the first image can be represented by a depth value image, that is, each pixel corresponds to an image with a depth value. The first image and the depth value image can be fused to obtain a fused image, and each pixel in the fused image corresponds to 4 channels, which are the original RGB channels in the first image and the channel of the depth value. The first neural network model can be pre-trained and used to segment multiple instances in the first image.

[0129] In one embodiment, the first image and the disparity value of each pixel in the first image may be input into the first neural network model. For example, the first image and the disparity value image may be fused. The disparity value image indicates an image in which each pixel corresponds to a disparity value.

[0130] In step S72, multiple instances in the first image are determined based on the identifiers in the third image, and different identifiers in the third image correspond to different instances in the first image.

[0131] In one embodiment, since the first image and the third image correspond to each other, multiple instances in the first image can be determined according to the identifiers in the third image. That is, the identifiers in the third image correspond to the instances in the first image. The third image can also be called a label image.

[0132] In one embodiment, Fig.11 As shown, Fig.11is an exemplary third image, where label 0 is used to identify a non-instance, and labels 1 to 3 are used to identify three different instances. The third image can be a grayscale image, such as Fig.11 As shown, different instances identified by different identifiers may have different grayscale values.

[0133] The present disclosure simultaneously inputs a first image and a depth value of each pixel in the first image into a first neural network model, wherein the depth value can provide additional speech information, so that the identification in the obtained third image is more accurate, that is, the instance segmentation is more accurate.

[0134] In some embodiments, Fig.12 is a flow chart of an image blurring method according to an exemplary embodiment. Fig.12 As shown, the following steps are included.

[0135] In step S81, the first image and the depth value of each pixel in the first image are input into the second neural network model to obtain an output fourth image, where the fourth image includes reference values ​​of each pixel in multiple first instances.

[0136] In one embodiment, the first image and the depth value of each pixel in the first image can be input into the second neural network model to obtain an output fourth image. Each pixel in the fourth image corresponds to a reference value. That is, the fourth image includes reference values ​​of each pixel in multiple first instances. The fourth image can be a mask image. Among them, the second neural network model can be the same as or different from the first neural network model. That is, when the second neural network model is the same as the first neural network model, the first image and the depth value of each pixel in the first image are input into the first neural network model, and the third image and the fourth image can be obtained at the same time. When the second neural network model and the first neural network model are different, the third image and the fourth image can be obtained respectively.

[0137] In one implementation, the third image and the fourth image may be obtained simultaneously with reference to the following formula 7.

[0138]

[0139] Wherein, M represents the third image, and L represents the fourth image. Instance represents a function for instance segmentation, I1 represents the first image, and D3 represents the third disparity value image. In the present disclosure, the image obtained by performing disparity calculation on the corrected first image and the corrected fifth image is referred to as the first disparity value image, and the disparity value corresponding to each pixel in the first disparity value image represents the disparity value between each pixel of the corrected first image and the corrected fifth image. The first disparity value image is subjected to an inverse transformation from the first image to the corrected first image, and the obtained disparity value image is referred to as the second disparity value image. Fig.13 As shown, it is a schematic diagram of the second disparity value image. The disparity value of each pixel in the second disparity value image represents the disparity value between each pixel of the first image and the third image. The second disparity value is denoised to obtain a third disparity value image, and the third disparity value image can be input into the first neural network model. Among them, the denoising process means that if there is a pixel in the second disparity value image whose disparity value is greatly different from the disparity values ​​of its surrounding pixels, for example, greater than a threshold value, the pixel is determined as noise and the disparity value of the pixel is set to 0.

[0140] In one implementation, the first image may be input into the first neural network model. For example, refer to the following formula 8.

[0141] {M,L}=Instance(I1)

[0142] Formula 8

[0143] Wherein, M represents the third image, L represents the fourth image, Instance represents a function for instance segmentation, and I1 represents the first image.

[0144] In one embodiment, the reference value of each pixel in the fourth image may include the following characteristics: the reference values ​​of each pixel in the same first instance are consistent in size. The reference values ​​between the first instance A and the first instance B are different. For the pixel in the connecting portion of the first instance A and the first instance B, the reference value is between the reference value of each pixel in the first instance A and the reference value of each pixel in the first instance B, and the closer to a first instance, the closer its reference value is to the reference value of each pixel in the first instance. For each pixel in the non-instance portion, the reference value is consistent and different from the reference value of each first instance. For each pixel in the connecting portion of the first instance and the non-instance portion, the closer to the first instance, the closer its reference value is to the reference value of each pixel in the first instance, and the closer to the non-instance portion, the closer its reference value is to the reference value of the non-instance portion.

[0145] In one embodiment, the reference values ​​of all instance parts may be set to be the same, and the reference values ​​of non-instance parts may be set to be the same. The reference values ​​of the pixels of the portion connecting the instance and the non-instance are between the reference values ​​of the instance part and the non-instance part. Moreover, the closer to the instance part, the closer the reference value is to the reference value of the instance part. The closer to the non-instance part, the closer the reference value is to the reference value of the non-instance part. The reference value may be a grayscale value, that is, the fourth image may be a grayscale image. Fig.14, which is an exemplary fourth image. The grayscale value of the instance portion in the fourth image may be 255, that is, the reference value is 255, and the grayscale value of the non-instance portion may be 0, that is, the reference value is 0. The reference value of the pixel point of the portion where the instance and the non-instance are connected, such as the edge of the hair, is between 0 and 255.

[0146] In step S82 , a second blurring intensity is determined based on the first blurring intensity and the reference value.

[0147] In one embodiment, the second blurring intensity may be determined based on the first blurring intensity and the reference value. For example, the greater the first blurring intensity, the greater the second blurring intensity. The smaller the difference between a pixel and the reference value of its adjacent pixel, the greater the second blurring intensity of the pixel.

[0148] In step S83, blurring is performed on the first instance based on the corresponding second blurring strength.

[0149] In one implementation, the first instance may be blurred based on the second blur strength.

[0150] The present disclosure obtains a reference value and adjusts the first blur strength according to the reference value to obtain a more accurate second blur strength, so that the blur effect is better. For example, blur processing can be performed by diffusion (scatter) or gathering (gather).

[0151] In some embodiments, the present disclosure provides a method for determining a second blurring intensity, comprising: determining a second blurring intensity of a first pixel point based on a first blurring intensity of the first pixel point and a difference between a reference value of the first pixel point and a reference value of at least one fifth pixel point.

[0152] In one embodiment, the first pixel point may represent any pixel point within a plurality of first instances, and the fifth pixel point represents the adjacent pixel points around the first pixel point. The fifth pixel point in the present disclosure may be the same as or different from the second pixel point. When the difference between the reference value of the first pixel point and the reference value of the fifth pixel point is small, it can be considered that the first pixel point and the fifth pixel point are both pixel points within a first instance, and the fifth pixel point can be used to determine the second blurring intensity of the first pixel point. When the difference between the reference value of the first pixel point and the reference value of the fifth pixel point is large, it can be considered that the first pixel point and the fifth pixel point are pixel points of different first instance contents, respectively. Or pixel points in the connected parts of different first instances. Or divided by the pixel points of the connected parts of the first instance and the non-instance. At this time, the fifth pixel point can be used less to determine the second blurring intensity of the first pixel point.

[0153] In one embodiment, the greater the first blur intensity of the first pixel point, the greater the second blur intensity; the smaller the difference between the reference value of the first pixel point and the reference value of at least one fifth pixel point, the greater the second blur intensity. Exemplarily, the second blur intensity can be determined with reference to the following formula 9.

[0154]

[0155] Wherein, P represents the first pixel point, and q represents the first pixel point and the fifth pixel point. That is, q includes p. The q in Formula 9 and the q in Formula 4 may be the same or different. I0(p) represents the second blurring intensity of pixel point p. I1(q) represents the first blurring intensity of q. B(p) represents the first blurring intensity of p. ∑ represents the summation symbol. q∈N3(B(p)) represents taking window N3 with the first pixel point as the center, and q is the pixel point in window N3. Moreover, the size of window N3 can be determined according to the first blurring intensity of p. For example, the greater the first blurring intensity of p, the greater N3, and the smaller the first blurring intensity of p, the smaller N3. s(p,q) represents the third weighting coefficient. It means multiplying the first blur intensity of each q by the third weighting coefficient and summing them. represents the sum of the third weighting coefficients of all q. That is, Formula 9 represents multiplying the first blurring intensity of each q by the third weighting coefficient, summing the result, and then dividing the sum by the sum of the third weighting coefficients of all q.

[0156] In one implementation, the third weighting coefficient in Formula 9 may be determined with reference to Formula 10.

[0157]

[0158] Wherein, represents the third weighting coefficient. e represents a natural constant, which is approximately equal to 2.71. Indicates e The power is the basic coefficient. P represents the first pixel, q represents the first pixel and the fifth pixel. B(p) represents the first blur intensity of p. |pq| represents the distance between p and q. The larger the first blur intensity of p, the larger the third weighting coefficient, that is, the greater the influence of q on the second blur intensity of p. The The smaller the distance between p and q, the larger the third weighting coefficient, that is, the greater the influence of q on the second blurring intensity of p, the larger the in formula 9. α(p,q) represents the matting coefficient, M(p) represents the reference value of p, and M(q) represents the reference value of q. When the values ​​of M(q) and M(p) are quite different, it can be indicated that p and q belong to different first instances. At this time, the α(p,q) value is smaller, which can indicate that the third weighting coefficient of p is determined by reducing the use of pixel points q in other first instances. R represents the maximum value of the reference value, for example, it can be 255, but it is not limited. β(q) represents the diffusion coefficient. The larger the β(q), the larger the α(p,q). A larger β(q) indicates that the q light has diffused. At this time, a larger α(p,q) value can indicate that the third weighting coefficient of p is determined by increasing the use of the pixel point q where the diffusion occurs. k β Represents a constant.

[0159] The present disclosure can determine a more accurate second blurring intensity by using a first blurring intensity of a first pixel point and a difference between a reference value of the first pixel point and a reference value of at least one fifth pixel point, thereby achieving a better blurring effect.

[0160] In some embodiments, the method for determining the second blurring intensity provided by the present disclosure includes: the number of third pixels is determined based on the first blurring intensity of the first pixel, and the greater the first blurring intensity of the first pixel, the greater the number of third pixels.

[0161] In one embodiment, the number of fifth pixels corresponding to different first pixels can be determined based on the first blur intensity of the first pixel. For example, the greater the first blur intensity of the first pixel, the greater the number of fifth pixels. Conversely, the smaller the first blur intensity of the first pixel, the smaller the number of fifth pixels.

[0162] The present disclosure flexibly determines the second blurring intensity of the first pixel point by determining the number of the fifth pixel point. The second blurring intensity is more accurate and the blurring effect is better.

[0163] Based on the same concept, an embodiment of the present disclosure also provides an image generating device.

[0164] It is understandable that, in order to realize the above functions, the image generation device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0165] It should be noted that those skilled in the art can understand that the various implementation methods / embodiments involved in the embodiments of the present disclosure can be used in conjunction with the aforementioned embodiments or can be used independently. Whether used alone or in conjunction with the aforementioned embodiments, the implementation principle is similar. In the implementation of the present disclosure, some embodiments are described in terms of implementation methods used together. Of course, those skilled in the art can understand that such examples are not limitations of the embodiments of the present disclosure.

[0166] Fig.15 is a block diagram of an image generating apparatus 100 according to an exemplary embodiment. Fig.15 The device 100 includes: an acquisition unit 101 and a processing unit 102.

[0167] The acquisition unit 101 is used to acquire a first image, wherein the first image includes a plurality of instances, and the depth values ​​of at least two of the plurality of instances are different. The processing unit 102 is used to determine a first depth value range of the instance to be displayed in the first image. The first instance outside the first depth value range is defocused. The second image is generated based on the defocused first image.

[0168] In one implementation, the processing unit 102 performs blur processing on the first instance outside the first depth value range in the following manner: in response to the number of first instances with different depth values ​​being multiple, first blur strengths corresponding to the multiple first instances are determined respectively, and different first instances correspond to different first blur strengths. Blurring is performed on the first instance based on the corresponding first blur strengths.

[0169] In one embodiment, before the processing unit 102 determines the first depth value range of the first image, the acquisition unit 101 is further used to: acquire the depth value of each pixel in the first image. The processing unit 102 determines the first blurring strengths corresponding to the multiple first instances respectively in the following manner: based on the depth value of each pixel in the multiple first instances, and the corresponding relationship between the depth value and the first blurring strength, the first blurring strengths corresponding to the multiple first instances are determined respectively.

[0170] In one embodiment, the processing unit 102 determines the first blur intensities corresponding to the plurality of first instances respectively based on the depth values ​​of the pixels in the plurality of first instances and the corresponding relationship between the depth values ​​and the first blur intensities in the following manner: calculating a first average value of the depth values ​​of the pixels in the same first instance. Based on the first average values ​​corresponding to the plurality of first instances and the corresponding relationship between the depth values ​​and the first blur intensities, determining the first blur intensities corresponding to the plurality of first instances respectively.

[0171] In one embodiment, the processing unit 102 determines the first blur intensity corresponding to the first instance based on the depth value of each pixel in the multiple first instances and the correspondence between the depth value and the first blur intensity in the following manner: determine the first pixel and at least one second pixel in the multiple first instances, the first pixel represents any pixel in the multiple first instances, and the second pixel represents a pixel adjacent to the first pixel. In response to the first pixel and the second pixel belonging to the same instance, calculate the second average value between the depth value of the first pixel and the depth value of the second pixel. Based on the second average value and the correspondence between the depth value and the first blur intensity, determine the first blur intensity of the first pixel.

[0172] In one embodiment, the processing unit 102 determines the first depth value range of the first image in the following manner: determine a third pixel and at least one fourth pixel in the first image, the third pixel represents a focus point, and the fourth pixel represents a pixel adjacent to the third pixel. In response to the third pixel and the fourth pixel belonging to the same instance, calculate a third average value between the depth value of the third pixel and the depth value of the fourth pixel. Based on the third average value and a preset offset, determine the first depth value range.

[0173] In one embodiment, the processing unit 102 determines multiple instances in the first image in the following manner: the first image and the depth value of each pixel in the first image are input into the first neural network model to obtain an output third image, wherein the third image includes multiple identifiers. Multiple instances in the first image are determined based on the identifiers in the third image, and different identifiers in the third image correspond to different instances in the first image.

[0174] In one embodiment, the processing unit 102 performs blurring processing on the first instance based on the corresponding first blurring strength in the following manner: inputting the first image and the depth value of each pixel in the first image into the second neural network model to obtain an output fourth image, wherein the fourth image includes a plurality of reference values ​​of each pixel in the first instance. Determine the second blurring strength based on the first blurring strength and the reference value. Perform blurring processing on the first instance based on the corresponding second blurring strength.

[0175] In one embodiment, the processing unit 102 determines the second blur intensity based on the first blur intensity and the reference value in the following manner: based on the first blur intensity of the first pixel, and the difference between the reference value of the first pixel and the reference value of at least one fifth pixel, the second blur intensity of the first pixel is determined. The first pixel represents any pixel in the plurality of first instances, and the fifth pixel represents a pixel adjacent to the first pixel. The greater the first blur intensity of the first pixel, the greater the second blur intensity. The smaller the difference between the reference value of the first pixel and the reference value of at least one fifth pixel, the greater the second blur intensity.

[0176] In one implementation, the number of the third pixel points is determined based on the first blurring intensity of the first pixel point, and the greater the first blurring intensity of the first pixel point, the greater the number of the fifth pixel points.

[0177] Fig.16 is a block diagram of an image generating device 200 according to an exemplary embodiment.

[0178] like Fig.16 As shown, apparatus 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .

[0179] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.

[0180] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0181] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.

[0182] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0183] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.

[0184] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0185] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0186] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0187] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0188] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0189] The present disclosure acquires an image, determines the depth range corresponding to the image, blurs the instances outside the depth range, and does not blur the instances within the depth range, thereby avoiding the blur leakage phenomenon caused by not blurring all instances.

[0190] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar thereto. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.

[0191] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.

[0192] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0193] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0194] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

Claims

1. An image generation method, characterized in that: include: Acquire a first image, wherein the first image includes a plurality of instances, and depth values ​​of at least two instances of the plurality of instances are different; Determining a first depth value range of an instance to be displayed in the first image; Performing blurring processing on the first instance outside the first depth value range; A second image is generated based on the blurred first image.

2. The method according to claim 1, characterized in that The blurring of the first instance outside the first depth value range includes: In response to the number of the first instances having different depth values ​​being multiple, respectively determining first blurring intensities corresponding to the multiple first instances, wherein different first instances correspond to different first blurring intensities; The first instance is blurred based on the corresponding first blur strength.

3. The method according to claim 2, characterized in that Before determining the first depth value range of the first image, the method further includes: Obtaining a depth value of each pixel in the first image; The respectively determining first blurring intensities corresponding to the plurality of first instances includes: Based on the depth value of each pixel point in the plurality of the first instances and the corresponding relationship between the depth value and the first blurring intensity, the first blurring intensity corresponding to the plurality of the first instances is determined respectively.

4. The method according to claim 3, characterized in that The determining, based on the depth value of each pixel point in the plurality of first instances and the corresponding relationship between the depth value and the first blurring intensity, respectively corresponding to the plurality of first instances includes: Calculating a first average value of depth values ​​of each pixel point in the same first instance; Based on the first average values ​​corresponding to the plurality of the first instances and the correspondence between the depth value and the first blurring intensity, the first blurring intensity corresponding to the plurality of the first instances is determined respectively.

5. The method according to claim 3, characterized in that: The determining, based on the depth values ​​of the pixels in the first instances and the corresponding relationship between the depth values ​​and the first blurring strengths, respectively corresponding to the first instances includes: Determine a first pixel point and at least one second pixel point in a plurality of the first instances, wherein the first pixel point represents any pixel point in the plurality of the first instances, and the second pixel point represents a pixel point adjacent to the first pixel point; In response to the first pixel and the second pixel belonging to the same instance, calculating a second average value between the depth value of the first pixel and the depth value of the second pixel; Based on the second average value and the corresponding relationship between the depth value and the first blurring intensity, a first blurring intensity of the first pixel is determined.

6. The method according to claim 3, characterized in that: The determining a first depth value range of the first image includes: Determine a third pixel point and at least one fourth pixel point in the first image, wherein the third pixel point represents a focus point, and the fourth pixel point represents a pixel point adjacent to the third pixel point; In response to the third pixel and the fourth pixel belonging to the same instance, calculating a third average value between the depth value of the third pixel and the depth value of the fourth pixel; Based on the third average value and a preset offset, the first depth value range is determined.

7. The method according to claim 3, characterized in that The multiple instances in the first image are determined in the following manner: Inputting the first image and the depth value of each pixel in the first image into a first neural network model to obtain an output third image, wherein the third image includes a plurality of identifiers; A plurality of instances in the first image are determined based on identifiers in the third image, wherein different identifiers in the third image correspond to different instances in the first image.

8. The method according to claim 3, characterized in that The blurring the first instance based on the corresponding first blurring strength includes: Inputting the first image and the depth value of each pixel in the first image into a second neural network model to obtain an output fourth image, wherein the fourth image includes reference values ​​of each pixel in a plurality of the first instances; determining a second blur intensity based on the first blur intensity and the reference value; The first instance is blurred based on a corresponding second blur strength.

9. The method according to claim 8, characterized in that The determining a second blurring intensity based on the first blurring intensity and the reference value comprises: Determine a second blurring intensity of the first pixel based on the first blurring intensity of the first pixel and a difference between a reference value of the first pixel and a reference value of at least one fifth pixel; The first pixel point represents any pixel point in the first instances, and the fifth pixel point represents a pixel point adjacent to the first pixel point; Among them, the greater the first blurring intensity of the first pixel point, the greater the second blurring intensity; the smaller the difference between the reference value of the first pixel point and the reference value of at least one of the fifth pixel points, the greater the second blurring intensity.

10. The method according to claim 9, characterized in that The number of the third pixel points is determined based on the first blurring intensity of the first pixel point. The greater the first blurring intensity of the first pixel point, the greater the number of the fifth pixel points.

11. An image generating device, characterized in that: include: An acquisition unit, configured to acquire a first image, wherein the first image includes a plurality of instances, and depth values ​​of at least two of the plurality of instances are different; A processing unit, configured to determine a first depth value range of an instance to be displayed in the first image; and perform blurring processing on the first instance outside the first depth value range; A second image is generated based on the blurred first image.

12. An electronic device, characterized in that: include: A memory for storing instructions; as well as A processor, configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 10.

13. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions are executed by the processor, the method according to any one of claims 1 to 10 is executed.

Images