An image blurring method, a training method of a depth prediction model, and an electronic device

Abstract

The application provides an image blurring method, a depth prediction model training method and an electronic device, which can improve the stability of depth information to reduce picture flickering in the video blurring process and improve the user experience. The method can first obtain a first image, input the first image into a depth prediction model to obtain depth information and probability information; wherein the depth information is used to indicate the predicted depth of each pixel point in the first image, and the probability information is used to indicate the confidence of the predicted depth of each pixel point; if the confidence of the depth information is greater than or equal to a first threshold, a second image is displayed; wherein the second image is an image obtained by blurring the first image based on the depth information of the first image, and the confidence of the depth information is determined according to the probability information.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image blurring method, a depth prediction model training method, and an electronic device. Background Art

[0002] With the continuous development of technology, most electronic devices such as mobile phones and tablets have a blur function. This function can make the focused subject area clear and the background area blurred when taking photos or videos, achieving a blur effect that highlights the subject and weakens the background. Electronic devices can distinguish the foreground and background of an image based on the depth information of the image, then keep the foreground clear and blur the background to achieve the ultimate blur effect.

[0003] Currently, a common method for determining depth information is to train a neural network model using training images and their labels, ultimately obtaining a converged neural network model, which is then used to predict the image's depth information. However, due to special scenarios such as occlusion and blur, as well as errors inherent in human annotation, the labels used in training often contain errors or ambiguities. This can lead to instability in the neural network model during the learning process, resulting in the converged neural network model outputting unstable and inaccurate depth information. This, in turn, can cause abnormal flickering in the image when using the blur function during video shooting, creating a poor user experience. Summary of the Invention

[0004] The embodiments of the present application provide an image blurring method, a depth prediction model training method, and an electronic device, which can reduce screen flickering during video blurring by improving the stability of depth information and enhance user experience.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides an image blurring method, which can first obtain a first image, which can be an image collected in real time by an electronic device; then input the first image into a depth prediction model to obtain depth information of the first image and probability information of the first image, wherein the depth information of the first image is used to indicate the predicted depth of each pixel in the first image, and the probability information of the first image is used to indicate the confidence of the predicted depth of each pixel in the first image; in addition, the method can also compare the confidence of the depth information with a first threshold, and the confidence of the depth information can be determined based on the probability information of the first image, such as the mean, variance, standard deviation, etc. of the confidence of the predicted depth of each pixel; wherein when the confidence of the depth information is greater than or equal to the first threshold, it indicates that the depth information obtained this time is more credible, and therefore the first image is blurred based on the depth information of the first image to obtain a second image and the second image is displayed; when the confidence of the depth information is less than the first threshold, it indicates that the depth information obtained this time is less credible, and therefore the second image is not displayed.

[0007] In this way, the device only displays the blurring effect of the image when the confidence level of the depth information of the acquired image is greater than or equal to the first threshold value, which can reduce the impact of some inaccurate depth information on the image blurring effect, thereby reducing the screen flickering phenomenon during the video blurring process and improving the user experience.

[0008] In an implementation provided in the first aspect, a depth prediction model is obtained by training a basic depth prediction model based on a training sample, wherein the training process includes: iteratively training the basic depth prediction model based on the training sample to generate a depth prediction model, the training sample including a training image and a label, the label being used to indicate the true depth of each pixel in the training image. During any iterative training process, the device can input the training image into the basic depth prediction model after the previous iteration to obtain the depth information and probability information of the training image; when the loss value of the training image meets the convergence condition (for example, it is less than a preset value), the iterative training is stopped and the depth prediction model is output, which is the basic depth prediction model after the previous iteration; when the loss value of the training image does not meet the convergence condition (for example, it is greater than or equal to a preset value), the model parameters of the basic depth prediction model after the previous iteration are adjusted according to the loss value of the training image and the iterative training is continued, and the loss value of the training image is determined according to the label, the depth information and probability information of the training image.

[0009] That is, in the process of iteratively training the basic depth prediction model, the loss value is determined from two dimensions of depth and confidence of depth, and the model parameters are adjusted based on the loss value, which can make the predicted depth closer and closer to the true depth, and the confidence of the predicted depth closer and closer to the confidence of the true depth, can reduce the influence of labels (i.e. inaccurate true depth) in special scenes such as occlusion and blur on the training process, make the training process more stable, and further make the depth prediction model obtained by the final training have higher stability, and the depth information finally output by the depth prediction model is also more stable.

[0010] In an implementation form of the first aspect, the loss value of the training image is determined according to the loss value of each pixel point in the training image, and the loss value of any one pixel point in the training image is determined according to the true depth, the predicted depth and the confidence of the predicted depth of the pixel point.

[0011] In an implementation form of the first aspect, the training image includes a first region and a second region, the first region is different from the second region, and the first region is a background hollow region in the training image. The background hollow region can refer to a background region within a certain foreground object in the image or a background region spaced between two foreground objects. The loss value of the training image is obtained by weighted sum of the loss value of the first region and the loss value of the second region, the weight coefficient corresponding to the first region is greater than the weight coefficient corresponding to the second region, the loss value of the first region is determined according to the loss value of each pixel point in the first region of the training image, the loss value of the second region is determined according to the loss value of each pixel point in the second region of the training image, and the loss value of any one pixel point in the training image is determined according to the true depth, the predicted depth and the confidence of the predicted depth of the pixel point.

[0012] It can be understood that the weight coefficient corresponding to the first region is greater than the weight coefficient corresponding to the second region, that is, the loss value of the first region has a greater influence on the loss value of the training image. In this way, the basic depth prediction model can be trained and learned for the background hollow region, so as to improve the accuracy of the predicted depth of the basic depth prediction model for the background hollow region in the iterative training process, and further reduce the missing virtual phenomenon.

[0013] In an implementation form of the first aspect, the loss value of any one pixel point in the training image is the similarity of the first probability distribution function and the second probability distribution function of the pixel point, the first probability distribution function is used to indicate the probability distribution of the true depth of the pixel point, and the second probability distribution function is used to indicate the probability distribution of the predicted depth of the pixel point.

[0014] In an implementation form of the first aspect, the first probability distribution function is determined according to the true depth of the pixel point, and the second probability distribution function is determined according to the predicted depth and the confidence of the predicted depth of the pixel point.

[0015] In an implementation provided by the first aspect, the first probability distribution function is a Dirkat distribution function constructed based on the true depth of the pixel point, and the second probability distribution function is a Gaussian distribution function constructed with the predicted depth of the pixel point as the mean and the confidence level of the predicted depth of the pixel point as the variance. The expression of the first probability distribution function is: P D (x) = δ(xx g ), x g is the true depth of the pixel. It can be understood that the first probability distribution function indicates that the true depth of the pixel is x g The probability is infinite, which can also be understood as the true depth of the pixel is x g The probability of is infinitely close to 1. The expression of the second probability distribution function is: x e is the predicted depth of the pixel, σ is the predicted depth x e confidence level.

[0016] In one implementation provided in the first aspect, the similarity between the first probability distribution function and the second probability distribution function is a KL divergence between the first probability distribution function and the second probability distribution function. The KL divergence, the first probability distribution function, and the second probability distribution function satisfy Formula 1:

[0017]

[0018] Among them, D KL (P D (x)||P θ (x)) is the KL divergence, P D (x) is the first probability distribution function, P θ (x) is the second probability distribution function.

[0019] It can be understood that the smaller the KL divergence of the first probability distribution function and the second probability distribution function, the higher the similarity between the first probability distribution function and the second probability distribution function, that is, the predicted depth is closer to the true depth, and the confidence of the predicted depth is closer to the confidence of the true depth.

[0020] In the second aspect, an embodiment of the present application provides a method for training a depth prediction model, the method comprising: iteratively training a basic depth prediction model based on training samples to generate a depth prediction model; wherein the training samples include labels and training images, and the labels are used to indicate the true depth of each pixel in the training image; during any iterative training process, the training image is input into the basic depth prediction model after the previous iteration, and the depth information and probability information of the training image are output; when the loss value of the training image meets the convergence condition, the iterative training is stopped; wherein the loss value of the training image is determined based on the label, the depth information and probability information of the training image; when the loss value of the training image does not meet the convergence condition, the model parameters of the basic depth prediction model after the previous iteration are adjusted according to the loss value of the training image, and the iterative training is continued. The depth prediction model trained by the embodiment of the present application can be applied to the image blurring method provided in the first aspect, and can also be applied to autonomous driving scenarios, robot navigation scenarios, etc., without specific limitation here.

[0021] In an implementation method provided in the second aspect, the loss value of the training image is determined based on the loss value of each pixel in the training image, and the loss value of any pixel in the training image is determined based on the true depth, predicted depth and confidence of the predicted depth of the pixel.

[0022] In an implementation method provided in the second aspect, the training image includes a first area and a second area, the first area is different from the second area, the first area is a background hollow area in the training image, the loss value of the training image is obtained by weighted summing the loss value of the first area and the loss value of the second area, the weight coefficient corresponding to the first area is greater than the weight coefficient corresponding to the second area, the loss value of the first area is determined according to the loss value of each pixel point in the first area of ​​the training image, the loss value of the second area is determined according to the loss value of each pixel point in the second area of ​​the training image, and the loss value of any pixel point in the training image is determined according to the true depth, predicted depth and confidence of the predicted depth of the pixel point.

[0023] In an implementation method provided in the second aspect, the loss value of any pixel point in the training image is the similarity between the first probability distribution function and the second probability distribution function of the pixel point, the first probability distribution function is used to indicate the probability distribution of the true depth of the pixel point, and the second probability distribution function is used to indicate the probability distribution of the predicted depth of the pixel point.

[0024] In an implementation manner provided by the second aspect, the first probability distribution function is determined based on the true depth of the pixel point, and the second probability distribution function is determined based on the predicted depth of the pixel point and the confidence of the predicted depth.

[0025] In an implementation form of the second aspect, the first probability distribution function is a Dirichlet distribution function constructed according to the real depth of the pixel point, and the second probability distribution function is a Gaussian distribution function constructed with the predicted depth of the pixel point as a mean value and a confidence degree of the predicted depth of the pixel point as a variance.

[0026] In an implementation form of the second aspect, the similarity between the first probability distribution function and the second probability distribution function is a KL divergence between the first probability distribution function and the second probability distribution function.

[0027] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and one or more processors; the memory and the processor are coupled; the memory is configured to store computer program codes, the computer program codes comprising computer instructions, when the computer instructions are executed by the electronic device, causing the electronic device to execute the method of the first aspect, the second aspect and any implementation form thereof.

[0028] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing computer instructions, when the computer instructions are run in the electronic device, causing the electronic device to execute the method of the first aspect, the second aspect and any implementation form thereof.

[0029] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer programs / instructions, when the computer programs / instructions are executed by the processor, realizing the steps of the method provided by the first aspect, the second aspect and any implementation form thereof.

[0030] The technical effects brought by any one of the second aspect to the fifth aspect can refer to the technical effects brought by different implementation forms of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 An ordinary photographing and photographing blur effect schematic diagram provided by an embodiment of the present application;

[0032] Figure 2 A blur effect schematic diagram provided by an embodiment of the present application;

[0033] Figure 3 Another blur effect schematic diagram provided by an embodiment of the present application;

[0034] Figure 4 A structure schematic diagram of an AI system provided by an embodiment of the present application;

[0035] Figure 5 A flowchart of a training method of a depth prediction model provided by an embodiment of the present application;

[0036] Figure 6 A schematic diagram of the network structure of a basic depth prediction model provided in an embodiment of the present application;

[0037] Figure 7 A schematic diagram of a process for determining the loss value of a training image using a basic depth prediction model provided in an embodiment of the present application;

[0038] Figure 8 A graph of a first probability distribution function and a second probability distribution function provided in an embodiment of the present application;

[0039] Figure 9 A terminal provided in an embodiment of the present application provides a depth image output by a basic depth prediction model before and after optimization training for a background hollowed-out area;

[0040] Figure 10 A flowchart of an image blurring method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. Among them, in the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and claims of the present application, the singular expressions "a", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in a kind of "or" relationship.

[0042] References to "one embodiment" or "some embodiments" etc. described in this specification mean that the specific features, structures or characteristics described in conjunction with the embodiment are included in one or more embodiments of the present application. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in another way. The term "connected" includes direct and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0043] In the embodiments of this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0044] The image blurring method provided in the embodiments of the present application can be applied to scenarios requiring environmental understanding, such as photography blurring, autonomous driving, and robot navigation. In blurring scenarios, depth can help electronic devices better distinguish between foreground and background, achieving a good blurring effect. These blurring scenarios include both photo blurring and video blurring. In autonomous driving and robot navigation, depth can help vehicles and robots better understand their surroundings, enabling path planning and obstacle avoidance.

[0045] Figure 1 The following diagram shows the effects of normal photography and blurring. Figure 1 As shown in (a) of FIG, the mobile phone can display an interface 101, which includes multiple shooting mode options, such as "photo", "portrait", "video", etc., where "photo" is selected. In addition, the interface 101 also includes an image 102. In response to the user's operation on "portrait", as shown in FIG. Figure 1 As shown in (b), the mobile phone can display an interface 103, which also includes shooting mode options such as "photo", "portrait", and "video", and "portrait" is selected. In addition, the interface 103 also includes an image 104, which includes a foreground 104a and a background 104b. Figure 1As can be seen from (a) and (b) of FIG. 1, the foreground 104a of the image 102 and the foreground 104a of the image 104 are both in a clear state, and the background 104b of the image 104 is in a blurred state, that is, the image 102 is an effect schematic diagram of ordinary photographing, and the image 104 is an effect schematic diagram of photographing blurring.

[0046] In the photographing blurring scenario, the mobile phone can calculate the depth of field according to the focusing position of the image 104 and the depth information of the image 104, and distinguish the foreground 104a and the background 104b of the image 104 according to the depth of field, and then perform different degrees of blurring processing on the region where the foreground 104a is located and the region where the background 104b is located (for example, keeping the clarity of the region where the foreground 104a is located unchanged and reducing the clarity of the region where the background 104b is located), so as to achieve the blurring effect that the clarity of the foreground 104a and the background 104b is inconsistent. It should be noted that the principle of the video blurring scenario is similar to that of the photographing blurring scenario, which will not be described here.

[0047] In the photographing blurring scenario, due to the instability of the depth calculation process or the inaccuracy of the calculation result, abnormal phenomena such as picture flickering or missing blurring often occur. Among them, picture flickering refers to the phenomenon that in the video blurring scenario, the difference between the depth information of the continuous frames is large, resulting in a large difference between the blurring effects of the continuous frames. For example, Figure 2 (a) and (b) of FIG. 2 respectively show the blurring effect schematic diagrams of the image 201 and the image 202, and the image 201 and the image 202 are two continuous images in the video being photographed by the mobile phone. As shown in (a) and (b) of FIG. 2, Figure 2 the foreground of the image 201 is clear and the background is blurred, and the foreground and the background of the image 202 are both clear, that is, the blurring effects of the image 201 and the image 202 are quite different, which gives the user a flickering picture feeling.

[0048] Missing blurring refers to the phenomenon that in the video blurring scenario or the photographing blurring scenario, due to the inaccuracy of the depth information, part of the region that should be blurred is not blurred. The missing blurring phenomenon often occurs in images with hollow regions. For example, Figure 3 (a) of FIG. 3 shows an effect schematic diagram in which the image 301 is normally blurred, Figure 3 (b) of FIG. 3 shows an effect schematic diagram in which the image 302 is missing blurred, and the image 301 and the image 302 are the same image. By comparing Figure 3 (a) and (b) of FIG. 3, it can be seen that the image 301 includes a region 301a, and the image 302 includes a region 302a, the region 301a and the region 302a are the same hollow region, the region 301a of the image 301 is normally blurred, and the region 302a of the image 302 is not blurred.

[0049] It can be seen that the blur effect presented by the mobile phone in the blurred scene is largely dependent on the stability of the depth information calculation process and the accuracy of the depth information. It should be noted that the above content is only used as an example to illustrate the blurred scene. The same principle applies to scenarios such as autonomous driving and robot navigation. In other words, the accuracy of path planning and obstacle avoidance is largely dependent on the stability of the depth information calculation process and the accuracy of the depth information.

[0050] like Figure 4 As shown, Figure 4 This is a schematic diagram of the structure of the AI ​​system provided in this application. Figure 4 As shown, the AI ​​system includes a data center and multiple terminals (such as Figure 4 The data center can communicate with the terminals via a network, which can be the Internet or other networks. The network can include one or more network devices, such as routers or switches.

[0051] A data center consists of one or more servers, such as Figure 4 The server 120 shown is, for example, an application server that supports application services, which can provide image services, video services, game services, other AI processing services based on video or images, etc. In an optional scenario, the server 120 refers to a server cluster deployed with multiple servers. The server cluster can have a rack that can establish communication for the multiple servers via a wired connection, such as a universal serial bus (USB) or a peripheral component interconnect express (PCIe) high-speed bus.

[0052] Server 120 can also obtain data from the terminal, perform AI processing on the data, and send the results of the AI ​​processing to the corresponding terminal. This AI processing can refer to using AI models to perform tasks such as depth prediction, object recognition, and target detection on the data, or it can refer to obtaining an AI model that meets the requirements based on samples collected by the terminal.

[0053] in addition, Figure 4 The data center shown may also include other physical devices with AI processing capabilities, such as mobile phones, tablets, or other devices.

[0054] Terminal can also be called terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The terminal can be a mobile phone (such as Figure 4The terminal 111 shown), a face-scanning payment device with a mobile payment function, a camera device with a data (such as image or video) acquisition and processing function (such as Figure 4 The terminal 112 shown in the figure may also be a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, a vehicle with data (such as image or video) acquisition and processing functions, etc. The embodiments of the present application do not limit the specific technology and specific device form adopted by the terminal device.

[0055] It is worth noting that the terminal can obtain the AI ​​model stored in the server 120, and then use the AI ​​model to perform various tasks on the information contained in the image. For example, the AI ​​model can be a model for performing operations such as depth prediction on the data.

[0056] For example, the AI ​​model deployed in terminal 112 can be used to implement functions such as depth prediction.

[0057] For example, the AI ​​model deployed in terminal 111 can realize functions such as automatically adjusting the local clarity of the image.

[0058] For example, the AI ​​model deployed in the terminal 111 implements functions such as face payment and object classification (such as product classification).

[0059] It should be noted that Figure 4 This is just a schematic diagram and should not be construed as limiting the present application. The embodiments of the present application do not limit the application scenarios of the terminal and the server.

[0060] In related technologies, it is possible to use Figure 4 The AI ​​system shown trains a basic depth prediction model to obtain a depth prediction model. The server 120 can input labeled training images into the basic depth prediction model to obtain depth information. The labels can be used to reflect the true depth of each pixel in the training image, and the depth information can be used to reflect the predicted depth of each pixel in the training image. The server 120 can then calculate a loss function based on the labels and the depth image, and adjust the parameters of the basic depth prediction model based on the loss function to ultimately obtain a depth prediction model.

[0061] In the above scheme, the loss function, depth image, and label satisfy the following formula:

[0062]

[0063]

[0064] Among them, D(y, y*) is the loss function, y is the depth image, y* is the label, y i is the true depth of the i-th pixel in the training image, is the predicted depth of the i-th pixel in the training image, and n is the number of pixels in the training image.

[0065] After the server 120 has trained the depth prediction model, it can port the depth prediction model to the terminal. The terminal can then input the image into the depth prediction model to obtain the image's depth information, and then perform operations such as blurring based on the depth information, and display it on the display screen. In other embodiments, the above training process can also be performed by the terminal.

[0066] However, due to special scenarios such as occlusion and blur, as well as errors in manual labeling, the labels used in the above training process often contain errors or ambiguities, which will cause errors in the loss function, resulting in errors and instability in the training process. The depth information output by the trained depth prediction model is not stable and accurate enough, which in turn leads to abnormal phenomena such as abnormal flickering or missing blur when shooting blurred scenes, or causes the navigation routes / obstacle avoidance routes planned in scenarios such as autonomous driving / robot navigation to be not accurate enough, giving users a bad experience.

[0067] To at least solve the above problems, the embodiments of the present application provide a method for training a depth prediction model and an image blurring method, which can improve the stability of the depth prediction process and the accuracy of the depth prediction results, and further reduce abnormal phenomena such as screen flickering and blurring when the method is applied to shooting blurred scenes. It should be noted that the method for training a depth prediction model and the image blurring method provided in the embodiments of the present application are inventions based on the same concept, and can also be understood as two parts in a system, or two stages of an overall process: such as the model training stage and the model application stage.

[0068] In the model training stage, an embodiment of the present application provides a training method for a depth prediction model, which uses depth information and probability information as the output of a basic depth prediction model, constructs a probability distribution of the predicted depth of each pixel using the depth information and probability information, and constructs a probability distribution of the true depth of each pixel using a label, and then uses the similarity between the probability distribution of the predicted depth and the probability distribution of the true depth as a loss value and adjusts the parameters of the basic depth prediction model based on the loss value to obtain a depth prediction model. Among them, using the similarity between the probability distribution of the predicted depth and the probability distribution of the true depth as a loss value can not only consider the error between the true value and the predicted value of the depth in the training stage, but also consider the error between the confidence of the true value and the confidence of the predicted value, that is, to train the basic depth prediction model from two dimensions of depth and depth confidence. This training process is more stable, so that the stability of the depth prediction model obtained by training is also higher, which can reduce the phenomenon of screen flickering in video blur scenes.

[0069] In the model application stage, an embodiment of the present application provides an image blurring method, which can input the acquired first image into the trained depth prediction model to obtain the depth information and probability information of the first image, and display the second image when the confidence level of the depth information is greater than or equal to a first threshold. The second image is an image after blurring the first image based on the depth information of the first image, and the confidence level of the depth information is determined according to the probability information.

[0070] The following first describes the above-mentioned model training stage with reference to the accompanying drawings.

[0071] Figure 5 A flow chart of a method for training a depth prediction model provided by an embodiment of the present application is shown. Figure 4 The server 120 or terminal in the embodiment can also be executed by other devices with computing capabilities. The embodiment of the present application takes the terminal execution as an example. Figure 5 The training method of the depth prediction model provided in the embodiment of the present application includes at least steps S510 to S540.

[0072] S510: The terminal iteratively trains the basic depth prediction model based on the training samples.

[0073] The training samples include training images with labels, where the labels are used to indicate the true depth of each pixel in the training image (which can be understood as the true value of the depth).

[0074] In the embodiment of the present application, the training image is input into the basic depth prediction model, and depth information and probability information can be obtained. The depth information is used to indicate the predicted depth (which can be understood as the predicted value of the depth) of each pixel point in the training image, and the probability information is used to indicate the confidence of the predicted depth of each pixel point in the training image.

[0075] For example, the depth information indicates that the predicted depth of the i th pixel point in the training image is 0.5, and the probability information indicates that the confidence of the predicted depth of the i th pixel point in the training image is 0.6, so the probability of the predicted depth of the i th pixel point in the training image being 0.5 is 0.6.

[0076] Figure 6 A network structure schematic diagram of the basic depth prediction model provided by the embodiment of the present application is shown. Please refer to Figure 6 The basic depth prediction model includes m feature extraction modules, m up-sampling modules and a prediction module. The process of outputting the depth information and the probability information by the basic depth prediction model is as follows: the input image is input into the feature extraction module 1 to obtain the down-sampling result 1, and the down-sampling result 1 is input into the up-sampling module 1 to obtain the up-sampling result 1; then the feature map obtained by fusing the up-sampling result 1 and the input image is input into the feature extraction module 2 to obtain the down-sampling result 2, and the down-sampling result 2 is input into the up-sampling module 2 to obtain the up-sampling result 2; then the feature map obtained by fusing the up-sampling result 2 and the input image is input into the feature extraction module 3 to obtain the down-sampling result 3; and so on, the down-sampling result m-1 is input into the up-sampling module m-1 to obtain the up-sampling result m-1, and the feature map obtained by fusing the up-sampling result m-1 and the input image is input into the feature extraction module m to obtain the down-sampling result m, and the down-sampling result m is input into the up-sampling module m to obtain the up-sampling result m. Finally, the feature map obtained by fusing the up-sampling result m and the input image is input into the prediction module to obtain the depth information and the probability information.

[0077] It should be noted that before fusing the up-sampling result and the input image, the input image can be subjected to convolution or pooling processing so that the size of the input image is the same as the size of the up-sampling result to be fused. For example, the size of the input image is 3×384×384, and the size of the up-sampling result 2 is 128×48×48, so before fusing the up-sampling result 2 and the input image, the input image can be subjected to convolution or pooling processing so that the size of the input image becomes 3×384×384, and then the up-sampling result 2 and the input image subjected to the convolution or pooling processing are fused.

[0078] Still as Figure 6As shown, each feature extraction module, prediction module, and upsampling module may include a different number of convolutional layers, pooling layers, and / or fully connected layers, etc. There is no restriction on the specific structure of the feature extraction module, prediction module, and upsampling module, as long as the corresponding functions can be achieved.

[0079] In an embodiment of the present application, the size of the feature maps output by each feature extraction network is different, which can extract richer detail features from the input image, which is conducive to improving the accuracy of the model output (i.e., depth information).

[0080] S520: The terminal determines whether the basic depth prediction model has converged.

[0081] If the basic depth prediction model does not converge, execute S530 ; if the basic depth prediction model converges, execute S540 .

[0082] In one possible design, the terminal can compare the loss value of the training image with a preset value to determine whether the basic depth prediction model has converged. If the loss value of the training image is less than or equal to the preset value, it indicates that the basic depth prediction model has converged; if the loss value of the training image is greater than the preset value, it indicates that the basic depth prediction model has not converged.

[0083] In an embodiment of the present application, the loss value of a training image can be determined based on the label, depth information, and probability information corresponding to the training image. Figure 7 The figure shows a flow chart of determining the loss value of a training image using the basic depth prediction model provided in the embodiment of the present application. Figure 7 , the basic depth prediction model can predict the training image to obtain depth information and probability information, and then the basic depth prediction model can construct a probability distribution function of the predicted depth based on the depth information and probability information, and construct a probability distribution function of the true depth based on the label, and finally calculate the loss value of the training image based on the probability distribution function of the predicted depth and the probability distribution function of the true depth.

[0084] Understandably, in Figure 7 In the flowchart shown, the loss value of the training image is calculated and output by the basic depth prediction model. In other embodiments, the loss value of the training image can also be calculated by the terminal using other algorithm modules, which is not specifically limited here.

[0085] In an embodiment of the present application, the terminal may first determine the loss value of each pixel in the training image, and then perform a weighted summation of the loss values ​​of all pixels to obtain the loss value of the training image. For any pixel in the training image, the terminal may determine the loss value of the pixel based on the true depth, predicted depth, and confidence level of the predicted depth. Figure 7The process shown is similar. For any pixel in the training image, the terminal can construct a probability distribution function of the predicted depth based on the predicted depth of the pixel and the confidence of the predicted depth, and construct a probability distribution function of the actual depth based on the actual depth of the pixel. Then, the loss value of the pixel is calculated based on the probability distribution function of the predicted depth and the probability distribution function of the actual depth.

[0086] In one possible design, for any pixel in the training image, the terminal can construct a Diktala distribution function (i.e., the probability distribution function of the true depth, also referred to as the first probability distribution function) based on the true depth of the pixel, and construct a Gaussian distribution function (i.e., the probability distribution function of the predicted depth, also referred to as the second probability distribution function) with the predicted depth of the pixel as the mean and the confidence of the predicted depth of the pixel as the variance, and then calculate the similarity between the first probability distribution function and the second probability distribution function as the loss value of the pixel.

[0087] Figure 8 The graph of the first probability distribution function and the second probability distribution function provided by the embodiment of the present application is shown. The expression of the first probability distribution function is: P D (x) = δ(xx g ), x g is the true depth of the pixel. It can be understood that the first probability distribution function indicates that the true depth of the pixel is x g The probability is infinite, which can also be understood as the true depth of the pixel is x g The probability is infinitely close to 1.

[0088] The expression of the second probability distribution function is: x e is the predicted depth of the pixel, σ is the predicted depth x e It can be understood that the second probability distribution function indicates that the predicted depth of the pixel point is Gaussian distributed, and the mean of the distribution is x e , with variance σ.

[0089] In the embodiment of the present application, the KL divergence (Kullback-Leibler divergence) can be used to measure the similarity (or difference) between the first probability distribution function and the second probability distribution function. The KL divergence can also be called relative entropy or information divergence. Wherein, the KL divergence, the first probability distribution function and the second probability distribution function satisfy Formula 1:

[0090]

[0091] Among them, D KL (P D (x)||Pθ (x)) is the KL divergence, P D (x) is the first probability distribution function, P θ (x) is the second probability distribution function.

[0092] Understandably, D KL (P D (x)||P θ The smaller (x)), the higher the similarity between the first probability distribution function and the second probability distribution function, that is, the closer the predicted depth is to the true depth, and the closer the confidence of the predicted depth is to the confidence of the true depth.

[0093] In this way, the loss value of the training image satisfies Equation 2:

[0094]

[0095] in, is the loss value of the training image, N is the number of pixels in the training image, D KLi (P D (xi)||P θ (xi)) is the KL divergence (i.e., loss value) of the i-th pixel in the training image, P D (xi) is the first probability distribution function of the i-th pixel in the training image, P θ (xi) is the second probability distribution function of the i-th pixel in the training image.

[0096] It should be noted that the above formula 2 is a formula used when the weight of each pixel is the same. When the weight of each pixel is different, the formula 2 can be adaptively adjusted according to the weight.

[0097] S530: The terminal updates the model parameters according to the loss value of the training image during the iteration process.

[0098] S540: The terminal outputs a depth prediction model.

[0099] It can be understood that the depth prediction model is a converged basic depth prediction model.

[0100] Among them, outputting the depth prediction model can be understood as the terminal storing the depth prediction model in a memory, or sending it to other devices, such as servers or other terminals, and no specific restrictions are made here.

[0101] It can be understood that, compared with the scheme for predicting depth information provided by related technologies, the embodiment of the present application updates the model parameters based on the loss value of the training image during the iteration process, and the loss value is calculated based on the probability distribution function of the predicted depth and the probability distribution function of the true depth, which can make the predicted depth output by the basic depth prediction model closer and closer to the true depth, and make the confidence of the predicted depth output by the basic depth prediction model closer and closer to the confidence of the true depth (close to 1), that is, the possible errors in the label and depth information are taken into account during the training process, which can reduce the impact of special scenes such as occlusion and blur on the training process, making the training process more stable, and thus making the depth prediction model finally trained have higher stability, and the depth information finally output by the depth prediction model is also more stable.

[0102] In one possible implementation, the training image includes a background hollowed-out area and a non-background hollowed-out area. The background hollowed-out area may refer to a background area within a foreground object or a background area between two foreground objects in the image, and the non-background hollowed-out area refers to areas other than the background hollowed-out area.

[0103] For example, the training image can be Figure 3 In the image 301 shown, the background hollowed-out area may be the area 301 a in the image 301 , and the non-background hollowed-out area may be all areas in the image 301 except the area 301 a .

[0104] For example, the training image can be Figure 9 The original image shown may include a background hollow area (a shadow area in the original image) and a non-background hollow area (an area other than the shadow area in the original image).

[0105] In addition, the label is also used to indicate the distribution of background hollow areas and non-background hollow areas. For example, the label includes the location information of the boundary of the background hollow area. The terminal can determine whether the pixel is located in the background hollow area or non-background hollow area by determining whether the pixel is within the boundary of the background hollow area. For another example, the label includes an attribute of each pixel in the training image, which is used to indicate whether the pixel is located in the background hollow area or non-background hollow area.

[0106] In this case, the terminal can calculate the loss value of the background hollow area and the loss value of the non-background hollow area respectively, and perform weighted summation of the loss value of the background hollow area and the loss value of the non-background hollow area according to the preset weight coefficient to obtain the loss value of the training image.

[0107] Among them, the loss value of the training image, the loss value of the background hollow area, and the loss value of the non-background hollow area can satisfy Formula 3:

[0108]

[0109] in, is the loss value of the training image, D KL_S1 is the loss value of the background hollow area, D KL_S2 is the loss value of the non-background hollow area, a1 and a2 are the preset weight coefficients, and a1>a2.

[0110] It should be noted that the loss value of the background hollow area and the loss value of the non-background hollow area can be calculated according to the method provided in S520, which will not be repeated here.

[0111] In one possible design, the non-background hollow area can be further divided into the main body area and the background non-hollow area. The label is used to indicate the distribution of the background hollow area, the main body area, and the background non-hollow area. Accordingly, the background hollow area, the main body area, and the background non-hollow area have different weights, and the weight coefficient of the background hollow area is greater than the weight coefficient of the main body area, and the weight coefficient of the background hollow area is greater than the weight coefficient of the background non-hollow area.

[0112] In one possible design, the above-mentioned depth information can be presented in the form of an image, namely a depth image. The terminal can mark the predicted depth of each pixel in the training image and use the marked training image as the depth information. For example, the training image can be Figure 9 The original image in the image, the depth information can be Figure 9 The depth image in .

[0113] It can be understood that by marking the predicted depth of each pixel in the image, the marked image can display the depth values ​​of different pixels in the image, so that users can quickly view the depth values ​​of different pixels from the depth image displayed by the terminal, thereby improving the quality of experience (QoE) of the user experience.

[0114] Figure 9 The depth images output by the terminal's basic depth prediction model before and after optimization training for the background hollowed-out area are shown. It can be seen that the depth image output by the optimized and trained basic depth prediction model has clearer and more distinct boundaries in the background hollowed-out area than the depth image output by the unoptimized basic depth prediction model.

[0115] That is, the loss values of different regions are weighted and calculated according to preset weight coefficients to obtain the loss value of the training image, and the weight coefficient of the background hollow region is greater than that of the non-background hollow region, so that the basic depth prediction model can be trained and learned for the background hollow region, thereby improving the accuracy of the predicted depth of the basic depth prediction model for the background hollow region, and further reducing the missing and virtual phenomenon.

[0116] The model application stage of the present application is described below.

[0117] Figure 10 A flowchart of an image blurring method provided by an embodiment of the present application is shown. The method can be executed by the server 120 or the terminal in Figure 4 , or by other devices with computing capabilities, and an embodiment of the present application takes the terminal as an example for description. Please refer to Figure 10 , the image blurring method provided by an embodiment of the present application includes at least steps S1001-S1005.

[0118] S1001, obtaining a first image.

[0119] The first image can be an image obtained by the terminal device in real time, any frame of image in a video obtained by the terminal device in real time, an image sent by other devices to the terminal device, any frame of image in a video sent by other devices to the terminal device, or a pre-stored image in the terminal device, etc., which is not limited here.

[0120] S1002, inputting the first image into a depth prediction model to obtain depth information and probability information.

[0121] It should be noted that the depth prediction model provided by an embodiment of the present application is a converged basic depth prediction model, so the network structure of the depth prediction model is the same as that of the aforementioned basic depth prediction model, which is not described here.

[0122] It can be understood that the depth prediction model used in the image blurring method provided by an embodiment of the present application is obtained based on the training method of the depth prediction model provided by Figure 5 , which is more stable, and further makes the depth prediction model obtained by the final training have higher stability, so that the depth information finally output by the depth prediction model is also more stable, i.e., the depth information of consecutive video frames in the video blurring scene is also more stable, which can reduce the flickering phenomenon caused by unstable depth information.

[0123] S1003, determining whether the confidence of the depth information is greater than or equal to a first threshold.

[0124] The confidence level of the depth information is related to the probability information. For example, the confidence level of the depth information may be the variance, standard deviation, mean, etc. of the confidence levels of the predicted depths of all pixels in the probability information.

[0125] S1004: If the confidence level of the depth information is greater than or equal to the first threshold, display the second image.

[0126] The second image is an image obtained by blurring the first image according to the depth information.

[0127] S1005: If the confidence level of the depth information is less than a first threshold, display the first image.

[0128] For example, Figure 1 or Figure 2 Taking the blurred scene shown in the figure as an example, the terminal can display a first interface including multiple shooting options. Figure 1 In the interface 101, multiple shooting options are, for example, "portrait", "photo", "video", etc. In response to the user's operation on the first shooting option, the terminal can input the first image into the depth prediction model to obtain depth information and probability information, and display the second image when the confidence level of the depth information is greater than or equal to the first threshold, such as image 104 or image 202, or directly display the first image when the confidence level of the depth information is less than the first threshold. Among them, the first shooting option may include an option that triggers the terminal to blur the image (i.e., the first image) captured in real time by the camera, such as "portrait", "video", etc.

[0129] That is to say, the first image is blurred and displayed only when the confidence level of the depth information of the first image is higher than the first threshold. This can further increase the stability of the depth information and further reduce the screen flickering between consecutive video frames in the video blur scenario.

[0130] In other possible designs, the above-mentioned depth prediction model can also be applied to autonomous driving scenarios, robot navigation scenarios, etc., without specific restrictions here.

[0131] An embodiment of the present application further provides a chip system, which includes at least one processor and at least one interface circuit. The processor and the interface circuit can be interconnected via lines. For example, the interface circuit can be used to receive signals from other devices (such as a memory of an electronic device). For another example, the interface circuit can be used to send signals to other devices (such as a processor). Exemplarily, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the electronic device can perform the various steps in the above embodiments. Of course, the chip system can also include other discrete devices, which is not specifically limited in the embodiment of the present application.

[0132] An embodiment of the present application further provides a computer storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device, the electronic device executes the various functions or steps executed by the terminal or server in the above-mentioned method embodiment.

[0133] An embodiment of the present application further provides a computer program product, which, when executed on a computer, enables the computer to execute the functions or steps executed by the terminal or server in the above method embodiment.

[0134] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0135] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0136] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0138] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0139] The above content is only a specific embodiment of this application, but the scope of protection of this application is not limited to this. Any changes or replacements within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for blurring an image, characterized in that: The method comprises: acquiring a first image; Inputting the first image into a depth prediction model to obtain depth information and probability information; wherein the depth information is used to indicate the predicted depth of each pixel in the first image, and the probability information is used to indicate the confidence level of the predicted depth of each pixel; If the confidence level of the depth information is greater than or equal to a first threshold, displaying a second image; wherein the second image is an image obtained by blurring the first image based on the depth information of the first image, and the confidence level of the depth information is determined according to the probability information; If the confidence level of the depth information is less than a first threshold, displaying the first image; The depth prediction model is obtained by training a basic depth prediction model based on training samples, wherein the training process includes: Iteratively training the basic depth prediction model based on the training samples to generate the depth prediction model; wherein the training samples include labels and training images, and the labels are used to indicate the true depth of each pixel in the training image; During any iterative training process, the training image is input into the basic depth prediction model after the previous iteration, and the depth information and probability information of the training image are output; When the loss value of the training image meets the convergence condition, the iterative training is stopped; wherein the loss value of the training image is determined according to the label, the depth information and the probability information of the training image, and the loss value of the training image is determined according to the loss value of each pixel in the training image, and the loss value of any pixel in the training image is the similarity between the first probability distribution function and the second probability distribution function of the pixel, the first probability distribution function is used to indicate the probability distribution of the true depth of the pixel, and the second probability distribution function is used to indicate the probability distribution of the predicted depth of the pixel; When the loss value of the training image does not meet the convergence condition, the model parameters of the basic depth prediction model after the last iteration are adjusted according to the loss value of the training image, and the iterative training is continued.

2. The method according to claim 1, characterized in that The training image includes a first area and a second area, the first area is different from the second area, the first area is the background hollow area in the training image, the loss value of the training image is obtained by weighted summing the loss value of the first area and the loss value of the second area, the weight coefficient corresponding to the first area is greater than the weight coefficient corresponding to the second area, the loss value of the first area is determined according to the loss value of each pixel point in the first area of ​​the training image, the loss value of the second area is determined according to the loss value of each pixel point in the second area of ​​the training image, and the loss value of any pixel point in the training image is determined according to the true depth, predicted depth and confidence of the predicted depth of the pixel point.

3. The method according to claim 1, characterized in that The first probability distribution function is determined according to the true depth of the pixel point, and the second probability distribution function is determined according to the predicted depth of the pixel point and the confidence of the predicted depth.

4. The method according to claim 3, characterized in that The first probability distribution function is a Dirkat distribution function constructed based on the true depth of the pixel point, and the second probability distribution function is a Gaussian distribution function constructed with the predicted depth of the pixel point as the mean and the confidence of the predicted depth of the pixel point as the variance.

5. The method according to any one of claims 1, 3-4, characterized in that: The similarity between the first probability distribution function and the second probability distribution function is a KL divergence between the first probability distribution function and the second probability distribution function.

6. A training method for a depth prediction model, characterized in that: The method comprises: Iteratively training a basic depth prediction model based on training samples to generate a depth prediction model; wherein the training samples include a label and a training image, and the label is used to indicate the true depth of each pixel in the training image; During any iterative training process, the training image is input into the basic depth prediction model after the previous iteration, and the depth information and probability information of the training image are output; When the loss value of the training image meets the convergence condition, the iterative training is stopped; wherein the loss value of the training image is determined according to the label, the depth information and the probability information of the training image, and the loss value of any pixel in the training image is the similarity between the first probability distribution function and the second probability distribution function of the pixel, the first probability distribution function is used to indicate the probability distribution of the true depth of the pixel, and the second probability distribution function is used to indicate the probability distribution of the predicted depth of the pixel; When the loss value of the training image does not meet the convergence condition, the model parameters of the basic depth prediction model after the last iteration are adjusted according to the loss value of the training image, and the iterative training is continued.

7. The method according to claim 6, characterized in that The training image includes a first area and a second area, the first area is different from the second area, the first area is the background hollow area in the training image, the loss value of the training image is obtained by weighted summing the loss value of the first area and the loss value of the second area, the weight coefficient corresponding to the first area is greater than the weight coefficient corresponding to the second area, the loss value of the first area is determined according to the loss value of each pixel point in the first area of ​​the training image, the loss value of the second area is determined according to the loss value of each pixel point in the second area of ​​the training image, and the loss value of any pixel point in the training image is determined according to the true depth, predicted depth and confidence of the predicted depth of the pixel point.

8. The method according to claim 6, characterized in that The first probability distribution function is determined according to the true depth of the pixel point, and the second probability distribution function is determined according to the predicted depth of the pixel point and the confidence of the predicted depth.

9. The method according to claim 8, characterized in that The first probability distribution function is a Dirkat distribution function constructed based on the true depth of the pixel point, and the second probability distribution function is a Gaussian distribution function constructed with the predicted depth of the pixel point as the mean and the confidence of the predicted depth of the pixel point as the variance.

10. The method according to any one of claims 6, 8-9, characterized in that: The similarity between the first probability distribution function and the second probability distribution function is a KL divergence between the first probability distribution function and the second probability distribution function.

11. An electronic device, characterized in that: The electronic device includes: a memory and one or more processors; the memory and the processor are coupled; the memory is used to store computer program code, and the computer program code includes computer instructions. When the computer instructions are executed by the electronic device, the electronic device executes the method as described in any one of claims 1-10.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are executed in an electronic device, the electronic device executes the method according to any one of claims 1 to 10.

Images