Image blurring method, depth prediction model training method and electronic equipment

Abstract

The invention provides an image blurring method, a training method of a depth prediction model and electronic equipment, which can achieve the effects of reducing a picture flicker phenomenon in a video blurring process and improving user experience by improving the stability of depth information. The method comprises the following steps: acquiring a first image, and inputting the first image into a depth prediction model to obtain depth information and probability information; wherein the depth information is used for indicating the predicted depth of each pixel point in the first image, and the probability information is used for indicating the confidence of the predicted depth of each pixel point; if the confidence degree of the depth information is greater than or equal to a first threshold, displaying a second image; wherein the second image is an image obtained by performing blurring processing on the first image based on the depth information of the first image, and the confidence coefficient of the depth information is determined according to the probability information.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image blurring method, a depth prediction model training method, and an electronic device. Background Art

[0002] With the continuous development of technology, most electronic devices such as mobile phones and tablets have a blur function. This function can make the focused subject area clear and the background area blurred when taking photos or videos, achieving a blur effect that highlights the subject and weakens the background. Electronic devices can distinguish the foreground and background of an image based on the depth information of the image, then keep the foreground clear and blur the background to achieve the ultimate blur effect.

[0003] Currently, a common method for determining depth information is to train a neural network model using training images and their labels, ultimately obtaining a converged neural network model, which is then used to predict the image's depth information. However, due to special scenarios such as occlusion and blur, as well as errors inherent in human annotation, the labels used in training often contain errors or ambiguities. This can lead to instability in the neural network model during the learning process, resulting in the converged neural network model outputting unstable and inaccurate depth information. This, in turn, can cause abnormal flickering in the image when using the blur function during video shooting, creating a poor user experience. Summary of the Invention

[0004] The embodiments of the present application provide an image blurring method, a depth prediction model training method, and an electronic device, which can reduce screen flickering during video blurring by improving the stability of depth information and enhance user experience.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides an image blurring method, which can first obtain a first image, which can be an image collected in real time by an electronic device; then input the first image into a depth prediction model to obtain depth information of the first image and probability information of the first image, wherein the depth information of the first image is used to indicate the predicted depth of each pixel in the first image, and the probability information of the first image is used to indicate the confidence of the predicted depth of each pixel in the first image; in addition, the method can also compare the confidence of the depth information with a first threshold, and the confidence of the depth information can be determined based on the probability information of the first image, such as the mean, variance, standard deviation, etc. of the confidence of the predicted depth of each pixel; wherein when the confidence of the depth information is greater than or equal to the first threshold, it indicates that the depth information obtained this time is more credible, and therefore the first image is blurred based on the depth information of the first image to obtain a second image and the second image is displayed; when the confidence of the depth information is less than the first threshold, it indicates that the depth information obtained this time is less credible, and therefore the second image is not displayed.

[0007] In this way, the device only displays the blurring effect of the image when the confidence level of the depth information of the acquired image is greater than or equal to the first threshold value, which can reduce the impact of some inaccurate depth information on the image blurring effect, thereby reducing the screen flickering phenomenon during the video blurring process and improving the user experience.

[0008] In an implementation provided in the first aspect, a depth prediction model is obtained by training a basic depth prediction model based on a training sample, wherein the training process includes: iteratively training the basic depth prediction model based on the training sample to generate a depth prediction model, the training sample including a training image and a label, the label being used to indicate the true depth of each pixel in the training image. During any iterative training process, the device can input the training image into the basic depth prediction model after the previous iteration to obtain the depth information and probability information of the training image; when the loss value of the training image meets the convergence condition (for example, it is less than a preset value), the iterative training is stopped and the depth prediction model is output, which is the basic depth prediction model after the previous iteration; when the loss value of the training image does not meet the convergence condition (for example, it is greater than or equal to a preset value), the model parameters of the basic depth prediction model after the previous iteration are adjusted according to the loss value of the training image and the iterative training is continued, and the loss value of the training image is determined according to the label, the depth information and probability information of the training image.

[0009] That is to say, in the process of iterative training of the basic depth prediction model, the present application determines the loss value from the two dimensions of depth and depth confidence and adjusts the model parameters based on the loss value, which can make the predicted depth closer and closer to the true depth, and the confidence of the predicted depth closer and closer to the confidence of the true depth. It can reduce the impact of labels (that is, inaccurate true depth) in special scenarios such as occlusion and blur on the training process, making the training process more stable, and thus making the depth prediction model finally trained have higher stability, and the depth information finally output by the depth prediction model is also more stable.

[0010] In an implementation method provided in the first aspect, the loss value of the training image is determined based on the loss value of each pixel in the training image, and the loss value of any pixel in the training image is determined based on the true depth, predicted depth and confidence of the predicted depth of the pixel.

[0011] In an implementation provided in the first aspect, the training image includes a first area and a second area, the first area being different from the second area, the first area being a background hollow area in the training image, and the background hollow area may refer to a background area within a foreground object in the image or a background area between two foreground objects, etc. The loss value of the training image is obtained by weighted summing the loss value of the first area and the loss value of the second area, the weight coefficient corresponding to the first area being greater than the weight coefficient corresponding to the second area, the loss value of the first area being determined based on the loss value of each pixel in the first area of ​​the training image, the loss value of the second area being determined based on the loss value of each pixel in the second area of ​​the training image, and the loss value of any pixel in the training image being determined based on the true depth, predicted depth, and confidence level of the predicted depth of the pixel.

[0012] Understandably, the weight coefficient corresponding to the first region is greater than the weight coefficient corresponding to the second region, which means that the loss value of the first region has a greater impact on the loss value of the training image. This allows the basic depth prediction model to be trained and learned for the background hollowed-out region, thereby improving the accuracy of the basic depth prediction model's predicted depth for the background hollowed-out region during the iterative training process, thereby reducing the phenomenon of missing depth.

[0013] In an implementation method provided in the first aspect, the loss value of any pixel point in the training image is the similarity between the first probability distribution function and the second probability distribution function of the pixel point, the first probability distribution function is used to indicate the probability distribution of the true depth of the pixel point, and the second probability distribution function is used to indicate the probability distribution of the predicted depth of the pixel point.

[0014] In an implementation manner provided by the first aspect, the first probability distribution function is determined according to the true depth of the pixel point, and the second probability distribution function is determined according to the predicted depth of the pixel point and the confidence of the predicted depth.

[0015] In an implementation provided by the first aspect, the first probability distribution function is a Dirkat distribution function constructed based on the true depth of the pixel point, and the second probability distribution function is a Gaussian distribution function constructed with the predicted depth of the pixel point as the mean and the confidence level of the predicted depth of the pixel point as the variance. The expression of the first probability distribution function is: P D (x) = δ(xx g ), x g is the true depth of the pixel. It can be understood that the first probability distribution function indicates that the true depth of the pixel is x g The probability is infinite, which can also be understood as the true depth of the pixel is x g The probability of is infinitely close to 1. The expression of the second probability distribution function is: x e is the predicted depth of the pixel, σ is the predicted depth x e confidence level.

[0016] In one implementation provided in the first aspect, the similarity between the first probability distribution function and the second probability distribution function is a KL divergence between the first probability distribution function and the second probability distribution function. The KL divergence, the first probability distribution function, and the second probability distribution function satisfy Formula 1:

[0017]

[0018] Among them, D KL (P D (x)||P θ (x)) is the KL divergence, P D (x) is the first probability distribution function, P θ (x) is the second probability distribution function.

[0019] It can be understood that the smaller the KL divergence of the first probability distribution function and the second probability distribution function, the higher the similarity between the first probability distribution function and the second probability distribution function, that is, the predicted depth is closer to the true depth, and the confidence of the predicted depth is closer to the confidence of the true depth.

[0020] In the second aspect, an embodiment of the present application provides a method for training a depth prediction model, the method comprising: iteratively training a basic depth prediction model based on training samples to generate a depth prediction model; wherein the training samples include labels and training images, and the labels are used to indicate the true depth of each pixel in the training image; during any iterative training process, the training image is input into the basic depth prediction model after the previous iteration, and the depth information and probability information of the training image are output; when the loss value of the training image meets the convergence condition, the iterative training is stopped; wherein the loss value of the training image is determined based on the label, the depth information and probability information of the training image; when the loss value of the training image does not meet the convergence condition, the model parameters of the basic depth prediction model after the previous iteration are adjusted according to the loss value of the training image, and the iterative training is continued. The depth prediction model trained by the embodiment of the present application can be applied to the image blurring method provided in the first aspect, and can also be applied to autonomous driving scenarios, robot navigation scenarios, etc., without specific limitation here.

[0021] In an implementation method provided in the second aspect, the loss value of the training image is determined based on the loss value of each pixel in the training image, and the loss value of any pixel in the training image is determined based on the true depth, predicted depth and confidence of the predicted depth of the pixel.

[0022] In an implementation method provided in the second aspect, the training image includes a first area and a second area, the first area is different from the second area, the first area is a background hollow area in the training image, the loss value of the training image is obtained by weighted summing the loss value of the first area and the loss value of the second area, the weight coefficient corresponding to the first area is greater than the weight coefficient corresponding to the second area, the loss value of the first area is determined according to the loss value of each pixel point in the first area of ​​the training image, the loss value of the second area is determined according to the loss value of each pixel point in the second area of ​​the training image, and the loss value of any pixel point in the training image is determined according to the true depth, predicted depth and confidence of the predicted depth of the pixel point.

[0023] In an implementation method provided in the second aspect, the loss value of any pixel point in the training image is the similarity between the first probability distribution function and the second probability distribution function of the pixel point, the first probability distribution function is used to indicate the probability distribution of the true depth of the pixel point, and the second probability distribution function is used to indicate the probability distribution of the predicted depth of the pixel point.

[0024] In an implementation manner provided by the second aspect, the first probability distribution function is determined based on the true depth of the pixel point, and the second probability distribution function is determined based on the predicted depth of the pixel point and the confidence of the predicted depth.

[0025] In an implementation method provided in the second aspect, the first probability distribution function is a Dirkat distribution function constructed based on the true depth of the pixel point, and the second probability distribution function is a Gaussian distribution function constructed with the predicted depth of the pixel point as the mean and the confidence of the predicted depth of the pixel point as the variance.

[0026] In an implementation manner provided in the second aspect, the similarity between the first probability distribution function and the second probability distribution function is a KL divergence between the first probability distribution function and the second probability distribution function.

[0027] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: a memory and one or more processors; the memory and the processor are coupled; the memory is used to store computer program code, the computer program code comprising computer instructions, and when the computer instructions are executed by the electronic device, the electronic device executes the method of the first aspect, the second aspect and any one of their implementation methods.

[0028] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed in an electronic device, the electronic device executes the method of the first aspect, the second aspect and any one of their implementation methods.

[0029] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method provided in the first aspect, the second aspect, and any one of their implementations.

[0030] Among them, the technical effects brought about by any design method in the second to fifth aspects can refer to the technical effects brought about by different implementation methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of a normal photo and a blurred photo effect provided in an embodiment of the present application;

[0032] Figure 2 A schematic diagram of a blur effect provided in an embodiment of the present application;

[0033] Figure 3 A schematic diagram of another blurring effect provided in an embodiment of the present application;

[0034] Figure 4 A schematic diagram of the structure of an AI system provided in an embodiment of the present application;

[0035] Figure 5 A flowchart of a method for training a depth prediction model provided in an embodiment of the present application;

[0036] Figure 6 A schematic diagram of the network structure of a basic depth prediction model provided in an embodiment of the present application;

[0037] Figure 7 A schematic diagram of a process for determining the loss value of a training image using a basic depth prediction model provided in an embodiment of the present application;

[0038] Figure 8 A graph of a first probability distribution function and a second probability distribution function provided in an embodiment of the present application;

[0039] Figure 9 A terminal provided in an embodiment of the present application provides a depth image output by a basic depth prediction model before and after optimization training for a background hollowed-out area;

[0040] Figure 10 A flowchart of an image blurring method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. Among them, in the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and claims of the present application, the singular expressions "a", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0042] References to "one embodiment" or "some embodiments" etc. described in this specification mean that the specific features, structures or characteristics described in conjunction with the embodiment are included in one or more embodiments of the present application. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in another way. The term "connected" includes direct and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0043] In the embodiments of this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0044] The image blurring method provided in the embodiments of the present application can be applied to scenarios requiring environmental understanding, such as photography blurring, autonomous driving, and robot navigation. In blurring scenarios, depth can help electronic devices better distinguish between foreground and background, achieving a good blurring effect. These blurring scenarios include both photo blurring and video blurring. In autonomous driving and robot navigation, depth can help vehicles and robots better understand their surroundings, enabling path planning and obstacle avoidance.

[0045] Figure 1 The following diagram shows the effects of normal photography and blurring. Figure 1 As shown in (a) of FIG, the mobile phone can display an interface 101, which includes multiple shooting mode options, such as "photo", "portrait", "video", etc., where "photo" is selected. In addition, the interface 101 also includes an image 102. In response to the user's operation on "portrait", as shown in FIG. Figure 1 As shown in (b), the mobile phone can display an interface 103, which also includes shooting mode options such as "photo", "portrait", and "video", and "portrait" is selected. In addition, the interface 103 also includes an image 104, which includes a foreground 104a and a background 104b. Figure 1As can be seen from (a) and (b), the foreground 104a of image 102 and image 104 are both clear, and the background 104b of image 104 is blurred, that is, image 102 is a schematic diagram of the effect of ordinary photography, and image 104 is a schematic diagram of the effect of blurred photography.

[0046] In the photo blurring scenario, the mobile phone can calculate the depth of field by combining the focus position of image 104 and the depth information of image 104, and distinguish the foreground 104a and background 104b of image 104 based on the depth of field. Then, the area where the foreground 104a is located and the area where the background 104b is located are blurred to varying degrees (for example, maintaining the clarity of the area where the foreground 104a is located and reducing the clarity of the area where the background 104b is located) to achieve a blurring effect in which the clarity of the foreground 104a and the background 104b are inconsistent. It should be noted that the principles of video blurring are similar to those of photo blurring and will not be further described here.

[0047] When shooting blurred scenes, abnormal phenomena such as abnormal flickering or blurring often occur due to unstable depth calculation process or inaccurate calculation results. Among them, abnormal flickering refers to the phenomenon that in the blurred scene of the video, the depth information between consecutive frames is greatly different, resulting in a large difference in the blur effect between consecutive frames. For example, Figure 2 (a) and (b) in FIG. 2 show schematic diagrams of blurring effects of image 201 and image 202, respectively. Image 201 and image 202 are two consecutive frames of images in a video being shot by a mobile phone. Figure 2 As shown, the foreground of image 201 is clear and the background is blurred, while the foreground and background of image 202 are both clear. That is, the blurring effects of image 201 and image 202 are quite different, which may give the user a feeling of flickering.

[0048] Blur leakage refers to the phenomenon that in a video blur scene or a photo blur scene, some areas that should be blurred are not blurred due to inaccurate depth information. Blur leakage phenomenon occurs more frequently in images with hollow areas. For example, Figure 3 (a) in FIG. 3 shows a schematic diagram of the effect of normal blurring of image 301. Figure 3 (b) in FIG. 3 shows a schematic diagram of the effect of image 302 being blurred, and image 301 and image 302 are the same image. Figure 3 As can be seen from (a) and (b), image 301 includes area 301a, and image 302 includes area 302a. Area 301a and area 302a are the same hollow area. Area 301a of image 301 is blurred normally, and area 302a of image 302 is not blurred.

[0049] It can be seen that the blur effect presented by the mobile phone in the blurred scene is largely dependent on the stability of the depth information calculation process and the accuracy of the depth information. It should be noted that the above content is only used as an example to illustrate the blurred scene. The same principle applies to scenarios such as autonomous driving and robot navigation. In other words, the accuracy of path planning and obstacle avoidance is largely dependent on the stability of the depth information calculation process and the accuracy of the depth information.

[0050] like Figure 4 As shown, Figure 4 This is a schematic diagram of the structure of the AI ​​system provided in this application. Figure 4 As shown, the AI ​​system includes a data center and multiple terminals (such as Figure 4 The data center can communicate with the terminals via a network, which can be the Internet or other networks. The network can include one or more network devices, such as routers or switches.

[0051] A data center consists of one or more servers, such as Figure 4 The server 120 shown is, for example, an application server that supports application services, which can provide image services, video services, game services, other AI processing services based on video or images, etc. In an optional scenario, the server 120 refers to a server cluster deployed with multiple servers. The server cluster can have a rack that can establish communication for the multiple servers via a wired connection, such as a universal serial bus (USB) or a peripheral component interconnect express (PCIe) high-speed bus.

[0052] Server 120 can also obtain data from the terminal, perform AI processing on the data, and send the results of the AI ​​processing to the corresponding terminal. This AI processing can refer to using AI models to perform tasks such as depth prediction, object recognition, and target detection on the data, or it can refer to obtaining an AI model that meets the requirements based on samples collected by the terminal.

[0053] in addition, Figure 4 The data center shown may also include other physical devices with AI processing capabilities, such as mobile phones, tablets, or other devices.

[0054] Terminal can also be called terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The terminal can be a mobile phone (such as Figure 4The terminal 111 shown), a face-scanning payment device with a mobile payment function, a camera device with a data (such as image or video) acquisition and processing function (such as Figure 4 The terminal 112 shown in the figure may also be a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, a vehicle with data (such as image or video) acquisition and processing functions, etc. The embodiments of the present application do not limit the specific technology and specific device form adopted by the terminal device.

[0055] It is worth noting that the terminal can obtain the AI ​​model stored in the server 120, and then use the AI ​​model to perform various tasks on the information contained in the image. For example, the AI ​​model can be a model for performing operations such as depth prediction on the data.

[0056] For example, the AI ​​model deployed in terminal 112 can be used to implement functions such as depth prediction.

[0057] For example, the AI ​​model deployed in terminal 111 can realize functions such as automatically adjusting the local clarity of the image.

[0058] For example, the AI ​​model deployed in the terminal 111 implements functions such as face payment and object classification (such as product classification).

[0059] It should be noted that Figure 4 This is just a schematic diagram and should not be construed as limiting the present application. The embodiments of the present application do not limit the application scenarios of the terminal and the server.

[0060] In related technologies, it is possible to use Figure 4 The AI ​​system shown trains a basic depth prediction model to obtain a depth prediction model. The server 120 can input labeled training images into the basic depth prediction model to obtain depth information. The labels can be used to reflect the true depth of each pixel in the training image, and the depth information can be used to reflect the predicted depth of each pixel in the training image. The server 120 can then calculate a loss function based on the labels and the depth image, and adjust the parameters of the basic depth prediction model based on the loss function to ultimately obtain a depth prediction model.

[0061] In the above scheme, the loss function, depth image, and label satisfy the following formula:

[0062]

[0063]

[0064] Among them, D(y, y*) is the loss function, y is the depth image, y* is the label, y i is the true depth of the i-th pixel in the training image, is the predicted depth of the i-th pixel in the training image, and n is the number of pixels in the training image.

[0065] After the server 120 has trained the depth prediction model, it can port the depth prediction model to the terminal. The terminal can then input the image into the depth prediction model to obtain the image's depth information, and then perform operations such as blurring based on the depth information, and display it on the display screen. In other embodiments, the above training process can also be performed by the terminal.

[0066] However, due to special scenarios such as occlusion and blur, as well as errors in manual labeling, the labels used in the above training process often contain errors or ambiguities, which will cause errors in the loss function, resulting in errors and instability in the training process. The depth information output by the trained depth prediction model is not stable and accurate enough, which in turn leads to abnormal phenomena such as abnormal flickering or missing blur when shooting blurred scenes, or causes the navigation routes / obstacle avoidance routes planned in scenarios such as autonomous driving / robot navigation to be not accurate enough, giving users a bad experience.

[0067] To at least solve the above problems, the embodiments of the present application provide a method for training a depth prediction model and an image blurring method, which can improve the stability of the depth prediction process and the accuracy of the depth prediction results, and further reduce abnormal phenomena such as screen flickering and blurring when the method is applied to shooting blurred scenes. It should be noted that the method for training a depth prediction model and the image blurring method provided in the embodiments of the present application are inventions based on the same concept, and can also be understood as two parts in a system, or two stages of an overall process: such as the model training stage and the model application stage.

[0068] In the model training stage, an embodiment of the present application provides a training method for a depth prediction model, which uses depth information and probability information as the output of a basic depth prediction model, constructs a probability distribution of the predicted depth of each pixel using the depth information and probability information, and constructs a probability distribution of the true depth of each pixel using a label, and then uses the similarity between the probability distribution of the predicted depth and the probability distribution of the true depth as a loss value and adjusts the parameters of the basic depth prediction model based on the loss value to obtain a depth prediction model. Among them, using the similarity between the probability distribution of the predicted depth and the probability distribution of the true depth as a loss value can not only consider the error between the true value and the predicted value of the depth in the training stage, but also consider the error between the confidence of the true value and the confidence of the predicted value, that is, to train the basic depth prediction model from two dimensions of depth and depth confidence. This training process is more stable, so that the stability of the depth prediction model obtained by training is also higher, which can reduce the phenomenon of screen flickering in video blur scenes.

[0069] In the model application stage, an embodiment of the present application provides an image blurring method, which can input the acquired first image into the trained depth prediction model to obtain the depth information and probability information of the first image, and display the second image when the confidence level of the depth information is greater than or equal to a first threshold. The second image is an image after blurring the first image based on the depth information of the first image, and the confidence level of the depth information is determined according to the probability information.

[0070] The following first describes the above-mentioned model training stage with reference to the accompanying drawings.

[0071] Figure 5 A flow chart of a method for training a depth prediction model provided by an embodiment of the present application is shown. Figure 4 The server 120 or terminal in the embodiment can also be executed by other devices with computing capabilities. The embodiment of the present application takes the terminal execution as an example. Figure 5 The training method of the depth prediction model provided in the embodiment of the present application includes at least steps S510 to S540.

[0072] S510: The terminal iteratively trains the basic depth prediction model based on the training samples.

[0073] The training samples include training images with labels, where the labels are used to indicate the true depth of each pixel in the training image (which can be understood as the true value of the depth).

[0074] In the embodiment of the present application, the training image is input into the basic depth prediction model to obtain depth information and probability information. The depth information is used to indicate the predicted depth of each pixel in the training image (which can be understood as the predicted value of the depth), and the probability information is used to indicate the confidence level of the predicted depth of each pixel in the training image.

[0075] For example, the depth information indicates that the predicted depth of the i-th pixel in the training image is 0.5, and the probability information indicates that the confidence of the predicted depth of the i-th pixel in the training image is 0.6, then the probability that the predicted depth of the i-th pixel in the training image is 0.5 is 0.6.

[0076] Figure 6 A schematic diagram of the network structure of a basic depth prediction model provided by an embodiment of the present application is shown. Figure 6 The basic depth prediction model includes m feature extraction modules, m upsampling modules, and a prediction module. The process of outputting depth information and probability information by the basic depth prediction model is as follows: input the input image into feature extraction module 1 to obtain downsampling result 1, input downsampling result 1 into upsampling module 1 to obtain upsampling result 1; then, the feature map obtained by fusing upsampling result 1 with the input image is input into feature extraction module 2 to obtain downsampling result 2, and input downsampling result 2 into upsampling module 2 to obtain upsampling result 2; then, the feature map obtained by fusing upsampling result 2 with the input image is input into feature extraction module 3 to obtain downsampling result 3; and so on, downsampling result m-1 is input into upsampling module m-1 to obtain upsampling result m-1, the feature map obtained by fusing upsampling result m-1 with the input image is input into feature extraction module m to obtain downsampling result m, and downsampling result m is input into upsampling module m to obtain upsampling result m. ​​Finally, the feature map obtained by fusing upsampling result m with the input image is input into the prediction module to obtain depth information and probability information.

[0077] It should be noted that before fusing the upsampled result with the input image, convolution or pooling can be performed on the input image to make the scale of the input image the same as the size of the upsampled result to be fused. For example, if the size of the input image is 3×384×384 and the size of the upsampled result 2 is 128×48×48, then before fusing the upsampled result 2 with the input image, convolution or pooling can be performed on the input image to make the size of the input image 3×384×384, and then the upsampled result 2 and the input image after convolution or pooling can be fused.

[0078] Still like Figure 6As shown, each feature extraction module, prediction module, and upsampling module may include a different number of convolutional layers, pooling layers, and / or fully connected layers, etc. There is no restriction on the specific structure of the feature extraction module, prediction module, and upsampling module, as long as the corresponding functions can be achieved.

[0079] In an embodiment of the present application, the size of the feature maps output by each feature extraction network is different, which can extract richer detail features from the input image, which is conducive to improving the accuracy of the model output (i.e., depth information).

[0080] S520: The terminal determines whether the basic depth prediction model has converged.

[0081] If the basic depth prediction model does not converge, execute S530 ; if the basic depth prediction model converges, execute S540 .

[0082] In one possible design, the terminal can compare the loss value of the training image with a preset value to determine whether the basic depth prediction model has converged. If the loss value of the training image is less than or equal to the preset value, it indicates that the basic depth prediction model has converged; if the loss value of the training image is greater than the preset value, it indicates that the basic depth prediction model has not converged.

[0083] In an embodiment of the present application, the loss value of a training image can be determined based on the label, depth information, and probability information corresponding to the training image. Figure 7 The figure shows a flow chart of determining the loss value of a training image using the basic depth prediction model provided in the embodiment of the present application. Figure 7 , the basic depth prediction model can predict the training image to obtain depth information and probability information, and then the basic depth prediction model can construct a probability distribution function of the predicted depth based on the depth information and probability information, and construct a probability distribution function of the true depth based on the label, and finally calculate the loss value of the training image based on the probability distribution function of the predicted depth and the probability distribution function of the true depth.

[0084] Understandably, in Figure 7 In the flowchart shown, the loss value of the training image is calculated and output by the basic depth prediction model. In other embodiments, the loss value of the training image can also be calculated by the terminal using other algorithm modules, which is not specifically limited here.

[0085] In an embodiment of the present application, the terminal may first determine the loss value of each pixel in the training image, and then perform a weighted summation of the loss values ​​of all pixels to obtain the loss value of the training image. For any pixel in the training image, the terminal may determine the loss value of the pixel based on the true depth, predicted depth, and confidence level of the predicted depth. Figure 7The process shown is similar. For any pixel in the training image, the terminal can construct a probability distribution function of the predicted depth based on the predicted depth of the pixel and the confidence of the predicted depth, and construct a probability distribution function of the actual depth based on the actual depth of the pixel. Then, the loss value of the pixel is calculated based on the probability distribution function of the predicted depth and the probability distribution function of the actual depth.

[0086] In one possible design, for any pixel in the training image, the terminal can construct a Diktala distribution function (i.e., the probability distribution function of the true depth, also referred to as the first probability distribution function) based on the true depth of the pixel, and construct a Gaussian distribution function (i.e., the probability distribution function of the predicted depth, also referred to as the second probability distribution function) with the predicted depth of the pixel as the mean and the confidence of the predicted depth of the pixel as the variance, and then calculate the similarity between the first probability distribution function and the second probability distribution function as the loss value of the pixel.

[0087] Figure 8 The graph of the first probability distribution function and the second probability distribution function provided by the embodiment of the present application is shown. The expression of the first probability distribution function is: P D (x) = δ(xx g ), x g is the true depth of the pixel. It can be understood that the first probability distribution function indicates that the true depth of the pixel is x g The probability is infinite, which can also be understood as the true depth of the pixel is x g The probability is infinitely close to 1.

[0088] The expression of the second probability distribution function is: x e is the predicted depth of the pixel, σ is the predicted depth x e It can be understood that the second probability distribution function indicates that the predicted depth of the pixel point is Gaussian distributed, and the mean of the distribution is x e , with variance σ.

[0089] In the embodiment of the present application, the KL divergence (Kullback-Leibler divergence) can be used to measure the similarity (or difference) between the first probability distribution function and the second probability distribution function. The KL divergence can also be called relative entropy or information divergence. Wherein, the KL divergence, the first probability distribution function and the second probability distribution function satisfy Formula 1:

[0090]

[0091] Among them, D KL (P D (x)||Pθ (x)) is the KL divergence, P D (x) is the first probability distribution function, P θ (x) is the second probability distribution function.

[0092] Understandably, D KL (P D (x)||P θ The smaller (x)), the higher the similarity between the first probability distribution function and the second probability distribution function, that is, the closer the predicted depth is to the true depth, and the closer the confidence of the predicted depth is to the confidence of the true depth.

[0093] In this way, the loss value of the training image satisfies Equation 2:

[0094]

[0095] in, is the loss value of the training image, N is the number of pixels in the training image, D KLi (P D (xi)||P θ (xi)) is the KL divergence (i.e., loss value) of the i-th pixel in the training image, P D (xi) is the first probability distribution function of the i-th pixel in the training image, P θ (xi) is the second probability distribution function of the i-th pixel in the training image.

[0096] It should be noted that the above formula 2 is a formula used when the weight of each pixel is the same. When the weight of each pixel is different, the formula 2 can be adaptively adjusted according to the weight.

[0097] S530: The terminal updates the model parameters according to the loss value of the training image during the iteration process.

[0098] S540: The terminal outputs a depth prediction model.

[0099] It can be understood that the depth prediction model is a converged basic depth prediction model.

[0100] Among them, outputting the depth prediction model can be understood as the terminal storing the depth prediction model in a memory, or sending it to other devices, such as servers or other terminals, and no specific restrictions are made here.

[0101] It can be understood that, compared with the scheme for predicting depth information provided by related technologies, the embodiment of the present application updates the model parameters based on the loss value of the training image during the iteration process, and the loss value is calculated based on the probability distribution function of the predicted depth and the probability distribution function of the true depth, which can make the predicted depth output by the basic depth prediction model closer and closer to the true depth, and make the confidence of the predicted depth output by the basic depth prediction model closer and closer to the confidence of the true depth (close to 1), that is, the possible errors in the label and depth information are taken into account during the training process, which can reduce the impact of special scenes such as occlusion and blur on the training process, making the training process more stable, and thus making the depth prediction model finally trained have higher stability, and the depth information finally output by the depth prediction model is also more stable.

[0102] In one possible implementation, the training image includes a background hollowed-out area and a non-background hollowed-out area. The background hollowed-out area may refer to a background area within a foreground object or a background area between two foreground objects in the image, and the non-background hollowed-out area refers to areas other than the background hollowed-out area.

[0103] For example, the training image can be Figure 3 In the image 301 shown, the background hollowed-out area may be the area 301 a in the image 301 , and the non-background hollowed-out area may be all areas in the image 301 except the area 301 a .

[0104] For example, the training image can be Figure 9 The original image shown may include a background hollow area (a shadow area in the original image) and a non-background hollow area (an area other than the shadow area in the original image).

[0105] In addition, the label is also used to indicate the distribution of background hollow areas and non-background hollow areas. For example, the label includes the location information of the boundary of the background hollow area. The terminal can determine whether the pixel is located in the background hollow area or non-background hollow area by determining whether the pixel is within the boundary of the background hollow area. For another example, the label includes an attribute of each pixel in the training image, which is used to indicate whether the pixel is located in the background hollow area or non-background hollow area.

[0106] In this case, the terminal can calculate the loss value of the background hollow area and the loss value of the non-background hollow area respectively, and perform weighted summation of the loss value of the background hollow area and the loss value of the non-background hollow area according to the preset weight coefficient to obtain the loss value of the training image.

[0107] Among them, the loss value of the training image, the loss value of the background hollow area, and the loss value of the non-background hollow area can satisfy Formula 3:

[0108]

[0109] in, is the loss value of the training image, D KL_S1 is the loss value of the background hollow area, D KL_S2 is the loss value of the non-background hollow area, a1 and a2 are the preset weight coefficients, and a1>a2.

[0110] It should be noted that the loss value of the background hollow area and the loss value of the non-background hollow area can be calculated according to the method provided in S520, which will not be repeated here.

[0111] In one possible design, the non-background hollow area can be further divided into the main body area and the background non-hollow area. The label is used to indicate the distribution of the background hollow area, the main body area, and the background non-hollow area. Accordingly, the background hollow area, the main body area, and the background non-hollow area have different weights, and the weight coefficient of the background hollow area is greater than the weight coefficient of the main body area, and the weight coefficient of the background hollow area is greater than the weight coefficient of the background non-hollow area.

[0112] In one possible design, the above-mentioned depth information can be presented in the form of an image, namely a depth image. The terminal can mark the predicted depth of each pixel in the training image and use the marked training image as the depth information. For example, the training image can be Figure 9 The original image in the image, the depth information can be Figure 9 The depth image in .

[0113] It can be understood that by marking the predicted depth of each pixel in the image, the marked image can display the depth values ​​of different pixels in the image, so that users can quickly view the depth values ​​of different pixels from the depth image displayed by the terminal, thereby improving the quality of experience (QoE) of the user experience.

[0114] Figure 9 The depth images output by the terminal's basic depth prediction model before and after optimization training for the background hollowed-out area are shown. It can be seen that the depth image output by the optimized and trained basic depth prediction model has clearer and more distinct boundaries in the background hollowed-out area than the depth image output by the unoptimized basic depth prediction model.

[0115] That is to say, the loss value of the training image is obtained by weighted calculation of the loss values ​​of different areas according to the preset weight coefficient, and the weight coefficient of the background hollow area is larger than the weight coefficient of the non-background hollow area. In this way, the basic depth prediction model can be trained and learned for the background hollow area, thereby improving the accuracy of the basic depth prediction model's predicted depth for the background hollow area, and thus reducing the false leakage phenomenon.

[0116] The following continues to describe the model application stage of this application.

[0117] Figure 10 The flowchart of an image blurring method provided by an embodiment of the present application is shown. Figure 4 The server 120 or terminal in the embodiment can also be executed by other devices with computing capabilities. The embodiment of the present application takes the terminal execution as an example. Figure 10 The image blurring method provided in the embodiment of the present application includes at least steps S1001 to S1005.

[0118] S1001, acquiring a first image.

[0119] Among them, the first image can be an image captured in real time by the terminal device, any frame image in the video captured in real time by the terminal device, an image sent to the terminal device by other devices, any frame image in the video sent to the terminal device by other devices, or an image pre-stored in the terminal device, etc., and no specific restrictions are made here.

[0120] S1002: Input the first image into a depth prediction model to obtain depth information and probability information.

[0121] It should be noted that the depth prediction model provided in the embodiment of the present application is a convergent basic depth prediction model. Therefore, the network structure of the depth prediction model is the same as the network structure of the aforementioned basic depth prediction model, which will not be repeated here.

[0122] It can be understood that the depth prediction model used in the image blurring method provided in the embodiment of the present application is based on Figure 5 The depth prediction model training method provided is obtained. This training process is more stable, which makes the depth prediction model finally trained have higher stability, so that the depth information finally output by the depth prediction model is also more stable, that is, the depth information of continuous video frames in the video blur scene is also more stable, which can reduce the screen flickering caused by unstable depth information.

[0123] S1003: Determine whether the confidence level of the depth information is greater than or equal to a first threshold.

[0124] The confidence level of the depth information is related to the probability information. For example, the confidence level of the depth information may be the variance, standard deviation, mean, etc. of the confidence levels of the predicted depths of all pixels in the probability information.

[0125] S1004: If the confidence level of the depth information is greater than or equal to the first threshold, display the second image.

[0126] The second image is an image obtained by blurring the first image according to the depth information.

[0127] S1005: If the confidence level of the depth information is less than a first threshold, display the first image.

[0128] For example, Figure 1 or Figure 2 Taking the blurred scene shown in the figure as an example, the terminal can display a first interface including multiple shooting options. Figure 1 In the interface 101, multiple shooting options are, for example, "portrait", "photo", "video", etc. In response to the user's operation on the first shooting option, the terminal can input the first image into the depth prediction model to obtain depth information and probability information, and display the second image when the confidence level of the depth information is greater than or equal to the first threshold, such as image 104 or image 202, or directly display the first image when the confidence level of the depth information is less than the first threshold. Among them, the first shooting option may include an option that triggers the terminal to blur the image (i.e., the first image) captured in real time by the camera, such as "portrait", "video", etc.

[0129] That is to say, the first image is blurred and displayed only when the confidence level of the depth information of the first image is higher than the first threshold. This can further increase the stability of the depth information and further reduce the screen flickering between consecutive video frames in the video blur scenario.

[0130] In other possible designs, the above-mentioned depth prediction model can also be applied to autonomous driving scenarios, robot navigation scenarios, etc., without specific restrictions here.

[0131] An embodiment of the present application further provides a chip system, which includes at least one processor and at least one interface circuit. The processor and the interface circuit can be interconnected via lines. For example, the interface circuit can be used to receive signals from other devices (such as a memory of an electronic device). For another example, the interface circuit can be used to send signals to other devices (such as a processor). Exemplarily, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the electronic device can perform the various steps in the above embodiments. Of course, the chip system can also include other discrete devices, which is not specifically limited in the embodiment of the present application.

[0132] An embodiment of the present application further provides a computer storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device, the electronic device executes the various functions or steps executed by the terminal or server in the above-mentioned method embodiment.

[0133] An embodiment of the present application further provides a computer program product, which, when executed on a computer, enables the computer to execute the functions or steps executed by the terminal or server in the above method embodiment.

[0134] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0135] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0136] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0137] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0138] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0139] The above content is only a specific embodiment of this application, but the scope of protection of this application is not limited to this. Any changes or replacements within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An image blurring method, characterized in that: The method comprises: acquiring a first image; Inputting the first image into a depth prediction model to obtain depth information and probability information; wherein the depth information is used to indicate the predicted depth of each pixel in the first image, and the probability information is used to indicate the confidence of the predicted depth of each pixel; If the confidence of the depth information is greater than or equal to a first threshold, a second image is displayed; wherein the second image is an image obtained by blurring the first image based on the depth information of the first image, and the confidence of the depth information is determined according to the probability information.

2. The method according to claim 1, characterized in that The depth prediction model is obtained by training a basic depth prediction model based on training samples, wherein the training process includes: Iteratively training the basic depth prediction model based on the training samples to generate the depth prediction model; wherein the training samples include labels and training images, and the labels are used to indicate the true depth of each pixel in the training image; In any iterative training process, the training image is input into the basic depth prediction model after the previous iteration, and the depth information and probability information of the training image are output; When the loss value of the training image meets the convergence condition, the iterative training is stopped; wherein the loss value of the training image is determined according to the label, the depth information and the probability information of the training image; When the loss value of the training image does not satisfy the convergence condition, the model parameters of the basic depth prediction model after the last iteration are adjusted according to the loss value of the training image, and the iterative training is continued.

3. The method according to claim 2, characterized in that The loss value of the training image is determined according to the loss value of each pixel in the training image, and the loss value of any pixel in the training image is determined according to the true depth, predicted depth and confidence of the predicted depth of the pixel.

4. The method according to claim 2, characterized in that: The training image includes a first area and a second area, the first area is different from the second area, the first area is a background hollow area in the training image, the loss value of the training image is obtained by weighted summing the loss value of the first area and the loss value of the second area, the weight coefficient corresponding to the first area is greater than the weight coefficient corresponding to the second area, the loss value of the first area is determined according to the loss value of each pixel in the first area of ​​the training image, the loss value of the second area is determined according to the loss value of each pixel in the second area of ​​the training image, and the loss value of any pixel in the training image is determined according to the true depth, predicted depth and confidence of the predicted depth of the pixel.

5. The method according to claim 3 or 4, characterized in that: The loss value of any pixel in the training image is the similarity between the first probability distribution function and the second probability distribution function of the pixel, the first probability distribution function is used to indicate the probability distribution of the true depth of the pixel, and the second probability distribution function is used to indicate the probability distribution of the predicted depth of the pixel.

6. The method according to claim 5, characterized in that The first probability distribution function is determined according to the true depth of the pixel point, and the second probability distribution function is determined according to the predicted depth of the pixel point and the confidence of the predicted depth.

7. The method according to claim 6, characterized in that The first probability distribution function is a Dirkat distribution function constructed according to the true depth of the pixel point, and the second probability distribution function is a Gaussian distribution function constructed with the predicted depth of the pixel point as the mean and the confidence of the predicted depth of the pixel point as the variance.

8. The method according to any one of claims 5 to 7, characterized in that: The similarity between the first probability distribution function and the second probability distribution function is the KL divergence between the first probability distribution function and the second probability distribution function.

9. A method for training a depth prediction model, characterized in that: The method comprises: Iteratively training a basic depth prediction model based on training samples to generate a depth prediction model; wherein the training samples include labels and training images, and the labels are used to indicate the true depth of each pixel in the training image; In any iterative training process, the training image is input into the basic depth prediction model after the previous iteration, and the depth information and probability information of the training image are output; When the loss value of the training image meets the convergence condition, the iterative training is stopped; wherein the loss value of the training image is determined according to the label, the depth information and the probability information of the training image; When the loss value of the training image does not satisfy the convergence condition, the model parameters of the basic depth prediction model after the last iteration are adjusted according to the loss value of the training image, and the iterative training is continued.

10. The method according to claim 9, characterized in that The loss value of the training image is determined according to the loss value of each pixel in the training image, and the loss value of any pixel in the training image is determined according to the true depth, predicted depth and confidence of the predicted depth of the pixel.

11. The method according to claim 9, characterized in that The training image includes a first area and a second area, the first area is different from the second area, the first area is a background hollow area in the training image, the loss value of the training image is obtained by weighted summing the loss value of the first area and the loss value of the second area, the weight coefficient corresponding to the first area is greater than the weight coefficient corresponding to the second area, the loss value of the first area is determined according to the loss value of each pixel in the first area of ​​the training image, the loss value of the second area is determined according to the loss value of each pixel in the second area of ​​the training image, and the loss value of any pixel in the training image is determined according to the true depth, predicted depth and confidence of the predicted depth of the pixel.

12. The method according to claim 10 or 11, characterized in that: The loss value of any pixel in the training image is the similarity between the first probability distribution function and the second probability distribution function of the pixel, the first probability distribution function is used to indicate the probability distribution of the true depth of the pixel, and the second probability distribution function is used to indicate the probability distribution of the predicted depth of the pixel.

13. The method according to claim 12, characterized in that The first probability distribution function is determined according to the true depth of the pixel point, and the second probability distribution function is determined according to the predicted depth of the pixel point and the confidence of the predicted depth.

14. The method according to claim 13, characterized in that The first probability distribution function is a Dirkat distribution function constructed according to the true depth of the pixel point, and the second probability distribution function is a Gaussian distribution function constructed with the predicted depth of the pixel point as the mean and the confidence of the predicted depth of the pixel point as the variance.

15. The method according to any one of claims 12 to 14, characterized in that: The similarity between the first probability distribution function and the second probability distribution function is the KL divergence between the first probability distribution function and the second probability distribution function.

16. An electronic device, characterized in that: The electronic device comprises: a memory and one or more processors; the memory and the processor are coupled; the memory is used to store computer program code, the computer program code comprises computer instructions, and when the computer instructions are executed by the electronic device, the electronic device executes the method as described in any one of claims 1-15.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed in an electronic device, the electronic device executes the method according to any one of claims 1 to 15.

Images