Image processing method, depth estimation model training method and electronic equipment

Abstract

The embodiment of the invention provides an image processing method, a depth estimation model training method and electronic equipment. The method comprises the following steps: acquiring a first image and a second image in a motion scene; the motion scene comprises a motion object; the moving object comprises a person in a moving state and / or other objects in the moving state except the person; inputting the collected first image and second image into a depth estimation model to obtain a first depth of the first image; input parameters of the depth estimation model comprise an image sequence, and the image sequence comprises a current frame and further comprises an image collected before the collection moment of the current frame and / or an image collected after the collection moment of the current frame; the depth estimation model is used for outputting the depth of the current frame based on the motion feature of the moving object in each frame of image in the image sequence; blurring the first image based on the first depth to obtain a blurred first image; the moving object in the blurred first image is not blurred.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and in particular to an image processing method, a depth estimation model training method, and an electronic device. Background Art

[0002] When a mobile phone or other electronic device shoots an image containing a subject (such as a person) and a background, blurring the background can usually better highlight the subject and improve the image display effect. The premise of blurring the background is to correctly identify the subject and the background. One of the key differences between the subject and the background is the different depths of the two. Therefore, the subject and the background can be better distinguished by calculating the depth of the subject and the depth of the background.

[0003] Monocular depth estimation is a method for calculating depth based on images taken by a single camera. It can accurately determine the depth of the subject and the background in scenes with relatively simple subjects. However, when the subject is more complex, such as in a moving scene, such as a person and an object in the person's hand, a person and an object being thrown by the person, etc., since the object is fused with the background, the general monocular depth estimation may determine the depth of the object to be the same as the background, resulting in a large error. Summary of the invention

[0004] To solve the above problems, the embodiments of the present application provide an image processing method, a depth estimation model training method and an electronic device, which can improve the accuracy of depth calculation, and thereby correctly blur the background in the image and avoid blurring the subject in the image.

[0005] In order to achieve the above purpose, the embodiment of the present application adopts the following technical solution:

[0006] In a first aspect, an image processing method is provided, which is applied to an electronic device. The electronic device collects a first image and a second image in a motion scene; the motion scene includes a moving object, wherein the moving object includes a person in motion and / or other objects other than the person in motion. The electronic device inputs the collected first image and the second image into a depth estimation model to obtain a first depth of the first image. The second image includes an image collected before the acquisition time of the first image, and / or an image collected after the acquisition time of the first image. The input parameters of the depth estimation model include an image sequence, the image sequence includes a current frame, and also includes images collected before the acquisition time of the current frame and / or images collected after the acquisition time of the current frame. The depth estimation model is used to output the depth of the current frame based on the motion features of the moving object in each frame of the image in the image sequence. The electronic device blurs the first image based on the first depth to obtain the blurred first image; the moving object in the blurred first image is not blurred.

[0007] In the present application, the electronic device is able to obtain motion features in multiple frames of images (first image and second image) in an image sequence, and based on the motion features of the multiple frames of images, refer to the depth of the moving object in the second image to correctly obtain the depth of the moving object in the first image, which helps to accurately identify the depth of the moving object in the moving scene, and then correctly blur the first image.

[0008] In a possible implementation of the first aspect, after the electronic device recognizes a shooting scene in which a person jumps, it continuously captures multiple images. The multiple images include multiple groups of first images and second images. The electronic device sequentially inputs the multiple groups of first images and second images into a depth estimation model in accordance with the order in which the multiple images are captured, and obtains multiple first depths corresponding to the multiple first images. In the case where the first image includes a person jumping, after obtaining the first depth, the electronic device corrects the depth of the person in the first image based on at least one first depth before the first depth and / or at least one first depth after the first depth; after correction, the depth of the person in the first depth image is consistent with the depth of the person in at least one first depth before the first depth and / or the depth of the person in at least one first depth after the first depth. In a shooting scene in which a person jumps, the electronic device can continuously capture multiple groups of first images and second images, and correspondingly generate multiple corresponding first depths. In the case where the first image includes a person jumping, the electronic device can correct the depth of the task based on the first image before and / or after the first image. After correction, the depth of the person in the first image is consistent with the depth of the portrait in the first image before and / or after the first image. In this way, the problem of underestimating the depth of the jump task in the conventional method can be avoided.

[0009] In a possible implementation manner of the first aspect, the other objects in the first image and the second image are objects that are in contact with the character and move with the character; or, the other objects are any one of objects thrown or projected by the character.

[0010] In a possible implementation of the first aspect, the depth estimation model includes a motion feature extraction module and a depth calculation module. The first image and the second image are input into the depth estimation model, and the motion feature extraction module extracts motion features based on the image features of the first image and the second image. The motion feature is the motion feature of a moving object in an image sequence composed of the first image and the second image; the depth calculation module estimates the depth of the first image based on the motion feature and outputs the first depth. Since the image sequence is input, the depth estimation model can obtain more image information. Moreover, since the motion features in the image sequence are extracted, it further helps the depth estimation model to correctly output the depth of the first image.

[0011] In a second aspect, a training method for a depth estimation model is provided. The method includes: obtaining training samples. The training samples include multiple groups of training images, and a group of training images includes a first training image and a second training image in a motion scene. The second training image includes an image captured before the acquisition time of the first training image, and / or an image captured after the acquisition time of the first training image. The motion scene includes a moving object. The moving object includes a person in motion, and / or other objects other than the person in motion. Based on the training samples, an initial model is trained to obtain a depth estimation model. Among them, the input parameters of the depth estimation model include an image sequence, the image sequence includes a current frame, and also includes images captured before the acquisition time of the current frame and / or images captured after the acquisition time of the current frame; the depth estimation model is used to output the depth of the current frame based on the motion characteristics of the moving object in each frame of the image sequence.

[0012] The present application uses an image sequence (a first training image and a second training image) to train an initial depth estimation model, the training model acquires motion features in the image sequence, and the training model correctly outputs the depth of the image based on the motion features.

[0013] In a possible implementation of the second aspect, the electronic device trains an initial depth estimation model based on a first loss. Specifically, the electronic device calculates the first loss. The first loss is the sum of at least one second loss and a third loss; wherein the second loss is used to calculate the difference between the second estimated depth and the third estimated depth of the moving object output by the depth estimation model; wherein the second estimated depth is the depth of the moving object in the estimated depth corresponding to the second training image output by the depth estimation model; the third estimated depth is the depth calculated based on the first estimated depth; the first estimated depth is the depth of the moving object in the estimated depth corresponding to the first training image output by the depth estimation model; the third loss is used to calculate the consistency of the depth of the pixels of the moving object in the first training image. The electronic device trains the initial model based on the value of the first loss to obtain a depth estimation model.

[0014] In a possible implementation manner of the second aspect, the second loss and the third loss are calculated based on the following formula:

[0015]

[0016]

[0017] in, is the depth of the i-th pixel in the moving object in the second training image output by the depth estimation model; is the depth of the i-th pixel of the moving object in the first training image output by the depth estimation model; S 2is the area of ​​the moving object in the second training image; S 1 is the area of ​​the moving object in the first training image; It is the depth of the i+1th pixel in the moving object in the first training image output by the depth estimation model, where i takes values ​​in [1, 2, ..., m] in sequence, and m is an integer greater than or equal to 2.

[0018] In a possible implementation of the second aspect, the electronic device inputs the training sample into the optical flow model, the optical flow model obtains at least one optical flow between the first training image and the second training image, and outputs at least one first optical flow map based on the at least one optical flow map of the first training image. Afterwards, the electronic device calculates a fourth loss, and the fourth loss is used to calculate the consistency between the depth of the at least one first optical flow map and the estimated depth of the first training image output by the depth estimation model. Finally, the electronic device trains the initial model based on the value of the first loss and the value of the fourth loss to obtain a depth estimation model. The electronic device uses the optical flow model to supervise the output of the depth estimation model, and trains the depth estimation model to extract the correct motion features.

[0019] In a possible implementation manner of the second aspect, the electronic device calculates the fourth loss based on the following formula:

[0020] Fourth loss = O 2→1 |D 2→1 -D 1 |.

[0021] Among them, D 2→1 is the depth corresponding to at least one first optical flow map; D 1 is the estimated depth of the first training image estimated by the depth estimation model; 2→1 It is a valid area composed of pixels whose positions change between the second training image and the first training image.

[0022] In a possible implementation of the second aspect, the electronic device further calculates a fifth loss. The fifth loss is used to calculate the difference between the estimated depth of the first image output by the depth estimation model and the true depth of the first image. The electronic device trains the initial model to obtain a depth estimation model based on the value of the first loss, the value of the fourth loss, and the value of the fifth loss.

[0023] In a third aspect, the present application provides an electronic device, comprising: a memory, and one or more processors; the memory is coupled to the processor; wherein the memory is used to store computer program code, and the computer program code includes computer instructions; when the computer instructions are executed by the processor, the electronic device executes the method described in any one of the first aspects.

[0024] In a fourth aspect, the present application provides a chip system, which can be applied to an electronic device including a memory. The chip system includes one or more interface circuits and one or more processors. The interface circuit and the processor are interconnected by a line. The interface circuit is used to receive a signal from the above-mentioned memory and send the signal to the processor, and the signal includes a computer instruction stored in the memory. When the processor executes the computer instruction, the electronic device executes the method as in the first aspect and any possible design thereof.

[0025] In a fifth aspect, the present application provides a computer-readable storage medium, the computer-readable storage medium comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method of the first aspect and any possible design thereof.

[0026] In a sixth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method of the first aspect and any possible design thereof.

[0027] It can be understood that the beneficial effects that can be achieved by the electronic device of any possible design of the third aspect, the chip system of the fourth aspect, the computer-readable storage medium of the fifth aspect, and the computer program product of the sixth aspect can be referred to as the beneficial effects in the first aspect, the second aspect and any possible design of them, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of capturing an image provided by an embodiment of the present application;

[0029] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0030] Figure 3 A schematic diagram of a software architecture of an electronic device provided in an embodiment of the present application;

[0031] Figure 4 A flowchart of an image processing method provided in an embodiment of the present application;

[0032] Figure 5 A schematic diagram of a display interface provided in an embodiment of the present application;

[0033] Figure 6 A schematic diagram of the structure of a depth estimation model provided in an embodiment of the present application;

[0034] Figure 7 A schematic diagram of a method for correcting the depth of a person in a first image provided by an embodiment of the present application;

[0035] Figure 8 A schematic diagram of a training sample provided in an embodiment of the present application;

[0036] Fig. 9 A schematic diagram of generating training samples provided in an embodiment of the present application;

[0037] Fig.10 A schematic diagram of an image processing process of an optical flow model provided in an embodiment of the present application;

[0038] Fig.11 A flowchart of a method for training a depth estimation model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The words "first", "second" and "third" in the embodiments of the present application are used to distinguish different objects rather than to limit a specific order. In addition, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0040] More and more terminal devices can provide a blurring camera function. Blurring refers to an image processing technology that blurs the image. For example, blurring can include portrait blurring. Portrait blurring can be for an image that includes a person, blurring the pixels other than the person in the image, so as to highlight the person in the image.

[0041] Optionally, the terminal device may perform blur processing based on the depth of the image. The depth of the image is the depth of each pixel in the image. The depth may be a numerical value. For example, the terminal device may apply a blur effect to pixels in the image whose depth values ​​are within a preset range based on the depth values ​​of the pixels in the image, i.e., perform blur processing.

[0042] In the embodiments of the present application, the depth of an image refers to the distance information between the pixels in the image and the camera. The depth of an image can be represented in the form of a depth map of the image, which can reflect the distance information between all pixels in the image and the camera. Figure 1 Different distance information is generally identified by different grayscale values. For example, the larger the grayscale value of a pixel in an image, the closer the pixel is to the camera, and the smaller the grayscale value of a pixel in an image, the farther the pixel is from the camera.

[0043] Please refer to Figure 1 , is a depth map provided in an embodiment of the present application. Figure 1As shown, the depth map includes a ball 101 and a person 102. The grayscale value of the pixels constituting the ball 101 in the depth map is greater than the grayscale value of the pixels constituting the person 102. Therefore, it can be determined Figure 1 The distance between the ball 101 and the camera is smaller than the distance between the person 102 and the camera, that is, the depth value of the ball 101 is smaller than the depth value of the person 102. At this time, when blurring, the ball 101 with a smaller depth value can be blurred to highlight the person 102 in the image.

[0044] It can be seen that identifying the depth of the image is a prerequisite for blurring. In the field of depth calculation, common depth calculation methods include binocular depth estimation in addition to monocular depth estimation. Binocular depth estimation refers to determining the depth of each pixel in an image based on images captured by two cameras. Exemplarily, in a binocular depth estimation algorithm, the electronic device can first perform stereo matching based on two images captured by two cameras to determine the disparity of the photographed object in the two frames of images. After that, the electronic device calculates the depth of the photographed object based on the disparity. Exemplarily, the photographed object can be, for example, Figure 1 Figure 102 and ball 101 in the figure. Parallax is the pixel difference of the object in the two images captured by the two cameras. In other words, binocular depth estimation requires matching the images captured by the two cameras first. Since the time and angle of the two cameras may be different, the above matching is very prone to errors in night scenes, sports scenes, and scenes with repeated textures, resulting in large errors in the depth estimation results.

[0045] For example, in night scenes, since there are many dark areas in the images taken in night scenes, the electronic device may make mistakes when matching images. The dark area refers to the area where the brightness of the pixels in the image is darker. In scenes with repeated textures, since the images include repeated textures, the electronic device may make matching errors at the repeated textures when matching images. In sports scenes, matching errors may occur due to the asynchronous image capture time.

[0046] The monocular depth estimation algorithm can use an image taken by a camera to estimate the distance of each pixel in the image relative to the camera. Although monocular depth estimation does not require stereo matching of two images to obtain the parallax of the photographed object, it can overcome the above problems of binocular depth estimation. However, since there is only one camera to capture the image, when the subject is complex, the depth of part of the subject may be determined to be the same as the depth of the background, resulting in large errors.

[0047] Part of the photographed subjects may be objects that are in physical contact with the person. For example, the objects that are in physical contact with the person may be handheld objects such as a racket held by the person, a flower held by the person, etc. Alternatively, part of the photographed subjects may include moving objects, objects thrown by the person, thrown objects, swinging objects, etc., such as a basketball thrown by the person, a volleyball thrown by the person, a football kicked by the person, etc.

[0048] Continue with Figure 1 For example, the subject of the shot image includes a person 102 and a ball 101. If the shot image is to be blurred, the ideal result should be to blur the background other than the person 102 and the ball 101, and highlight the overall image of the person 102 and the ball 101. However, the general monocular depth calculation method may determine the depth of the ball 101 to be the same as the depth of the background when calculating the depth of the image, resulting in a large error. In this way, the ball 101 may also be blurred during subsequent blurring, affecting the user's experience. In other words, in a sports scene, the portrait and moving objects in the image do not need to be blurred. However, the conventional monocular depth algorithm cannot identify the depth of the moving object as a depth that does not require blurring, which leads to a poor blurring display effect. For example, in a scene where a user plays football, the conventional monocular depth algorithm will identify the football as an object that needs to be blurred. In the final displayed image, only the user is clearly displayed, and the look and feel of the entire image is relatively abrupt.

[0049] In order to solve the above problems, the embodiments of the present application provide an image processing method, a depth estimation model training method and an electronic device, which can accurately identify the more complex shooting subjects in the image, which is conducive to the subsequent accurate blurring of the background outside the shooting subject, thereby improving the display effect and enhancing the user experience.

[0050] In the embodiments of the present application, the electronic device may be a portable terminal, such as a mobile phone, a tablet computer, a wearable device (such as a smart watch), a vehicle-mounted device, etc., or may be a cloud server, which is not limited here.

[0051] As an example, see Figure 2 , is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The monocular depth estimation method provided in an embodiment of the present application can be applied to Figure 2 In the electronic device 200 shown.

[0052] like Figure 2 As shown, the electronic device 200 may include a processor 201, a memory 202, a camera 203, etc.

[0053] The processor 201 may include one or more processing units, for example, the processor 201 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video stream codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated in one or more processors 201.

[0054] The memory 202 may be independent of the processor 201 or may be integrated into the processor 201. The memory 202 is used to store instructions and data.

[0055] When the processor 201 executes the instructions stored in the memory 202, the electronic device 200 can execute the monocular depth estimation method provided in the embodiment of the present application.

[0056] The camera 203 is used to take photos and videos. In the embodiment of the present application, the camera 203 can be used to take images including a subject and a background.

[0057] It should be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 200. In other embodiments, the electronic device 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0058] Above through Figure 2 The hardware structure of the electronic device provided in the embodiment of the present application is introduced, and the software architecture of the electronic device provided in the embodiment of the present application is exemplarily described below.

[0059] Please refer to Figure 3 , is a schematic diagram of the software architecture of an electronic device provided in an embodiment of the present application. The software system of the electronic device 200 can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes the Android system of the layered architecture as an example to illustrate the software structure of the electronic device 200. The software structure is as follows Figure 3 shown.

[0060] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into six layers, from top to bottom: application layer, application framework layer, system library, and driver layer.

[0061] Among them, the application layer can include a series of application packages.

[0062] The application package can include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, social networking and other applications.

[0063] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 2 As shown, the application framework layer may include a content provider, a view system, a resource manager, a notification manager, an input system, a depth estimation module, and a camera service management.

[0064] The input system is used to monitor the input module of the mobile phone (such as the touch screen driver), and convert the parameters input by the input module into usable events and pass them to the relevant modules in the upper layer. For example, the input system is used to monitor the touch screen of the mobile phone through the touch screen driver, and convert the touch parameters generated by the touch operation input by the touch screen into usable events and pass them to the upper layer APP.

[0065] Content providers are used to store and retrieve data and make it accessible to applications. The data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0066] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build the display interface of an application.

[0067] The notification manager enables applications to display notification information in the status bar, which can be used to convey informational messages and disappear automatically after a short stay without user interaction.

[0068] The depth estimation module is used to call a depth estimation model to output a first depth of the first image based on the first image and the second image.

[0069] The camera service is used to respond to a call instruction of the camera APP and drive the camera to capture the first image and the second image through the camera driver.

[0070] Android Runtime includes core libraries and virtual machines. Android Runtime is responsible for scheduling and management of the Android system. The core library consists of two parts: one is the function that the Java language needs to call, and the other is the Android core library. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0071] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0072] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0073] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0074] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0075] A 2D graphics engine is a drawing engine for 2D drawings.

[0076] The kernel layer may include touch screen driver, display driver, camera driver, audio driver, etc.

[0077] The image processing method provided in the embodiment of the present application is applied to an electronic device. The hardware structure of the electronic device can be as follows: Figure 2 As shown, the software architecture of the electronic device can be as follows Figure 3 As shown. Exemplarily, the monocular depth estimation method can be applied to camera applications in electronic devices. The following is a specific introduction using a mobile phone as an example of the electronic device.

[0078] Please refer to Figure 4 , is a flow chart of an image processing method provided in an embodiment of the present application. Figure 4 As shown, the method may include the following process:

[0079] S401, the mobile phone collects images.

[0080] Among them, the image includes a first image and a second image captured for the same image. The second image is at least one frame of image before the first image. Or, the second image is at least one frame of image after the first image. Or, the second image is at least one frame of image before the first image and at least one frame of image after the first image. It should be understood that the before and after here refers to the before and after the acquisition time. The image whose acquisition time is before the acquisition time of the first image is the image before the first image. The image whose acquisition time is after the acquisition time of the first image is the image after the first image. That is to say, the first image and the second image can be a continuous sequence of images. The second image can be several frames of images captured before the acquisition time of the first image. The second image can be several frames of images captured after the acquisition time of the first image. The second image can also be several frames of images captured before the acquisition time of the first image and several frames of images captured after the acquisition time of the first image. The embodiment of the present application does not make specific limitations on this.

[0081] Optionally, the mobile phone may capture an image in response to the user's first operation. In one example, the mobile phone may be installed with a camera APP. The camera APP may display the following: Figure 5 The preview interface 501 shown. The first operation may be that the user clicks the shutter button 502 in the preview interface 501 of the camera APP. In response to the user clicking the shutter button once in the preview interface 501, the mobile phone may capture the first image and the second image. The first image and the second image are captured at different times.

[0082] In another example, the first operation may also be a first voice control instruction. Figure 5 In the preview interface 501 shown, in response to the user's first voice control instruction, the mobile phone captures the first image and the second image. For example, the first voice control instruction may be "take a photo". In other words, the first operation is used to trigger the mobile phone to capture the first image and the second image. The embodiment of the present application does not impose any limitation on the specific form of the first operation.

[0083] Exemplarily, the mobile phone may capture two images, including a first image and a second image before the first image, or including a first image and a second image after the first image. The mobile phone may also capture three images, including a second image before the first image, a first image, and a second image after the first image.

[0084] Optionally, after identifying a shooting scene in which a person jumps, the mobile phone may continuously capture multiple images, the multiple images including multiple groups of images, and a group of images including a first image and a second image. Exemplarily, after identifying a shooting scene in which a person jumps, the mobile phone may capture N groups of images, each of the N groups of images including a first image and a second image. N is an integer greater than or equal to 4. Exemplarily, the mobile phone may capture the first image and the second image in response to a first operation. Afterwards, the mobile phone may identify whether it is a shooting scene in which a person jumps based on the first image and / or the second image. After the mobile phone identifies a shooting scene in which a person jumps, the mobile phone may continue to capture images until N groups of images are captured.

[0085] S402: The mobile phone outputs the first image and the second image to a depth estimation model to obtain a first depth of the first image.

[0086] The depth estimation model has the ability to output the first depth of the first image according to the input first image and second image. Specifically, the training method of the depth estimation model can refer to the following embodiments.

[0087] The first depth may be a first depth map, and the first depth includes the depth of each pixel in the first image.

[0088] The following introduces the present solution by taking the example of a mobile phone obtaining the depth of a first image based on a first image, a frame of image before the first image, and a frame of image after the first image. Exemplarily, the three images are image a-1 (second image), image n (first image), and image a+1 (second image). The mobile phone can obtain the first depth of image a based on these three images. Among them, image a-1 is the previous frame of image a, and image a+1 is the next frame of image a.

[0089] Specifically, the mobile phone can input these three images into a depth estimation model, and the depth estimation model has the ability to output the first depth of the first image. The mobile phone can input image a, image a+1, and image a-1 into the depth estimation model to obtain the first depth of image a. The first depth of image a includes the depth of each pixel in image a.

[0090] Optionally, the depth estimation model is also capable of outputting the first depth of the first image and the second depth of the second image. For example, the depth estimation model is also capable of outputting the second depth of the image a-1 and the second depth of the image a+1.

[0091] Figure 6 A structural diagram of a depth estimation model is given. Figure 6 As shown, the depth estimation model includes a feature extraction module, a motion feature extraction module, and a depth calculation module.

[0092] The feature extraction module can extract image features of the input image, wherein the image features are used to characterize the input image. Exemplarily, the image features include color features, texture features, shape features, and position features of each pixel in the image. Exemplarily, the feature extraction module can extract image features f of image a a , image feature f of image a+1 a+1 , taking the image feature f of image a-1 a-1 The feature extraction module can be, for example, a convolutional neural network, a graph neural network, etc.

[0093] The motion feature extraction module is used to extract motion features from the three images. The motion features may be features of a moving object in an image sequence formed by the first image and the second image, and the moving object may include a person in motion and / or other objects in motion other than a person. Exemplarily, the motion features include shape features, motion trajectory features, etc. of the moving object in the image sequence.

[0094] The motion feature extraction module may be, for example, a cross-attention mechanism (Cross-Attention ion) model. Exemplarily, the motion feature extraction module may be based on the image feature f a , image feature f a+1 , image feature f a-1 , extract the motion features of moving objects in the image sequence composed of image a-1, image a, and image a+1.

[0095] The depth calculation module is used to generate a first depth based on motion features. The depth calculation module can generate a first depth based on the image features of the first image and the second image, and the motion features in the first image and the second image. Since the image features of the first image and the second image include more image information than a frame of image, it helps to estimate the accuracy of the image depth. And since the motion features of the image sequence (the first image and the second image) are extracted, the depth calculation module can refer to the depth of the moving object in the image sequence to calculate the depth of the moving object in the depth of the first image. Compared with using only one image to calculate the depth of the moving object therein, this solution can improve the accuracy of the depth estimation of the moving object in the first image based on the motion features in the frames before and / or after the first image, and refer to the depth of the moving object in the frames before and / or after the first image.

[0096] Optionally, the depth calculation module may also generate a first depth and a second depth.

[0097] Optionally, the mobile phone can also correct the depth of the person in the first image based on several frames of the first image before and after the first image. This is especially true in scenes where people are jumping. For example, in scenes where people are jumping, conventional depth estimation algorithms may underestimate the depth of the person in the image where the person is off the ground, thereby causing blurring errors. Therefore, after the mobile phone obtains the first depth map of the first image based on the depth estimation model, it can correct the depth of the person in the first image based on at least one frame of the first image before the first image, and / or at least one frame of the first image after the first image, so that the depth of the person in the corrected first image is consistent with the depth of the person in at least one frame of the first image before the first image and / or the depth of the person in at least one frame of the first image after the first image. For example, if Figure 7 As shown, Figure 7 FIG. a shows a depth diagram of a human portrait before correction. Figure 7 As shown in a, when the person jumps, the person is no longer in contact with the ground. Conventional depth estimation algorithms may underestimate the depth of the jumping person.

[0098] To this end, after the mobile phone has captured N groups of first images and second images, the mobile phone can generate a first depth of the first image based on a group of first images and second images. In this way, the mobile phone can generate N first depths. In a scenario where the first image includes a jumping person, the mobile phone can correct the depth of the portrait in the first image based on the first depth of the first image before the first image. Alternatively, in a scenario where the first image includes a jumping person, the mobile phone can correct the depth of the portrait in the first image based on the first depth of the first image after the first image. Alternatively, in a scenario where the first image includes a jumping person, the mobile phone can correct the depth of the portrait in the first image based on the first depth of the first image before the first image and the first depth of the first image after the first image.

[0099] For example, Figure 7 b in FIG. 4 shows a correction method. The mobile phone collects 5 images, and the mobile phone generates the depth of the first image before the first image based on images 1 to 3 using the method shown in S402. Figure 1 The mobile phone generates the depth of the first image based on images 2 to 4 using the method shown in S402. Figure 2 The mobile phone generates the depth of the first image after the first image using the method shown in S402 based on images 3 to 5. Figure 3 In the case where the first image includes a jumping person, the mobile phone can Figure 1 and depth Figure 3 Depth correction for portraits Figure 2 Specifically, the phone obtains the depth of the portrait. Figure 1 Depth 1 and depth of portrait in Figure 3Depth 3 of the portrait in the middle. For example, the mobile phone can obtain the depth Figure 1 The depth of some or all pixels in the portrait area is calculated and the average is calculated to obtain the depth of the portrait 1. Alternatively, the mobile phone can obtain the depth Figure 1 The mode of the depth of some or all pixels in the portrait area is 1. The mobile phone can obtain the depth Figure 3 The depth of some or all pixels in the portrait area is calculated and the average is calculated to obtain the depth of the portrait 3. Alternatively, the mobile phone can obtain the depth Figure 3 The mode of the depths of some or all pixels in the portrait area in the first depth map is the portrait depth 3. The mobile phone averages the portrait depth 1 and the portrait depth 2, and assigns the pixel depth of the portrait area in the first depth map to the average value, thereby completing the correction of the portrait depth in the first image. The corrected depth is as follows: Figure 7 As shown in b.

[0100] For another example, the mobile phone collects 4 images, and the mobile phone generates the depth of the first image before the first image using the method shown in S402 based on images 1 to 3. Figure 1 The mobile phone generates the depth of the first image based on images 2 to 4 using the method shown in S402. Figure 2 In the case where the first image includes a jumping person, the mobile phone can Figure 1 Depth correction for portraits Figure 2 For example, the phone will Figure 1 The mean or mode of the depth of the portrait is assigned to the depth Figure 2 Alternatively, the mobile phone generates the depth of the first image based on images 1 to 3 using the method shown in S402. Figure 1 The mobile phone generates the depth of the first image after the first image using the method shown in S402 based on images 2 to 4. Figure 2 In the case where the first image includes a jumping person, the mobile phone can Figure 2 Depth correction for portraits Figure 1 Depth of the portrait.

[0101] For another example, the mobile phone collects four images, namely, mobile phone image 1 and image 2, and the method shown in S402 is used to generate the depth of the first image before the first image. Figure 1 The mobile phone generates the depth of the first image based on image 2 and image 3 using the method shown in S402. Figure 2 The mobile phone generates the depth of the first image after the first image is generated by the method shown in S402 based on the image 3 and the image 4. Figure 3 In the case where the first image includes a jumping person, the mobile phone can Figure 1 Depth and / or depth of human figures Figure 3Depth correction for portraits Figure 2 The depth of the portrait in the image. The specific correction method is mentioned above and will not be repeated here.

[0102] S403: The mobile phone blurs the first image based on the first depth of the first image.

[0103] Exemplarily, the mobile phone stores the correspondence between different depths and different blur coefficients. The mobile phone can obtain the corresponding blur coefficient based on the different depths, and use the blur coefficient to blur the pixels of the corresponding depth. For another example, the mobile phone can obtain the shooting focus, which corresponds to a depth range (depth of field). The mobile phone may not blur the pixels within the depth range, and the mobile phone may blur the pixels outside the depth range. Among them, the shooting focus may be the position for capturing the focus of the object to be photographed. Exemplarily, in response to the user in the following example, Figure 5 By selecting the focus on the preview image displayed on the preview interface 501 shown in FIG. 5 , the mobile phone obtains the current shooting focus. Figure 5 In the preview interface 501 shown, based on the auto-focus algorithm inside the mobile phone, the mobile phone can obtain the current shooting focus and capture an image based on the shooting focus.

[0104] The subject in the blurred first image is not blurred, and the subject includes a moving object.

[0105] Furthermore, the mobile phone may store the blurred first image in the mobile phone. Optionally, the mobile phone may also display the blurred first image.

[0106] In the foregoing, an image blur processing method provided by an embodiment of the present application is introduced in conjunction with the accompanying drawings. The following describes the training process of the depth estimation model. The training process of the depth estimation model includes a sample construction process and a training process. The sample construction process may refer to constructing training samples for training the depth estimation model, and the training process may refer to training the depth estimation model using the training samples.

[0107] In the present application, the training images in the training samples all include people and moving objects. And the positions of the people and / or moving objects in the training images are relatively changed. For example, the position of the portrait in the training image is constant or changed. The position of the moving object in the training image is changed. Or, the position and size of the moving object in the training image are changed. The backgrounds of multiple training images of a training sample are the same.

[0108] The initial depth estimation model can be configured in a mobile phone, and the mobile phone can train the initial depth estimation model before leaving the factory and obtain the trained depth estimation model. Alternatively, the initial depth estimation model can be configured in a computer, and the depth estimation model is trained by the computer. The computer configures the trained depth estimation model in the mobile phone before the mobile phone leaves the factory. Alternatively, the initial depth estimation model can be configured in a computer, and the depth estimation model is trained by the computer. The computer can push the trained depth estimation model to the mobile phone, and the mobile phone will configure the depth estimation model in the mobile phone if the user agrees to receive it. This application does not make specific limitations on the electronic device for training the depth estimation model. The following takes the training of the depth estimation model by a mobile phone as an example to introduce this solution.

[0109] First, let’s introduce the sample construction process.

[0110] Example 1: The training samples include multiple groups of images, and one group of images includes a first training image and a second training image in a motion scene. The second training image includes an image captured before the capture time of the first training image, and / or an image captured after the capture time of the first training image. In other words, the first training image and the second training image can be an image sequence. The mobile phone can capture the image sequence to obtain a group of training images. The motion scene includes a moving object; the moving object includes a person in motion, and / or other objects in motion except a person.

[0111] Example 2: A frame of training image in a motion scene is obtained, and the mobile phone can identify the moving object in the frame of training image. The mobile phone generates multiple sets of training images and obtains training samples by modifying the position or position and size of the moving object in the training image.

[0112] The following takes the training sample including a set of training images, and a set of training images including three training images as an example to introduce the sample construction process. Figure 8 As shown, the mobile phone recognizes the football in training image 1, moves the position of the football and modifies the size of the football to generate training image 2 and training image 3. Specifically, the mobile phone moves the football away from the camera and reduces the size of the football to obtain training image 2. The mobile phone moves the football away from the camera again and reduces the size of the football again to obtain training image 3. Training image 1, training image 2, and training image 3 are one training sample.

[0113] Example 3: After obtaining a training sample, in order to further expand the training sample, the mobile phone can replace the background image of the training image to obtain a new image, and use the new image as another training sample.

[0114] like Figure 8As shown, the mobile phone can replace the background image of each training sample in the training sample to generate a new training sample. The new training sample includes training image 4, training image 5, and training image 6. Optionally, when generating a new training sample, the mobile phone can blur the portrait to simulate the image captured in a motion scene. Figure 8 In training image 6, the phone blurs the portrait.

[0115] Figure 8 In the example given, the training image includes a moving person and other objects in motion other than the person, such as a football. Optionally, the training image may only include a moving person.

[0116] Example 4: For training images that only include people, the mobile phone combines people with multiple backgrounds to generate multiple sets of training images. Fig. 9 As shown, for training image a, the mobile phone can first obtain a mask image of the portrait area. Then, the mobile phone synthesizes training image b based on the mask image and the background image. Similarly, in order to simulate a motion scene, the mobile phone can blur the portrait in the synthesized training image b. Training image a and training image b can be a set of training images.

[0117] The mobile phone inputs the training samples into the depth estimation model, and the depth estimation model can output the estimated depth corresponding to each training image in the training samples. Specifically, taking the training samples including three training images as an example, the feature extraction module in the depth estimation model can extract the image features corresponding to the training samples, and extract the motion features in the three training images through the motion feature extraction module. After that, the mobile phone can output the estimated depth of each training sample through the depth estimation model.

[0118] Exemplarily, the training samples include the second training image n-1, the first training image n, and the second training image n+1, and the mobile phone can obtain the estimated depth corresponding to the training image based on the depth estimation model. Afterwards, the mobile phone can optimize the model parameters of the depth estimation model based on the estimated depth of each training image output by the depth estimation model.

[0119] Specifically, the mobile phone can calculate a first loss, which is the sum of at least one second loss and a third loss. The second loss is used to calculate the correctness of the first estimated depth of the first training image output by the depth estimation model and the second estimated depth of the second training image. The second loss is the difference between the second estimated depth of the moving object in the second training image output by the depth estimation model and the third estimated depth calculated based on the first estimated depth. The first estimated depth is the depth of the moving object in the first training image output by the depth estimation model. The third estimated depth is the depth calculated based on the first estimated depth. The first loss is used to improve the depth transferability of the moving object and the internal consistency of the object.

[0120] That is, the second loss = |d 2 -d 3 |,d 2 is the second estimated depth of the moving object in the second training image output by the depth estimation model, d 3 Based on d 1 Calculated depth.

[0121] For example, the third estimated depth is but Among them, S 2 is the area of ​​the moving object in the second training image. 1 is the area of ​​the moving object in the first training image.

[0122] Among them, the second estimated depth d 2 It can be the average of the depths of multiple pixels constituting the moving object in the second training image. Accordingly, the first estimated depth d 1 It can be the average of the depths of multiple pixels constituting the moving object in the first training image. Or, the second estimated depth d 2 It can be the mode of the depths of the multiple pixels constituting the moving object in the second training image. 1 It may be the mode of the depths of a plurality of pixels constituting the moving object in the first training image.

[0123] Take the training images as training image n-1, training image n, and training image n+1 as an example. Among them, d n+1 It is the second estimated depth of the moving object in the training image n+1 output by the depth estimation model. n is the first estimated depth of the moving object in the training image n output by the depth estimation model. n-1 S is the second estimated depth of the moving object in the training image n-1 output by the depth estimation model. n S is the area of ​​the moving object in the training image n.n+1 S is the area of ​​the moving object in the training image n+1. n-1 is the area of ​​the moving object in the training image n-1.

[0124] Optionally, in order to reduce the amount of calculation, the mobile phone can sample the depth of m pixels on the moving object in the second training image, obtain the depth of m pixels at the same position in the first training image, and calculate the second loss based on the depth of m pixels.

[0125] in, is the depth of the i-th pixel in the moving object in the second training image estimated by the depth estimation model; is the depth of the i-th pixel in the moving object in the first training image estimated by the depth estimation model. Where i is in [1, 2, ..., m], and M is an integer greater than 2.

[0126] Take the training images as training image n-1, training image n, and training image n+1 as an example.

[0127] in, is the depth of the i-th pixel in the moving object in the training image n-1 estimated by the depth estimation model. is the depth of the i-th pixel in the moving object in the training image n estimated by the depth estimation model. is the depth of the i-th pixel in the moving object in the training image n+1 estimated by the depth estimation model.

[0128] For use based on calculate Theoretical value (third depth). For use based on calculate The theoretical value (third depth). This theoretical value should be the same or similar to the estimated value output by the depth estimation model. Therefore, Should be zero or a small value. Should be zero or smaller.

[0129] The third loss is used to calculate the consistency of the depth of the pixels of the moving object in the first training image output by the depth estimation model. The third loss is the difference in depth between the pixels of the moving object in the first training image. It should be understood that the depth of the object in a frame of image is consistent, that is, the depth of each pixel of the object in a frame of image should be consistent.

[0130] Theoretically, in the same frame image, the depth values ​​of each pixel of the moving object are the same. Therefore, the third loss should be zero or a smaller value. is the depth of the i+1th pixel of the moving object in the first training image output by the depth estimation model.

[0131] but,

[0132] The mobile phone optimizes the parameters of the depth estimation model based on the first loss until the first loss approaches zero or a minimum value, thereby obtaining a trained depth estimation model.

[0133] Optionally, in order to train the depth estimation model to extract the correct motion features, the mobile phone can use a second neural network model to supervise the depth estimation model so that the depth estimation model can extract the correct motion features. The second neural network model can be an optical flow model, which can be a trained model, and the input of the optical flow model is the training sample.

[0134] The optical flow model includes an optical flow extraction module, an optical flow mapping module, and an effective area recognition module. The optical flow extraction module is used to extract the optical flow between two adjacent frames of images, wherein the optical flow can represent the motion offset of each pixel of one frame of the two images, wherein the motion offset includes the motion direction and the motion displacement.

[0135] The optical flow mapping module is used to map the optical flow in another frame image based on the extracted optical flow to obtain an optical flow map.

[0136] For example, taking the training samples including training image n-1, training image n, and training image n+1 as an example, combined with Fig.10 Introduce this program.

[0137] The mobile phone inputs three frames of images into the optical flow model at the same time, and the optical flow model can output the optical flow map after optical flow mapping, such as outputting the optical flow corresponding to the training image n-1 and the training image n Figure 1 , the optical flow corresponding to the training image n and the training image n+1 Figure 2 .

[0138] Specifically, the optical flow model can obtain the optical flow F between the training image n-1 and the training image n n-1→n The optical flow F n-1→n Including the motion offset of each pixel in training image n-1 and training image n. The optical flow model can obtain the optical flow F between training image n and training image n+1 n→n+1 The optical flow F n→n+1 Includes the motion offset of each pixel in training image n and training image n+1.

[0139] Afterwards, the optical flow model can be used to perform optical flow mapping (warp). The optical flow model can be based on the optical flow F n-1→n Map the positions of pixels in training image n-1 to get the optical flow Figure 1 The optical flow model can be based on the optical flow F n→n+1 Map the positions of pixels in the training image n+1 to get the optical flow Figure 2 .

[0140] For example, for optical flow Figure 1 For each pixel in the optical flow Figure 1 Medium pixel (x 1 ,y 1 )=(x n-1 +Δx,y n-1 +Δy). Among them, (x 1 ,y 1 ) is the pixel in the optical flow Figure 1 The position in (x n-1 ,y n-1 ) is the position of the pixel in the training image n-1. Optical flow F n-1→n (x, y) = (Δx, Δy). That is, add the corresponding optical flow to each pixel of the training image n-1 to get the optical flow Figure 1 For optical flow Figure 2 For each pixel in the optical flow Figure 2 Pixels (x 2 ,y 2 )=(x n+1 +Δx,y n+1 +Δy). Among them, (x 2 ,y 2 ) is the pixel in the optical flow Figure 2 The position in (x n+1 ,y n+1 ) is the position of the pixel in the training image n+1. Among them, the optical flow F n→n+1 (x, y) = (Δx, Δy). That is, add the corresponding optical flow to each pixel of the training image n+1 to get the optical flow Figure 2 In theory, optical flow Figure 2 is the same or very close to the training image n. Figure 1 is the same or very close to the training image n.

[0141] Therefore, the mobile phone can calculate the fourth loss. The fourth loss is used to calculate the consistency between the depth of the at least one optical flow map and the estimated depth of the first training image output by the depth estimation model. As in the example above, the fourth loss can calculate the optical flow Figure 1 The consistency between the depth of the first training image and the estimated depth, and the optical flow Figure 2 The consistency between the depth of and the estimated depth of the first training image.

[0142] Exemplarily, the mobile phone may calculate the fourth loss based on the following formula.

[0143] Fourth loss = O 2→1 |D 2→1 -D 1 |. Among them, D 2→1 is the depth of the first optical flow map; D 1 is the estimated depth of the first training image estimated by the depth estimation model; 2→1 The effective area composed of pixels whose positions change between the second training image and the first training image is the optical flow F 2→1 effective area.

[0144] Taking the previous example as an example, the fourth loss = O n-1→n |D n-1→n -D n |+O n+1→n |D n+1→n -D n |.

[0145] Among them, D n-1→n Optical Flow Figure 1 The depth of (first optical flow map), D n is the estimated depth of the first image n estimated by the depth estimation model, D n+1→n It is optical flow Figure 2 The depth of (the first optical flow map). n-1→n is the effective area of ​​optical flow 1, that is, the area composed of pixels whose positions have changed in training image n-1 and image n. n+1→n is the effective area of ​​optical flow 2, that is, the area composed of pixels whose positions have changed in training image n and image n+1. n-1→n |D n-1→n -D n |Used to calculate optical flow Figure 1 The difference between each pixel in the valid area of ​​the training image n should be zero or a minimum value. n+1→n |D n+1→n -D n |Used to calculate optical flow Figure 2 The difference between each pixel in the valid area of ​​the training image n should be zero or a minimum value. The fourth loss is used to improve the inter-frame consistency of the overall unoccluded area between frames.

[0146] optical flow Figure 1 Depth D n-1→n The depth D of the training image n can be n With optical flow F n-1→n Calculated. Optical flow Figure 2 Depth D n+1→nThe depth D of the training image n can be n With optical flow F n→n+1 Calculated.

[0147] The mobile phone optimizes the parameters of the depth estimation model based on the first loss and the fourth loss. Specifically, the mobile phone can calculate the first total loss. The second total loss = the first loss + the α fourth loss. Among them, α is a coefficient, and α can be a decimal less than 1 and greater than zero. The mobile phone optimizes the parameters of the depth estimation model based on the first total loss until the value of the first total loss tends to zero or a minimum value, and obtains a trained depth estimation model.

[0148] Optionally, the mobile phone may further calculate a fifth loss. The fifth loss is used to measure the correctness of the first depth of the first image output by the depth estimation model. Exemplarily, the fifth loss is used to calculate the difference between the estimated depth of the first image output by the depth estimation model and the true depth of the first image. The true depth of the first image is known.

[0149] The fifth loss can be calculated based on the following formula: in, It can be the real depth of the first image.

[0150] The mobile phone optimizes the parameters of the depth estimation model based on the first loss, the fourth loss and the fifth loss. Specifically, the mobile phone can calculate the second total loss. The second total loss = the first loss + the α fourth loss + the β fifth loss. Among them, β is a coefficient, and β can be a decimal less than 1 and greater than zero. The mobile phone optimizes the parameters of the depth estimation model based on the second total loss until the value of the second total loss tends to zero or is a minimum value, and obtains a trained depth estimation model.

[0151] The following example introduces the present solution by taking the training sample including two groups of training images. One group of training images includes image b-1, image b, and image b+1. The other group of training images includes image b, image b+1, and image b-1. That is, the training sample is an image sequence including 4 consecutive frames of images.

[0152] like Fig.11 As shown, the mobile phone inputs the above image sequence into the depth estimation model. The feature extraction module extracts the image features f b-1 , image feature f b , image feature f b+1 , image feature f b+2 Afterwards, the motion feature extraction module is based on the image feature f b-1 , image feature f b , image feature f b+1 Extract the motion feature b, based on the image feature f b , image feature f b+1 , image feature f b+2Extract the motion feature b+1. The depth calculation module outputs the depth D corresponding to the image sequence b-1 , depth D b , depth D b+1 , depth D b+2 Afterwards, the phone uses the depth D b-1 The depth d of the moving object b-1 , depth D b The depth d of the moving object b , depth D b+1 The depth d of the moving object b+1 Calculate the first loss b Based on the depth D b The depth d of the moving object b , depth D b+1 The depth d of the moving object b+1 , depth D b+2 The depth d of the moving object b+2 , calculate the first loss b+1 .

[0153]

[0154]

[0155] in, is the depth of the i-th pixel in the moving object in the training image b+1 estimated by the depth estimation model. The depth of the i-th pixel in the moving object in the training image b estimated by the depth estimation model. The depth of the i-th pixel in the moving object in the training image b-1 estimated by the depth estimation model. The depth of the i+1th pixel in the moving object in the training image b estimated by the depth estimation model. The depth of the i-th pixel in the moving object in the training image b estimated by the depth estimation model. The depth of the i+1th pixel in the moving object in the training image b+1 estimated by the depth estimation model. The depth of the i-th pixel of the moving object in the training image b+2 estimated by the depth estimation model. S b-1 S is the area of ​​the moving object in the training image b-1. b is the area of ​​the moving object in the training image b. b+1 is the area of ​​the moving object in the training image b+1. b+2 is the area of ​​the moving object in the training image b+2.

[0156] At the same time, the mobile phone inputs the above image sequence into the optical flow model. The optical flow model is based on the optical flow F between the training image b-1 and image b. b-1→b , map image b-1 and output optical flow Figure 1 , based on the optical flow F between image b and training image b+1 b→b+1 , map image b+1 and output optical flow Figure 2 , 1, based on the optical flow F between image b and training image b+1 b→b+1 , map image b and output optical flow Figure 3 , based on the optical flow F between image b+1 and training image b+2 b+1→b+2 , map image b+2 and output optical flow Figure 4 .

[0157] After that, the phone can be based on optical flow Figure 1 Depth D b-1→b , the depth D of image b b , optical flow Figure 2 Depth D b+1→b Calculate the fourth loss b . Mobile phone based on optical flow Figure 3 Depth D b→b+1 , the depth D of image b+1 b+1 , optical flow Figure 4 Depth D b+2→b+1 Calculate the fourth loss b+1 .

[0158] The fourth loss b =O b-1→b |D b-1→b -D b |+O b+1→b |D b+1→b -D b |.

[0159] The fourth loss b+1 =O b→b+1 |D b→b+1 -D b+1 |+O b+2→b+1 |D b+2→b+1 -D b+1 |. Among them, O b-1→b The effective area of ​​optical flow 1 is the area consisting of pixels whose positions have changed in training image b-1 and image b. b+1→b The effective area of ​​optical flow 2 is the area consisting of pixels whose positions have changed in training image b+1 and image b. →b+1 Ob is the effective area of ​​optical flow 3, that is, the area composed of pixels whose positions have changed in training image b and image b+1. +2→b+1 It is the effective area of ​​optical flow 4, that is, the area composed of pixels whose positions have changed in training image b+1 and image b+2.

[0160] Mobile phone calculation fifth loss b and the fifth loss b+1 .

[0161] in, is the true depth of image b. is the true depth of image b+1.

[0162] After that, the phone calculates the second total loss b = First loss b +α fourth loss b +β fifth loss b . Mobile phone calculation of the second total loss b+1 = First loss b+1 +α fourth loss b+1 +β fifth loss b+1 . Fig.11 In the example given, the second total loss is calculated only twice. In actual training, the phone will calculate multiple second total losses. The phone continuously optimizes the model parameters of the depth estimation model so that the second total loss approaches zero or is a minimum value, completing the model training.

[0163] The image processing method proposed in the embodiment of the present application is used for blurring portraits in motion scenes. The depth of the human body, handheld objects, and moving objects in the image is effectively improved, and the false blur is significantly reduced. The depth estimation model in the embodiment of the present application can be a monocular depth estimation model, so there is no need to introduce a second camera device, saving implementation costs. The depth estimation model in the embodiment of the present application is superior to the binocular depth estimation model and the conventional monocular depth estimation in motion scenes and repeated texture scenes. The depth estimation model in the embodiment of the present application has a lightweight network structure and is easy to deploy on the mobile terminal.

[0164] An embodiment of the present application also provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0165] The embodiments of the present application also provide a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the method in the above-mentioned embodiments.

[0166] In addition, the embodiments of the present application also provide a device, which may be a chip, a component or a module, and the device may include a connected processor and a memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor may execute the computer-executable instructions stored in the memory, so that the chip executes the methods in the above-mentioned method embodiments. It should be noted that all relevant contents of each step involved in the above-mentioned method embodiments can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0167] Among them, the electronic device, computer storage medium, computer program product or chip provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0168] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of electronic equipment. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0169] The embodiment of the present application can divide the functional modules of the devices involved therein according to the above method examples. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0170] The functions or actions or operations or steps in the above embodiments can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or may include one or more servers, data centers and other data storage devices that can be integrated with the medium. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0171] Although the present application has been described in conjunction with specific features and embodiments thereof, it is obvious that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely exemplary illustrations of the present application as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art may make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An image processing method, characterized in that: The method comprises: A first image and a second image are captured in a motion scene; the motion scene includes a moving object; the moving object includes a person in motion and / or other objects in motion except the person; Inputting the first image and the second image captured into a depth estimation model to obtain a first depth of the first image; the second image includes an image captured before the capture time of the first image, and / or an image captured after the capture time of the first image; the input parameters of the depth estimation model include an image sequence, the image sequence includes a current frame, and also includes images captured before the capture time of the current frame and / or images captured after the capture time of the current frame; the depth estimation model is used to output the depth of the current frame based on the motion characteristics of the moving object in each frame of the image sequence; The first image is blurred based on the first depth to obtain a blurred first image; in the blurred first image, the moving object is not blurred.

2. The method according to claim 1, characterized in that The collecting of the first image and the second image in the motion scene comprises: After identifying a shooting scene in which a person jumps, continuously capturing a plurality of images; the plurality of images include a plurality of groups of the first images and the second images; The step of inputting the collected first image and the second image into a depth estimation model to obtain a first depth of the first image includes: Inputting multiple groups of the first images and the second images into a depth estimation model in sequence according to the acquisition order of the multiple images to obtain multiple first depths corresponding to the multiple first images; In the case that the first image includes a jumping character, after obtaining the first depth, the depth of the character in the first image is corrected based on at least one first depth before the first depth and / or at least one first depth after the first depth; after the correction, the depth of the character in the first depth is consistent with the depth of the character in at least one first depth before the first depth and / or the depth of the character in at least one first depth after the first depth.

3. The method according to claim 1, characterized in that The other object is an object that contacts the character and moves with the character; or, the other object is any one of a thrown object or a thrown object of the character.

4. The method according to claim 1, characterized in that: The depth estimation model includes a motion feature extraction module and a depth calculation module; The step of inputting the collected first image and the second image into a depth estimation model to obtain a first depth of the first image includes: The first image and the second image are input into the depth estimation model, and the motion feature extraction module extracts motion features based on image features of the first image and the second image; the motion features are motion features of moving objects in an image sequence formed by the first image and the second image; The depth calculation module estimates the depth of the first image based on the motion feature and outputs the first depth.

5. A method for training a depth estimation model, characterized in that: The method comprises: Acquire training samples; the training samples include multiple groups of training images, one group of training images includes a first training image and a second training image in a motion scene; the second training image includes an image acquired before the acquisition time of the first training image, and / or an image acquired after the acquisition time of the first training image; the motion scene includes a moving object; the moving object includes a person in motion, and / or other objects in motion except the person; Based on the training samples, an initial model is trained to obtain a depth estimation model; wherein, the input parameters of the depth estimation model include an image sequence, the image sequence includes a current frame, and also includes images captured before the acquisition time of the current frame and / or images captured after the acquisition time of the current frame; the depth estimation model is used to output the depth of the current frame based on the motion features of the moving objects in each frame of the image sequence.

6. The method according to claim 5, characterized in that The step of training an initial model based on the training sample to obtain a depth estimation model includes: Inputting the training sample into the depth estimation model, the depth estimation model outputting the estimated depth corresponding to the training image in the training sample; Calculate a first loss; the first loss is the sum of at least one second loss and a third loss; wherein the second loss is used to calculate the difference between the second estimated depth and the third estimated depth of the moving object output by the depth estimation model; wherein the second estimated depth is the depth of the moving object in the estimated depth corresponding to the second training image output by the depth estimation model; the third estimated depth is the depth calculated based on the first estimated depth; the first estimated depth is the depth of the moving object in the estimated depth corresponding to the first training image output by the depth estimation model; The third loss is used to calculate the consistency of the depth of the pixels of the moving object in the first training image; Based on the value of the first loss, the initial model is trained to obtain a depth estimation model.

7. The method according to claim 6, characterized in that The second loss and the third loss are calculated based on the following formula: in, is the depth of the i-th pixel in the moving object in the second training image output by the depth estimation model; is the depth of the i-th pixel in the moving object in the first training image output by the depth estimation model; S2 is the area of ​​the moving object in the second training image; S1 is the area of ​​the moving object in the first training image; It is the depth of the i+1th pixel in the moving object in the first training image output by the depth estimation model, where i takes values ​​in [1, 2, ..., m] in sequence, and m is an integer greater than or equal to 2.

8. The method according to claim 6, characterized in that The method further comprises: Inputting the training sample into an optical flow model, the optical flow model acquiring at least one optical flow between the first training image and the second training image, and outputting the at least one first optical flow map based on the at least one optical flow mapping the first training image; Calculating a fourth loss; the fourth loss is used to calculate the consistency between the depth of the at least one first optical flow map and the estimated depth of the first training image output by the depth estimation model; Based on the value of the first loss, training the initial model to obtain a depth estimation model includes: Based on the value of the first loss and the value of the fourth loss, the initial model is trained to obtain a depth estimation model.

9. The method according to claim 8, characterized in that The fourth loss is calculated based on the following formula: Fourth loss = O 2→1 |D 2→1 -D1|; Among them, D 2→1 is the depth corresponding to the at least one first optical flow map; D1 is the estimated depth of the first training image estimated by the depth estimation model; 2→1 It is a valid area composed of pixels whose positions change between the second training image and the first training image.

10. The method according to any one of claims 6 to 9, characterized in that: The method further comprises: Calculating a fifth loss; the fifth loss is used to calculate the difference between the estimated depth of the first image output by the depth estimation model and the actual depth of the first image; The step of training the initial model based on the value of the first loss to obtain a depth estimation model includes: Based on the value of the first loss, the value of the fourth loss and the value of the fifth loss, the initial model is trained to obtain a depth estimation model.

11. An electronic device, characterized in that: include: One or more processors, one or more memories; the one or more memories are coupled to the one or more processors, and the one or more memories store a computer program; when the one or more processors execute the computer program, the electronic device executes the method described in any one of claims 1-10.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

13. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 10 when being executed by a processor.

Images