Image processing method, device and storage medium

Abstract

An embodiment of the present application provides an image processing method, device and storage medium, which relate to the field of image processing technology. The method includes a terminal device using an image acquisition device to obtain an image to be processed; the image to be processed is a two-dimensional image; the terminal device inputs the image to be processed into a depth estimation model, and the depth estimation model is used to make the depth information of the image to be processed converge to multiple target depth values ​​respectively; the target depth value corresponds to the interval in the image to be processed one-to-one; the interval is related to the depth range of the object in the image to be processed; the depth range of different intervals is different; the terminal device uses the depth estimation model to output the depth information of the image to be processed, so that the target depth values ​​of different intervals in the image to be processed are different, so when the depth estimation model converges the depth information of the image to be processed to multiple different target depth values ​​in different intervals, the depth information of the image to be processed predicted by the depth estimation model is closer to the actual value.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device and storage medium. Background Art

[0002] Terminal devices that use augmented reality (AR) technology can also be called AR devices. AR devices can use monocular depth estimation methods to obtain the depth information of images. The images are images that represent real scenes. AR devices fuse virtual objects with images based on the acquired depth information and display the fused images. Virtual objects can be objects represented by images. For example, virtual objects can be animals, buildings, people, and scenery represented by images. Among them, the monocular depth estimation method is a method of predicting the depth information of each pixel in an image through an image. The depth information is the distance between an object in a real scene and the image acquisition device used to shoot the real scene.

[0003] Currently, the depth information of images obtained by AR devices through monocular depth estimation methods has poor accuracy. As a result, when the AR device fuses virtual objects and images based on the obtained depth information, problems such as the position of the virtual object deviating from the preset position and / or the occlusion boundary between the virtual object and the real object may occur. Summary of the Invention

[0004] The embodiments of the present application provide an image processing method, device, and storage medium, which relate to the field of image processing technology and help improve the accuracy of monocular depth estimation.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, which includes: a terminal device uses an image acquisition device to obtain an image to be processed; the image to be processed is a two-dimensional image; the terminal device inputs the image to be processed into a depth estimation model, and the depth estimation model is used to make the depth information of the image to be processed converge to multiple target depth values ​​respectively; the target depth values ​​correspond one-to-one to intervals in the image to be processed; the intervals are related to the depth range of objects in the image to be processed; the depth ranges of different intervals are different; the terminal device uses the depth estimation model to obtain the depth information of the image to be processed.

[0006] In an embodiment of the present application, the depth estimation model converges the depth information of the image to be processed to multiple target depth values ​​when performing depth prediction on the image, and one target depth value corresponds to an interval in the image, and the depth ranges of different intervals are different. In this way, the depth information of the pixels in the image will not converge to the average value, so that the depth information of the farther pixels in the image and the depth information of the closer pixels obtained according to the depth estimation model of the present application are closer to the true value, thereby helping to improve the accuracy of the terminal device when estimating the depth information in the image using the monocular depth estimation method.

[0007] In one possible implementation, the depth estimation model is obtained by iteratively training a neural network model based on multiple sample data and a loss function until convergence; the sample data includes a sample image, the labeled depth information marked by the sample image, and the number of labeled intervals marked by the sample image; the input of the neural network model includes the sample image; the output of the neural network model includes the predicted depth information of the sample image and the predicted depth feature value of each interval in the sample image; the loss function is used to characterize the similarity between the labeled depth information and the predicted depth information, and is used to characterize the similarity between the predicted depth feature value and the labeled depth feature value; the labeled depth feature value includes multiple first depth values; the first depth value corresponds one-to-one to the first interval in the sample image; the first interval is obtained by dividing the sample image according to the number of labeled intervals.

[0008] In another possible implementation, the neural network model includes a feature extraction model and an interval partitioning network; the input of the feature extraction model includes a sample image; the output of the feature extraction model includes a feature map of the sample image; the feature map is used to characterize the pixel-level features of the sample image; the input of the interval partitioning network is a feature map; the output of the interval partitioning network includes an attention feature map and the number of interval partitions; the attention feature map includes depth-related features in the sample image; the predicted depth information is obtained by inputting the number of interval partitions and the attention feature map into other layers of the neural network model; the other layers are the neural network layers in the neural network model except the feature extraction model and the interval partitioning network; the predicted depth feature values ​​include multiple second depth values; the second depth values ​​correspond one-to-one to the second interval in the sample image; the second interval is obtained by dividing the sample image according to the number of interval partitions.

[0009] In another possible implementation, the feature extraction model includes an encoding structure and a decoding structure. The encoding structure takes a sample image as input and outputs a feature vector representing the semantic information of the sample image. The decoding structure takes a feature vector as input and outputs a feature map. The encoding and decoding structures are connected using skip connections. Skip connections can combine pixel-level feature maps with high-level semantics, reducing feature loss.

[0010] In another possible implementation, the interval partitioning network is a network model based on the attention mechanism.

[0011] In another possible implementation, the method further includes: the terminal device obtains multiple sample data; the terminal device inputs the sample image into the neural network model to obtain predicted depth information of the sample image; the terminal device obtains a predicted depth map of the sample image based on the predicted depth information, and obtains a labeled depth map of the sample image based on the labeled depth information; the terminal device determines a first loss between a first normal vector in the predicted depth map and a corresponding second normal vector in the labeled depth map; the terminal device determines a second loss based on the labeled depth information and the predicted depth information of the sample image; the terminal device determines a third loss based on multiple first depth values ​​in the sample image and multiple second depth values ​​in the sample image; the terminal device weights and aggregates the first loss, the second loss, and the third loss to obtain a total loss; the terminal device iteratively updates the weight parameters of the neural network model based on the total loss to obtain a depth estimation model. In this way, the training of the depth estimation model not only focuses on the loss of the predicted depth information and the labeled depth information, but also focuses on the loss of the normal vector in the predicted depth map and the normal vector in the labeled depth map, as well as the loss of depth values ​​in each interval. Therefore, the depth information obtained by the trained depth estimation model for depth estimation is closer to the true value.

[0012] In another possible implementation, the first depth value is the middle value of the first depth range; the first depth range is the depth range corresponding to the first interval; the second depth value is the middle value of the second depth range; and the second depth range is the depth range corresponding to the second interval. In this way, the middle value of the depth range better reflects the mean value of the depth range, making the depth information predicted in each interval of the image closer to the true value.

[0013] In another possible implementation, before the terminal device inputs the image to be processed into the depth estimation model, the method also includes: the terminal device receives a special effect selection operation; the terminal device displays an icon of at least one virtual object in response to the special effect selection operation; the terminal device receives a trigger operation for the icon of the virtual object to be added; the terminal device marks the virtual object to be added on the interface of the terminal device in response to the trigger operation; the terminal device receives a position selection operation for the target position in the image to be processed; the above-mentioned terminal device uses the depth estimation model to obtain the depth information of the image to be processed, including: the terminal device uses the depth estimation model to obtain the depth information of the image to be processed in response to the position selection operation.

[0014] In another possible implementation, the method further includes: the terminal device fusing the image to be processed, its depth information, the target location, and the virtual object to be added to obtain a target image; the target image is the image after the virtual object to be added is added to the target location of the image to be processed; and the terminal device displays the target image. In this way, because the depth information of the image to be processed is more accurately predicted, the position of the virtual object in the image fused with the virtual object is more precise, and the occlusion boundary between the virtual object and the real object is more accurate.

[0015] In a second aspect, an embodiment of the present application provides an image processing device, comprising a storage module and a processing module; the storage module is used to store a depth estimation model; the depth estimation model is used to make the depth information of a two-dimensional image converge to multiple target depth values ​​respectively; the target depth values ​​correspond one-to-one to intervals in the two-dimensional image; the intervals are related to the depth range of objects in the two-dimensional image; the depth ranges of different intervals are different; the processing module is used to: obtain a two-dimensional image using an image acquisition device; input the two-dimensional image into the depth estimation model; and obtain the depth information of the image to be processed using the depth estimation model.

[0016] Optionally, the depth estimation model is obtained by iteratively training a neural network model based on multiple sample data and a loss function until convergence; the sample data includes a sample image, the labeled depth information marked by the sample image, and the number of labeled intervals marked by the sample image; the input of the neural network model includes the sample image; the output of the neural network model includes the predicted depth information of the sample image and the predicted depth feature values ​​of each interval in the sample image; the loss function is used to characterize the similarity between the labeled depth information and the predicted depth information, and is used to characterize the similarity between the predicted depth feature values ​​and the labeled depth feature values; the labeled depth feature values ​​include multiple first depth values; the first depth value corresponds one-to-one to the first interval in the sample image; the first interval is obtained by dividing the sample image according to the number of labeled intervals.

[0017] Optionally, the neural network model includes a feature extraction model and an interval partitioning network; the input of the feature extraction model includes a sample image; the output of the feature extraction model includes a feature map of the sample image; the feature map is used to characterize the pixel-level features of the sample image; the input of the interval partitioning network is a feature map; the output of the interval partitioning network includes an attention feature map and the number of interval partitions; the attention feature map includes depth-related features in the sample image; the predicted depth information is obtained by inputting the number of interval partitions and the attention feature map into other layers of the neural network model; the other layers are the neural network layers in the neural network model except the feature extraction model and the interval partitioning network; the predicted depth feature values ​​include multiple second depth values; the second depth values ​​correspond one-to-one to the second interval in the sample image; the second interval is obtained by dividing the sample image according to the number of interval partitions.

[0018] Optionally, the feature extraction model includes an encoding structure and a decoding structure; the input of the encoding structure is a sample image; the output of the encoding structure is a feature vector used to represent the semantic information of the sample image; the input of the decoding structure is a feature vector; the output of the decoding structure is a feature map; the encoding structure and the decoding structure are connected using a jump connection.

[0019] Optionally, the interval partitioning network is a network model based on an attention mechanism.

[0020] Optionally: the processing module is also used to obtain multiple sample data; input the sample image into the neural network model to obtain the predicted depth information of the sample image; obtain the predicted depth map of the sample image based on the predicted depth information, and obtain the labeled depth map of the sample image based on the labeled depth information; determine the first loss between the first normal vector in the predicted depth map and the corresponding second normal vector in the labeled depth map; determine the second loss based on the labeled depth information and the predicted depth information of the sample image; determine the third loss based on the multiple first depth values ​​in the sample image and the multiple second depth values ​​in the sample image; weightedly summarize the first loss, the second loss and the third loss to obtain the total loss; iteratively update the weight parameters of the neural network model based on the total loss to obtain a depth estimation model.

[0021] Optionally, the first depth value is the middle value of the first depth range; the first depth range is the depth range corresponding to the first interval; the second depth value is the middle value of the second depth range; and the second depth range is the depth range corresponding to the second interval.

[0022] Optionally, before the terminal device inputs the image to be processed into the depth estimation model, the processing module is also used to: receive a special effect selection operation; display an icon of at least one virtual object in response to the special effect selection operation; receive a trigger operation for the icon of the virtual object to be added; mark the virtual object to be added on the interface of the terminal device in response to the trigger operation; receive a position selection operation for the target position in the image to be processed; the processing module is specifically used to: obtain depth information of the image to be processed using the depth estimation model in response to the position selection operation.

[0023] The processing module is also used to: obtain a target image based on the image to be processed, the depth information of the image to be processed, the target position and the virtual object to be added; the target image is the image after the virtual object to be added is added to the target position of the image to be processed; the image processing device also includes a display module for displaying the target image.

[0024] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the image processing method described in the first aspect or any possible implementation of the first aspect.

[0025] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is run on a computer, the computer executes the image processing method described in the first aspect or any possible implementation of the first aspect.

[0026] In a fifth aspect, an embodiment of the present application provides a computer program product comprising a computer program, which, when the computer program runs on a computer, enables the computer to execute the image processing method described in the first aspect or any possible implementation of the first aspect.

[0027] In a sixth aspect, the present application provides a chip or chip system, comprising at least one processor and a communication interface, wherein the communication interface and the at least one processor are interconnected via a circuit, and the at least one processor is configured to execute a computer program or instruction to perform the image processing method described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip may be an input / output interface, a pin, or a circuit.

[0028] In one possible implementation, the chip or chip system described above in this application further includes at least one memory, in which instructions are stored. The memory may be a storage unit within the chip, such as a register, a cache, etc., or a storage unit of the chip (e.g., a read-only memory, a random access memory, etc.).

[0029] It should be understood that the second to sixth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of the structure of a terminal device applicable to an embodiment of the present application;

[0031] Figure 2 A software structure block diagram of a terminal device to which the embodiments of the present application are applicable;

[0032] Figure 3 A schematic diagram of a learning process of a depth estimation model in an image processing method provided in an embodiment of the present application;

[0033] Figure 4 A flowchart of an image processing method provided in an embodiment of the present application;

[0034] Figure 5 A schematic diagram of a display interface of a terminal device applicable to an embodiment of the present application;

[0035] Figure 6 A schematic diagram of a shooting preview interface of a terminal device applicable to an embodiment of the present application;

[0036] Figure 7 A schematic diagram of an image preview interface of a terminal device applicable to an embodiment of the present application;

[0037] Figure 8A schematic diagram of the operation of selecting a virtual object provided in an embodiment of the present application;

[0038] Figure 9 A schematic diagram of the operation of selecting a location to add a virtual object provided in an embodiment of the present application;

[0039] Figure 10 This is a schematic diagram of the operation of adding AR special effects provided in an embodiment of the present application;

[0040] Figure 11 A flowchart of another image processing method provided in an embodiment of the present application;

[0041] Figure 12 A flowchart of another image processing method provided in an embodiment of the present application;

[0042] Figure 13 A schematic diagram of an image to be processed applicable to an embodiment of the present application;

[0043] Figure 14 A schematic diagram of a feature map of an image to be processed obtained in an embodiment of the present application;

[0044] Figure 15 A schematic diagram of a depth map obtained by the image processing method provided in an embodiment of the present application;

[0045] Figure 16 A schematic diagram of an image processing device provided in an embodiment of the present application;

[0046] Figure 17 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] To facilitate a clear description of the technical solutions of the embodiments of the present application, some of the terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0048] 1) Neural Networks

[0049] A neural network can be composed of neural units. Specifically, it can be understood as a neural network with an input layer, a hidden layer, and an output layer. Generally, the first layer is the input layer, the last layer is the output layer, and all layers in between are hidden layers. A neural network with many hidden layers is called a deep neural network (DNN). The operation of each layer in a neural network can be described mathematically. From a physical perspective, the operation of each layer in a neural network can be understood as transforming the input space (a set of input vectors) to the output space (i.e., from the row space of a matrix to the column space). These five operations include: 1. Dimensionality increase / reduction; 2. Scaling / reduction; 3. Rotation; 4. Translation; and 5. Warping. The goal of training a neural network is to ultimately obtain the weight matrices for all layers of the trained neural network. Therefore, the process of training a neural network is essentially about learning how to control spatial transformations, more specifically, learning the weight matrices.

[0050] It should be noted that in the embodiments of the present application, the learning models (also referred to as learners, models, etc.) used in machine learning tasks (such as active learning, supervised learning, unsupervised learning, semi-supervised learning, etc.) are essentially neural networks.

[0051] 2) Loss function

[0052] In the process of training a neural network, because we want the output of the neural network to be as close as possible to the value we actually want to predict, we can compare the current network's predicted value with the desired target value, and then update the weight matrix of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, which is to pre-configure the parameters for each layer in the neural network). For example, if the network's predicted value is too high, the weight matrix is ​​adjusted to make it predict a lower value, and this adjustment is continued until the neural network can predict the desired target value. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value." This is the loss function or objective function, which is an important equation used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the neural network becomes a process of minimizing this loss as much as possible.

[0053] During neural network training, the back propagation (BP) algorithm can be used to correct the size of the parameters in the initial neural network model, reducing the reconstruction error loss of the neural network model. Specifically, forward propagation of the input signal to the output generates error loss. This error loss information is then backpropagated to update the parameters in the initial neural network model, thereby converging the error loss. The BP algorithm is a backpropagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0054] 3) Features

[0055] A feature is an input variable, i.e. the x variable in a simple linear regression. A simple machine learning task might use a single feature, while a more complex machine learning task might use millions of features.

[0056] 4) Other terms

[0057] In the embodiments of this application, terms such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. For example, the terms "first chip" and "second chip" are used solely to distinguish between different chips and do not define their order. Those skilled in the art will understand that terms such as "first" and "second" do not define the quantity or execution order, and do not necessarily define differences.

[0058] It should be noted that in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0059] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, c can be single or multiple.

[0060] When a terminal device uses depth information estimated using a monocular depth estimation method to fuse a virtual object with an image of a real scene, the position of the virtual object may deviate from its preset position, or the occlusion boundary between the virtual object and a real object in the real scene may be inaccurate. For example, when a user uses a terminal device to take an AR photo, the user selects a dinosaur as the virtual object to be added to the captured image. In the captured image, the occlusion boundary between the dinosaur and the trees in the real scene next to the dinosaur is inaccurate.

[0061] This is because the regression-based algorithm used in current monocular depth estimation tends to converge the depth information to an average value. Therefore, in the depth information of the image estimated by the monocular depth estimation method, the depth information of farther pixels and the depth information of closer pixels are estimated inaccurately, which leads to problems such as position deviation of the virtual object and / or inaccurate occlusion boundary between the virtual object and the real object when the virtual object is fused with the image of the real scene.

[0062] In view of this, an embodiment of the present application provides an image processing method, which can utilize a trained depth estimation model including an interval division network. The depth estimation model obtains depth information of pixels in the image based on the probability that the pixels in the image belong to each interval. Since the depth information of the pixels in the image is divided into possible intervals, the depth information of the pixels in the image is obtained based on the probability that the pixels belong to different intervals, and the depth information of the pixels in the image will not converge to the average value. In this way, the depth information of farther pixels and the depth information of closer pixels in the image are closer to the true value, thereby helping to improve the accuracy of the terminal device when estimating the depth information in the image using the monocular depth estimation method.

[0063] The image processing method provided in the embodiment of the present application can be applied to a terminal device. In the embodiment of the present application, the terminal device can also be referred to as a terminal, user equipment (UE), mobile station (MS) or mobile terminal (MT), etc. The terminal device can be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a desktop computer, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a vehicle-mounted terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, etc. The embodiment of the present application does not limit the specific technology and specific device form adopted by the terminal device.

[0064] In order to better understand the embodiments of the present application, the structure of the terminal device to which the embodiments of the present application are applicable is described below. Figure 1 FIG. 1 is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application. Figure 1 The terminal device 10 shown may include a processor 110, a memory 120, a universal serial bus (USB) interface 130, a power supply 140, a communication module 150, an audio module 170, a sensor module 180, a button 190, a camera 191, and a display screen 160. The sensor module 180 may include a pressure sensor 180A, a fingerprint sensor 180B, a touch sensor 180C, etc.

[0065] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the terminal device 10. In other embodiments of the present application, the terminal device 10 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0066] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a digital signal processor (DSP), a baseband processor, etc. The different processing units may be independent devices or integrated into one or more processors.

[0067] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0068] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0069] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0070] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C bus lines. The processor 110 may be coupled to the touch sensor 180C, the charger, the flash, the camera 191, and the like via different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180C via the I2C interface, enabling communication between the processor 110 and the touch sensor 180C via the I2C bus interface, thereby implementing the touch function of the terminal device 10.

[0071] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface, enabling the function of answering calls through a Bluetooth headset.

[0072] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0073] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface, enabling the function of playing music through Bluetooth headphones.

[0074] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display 160 and the camera 191. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 110 and the camera 191 communicate via the CSI interface to implement the camera function of the terminal device 10. The processor 110 and the display 160 communicate via the DSI interface to implement the display function of the terminal device 10.

[0075] The USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the terminal device 10, or to transfer data between the terminal device 10 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as AR devices.

[0076] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is merely an illustrative illustration and does not constitute a structural limitation on the terminal device 10. In other embodiments of the present application, the terminal device 10 may also adopt a different interface connection method from the above embodiment, or a combination of multiple interface connection methods.

[0077] The power supply 140 supplies power to the terminal device 10 .

[0078] The communication module 150 can use any transceiver or other device for communicating with other devices or communication networks, such as wide area networks (WAN), local area networks (LAN), etc.

[0079] The terminal device 10 implements display functions through a GPU, display screen 160, and an application processor. The GPU is a microprocessor for image processing that connects the display screen 160 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0080] The display screen 160 is used to display images, videos, etc. The display screen 160 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, the terminal device 10 may include one or N display screens 160, where N is a positive integer greater than 1.

[0081] The terminal device 10 can implement the shooting function through the ISP, camera 191, video codec, GPU, display screen 160 and application processor.

[0082] The ISP processes data fed back by camera 191. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 191.

[0083] The camera 191 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 10 may include 1 or N cameras 191, where N is a positive integer greater than 1. Exemplarily, the camera 191 may be used to perform Figure 4 As shown in S400 , an image to be processed is obtained.

[0084] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the terminal device 10 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0085] The memory 120 can be used to store one or more computer programs, which include instructions. The processor 110 can enable the terminal device 10 to perform various functional applications and data processing by running the above instructions stored in the memory 120. The memory 120 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system; the program storage area can also store one or more applications (such as a gallery, contacts, etc.). Exemplarily, the program storage area of ​​the memory 120 can store a depth estimation model trained by the method provided in the embodiment of the present application.

[0086] The data storage area can store data (such as photos, etc.) created during the use of the terminal device 10. In addition, the memory 120 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. In some embodiments, the processor 110 can execute instructions stored in the memory 120 and / or instructions stored in a memory provided in the processor 110 to enable the terminal device 10 to perform various functional applications and data processing.

[0087] The terminal device 10 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0088] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0089] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be located on display screen 160. There are many types of pressure sensors 180A, such as resistive, inductive, and capacitive. A capacitive pressure sensor can include at least two parallel plates made of conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Terminal device 10 determines the intensity of the pressure based on the change in capacitance. When a touch operation is applied to display screen 160, terminal device 10 detects the intensity of the touch operation based on pressure sensor 180A. Terminal device 10 can also calculate the location of the touch based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch location but with different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with an intensity less than a first pressure threshold is applied to a short message application icon, an instruction to view short messages is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to a short message application icon, an instruction to create a new short message is executed.

[0090] The fingerprint sensor 180B is used to collect fingerprints. The terminal device 10 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint answering calls, etc.

[0091] Touch sensor 180C, also known as a "touch device," can be provided on display screen 160. The touch sensor 180C and display screen 160 form a touch screen, also known as a "touch screen." Touch sensor 180C is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to an application processor to determine the type of touch event. Visual output related to the touch operations can be provided via display screen 160. In other embodiments, touch sensor 180C can also be provided on the surface of terminal device 10, at a location different from that of display screen 160.

[0092] The buttons 190 include a power button, a volume button, etc. The buttons 190 may be mechanical buttons or touch buttons. The terminal device 10 may receive key inputs and generate key signal inputs related to user settings and function control of the terminal device 10.

[0093] The software system of the terminal device 10 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present application, the Android system with a layered architecture is used as an example to illustrate the software structure of the terminal device 10. Figure 2The following is a software structure diagram of a terminal device applicable to embodiments of the present application. The layered architecture divides the software system of the terminal device 10 into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system can be divided into five layers: application layer (applications), application framework layer (application framework), Android runtime (Android runtime) and system library, hardware abstract layer (HAL), and kernel layer (kernel).

[0094] The application layer can include a series of application packages, and the application layer runs applications by calling the application programming interface (API) provided by the application framework layer. Figure 5 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, game, etc.

[0095] The application framework layer provides API and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, a database, and the like.

[0096] The window manager is used to manage window programs. It can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, and so on. The content provider is used to store and retrieve data and make this data accessible to applications. This data can include video, images, audio, calls made and received, browsing history and bookmarks, phone books, and so on. The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon can include a view for displaying text and a view for displaying images. The phone manager is used to provide communication functions for the terminal device 10. For example, it manages call status (including connected and hung up). The resource manager provides various resources to applications, such as localized strings, icons, images, layout files, video files, and so on. The notification manager enables applications to display notification information in the status bar. This can be used to convey notification-type messages and can automatically disappear after a short period of time without user interaction. For example, the notification manager is used to notify download completion, message reminders, and so on. The notification manager can also be a notification that appears in the system's top status bar in the form of an icon or scrolling text bar, such as a notification from an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message may be displayed in the status bar, a notification sound may be emitted, the terminal device 10 may vibrate, an indicator light may flash, etc. A database can be used to organize, store, and manage data according to a data structure.

[0097] The Android runtime consists of core libraries and a virtual machine (VM). The Android runtime is responsible for scheduling and management of the Android system. The core libraries consist of two parts: one containing the Java language's callable functions and the other the Android core library. The application layer and the application framework layer run in the VM. The VM executes the Java files in the application and framework layers as binary files. The VM is responsible for managing object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0098] The system library may include multiple functional modules, such as a surface manager, media libraries, a 3D graphics processing library, such as the Open Graphics Library (OpenGL), a 2D graphics engine (such as SGL), and the XX model provided in the embodiments of the present application.

[0099] The surface manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of various common audio and video formats, as well as static image files. The media library supports a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. The 3D graphics processing library implements 3D graphics drawing, rendering, compositing, and layer processing. The 2D graphics engine is the drawing engine for 2D graphics.

[0100] The hardware abstraction layer can contain multiple library modules, such as sensor library modules. The Android system can load the corresponding library modules for the device hardware, thereby enabling the application framework layer to access the device hardware.

[0101] The kernel layer is the layer between hardware and software. The kernel layer is used to drive the hardware to make it work. The kernel layer includes at least display drivers, etc., which are not limited in this embodiment of the application.

[0102] The image processing method provided in the embodiment of the present application includes a learning process of a depth estimation model and a process of estimating image depth based on the depth estimation model.

[0103] Figure 3 A schematic diagram of a learning process of a depth estimation model in an image processing method provided in an embodiment of the present application is provided. Figure 3 The learning process of the depth estimation model shown may include the following steps:

[0104] S300: The terminal device obtains multiple sample data.

[0105] In the embodiment of the present application, the sample data includes a sample image, a depth map of the sample image, and the number of interval divisions of the sample image. The number of interval divisions is the number of sub-ranges after the depth range of the image is divided into multiple sub-ranges.

[0106] In a possible implementation, a plurality of sample images including depth information are obtained from a public data set, the sample images and depth maps of the sample images are obtained, and the number of interval divisions of the sample images is marked.

[0107] Exemplarily, the sample data obtained by the terminal device includes: 25k sample data sampled from a public indoor scene dataset, and 25k sample data sampled from a public outdoor scene dataset. The indoor scene dataset may include the indoor datasets in the New York University Depth Datasets v2 (NYUDv2) and the Yonsei University Digital Image Media Lab datasets (DIML); the outdoor scene dataset includes the Karlsruhe Institute of Technology and Toyota Technological Institute datasets (KITTI) dataset and the outdoor dataset in DIML, and the number of interval divisions of the sample image is annotated to obtain 50k sample data. In this way, obtaining sample data from different public datasets can enrich the scenes included in the sample image.

[0108] S301: The terminal device processes the sample data into a preset size.

[0109] In the embodiment of the present application, the preset size of the image processing in the sample data can be determined according to the resolution of the terminal device to which the depth estimation model is applicable.

[0110] For example, if the resolution of the terminal device to which the depth estimation model is applicable is 640*480 pixels, then the terminal device processes the image in the sample data into 640*480 pixels to obtain a plurality of sample data with a size of 640*480 pixels.

[0111] S302: The terminal device performs data enhancement on the processed multiple sample data.

[0112] In an embodiment of the present application, the data enhancement method includes at least one of rotation, flipping, scaling, cropping, translation, color change, contrast transformation, scale transformation or noise perturbation.

[0113] In a possible implementation, the terminal device applies at least one data enhancement method selected from the group consisting of rotation, flipping, scaling, cropping, translation, color change, contrast transformation, scale transformation, or noise perturbation to the processed sample data to obtain multiple new sample data. The terminal device may also interpolate the obtained new sample data to obtain enhanced sample data.

[0114] It can be understood that after the terminal device performs data enhancement processing on the processed multiple sample data, the amount of sample data increases and the distribution of the sample data is more even.

[0115] S303: The terminal device trains a depth estimation model according to the multiple sample data after data enhancement until convergence.

[0116] In an embodiment of the present application, the depth estimation model includes a feature extraction model and an interval partitioning network. Among them, the feature extraction model includes an encoding structure and a decoding structure. The input of the encoding structure is an image, and the output is a feature vector used to characterize the semantic information of the input image. The input of the decoding structure is the feature vector output by the encoding structure, and the output of the decoding structure is a feature map of the same size as the input image. The feature map is used to characterize the pixel-level features of the input image. Exemplarily, the output of the decoding structure is a feature matrix of 640*480*100.

[0117] In an embodiment of the present application, the coding structure can be constructed based on ResNet, ResNeXt, ResNeSt or Unet. For example, the terminal device can remove the last two layers of ResNet101 to obtain the coding structure.

[0118] In the embodiment of the present application, a skip connection is used between the encoding structure and the decoding structure. Exemplarily, the skip connection can be a full-scale skip connection. The full-scale skip connection can combine pixel-level feature maps with high-level semantics.

[0119] In the embodiment of the present application, the input of the interval partitioning network is the feature map output by the decoding structure, and the output of the interval partitioning network is the attention feature map and the number of interval partitions. The attention feature map includes various features related to depth information in the image.

[0120] In a possible implementation, the interval partitioning network is an attention-based network model. This attention-based network model incorporates an attention model into the neural network structure of the network model. The added attention model can be used to capture the correlation between hidden features in the encoding structure and hidden features in the decoding structure, providing the decoding structure with a mechanism to access specific relevant locations in the input feature vector.

[0121] The network model based on the attention mechanism can obtain image features that are strongly correlated with image depth information, so the use of the attention mechanism can improve the accuracy of depth estimation.

[0122] In a possible implementation, the terminal device trains a depth estimation model based on multiple sample data through the following steps.

[0123] Step 1: The terminal device predicts the depth information of the sample data according to the current depth estimation model to obtain a depth map including the predicted depth information.

[0124] Step 2: The terminal device determines the image loss based on the depth map of the sample image in the sample data and the depth map predicted by the model.

[0125] In the embodiment of the present application, the image loss satisfies the formula: Loss = VNL + α*SSI + β*L BIN Among them, VNL is the normal vector loss value, and VNL satisfies the following formula:

[0126]

[0127] Among them, n i is the predicted normal vector. The predicted normal vector refers to the normal vector obtained by constructing a plane using the first set of points in the predicted depth image of the sample image. ni* is the labeled normal vector. The labeled normal vector refers to the normal vector obtained by constructing a plane using the second set of points in the depth image of the sample data. The first and second point sets are both sets of three-dimensional points corresponding to the same pixel in the sample image. N is the preset number of normal vectors.

[0128] It should be noted that three points are randomly selected in the first point set, the distance between any two points in the three points is greater than or equal to the preset threshold, and the angle between any two sides formed by the selected three points is within the preset range, the three points constitute a plane, and the plane corresponds to a normal vector. In the second point set, three points at the pixel positions corresponding to the three points in the first point set are selected to constitute another plane, and the plane corresponds to the marked normal vector.

[0129] For example, three points randomly obtained from the first point set are PA, PB and PC, PA, PB and PC can be used to construct the first plane, and the normal vector n of the first plane is obtained. i PA1, PB1, and PC1 are the three points in the second point set that correspond to PA, PB, and PC, respectively. PA1 and PA correspond to the same pixel in the sample image, PB1 and PB correspond to the same pixel in the sample image, and PC1 and PC correspond to the same pixel in the sample image. PA1, PB1, and PC1 can be used to construct the second plane and obtain the normal vector ni* of the second plane.

[0130] Formula Loss = VNL + α*SSI + β*L BIN Where α and β are empirical values ​​set based on experience. SSI is the depth loss. SSI satisfies the following formula:

[0131]

[0132] Where N is the number of pixels, di is the depth information of the pixel in the sample image predicted by the current depth estimation model, and di* is the depth information of the corresponding pixel in the depth map. h satisfies the following formula:

[0133]

[0134] in, T, d i The depth information of the pixel of the sample image predicted by the current depth estimation model, d i * is the depth information of the corresponding pixel in the depth map.

[0135] L BIN is the aggregate loss of the intermediate values ​​of the depth information in the interval, L BIN Satisfies the following formula:

[0136] L BIN =chamfer(X,c(b))+chamfer(c(b),X)

[0137] Wherein, X is the set of intermediate depth values ​​of the interval of the sample image predicted by the depth estimation model, and c(b) is the set of intermediate depth values ​​of the interval of the sample image in the sample data.

[0138] For example, the sample image is annotated with the following intervals: [0-1], [1-2], [2-3], [3-4], and [4-5]. The middle depth value of [0-1] is 0.5, the middle depth value of [1-2] is 1.5, and so on. The set of middle depth values ​​in the intervals of the sample image is (0.5, 1.5, 2.5, 3.5, and 4.5). The predicted number of interval divisions of the sample image includes [0-0.5], [0.5-1], [1-1.5], [1.5-2], [2-2.5], [2.5-3], [3-3.5], [3.5-4], [4-4.5], [4.5-5], and the set of intermediate depth values ​​in the intervals divided by the predicted number of interval divisions is (0.25, 0.75, 1.25, 1.75, 2.25, 2.75, 3.25, 3.75, 4.25, 4.75).

[0139] Step 3: The terminal device backpropagates the determined image loss in the current depth estimation model to update the parameters in the model.

[0140] It is understandable that the terminal device repeatedly executes the above steps 1 to 3 according to multiple different sample data until the depth estimation model converges.

[0141] It should be noted that the model convergence condition in the embodiment of the present application is preset. Exemplarily, the model convergence condition may be that the image loss is less than a loss threshold.

[0142] After the terminal device has trained the depth estimation model, it can perform depth estimation of the monocular image based on the depth estimation model.

[0143] It should be noted that the terminal device used to train the depth estimation model can be a computer, such as a laptop or desktop computer, etc. The embodiment of the present application can also use a server to train the depth estimation model, which is not limited in the embodiment of the present application.

[0144] The trained depth estimation model can be transplanted into a mobile phone to perform depth estimation of a monocular image, or can be transplanted into a vehicle-mounted terminal to perform depth estimation of a monocular image.

[0145] like Figure 4 FIG. 1 is a flow chart of an image processing method provided in an embodiment of the present application. Figure 4 The image processing method shown includes the following steps:

[0146] S400: The terminal device obtains the image to be processed.

[0147] In the embodiment of the present application, the image to be processed may be any two-dimensional image acquired by the image acquisition device of the terminal device.

[0148] In one possible implementation, the terminal device detects a touch operation of a user's finger (or stylus, etc.) on an icon for acquiring an image to be processed, and in response to the touch operation, the terminal device opens a shooting preview interface corresponding to the icon, and the acquired image to be processed is displayed in the preview interface of the terminal device.

[0149] For example, when the terminal device detects that the user's finger touches Figure 5 After the operation of the camera icon 501 shown, in response to the user's finger touching the camera icon 501, the camera application is opened and the shooting preview interface is entered. For example, the preview interface displayed by the terminal device can be specifically Figure 6 Preview interface 502 is shown. Preview interface 502 includes an image to be processed 502A in a preview frame. Preview interface 502 also includes an AR special effect control 502B and a capture button 502C. AR special effect control 502B is used to add AR special effects to the image to be processed, and capture button 502C is used to trigger the terminal device to capture the image in the current preview frame, or to trigger the terminal device to start or stop video capture.

[0150] Here, in conjunction with the capture and photo shooting scene, the workflow of the software and hardware of the terminal device 10 is exemplified. When the touch sensor 180C receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into an original input operation (including touch coordinates, timestamp of the touch operation, and other information). The original input operation is stored in the kernel layer. The application framework layer obtains the original input operation from the kernel layer and identifies the control corresponding to the original input operation. Taking the touch operation as a single-click operation as an example, and the control corresponding to the single-click operation is a camera application icon as an example, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture a static image or video through the camera 191.

[0151] In another possible implementation, the terminal device receives a trigger operation from the user on an editing control of the image to be processed, and displays the image to be processed in a preview box of the interface in response to the trigger operation.

[0152] Exemplarily, the terminal device receives a trigger operation of the user on the editing control of the image to be processed, and responds to the trigger operation in the following manner: Figure 7 The preview frame of the preview interface 503 shown displays an image 503A to be processed. The image 503A includes a person 503A1 and trees 503A2 in the background. The preview interface 503 also includes an AR effect control 503B and a save button 503D. The AR effect control 503B is used to add AR effects to the image to be processed, and the save button 503D is used to trigger the terminal device to save the image in the current preview frame.

[0153] S401: The terminal device receives a selection operation of a virtual object, and determines a virtual object to be added in response to the selection operation.

[0154] In the embodiment of the present application, the virtual object may be a cartoon image, a virtual animal, a virtual person, etc.

[0155] In a possible implementation, the terminal device receives a special effect selection operation and displays at least one virtual object in the interface in response to the special effect selection operation. The terminal device receives a user's selection trigger operation on a virtual object icon and determines the virtual object to be added corresponding to the selection trigger operation in response to the selection trigger operation.

[0156] For example, Figure 8The terminal device shown receives a user's selection operation on AR special effect control 503B in preview interface 503. In response to the special effect selection operation, the virtual object icons displayed in the preview interface include a first virtual object icon 503B1 and a second virtual object icon 503B2. The terminal device receives the user's selection trigger operation on first virtual object icon 503B1 and, in response to the selection trigger operation, determines that the virtual object to be added corresponding to the selection trigger operation is a dinosaur.

[0157] S402: The terminal device determines the position of the virtual object to be added in the image to be processed.

[0158] In the embodiment of the present application, the position of the virtual object to be added in the image to be processed may be the coordinates of the virtual object to be added in the image coordinate system of the image to be processed.

[0159] In a possible implementation, the terminal device receives a user's triggering operation on a location for adding a virtual object to be added, and determines the location of the virtual object to be added in the image to be processed in response to the triggering operation.

[0160] For example, Figure 9 The terminal device shown receives the user's Figure 9 A touch operation is shown on a position L1 of the image to be processed 503A in the preview frame of the preview interface 503 , and the coordinates of the position L1 in the image coordinate system are determined in response to the touch operation.

[0161] S403: The terminal device estimates depth information of the image to be processed according to the depth estimation model.

[0162] In a possible implementation, the depth estimation model obtains the depth information of the image to be processed through the following steps:

[0163] Step 1: The image to be processed is input into the depth estimation model and then passes through the encoder of the feature extraction model to obtain a feature vector representing the semantic information of the image to be processed.

[0164] Exemplarily, a 640*480 image to be processed is input into the encoder of the feature extraction model to obtain a 7*7*1024 feature vector.

[0165] Step 2: The feature vector representing the semantic information of the image to be processed is passed through the decoder of the feature extraction model to obtain a feature map of the same size as the image to be processed.

[0166] Step 3: After the feature map output in step 2 passes through the interval partitioning network, it outputs the attention feature map and the median value of the depth information in each interval of the image to be processed.

[0167] Step 4: The remaining layers of the depth estimation model obtain the median value of the depth information in each interval and the probability that each pixel in the image to be processed belongs to each interval according to the number of interval divisions of the image to be processed and the attention feature map. Then, the depth information of the image to be processed is obtained based on the median value of the depth information in each interval and the probability that each pixel in the image to be processed belongs to each interval.

[0168] For example, if the number of intervals of the image to be processed is 5, the median values ​​of the depth information in each interval of the image to be processed are 0.5, 1.5, 2.5, 3.5 and 4.5. The probability that the pixel of the image to be processed belongs to the interval corresponding to the median value 0.5 is 0.8, and the probability that it belongs to the interval corresponding to the other median values ​​is 0.05. Then, the depth estimation model predicts that the depth information of the pixel of the image to be processed is: 0.5*0.8+1.5*0.05+2.5*0.05+3.5*0.05+4.5*0.05=1m. The depth information of the remaining pixels of the image to be processed can also be obtained in this way, which will not be repeated here.

[0169] It should be noted that the probability of each pixel belonging to each interval may be different.

[0170] In the embodiment of the present application, during the training process of the depth estimation model used in depth prediction of the image to be processed, not only the loss of the normal vector between the predicted depth map and the depth map of the sample image, and the loss of the depth information between the predicted depth information and the depth map are considered, but also the loss of the set of intermediate depth values ​​of each interval after the interval partitioning network divides the depth information of the image to be processed into intervals is considered. Therefore, when the depth estimation model trained according to the method provided in the embodiment of the present application is used to predict the depth of the image to be processed, the depth information of the image to be processed can be divided into multiple intervals according to the feature map of the image to be processed that affects the depth information, and the depth information of the pixel point estimated based on the probability that each pixel point belongs to each interval is closer to the true depth information.

[0171] S404: The terminal device performs fusion according to the image to be processed, the depth information of the image to be processed, the position of the virtual object to be added in the image to be processed, and the virtual object to be added to obtain a target image.

[0172] In a possible implementation, the terminal device determines the occlusion relationship between the object in the image to be processed and the virtual object to be added based on the depth information of the image to be processed, the virtual object to be added, and the position of the virtual object to be added in the image to be processed, and fuses the image to be processed and the virtual object to be added into a target image.

[0173] Based on the example of the virtual object to be added selected in S401 and the position of the virtual object to be added selected in S402 in the image to be processed, as shown in FIG. Figure 10 The terminal device shown receives a selection trigger operation of the user on the first virtual object icon 503B1, and in response to the selection trigger operation, determines that the virtual object to be added corresponding to the selection trigger operation is a dinosaur. The terminal device receives a touch operation of the user on the position L1 of the image to be processed 503A in the preview box of the preview interface 503, and in response to the touch operation, determines the coordinates of the position L1 in the image coordinate system. Then, the terminal device displays the target image after the image to be processed 503A and the virtual object to be added corresponding to the first virtual object icon 503B1 are fused.

[0174] In the embodiment of the present application, since the depth information of the image to be processed estimated by the terminal device according to the depth estimation model in S403 is closer to the real depth information, the occlusion boundary between the virtual object and the real object in the target image fused by the terminal device is more accurate.

[0175] like Figure 11 FIG. 1 is a flow chart of another image processing method provided in an embodiment of the present application. Figure 11 The image processing method shown can be applied to terminal equipment. Figure 11 The image processing method shown includes the following steps:

[0176] S110: Build a dataset.

[0177] For possible implementation methods and examples, please refer to the description in S300-S302 and will not be repeated here.

[0178] S111: Build a feature extraction model based on convolutional neural network.

[0179] In the embodiment of the present application, the feature extraction model refers to the relevant description of the feature extraction model in S303 and will not be repeated here.

[0180] S112: Constructing an interval partitioning network.

[0181] In the embodiment of the present application, the interval division network refers to the relevant description of the interval division network in S303 and is not repeated here.

[0182] S113: Use the data set to train the entire network model.

[0183] In the embodiment of the present application, the data set is the data set constructed in S110. The entire network model is a depth estimation model including a feature extraction model and an interval partitioning network.

[0184] S114: Use the trained model to perform depth estimation on the input image to obtain a depth map.

[0185] In the embodiment of the present application, the trained model is a depth estimation model obtained by training the depth estimation model using a data set until convergence.

[0186] For possible implementation methods, please refer to the description in S403 and will not be repeated here.

[0187] In the embodiment of the present application, during the training process of the depth estimation model used for depth estimation of the input image, not only the loss of the normal vector of the depth map of the predicted depth map and the sample image, and the loss of the depth information of the predicted depth information and the depth map are considered, but also the loss of the set of intermediate depth values ​​of each interval after the interval partitioning network divides the depth information of the input image into intervals is considered. Therefore, when the depth estimation model trained according to the method provided in the embodiment of the present application performs depth prediction on the input image, the depth information of the input image can be divided into multiple intervals according to the feature map of the input image that affects the depth information, and the depth information of the pixel point estimated based on the probability that each pixel point belongs to each interval is closer to the true depth information.

[0188] like Figure 12 FIG. 1 is a flow chart of another image processing method provided in an embodiment of the present application. Figure 12 The image processing method shown can be applied to terminal equipment. Figure 12 The image processing method shown includes the following steps:

[0189] S120: Acquire the image to be processed.

[0190] In a possible implementation, the terminal device obtains an image to be processed of the environment in which the terminal device is located, which is captured by an image capture device.

[0191] In another possible implementation, the terminal device reads the image to be processed from a gallery.

[0192] For example, the image to be processed is Figure 13 shown.

[0193] S121: The image to be processed is input into the trained feature extraction model to obtain a feature map.

[0194] In the embodiment of the present application, the trained feature extraction model is a feature extraction model in a depth estimation model that has been trained to convergence. The feature map is the output of the feature extraction model.

[0195] Based on the example of the image to be processed in S120, the feature map obtained is as follows Figure 14 shown.

[0196] S122: According to the trained interval division network, the number of interval divisions and the probability that each pixel in the image to be processed belongs to each interval are obtained.

[0197] In the embodiment of the present application, the trained interval partitioning network is an interval partitioning network in a depth estimation model that is trained to convergence.

[0198] In a possible implementation, the feature map input trained interval partitioning network can output the attention feature map and the number of interval partitions, and predict the probability of each pixel in the image to be processed belonging to each interval based on the attention feature map and the number of interval partitions.

[0199] S123: Obtain depth information of the image to be processed according to the number of interval divisions and the probability that each pixel in the image to be processed belongs to each interval.

[0200] For example, if the number of intervals of the image to be processed is 5, the median values ​​of the depth information in each interval of the image to be processed are 0.5, 1.5, 2.5, 3.5, and 4.5. The probability that a pixel of the image to be processed belongs to the interval corresponding to the median value 0.5 is 0.8, and the probability that it belongs to the interval corresponding to the other median values ​​is 0.05. Then, the depth information of the pixel is: 0.5*0.8+1.5*0.05+2.5*0.05+3.5*0.05+4.5*0.05=1m. The depth information of the remaining pixels of the image to be processed is obtained in a similar manner and will not be repeated here.

[0201] S124: Outputting a depth map of the image to be processed according to the depth information of the image to be processed.

[0202] For example, the depth map of the image to be processed is as follows: Figure 15 shown.

[0203] In an embodiment of the present application, the depth information of the image to be processed can be divided into multiple intervals based on the feature map that affects the depth information of the image to be processed, and the feature extraction model for obtaining the feature map that affects the depth information and the interval division network for dividing the depth information of the image to be processed into multiple intervals are both networks in the depth estimation model trained to convergence. Therefore, the number of interval divisions obtained and the probability of the pixels of the image to be processed in each interval are closer to the actual situation. Therefore, the depth information of the pixel obtained based on the probability of the pixel in each interval is closer to the actual depth information, and the depth map obtained is more accurate.

[0204] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of method. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily appreciate that, in combination with the method steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0205] In the embodiment of the present application, the functional modules of the device for implementing the image processing method can be divided according to the above-mentioned method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. For example, the functions of the target application, the drawing interface, and the display engine are integrated into the display control unit. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0206] like Figure 16 FIG. 1 is a schematic structural diagram of an image processing device provided in an embodiment of the present application. Figure 16 The terminal device 1600 shown includes a storage module 16001 and a processing module 16002. The storage module 16001 is used to store a depth estimation model; the depth estimation model is used to make the depth information of the two-dimensional image converge to multiple target depth values; the target depth value corresponds to the interval in the two-dimensional image; the interval is related to the depth range of the object in the two-dimensional image; the depth range of different intervals is different; the processing module 16002 is used to: use an image acquisition device to obtain a two-dimensional image; input the two-dimensional image into the depth estimation model; and use the depth estimation model to obtain the depth information of the image to be processed. For example, combined with Figure 3 The storage module 16001 can be used to store the depth estimation model. The processing module 16002 can be used to execute S300 to S303. Figure 4 , the processing module 16002 can be used to execute S400 to S404. Figure 12 , the processing module 16002 can be used to execute S120 to S124.

[0207] Optionally, the depth estimation model is obtained by iteratively training a neural network model based on multiple sample data and a loss function until convergence; the sample data includes a sample image, the labeled depth information marked by the sample image, and the number of labeled intervals marked by the sample image; the input of the neural network model includes the sample image; the output of the neural network model includes the predicted depth information of the sample image and the predicted depth feature values ​​of each interval in the sample image; the loss function is used to characterize the similarity between the labeled depth information and the predicted depth information, and is used to characterize the similarity between the predicted depth feature values ​​and the labeled depth feature values; the labeled depth feature values ​​include multiple first depth values; the first depth value corresponds one-to-one to the first interval in the sample image; the first interval is obtained by dividing the sample image according to the number of labeled intervals.

[0208] Optionally, the neural network model includes a feature extraction model and an interval partitioning network; the input of the feature extraction model includes a sample image; the output of the feature extraction model includes a feature map of the sample image; the feature map is used to characterize the pixel-level features of the sample image; the input of the interval partitioning network is a feature map; the output of the interval partitioning network includes an attention feature map and the number of interval partitions; the attention feature map includes depth-related features in the sample image; the predicted depth information is obtained by inputting the number of interval partitions and the attention feature map into other layers of the neural network model; the other layers are the neural network layers in the neural network model except the feature extraction model and the interval partitioning network; the predicted depth feature values ​​include multiple second depth values; the second depth values ​​correspond one-to-one to the second interval in the sample image; the second interval is obtained by dividing the sample image according to the number of interval partitions.

[0209] Optionally, the feature extraction model includes an encoding structure and a decoding structure; the input of the encoding structure is a sample image; the output of the encoding structure is a feature vector used to represent the semantic information of the sample image; the input of the decoding structure is a feature vector; the output of the decoding structure is a feature map; the encoding structure and the decoding structure are connected using a jump connection.

[0210] Optionally, the interval partitioning network is a network model based on an attention mechanism.

[0211] Optional: processing module 16002 is also used to obtain multiple sample data; input the sample image into the neural network model to obtain predicted depth information of the sample image; obtain a predicted depth map of the sample image based on the predicted depth information, and obtain a labeled depth map of the sample image based on the labeled depth information; determine a first loss between a first normal vector in the predicted depth map and a corresponding second normal vector in the labeled depth map; determine a second loss based on the labeled depth information and the predicted depth information of the sample image; determine a third loss based on multiple first depth values ​​in the sample image and multiple second depth values ​​in the sample image; weightedly summarize the first loss, the second loss and the third loss to obtain a total loss; iteratively update the weight parameters of the neural network model based on the total loss to obtain a depth estimation model.

[0212] Optionally, the first depth value is the middle value of the first depth range; the first depth range is the depth range corresponding to the first interval; the second depth value is the middle value of the second depth range; and the second depth range is the depth range corresponding to the second interval.

[0213] Optionally, the processing module 16002 is also used to: receive a special effect selection operation; display an icon of at least one virtual object in response to the special effect selection operation; receive a trigger operation for the icon of the virtual object to be added; mark the virtual object to be added on the interface of the terminal device in response to the trigger operation; receive a position selection operation for the target position in the image to be processed; the processing module 16002 is specifically used to: obtain depth information of the image to be processed using a depth estimation model in response to the position selection operation.

[0214] The processing module 16002 is also used to: obtain a target image based on the image to be processed, the depth information of the image to be processed, the target position and the virtual object to be added; the target image is the image after the virtual object to be added is added to the target position of the image to be processed; the image processing device 1600 also includes a display module 16001, which is used to display the target image.

[0215] In one example, combining Figure 1 The function of the storage module 16001 can be Figure 1 The memory 120 shown in FIG. 1 is implemented, and the function of the processing module 16002 can be realized by Figure 1 The processor 110 is shown to call the computer program stored in the memory 120. The function of the display module 16003 can be implemented by the display screen 160.

[0216] like Figure 17 FIG2 is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip 1700 includes one or more (including two) processors 17001, a communication line 17002 and a communication interface 17003. Optionally, the chip 1700 also includes a memory 17004.

[0217] In some embodiments, memory 17004 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.

[0218] The method described in the above embodiment of the present application can be applied to the processor 17001, or implemented by the processor 17001. The processor 17001 may be an integrated circuit chip with signal processing capabilities. During the implementation process, the various steps of the above method can be completed by the hardware integrated logic circuit in the processor 17001 or the instructions in the form of software. The above-mentioned processor 17001 can be a general-purpose processor (for example, a microprocessor or a conventional processor), a digital signal processor (digital signal processing, DSP), an application specific integrated circuit (application specific integrated circuit, ASIC), a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates, transistor logic devices or discrete hardware components. The processor 17001 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application.

[0219] The steps of the method disclosed in the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be implemented by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as a random access memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable read-only memory (EEPROM). The storage medium is located in the memory 17004, and the processor 17001 reads the information in the memory 17004 and completes the steps of the above method in combination with its hardware.

[0220] The processor 17001 , the memory 17004 , and the communication interface 17003 can communicate with each other via the communication line 17002 .

[0221] In the above embodiment, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product, wherein the computer program product may be pre-written in the memory or downloaded and installed in the memory in the form of software.

[0222] The present application also provides a computer program product including one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated. For example, the available medium can include magnetic media (e.g., floppy disk, hard disk or tape), optical media (e.g., digital versatile disc (DVD)), or semiconductor media (e.g., solid state disk (SSD)).

[0223] An embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform any one of the above-mentioned image processing methods.

[0224] The present application also provides a computer-readable storage medium. The methods described in the above embodiments can be implemented in whole or in part via software, hardware, firmware, or any combination thereof. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one location to another. The storage medium can be any target medium that can be accessed by a computer.

[0225] As one possible design, computer-readable media may include compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM or other optical disc storage; computer-readable media may include magnetic disk storage or other magnetic disk storage devices. Moreover, any connecting line may also be appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave are included in the definition of medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically using lasers.

[0226] The above combinations should also be included within the scope of computer-readable media. The above description is only a specific embodiment of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by any person skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An image processing method, characterized in that: The method comprises: The terminal device uses an image acquisition device to obtain an image to be processed; the image to be processed is a two-dimensional image; The terminal device inputs the image to be processed into a depth estimation model, wherein the depth estimation model is used to make the depth information of the image to be processed converge to a plurality of target depth values; the target depth values ​​correspond to intervals in the image to be processed; the intervals are related to the depth range of objects in the image to be processed; and the depth ranges of different intervals are different; The terminal device obtains depth information of the image to be processed using the depth estimation model; The depth estimation model is obtained by iteratively training a neural network model according to a plurality of sample data and a loss function until convergence; The sample data includes a sample image, annotation depth information annotated on the sample image, and the number of annotation intervals annotated on the sample image; The input of the neural network model includes the sample image; the output of the neural network model includes the predicted depth information of the sample image and the predicted depth feature value of each interval in the sample image; The loss function is used to characterize the similarity between the annotated depth information and the predicted depth information, and is used to characterize the similarity between the predicted depth feature value and the annotated depth feature value; the annotated depth feature value includes a plurality of first depth values; the first depth values ​​correspond one-to-one to a first interval in the sample image; the first interval is obtained by dividing the sample image according to the number of the annotated intervals; The neural network model includes a feature extraction model and an interval partitioning network; The input of the feature extraction model includes the sample image; the output of the feature extraction model includes a feature map of the sample image; the feature map is used to characterize pixel-level features of the sample image; The input of the interval partitioning network is the feature map; the output of the interval partitioning network includes an attention feature map and the number of interval partitions; the attention feature map includes depth-related features in the sample image; The predicted depth information is obtained by inputting the interval division number and the attention feature map into other layers of the neural network model; the other layers are neural network layers in the neural network model excluding the feature extraction model and the interval division network; The predicted depth feature value includes a plurality of second depth values; the second depth values ​​correspond one-to-one to a second interval in the sample image; the second interval is obtained by dividing the sample image according to the number of interval divisions; The interval partitioning network is a network model based on the attention mechanism.

2. The image processing method according to claim 1, wherein: The feature extraction model includes an encoding structure and a decoding structure; the input of the encoding structure is the sample image; the output of the encoding structure is a feature vector for representing the semantic information of the sample image; The input of the decoding structure is the feature vector; the output of the decoding structure is the feature map; The encoding structure and the decoding structure are connected using a skip connection.

3. The image processing method according to claim 1, wherein: The method further comprises: The terminal device acquires the plurality of sample data; The terminal device inputs the sample image into the neural network model to obtain predicted depth information of the sample image; The terminal device obtains a predicted depth map of the sample image according to the predicted depth information, and obtains a labeled depth map of the sample image according to the labeled depth information; The terminal device determines a first loss between a first normal vector in the predicted depth map and a corresponding second normal vector in the annotated depth map; The terminal device determines a second loss according to the annotated depth information and the predicted depth information of the sample image; The terminal device determines a third loss according to the plurality of first depth values ​​in the sample image and the plurality of second depth values ​​in the sample image; The terminal device performs weighted aggregation of the first loss, the second loss, and the third loss to obtain a total loss; The terminal device iteratively updates the weight parameters of the neural network model according to the total loss to obtain the depth estimation model.

4. The image processing method according to claim 3, wherein: The first depth value is a middle value of a first depth range; the first depth range is the depth range corresponding to the first interval; The second depth value is a middle value of the second depth range; the second depth range is the depth range corresponding to the second interval.

5. The image processing method according to any one of claims 1 to 4, characterized in that: Before the terminal device inputs the image to be processed into the depth estimation model, the method further includes: The terminal device receives a special effect selection operation; The terminal device displays an icon of at least one virtual object in response to the special effect selection operation; The terminal device receives a triggering operation for an icon of a virtual object to be added; The terminal device marks the virtual object to be added on the interface of the terminal device in response to the trigger operation; The terminal device receives a position selection operation for a target position in the image to be processed; The terminal device obtains depth information of the image to be processed by using the depth estimation model, including: The terminal device obtains depth information of the image to be processed using the depth estimation model in response to the position selection operation.

6. The image processing method according to claim 5, characterized in that The method further comprises: The terminal device obtains a target image by fusing the image to be processed, the depth information of the image to be processed, the target position, and the virtual object to be added; the target image is an image after the virtual object to be added is added to the target position of the image to be processed; The terminal device displays the target image.

7. An image processing device, characterized in that including a storage module and a processing module; The storage module is configured to store a depth estimation model; the depth estimation model is configured to cause depth information of a two-dimensional image to converge to a plurality of target depth values; the target depth values ​​correspond to intervals in the two-dimensional image in a one-to-one manner; the intervals are related to the depth range of an object in the two-dimensional image; and the depth ranges of different intervals are different; The processing module is configured to: acquire the two-dimensional image using an image acquisition device; input the two-dimensional image into the depth estimation model; and acquire depth information of the two-dimensional image using the depth estimation model; The depth estimation model is obtained by iteratively training a neural network model according to a plurality of sample data and a loss function until convergence; The sample data includes a sample image, annotation depth information annotated on the sample image, and the number of annotation intervals annotated on the sample image; The input of the neural network model includes the sample image; the output of the neural network model includes the predicted depth information of the sample image and the predicted depth feature value of each interval in the sample image; The loss function is used to characterize the similarity between the annotated depth information and the predicted depth information, and is used to characterize the similarity between the predicted depth feature value and the annotated depth feature value; the annotated depth feature value includes a plurality of first depth values; the first depth values ​​correspond one-to-one to a first interval in the sample image; the first interval is obtained by dividing the sample image according to the number of the annotated intervals; The neural network model includes a feature extraction model and an interval partitioning network; The input of the feature extraction model includes the sample image; the output of the feature extraction model includes a feature map of the sample image; the feature map is used to characterize pixel-level features of the sample image; The input of the interval partitioning network is the feature map; the output of the interval partitioning network includes an attention feature map and the number of interval partitions; the attention feature map includes depth-related features in the sample image; The predicted depth information is obtained by inputting the interval division number and the attention feature map into other layers of the neural network model; the other layers are neural network layers in the neural network model excluding the feature extraction model and the interval division network; The predicted depth feature value includes a plurality of second depth values; the second depth values ​​correspond one-to-one to a second interval in the sample image; the second interval is obtained by dividing the sample image according to the number of interval divisions; The interval partitioning network is a network model based on the attention mechanism.

8. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the image processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, which, when executed, enable a computer to execute the image processing method according to any one of claims 1 to 6.

Images