Image processing method and device, terminal equipment, storage medium and product

Abstract

The present disclosure relates to an image processing method, device, terminal equipment, storage medium and product. The method comprises: obtaining a first image, wherein the first image is a monocular image; performing at least one down-sampling processing on the first image to obtain at least one second image; wherein the at least one second image and the first image constitute an image pyramid; inputting the image pyramid into a depth estimation model to obtain a third image, wherein the third image is a depth image of the first image. Through the method of the present disclosure, a depth image with more accurate depth information can be obtained.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image processing method, apparatus, terminal device, storage medium, and product. Background Technology

[0002] Depth information in images is crucial for various applications, including background blurring, image segmentation, and object detection in captured images and videos, making it one of the most important pieces of information in an image. Currently, depth information from monocular images is primarily obtained using neural network models. However, the accuracy of depth information obtained through neural network models is insufficient. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides an image processing method, apparatus, terminal device, storage medium, and product that can obtain depth images with more accurate depth information.

[0004] According to a first aspect of the present disclosure, an image processing method is provided, the method comprising: acquiring a first image, wherein the first image is a monocular image; performing at least one downsampling process on the first image to obtain at least one second image; wherein the at least one second image and the first image constitute an image pyramid; and inputting the image pyramid into a depth estimation model to obtain a third image, wherein the third image is a depth image of the first image.

[0005] In some embodiments, the depth estimation model includes a first module and a second module; wherein, inputting an image pyramid into the depth estimation model to obtain a third image includes: inputting the image pyramid into the first module for structural feature extraction to obtain a first feature image; wherein the number of first feature images is the same as the number of second images; and inputting the first feature images into the second module for depth feature extraction to obtain a third image.

[0006] In some embodiments, the first module includes a first submodule and a second submodule; wherein, inputting the image pyramid into the first module for structural feature extraction to obtain a first feature image includes: inputting the first image into the first submodule for structural feature extraction to obtain a second feature image; and inputting the second feature image and at least one second image into the second submodule for structural feature extraction to obtain the first feature image.

[0007] In some embodiments, the second submodule includes N first units, and the number of second images is N, where N is a positive integer; wherein, inputting the second feature image and at least one second image into the second submodule for row structure feature extraction to obtain a first feature image includes: stitching a third feature image and a second image of the same size as the three feature images in the channel dimension to obtain a first stitched image, wherein the different second images have different sizes, and the third feature image is either the first feature image or the second feature image output by the (i-1)th first unit; inputting the first stitched image into the i-th first unit for structure feature extraction to obtain the first feature image output by the i-th first unit, where i is an integer from 1 to N.

[0008] In some embodiments, each of the N first units includes a first convolution sub-unit, a spatial aggregation sub-unit, and a channel aggregation sub-unit; wherein, inputting the first stitched image into the i-th first unit for structural feature extraction to obtain the first feature image output by the i-th first unit includes: inputting the first stitched image into the first convolution sub-unit included in the i-th first unit for convolution processing to obtain a fourth feature image; inputting the fourth feature image into the spatial aggregation sub-unit included in the i-th first unit for spatial feature aggregation to obtain a fifth feature image; and inputting the fifth feature image into the channel aggregation sub-unit included in the i-th first unit for channel feature aggregation to obtain the first feature image output by the i-th first unit.

[0009] In some embodiments, the second module includes a third submodule and a fourth submodule; wherein, inputting the first feature image into the second module for depth feature extraction to obtain the third image includes: inputting the first feature image into the third submodule for depth feature extraction to obtain a sixth feature image; and inputting the sixth feature image into the fourth submodule for image generation to obtain the third image.

[0010] In some embodiments, the number of first feature images is multiple; the third submodule includes M second units, where M is a positive integer; wherein, inputting the first feature image into the third submodule for depth feature extraction to obtain the sixth feature image includes: stitching the seventh feature image and the first feature image of the same size as the seventh feature image in the channel dimension to obtain a second stitched image, wherein the seventh feature image is a feature image obtained by upsampling the last first feature image output by the first module or the eighth feature image output by the (j-1)th second unit; inputting the second stitched image into the jth second unit for depth feature extraction to obtain the eighth feature image output by the jth second unit, wherein j is an integer from 1 to M; and determining the eighth feature image output by the Mth second unit as the sixth feature image.

[0011] In some embodiments, the first image includes at least one frame of a monocular image from a monocular video stream.

[0012] According to a second aspect of the present disclosure, an image processing apparatus is provided, the apparatus comprising: an acquisition module configured to acquire a first image, wherein the first image is a monocular image; a processing module configured to perform at least one downsampling process on the first image to obtain at least one second image; wherein the at least one second image and the first image constitute an image pyramid; and an input module configured to input the image pyramid into a depth estimation model to obtain a third image, wherein the third image is a depth image of the first image.

[0013] In some embodiments, the depth estimation model includes a first module and a second module; wherein the input module is configured to: input an image pyramid into the first module for structural feature extraction to obtain a first feature image; wherein the number of first feature images is the same as the number of second images; and input the first feature images into the second module for depth feature extraction to obtain a third image.

[0014] In some embodiments, the first module includes a first submodule and a second submodule; wherein the input module is configured to: input a first image into the first submodule for structural feature extraction to obtain a second feature image; and input the second feature image and at least one second image into the second submodule for structural feature extraction to obtain a first feature image.

[0015] In some embodiments, the second submodule includes N first units, and the number of second images is N, where N is a positive integer; wherein, the input module is configured to: stitch the third feature image and the second image with the same size as the three feature images in the channel dimension to obtain a first stitched image, wherein different second images have different sizes, and the third feature image is either the first feature image or the second feature image output by the (i-1)th first unit; input the first stitched image into the i-th first unit for structural feature extraction to obtain the first feature image output by the i-th first unit, where i is an integer from 1 to N.

[0016] In some embodiments, each of the N first units includes a first convolution sub-unit, a spatial aggregation sub-unit, and a channel aggregation sub-unit; wherein, the input module is configured to: input the first stitched image into the first convolution sub-unit included in the i-th first unit for convolution processing to obtain a fourth feature image; input the fourth feature image into the spatial aggregation sub-unit included in the i-th first unit for spatial feature aggregation to obtain a fifth feature image; and input the fifth feature image into the channel aggregation sub-unit included in the i-th first unit for channel feature aggregation to obtain the first feature image output by the i-th first unit.

[0017] In some embodiments, the second module includes a third submodule and a fourth submodule; wherein the input module is configured to: input the first feature image into the third submodule for depth feature extraction to obtain a sixth feature image; and input the sixth feature image into the fourth submodule for image generation to obtain a third image.

[0018] In some embodiments, the number of first feature images is multiple; the third submodule includes M second units, where M is a positive integer; wherein, the input module is configured to: stitch the seventh feature image and the first feature image with the same size as the seventh feature image in the channel dimension to obtain a second stitched image, wherein the seventh feature image is a feature image obtained by upsampling the last first feature image output by the first module or the eighth feature image output by the (j-1)th second unit; input the second stitched image into the jth second unit for depth feature extraction to obtain the eighth feature image output by the jth second unit, wherein j is an integer from 1 to M; and determine the eighth feature image output by the Mth second unit as the sixth feature image.

[0019] In some embodiments, the first image includes at least one frame of a monocular image from a monocular video stream.

[0020] According to a third aspect of the present disclosure, a terminal device is provided, comprising: a processor; and a memory for storing computer programs or instructions; wherein the processor executes the computer programs or instructions to implement the steps of the method described in the first aspect.

[0021] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, which stores executable instructions or a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0022] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program or instructions, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0023] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0024] In this disclosure, performing at least one downsampling process on the first image yields an image pyramid composed of the first and second images. Since the image pyramid comprises multiple images of different sizes, inputting the image pyramid into a depth estimation model allows for feature extraction from images of different sizes, resulting in more depth features in the first image and thus making the depth information in the obtained depth image more accurate.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0027] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment.

[0028] Figure 2 This is a flowchart illustrating an image processing method according to another exemplary embodiment.

[0029] Figure 3 This is a flowchart illustrating a model training method according to an exemplary embodiment.

[0030] Figure 4 This is a schematic diagram of the structure of a first submodule according to an exemplary embodiment.

[0031] Figure 5 This is a schematic diagram of the structure of a first unit according to an exemplary embodiment.

[0032] Figure 6 This is a schematic diagram of the structure of a spatial aggregate subunit according to an exemplary embodiment.

[0033] Figure 7 This is a schematic diagram of the structure of a channel aggregation subunit according to an exemplary embodiment.

[0034] Figure 8 This is a schematic diagram illustrating the structure of a fourth submodule according to an exemplary embodiment.

[0035] Figure 9 This is a schematic diagram of the structure of a second unit according to an exemplary embodiment.

[0036] Figure 10 This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment.

[0037] Figure 11 This is a structural block diagram of a terminal device according to an exemplary embodiment. Detailed Implementation

[0038] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0039] Depth information in images is crucial for various applications, including background blurring, image segmentation, and object detection in captured images and videos. Currently, depth information from monocular images is primarily obtained using neural network models. However, the accuracy of depth information acquired through neural network models is insufficient. Furthermore, existing neural network models for obtaining depth information from monocular images typically have a large number of parameters, making them difficult to deploy on terminal devices and thus hindering the timely acquisition of depth information from monocular images captured by these devices.

[0040] To address the aforementioned problems, this disclosure provides an image processing method. This method can be executed by a terminal device. Exemplarily, the terminal device executing the image processing method provided in this disclosure can be a mobile terminal or a fixed terminal. The mobile terminal may include smartphones, tablets, laptops, digital cameras, and wearable electronic devices, etc. The fixed terminal may include desktop computers, all-in-one computers, etc.

[0041] Figure 1 This is a schematic flowchart illustrating an image processing method according to an exemplary embodiment. In the following, a terminal device will be used as the execution subject, with reference to... Figure 1 The steps shown illustrate the image processing method provided in this disclosure. This image processing method mainly includes the following steps:

[0042] In step 101, the first image is acquired.

[0043] Understandably, the terminal device can acquire a monocular image of the target object and designate the acquired monocular image as the first image. Alternatively, the terminal device can also acquire a monocular image from an image acquisition device connected to the terminal device and use the obtained monocular image as the first image.

[0044] In some embodiments, a monocular image can be understood as an image captured by a single camera.

[0045] In some embodiments, the first image may be represented using color information from any color space, and this disclosure does not limit this.

[0046] For example, the first image can be represented using color information in the LAB color space; the first image can also be represented using color information in the RGB color space. In LAB, L stands for Lightness, A for Red-Green Axis, and B for Yellow-Blue Axis; in RGB, R stands for Red, G for Green, and B for Blue.

[0047] In step 102, the first image is downsampled at least once to obtain at least one second image.

[0048] Understandably, after obtaining the first image through step 101, the terminal device can perform at least one downsampling process on the first image. At this time, the terminal device can obtain at least one second image.

[0049] In some embodiments, at least one second image and the first image together constitute an image pyramid.

[0050] In some embodiments, the image pyramid includes multiple layers, each layer being an image, and the size of the image in each layer from the bottom to the top of the image pyramid is larger than the size of the adjacent layer above it.

[0051] For example, the size of the image in each layer of the image pyramid is two, three, five, or so times the size of the image in the adjacent layer above.

[0052] In some embodiments, the number of times the terminal device downsamples the first image can be determined as needed, and this disclosure does not limit this.

[0053] For example, the number of times the terminal device performs downsampling processing on the first image can be any value from 2 to 5. For instance, the number of times the terminal device performs downsampling processing on the first image can be 2 times, 3 times, 5 times, etc.

[0054] In some embodiments, after obtaining the first image, the terminal device may perform a downsampling process on the first image to obtain a first second image. Then, the terminal device may perform a downsampling process on the first second image to obtain a second second image. After obtaining the second second image, the terminal device may perform a downsampling process on the second second image to obtain a third second image. Thus, the first image, the first second image, the second second image, and the third second image can constitute a four-layer image pyramid.

[0055] For example, Figure 2 This is a flowchart illustrating an image processing method according to another exemplary embodiment. For example... Figure 2As shown, after obtaining the first image 201, the electronic device can downsample the first image 201 to obtain the second image 202. After obtaining the first second image 202, the electronic device can downsample the first second image 202 to obtain the second second image 203. After obtaining the second second image 203, the electronic device can downsample the second second image 203 to obtain the third second image 204. The size of the first second image 202 is half the size of the first image 201, the size of the second second image 203 is half the size of the first second image 202, and the size of the third second image 204 is half the size of the second second image 203. At this time, the first image 201, the first second image 202, the second second image 203, and the third second image 204 can form a four-layer image pyramid.

[0056] In step 103, the image pyramid is input into the depth estimation model to obtain the third image.

[0057] Understandably, after obtaining the image pyramid in step 102, the terminal device can input the first and second images included in the image pyramid into the trained depth estimation model to obtain the third image output by the trained depth estimation model. At this time, the terminal device can use the obtained third image as the depth image of the first image.

[0058] In some embodiments, the electronic device can store the depth value of each pixel in the first image as the grayscale value of that pixel. In this case, the third image displayed on the screen by the electronic device is a grayscale image with the same size as the first image.

[0059] For example, the third image obtained by the electronic device can be Figure 2 The depth image 205 shown.

[0060] In some embodiments, the depth estimation model can be any type of lightweight neural network model, and this disclosure does not limit this.

[0061] For example, the depth estimation module can be a MobileNet model, a ShuffleNet model, a SqueezeNet model, etc.

[0062] In some embodiments, the number of parameters in the depth estimation model is less than a parameter threshold.

[0063] In some embodiments, the parameter threshold can be any value from 0.1 megabytes (M) to 10 megabytes (M). For example, the parameter threshold can be 0.1M, 1M, 3M, 5M, 8M, 10M, etc.

[0064] In some embodiments, before using the trained depth estimation model, the terminal device may first train the depth estimation model and input the image pyramid composed of the first image and the second image into the trained depth estimation model to obtain the depth image of the first image.

[0065] In some embodiments, the terminal device can simulate a real-world scene using a simulation engine to generate a simulated video sequence of at least one scene, wherein the simulated images included in the simulated video sequence have depth labels. The terminal device can then use the simulated images included in the obtained simulated video sequence and the depth labels of the simulated images as a training set to train a depth estimation model, thereby obtaining a trained depth estimation model.

[0066] In some embodiments, the terminal device may also perform data augmentation on the simulated video sequence, including simulated images, and train a depth estimation model based on the data-augmented simulated images.

[0067] In some embodiments, the terminal device may enhance the number of simulated images included in the simulated video sequence in ways including but not limited to: rotating, cropping, and color-changing the simulated images included in the simulated video sequence.

[0068] For example, Figure 3 This is a flowchart illustrating a model training method according to an exemplary embodiment. In the following text, reference will be made to... Figure 3 The training process of the depth estimation model is further described below:

[0069] In step 301, the terminal device generates a simulated video sequence with dense depth labels through a simulation engine.

[0070] In step 302, the terminal device performs data augmentation on the simulated images included in the simulated video sequence to obtain data-augmented simulated images.

[0071] In step 303, the terminal device trains the depth estimation model based on the data-augmented simulated image and the depth label of the simulated image through supervised learning to obtain the trained depth estimation model.

[0072] In some embodiments, if each pixel of the simulated image has a depth label or the number of pixels with depth labels in the simulated image is greater than or equal to the total number of pixels in the simulated image, then the simulated image is determined to have dense depth labels.

[0073] In this embodiment of the disclosure, compared to training the depth estimation model using images of the acquired real scene, generating training data through a simulation engine can reduce the difficulty of obtaining depth labels. Furthermore, using a simulation engine to generate training data can also increase the diversity of the training data, thereby enabling the trained depth estimation model to have stronger generalization ability.

[0074] In this embodiment, the depth estimation model uses a parameter count less than a parameter count threshold. Therefore, this depth estimation model is a lightweight neural network model with fewer parameters, which can be deployed on a terminal device. This enables real-time processing of monocular images acquired by the terminal device, allowing for the timely acquisition of depth images from the monocular images.

[0075] Furthermore, by performing at least one downsampling process on the first image, an image pyramid composed of the first and second images can be obtained. Since the image pyramid includes multiple images of different sizes, inputting the image pyramid into the depth estimation model allows for feature extraction from images of different sizes, thereby obtaining more depth features from the first image and making the depth information included in the resulting depth image more accurate.

[0076] In some embodiments, the depth estimation model includes a first module and a second module. Step 103 includes: inputting the image pyramid into the first module for structural feature extraction to obtain a first feature image; and inputting the first feature image into the second module for depth feature extraction to obtain a third image.

[0077] In some embodiments, the number of first feature images is the same as the number of second images.

[0078] Understandably, the depth estimation model includes a first module and a second module. After obtaining an image pyramid composed of the first image and the second image, the electronic device can input the image pyramid into the first module to extract structural features from the images included in the image pyramid. Based on this, the terminal device can obtain a first feature image output by the first module. After obtaining the first feature image, the electronic device can input the first feature image into the second module for depth feature extraction, and generate a depth image of the first image based on the depth features extracted from the first feature image.

[0079] In some embodiments, after the electronic device inputs a second image into the first module, the first module outputs a first feature image for each second image. Therefore, the electronic device can obtain the same number of first feature images as the number of second images.

[0080] In some embodiments, the size of each first feature image output by the first module is the same as the size of a second image in the image pyramid.

[0081] In some embodiments, the structural features extracted by the electronic device from the first image included in the image pyramid by the first module include, but are not limited to, edge features, texture features, shape features, color features, gradient features, etc.

[0082] In some embodiments, since the number of first feature images output by the first module may be multiple, after obtaining multiple first feature images, the electronic device may first perform upsampling processing on at least a portion of the first feature images, and then input the first feature images into the second module.

[0083] For example, the electronic device may upsample the last first feature image output by the first module so that the size of the last first feature image is the same as the size of the penultimate first feature image output by the first module.

[0084] In some embodiments, the first module includes a first submodule and a second submodule; wherein, inputting the image pyramid into the first module for structural feature extraction to obtain a first feature image includes: inputting the first image into the first submodule for structural feature extraction to obtain a second feature image; and inputting the second feature image and at least one second image into the second submodule for structural feature extraction to obtain the first feature image.

[0085] Understandably, the first module of the depth estimation model can include a first submodule and a second submodule. Based on this, the electronic device can input the first image included in the image pyramid into the first submodule to extract structural features from the first image. The electronic device can then obtain a second feature image output by the first submodule. After obtaining the second feature image, the electronic device can input the second feature image and all the second images included in the image pyramid into the second submodule to extract structural features. The electronic device can then obtain a first feature image output by the second submodule.

[0086] In some embodiments, the size of the second feature image output by the first submodule is the same as the size of a second image in the image pyramid.

[0087] For example, the size of the second feature map output by the first submodule is the same as the size of the second image, which is the largest in the image pyramid.

[0088] In some embodiments, the first submodule may include a convolution unit and a downsampling unit. The convolution unit is used to perform convolution processing on the first image, and the downsampling unit is used to perform downsampling processing on the output of the convolution unit.

[0089] In some embodiments, the number of convolutional units, the size of the convolutional kernel, the number of downsampling units, and the downsampling algorithm used by the downsampling units in the first submodule can all be determined as needed, and the embodiments disclosed herein do not limit these aspects.

[0090] For example, Figure 4 This is a schematic diagram illustrating the structure of a first submodule according to an exemplary embodiment. For example... Figure 4 As shown, the first submodule may include two convolutional units (see reference). Figure 4 The first convolutional unit 401 and the second convolutional unit 403 in the system and a downsampling unit 402.

[0091] In some embodiments, the convolution kernel used by the first convolution unit 401 and the convolution kernel used by the second convolution unit 403 may be the same or different, and this disclosure does not limit this.

[0092] For example, the convolution kernels sampled by the first convolution unit and the second convolution unit can both be 3×3 convolution kernels, 4×4 convolution kernels, 5×5 convolution kernels, etc.

[0093] In some embodiments, the downsampling algorithm used by the downsampling unit 402 includes, but is not limited to: bilinear downsampling algorithm, trilinear downsampling algorithm, interlaced sampling algorithm, random sampling algorithm, Gaussian downsampling algorithm, etc.

[0094] In some embodiments, based on Figure 4 In the first submodule shown, after the electronic device inputs the first image into the depth estimation model, it can perform convolution processing on the first image through the first convolution unit 401 to obtain the ninth feature image. Then, the electronic device can perform bilinear downsampling processing on the ninth feature image through the downsampling unit 402 to obtain a tenth feature image with a smaller size than the ninth feature image. Afterwards, the electronic device performs convolution processing on the tenth feature image through the second convolution unit 403. At this point, the electronic device can obtain the second feature image.

[0095] In some embodiments, the step size used by the first convolution unit 401 and the second convolution unit 403 when performing convolution operations on the input image can be determined as needed, and this disclosure does not limit this.

[0096] For example, the step size used by the first convolution unit 401 and the second convolution unit 403 when performing convolution operations on the input image can be 1.

[0097] In some embodiments, after obtaining the second feature image, the electronic device can search for a second image with the same size as the second feature image from the image pyramid. Then, the electronic device can stitch the second feature image and the second image with the same size along the channel dimension to obtain a first stitched image. Next, the electronic device can input the obtained first stitched image into a second submodule for structural feature extraction, at which point the electronic device can obtain the first first feature image output by the second submodule. After obtaining the first first feature image, the electronic device can stitch the first first feature image and the second image with the same size as the first first feature image to obtain a second stitched image. After obtaining the second first stitched image, the electronic device can input the second first stitched image into the second submodule for structural feature extraction to obtain the second first feature image output by the second submodule. Then, the electronic device can stitch together the second first feature image and the second image with the same size as the second first feature image, and input the stitched third first stitched image into the second sub-module, repeating the above process until the number of first feature images is the same as the number of second images included in the image pyramid.

[0098] For example, suppose the image pyramid includes three second images, located at layers 2 through 4 of the image pyramid. Based on this, after obtaining the second feature image, the electronic device can stitch the second feature image with the second image at layer 2 of the image pyramid (the size of the second feature image is the same as the size of the second image at layer 2) to obtain a first stitched image. Then, the electronic device can input the first stitched image into a second submodule for structural feature extraction to obtain the first first feature image output by the second submodule. Next, the electronic device can stitch the first first feature image with the second image at layer 3 (the size of the first first feature image is the same as the size of the second image at layer 3) to obtain a second stitched image. Then, the electronic device can input the second stitched image into the second submodule for structural feature extraction to obtain a second first feature image. After obtaining the second first feature image, the electronic device can stitch the second first feature image with the second image of the fourth layer (the size of the second first feature image is the same as the size of the second image of the fourth layer) to obtain the third stitched first image. Then, the electronic device can input the third stitched first image into the second submodule for structural feature extraction to obtain the third first feature image. In this way, the electronic device can obtain three first feature images.

[0099] In some embodiments, the second submodule includes N first units, and the number of second images is N; N is a positive integer. Inputting the second feature image and at least one second image into the second submodule for row structure feature extraction to obtain a first feature image includes: stitching a third feature image and a second image of the same size as the third feature image along the channel dimension to obtain a first stitched image; inputting the first stitched image into the i-th first unit for structure feature extraction to obtain the first feature image output by the i-th first unit, where i is an integer from 1 to N.

[0100] In some embodiments, the different second images have different sizes, and the third feature image is either the first feature image or the second feature image output by the (i-1)th first unit.

[0101] Understandably, the second submodule in the depth estimation model can include N first units. Based on this, the electronic device can perform N downsampling operations on the first image to obtain N second images, each with a different size. After obtaining the N second images, for the i-th first unit among the N first units, if i is 1, the electronic device can use the second feature image as the third feature image and determine a second image with the same size as the second feature image from the N second images. Then, the electronic device can stitch the second feature image and the second image with the same size along the channel dimension to obtain a first stitched image. Next, the electronic device can input the first stitched image into the second submodule to extract structural features from the first first unit within the second submodule. Based on this, the electronic device can obtain the first feature image output by the first first unit. If i is greater than 1, the electronic device can obtain the first feature image output by the (i-1)-th first unit and use the first feature image output by the (i-1)-th first unit as the third feature image. Next, the electronic device can determine a second image from N second images that has the same size as the first feature image output by the (i-1)th first unit. Then, the electronic device can stitch the first feature image output by the (i-1)th first unit and the second image with the same size in the channel dimension to obtain a first stitched image. After obtaining the first stitched image, the electronic device can input the first stitched image into the i-th first unit included in the second submodule for structural feature extraction to obtain the first feature image output by the i-th first unit. Based on the above steps, the electronic device can obtain the first feature image output by each of the N first units.

[0102] In some embodiments, the specific value of N can be determined as needed, and this disclosure does not limit this.

[0103] For example, the value of N can be any integer from 2 to 10. For instance, the value of N can be 2, 3, 5, 7, 9, 10, etc.

[0104] In some embodiments, such as Figure 2 As shown, the second submodule includes three first units (see reference). Figure 2 (Units a, b, and c in the module). After obtaining the second feature image output by the first submodule 206, the electronic device can stitch the second feature image and the second image 202 together in the channel dimension to obtain a first stitched image. Then, the electronic device can input the first stitched image into unit a of the second submodule 207 for structural feature extraction to obtain the first feature image output by unit a. After obtaining the first feature image output by unit a, the electronic device stitches the first feature image output by unit a with the second image 203 together in the channel dimension to obtain a second stitched image. Then, the electronic device can input the second stitched image into unit b for structural feature extraction to obtain the first feature image output by unit b. After obtaining the first feature image output by unit b, the electronic device stitches the first feature image output by unit b with the second image 204 together in the channel dimension to obtain a third stitched image. Then, the electronic device can input the third stitched image into unit c for structural feature extraction to obtain the first feature image output by unit c. Based on this, the electronic device can obtain three first feature images.

[0105] It should be noted that the size of the second image 202 is the same as the size of the second feature image; the size of the second image 203 is the same as the size of the first feature image output by unit a; and the size of the second image 204 is the same as the size of the first feature image output by unit b.

[0106] In some embodiments, stitching two images along the channel dimension can be understood as merging the two images along the channel dimension. For example, if image A has dimensions (H, W, C1) and image B has dimensions (H, W, C2), the stitched image obtained by stitching images A and B along the channel dimension is (H, W, C1+C2). H is the height of images A and B, W is the width of images A and B, C1 is the number of channels in image A, and C2 is the number of channels in image B.

[0107] In the embodiments of this disclosure, stitching together the first feature image and the second image, or stitching together the second feature image and the second image, can enrich the features in the resulting first stitched image, thereby enabling the first feature image generated based on the first stitched image to contain more features.

[0108] Furthermore, stitching the first feature image and the second image together in the channel dimension, or stitching the second feature image and the second image together, will not destroy the spatial structure of the first feature image and the second feature image, thus allowing more features to be retained in the first stitched image obtained after stitching.

[0109] In some embodiments, each first unit includes a first convolution sub-unit, a spatial aggregation sub-unit, and a channel aggregation sub-unit; inputting the first stitched image into the i-th first unit for structural feature extraction to obtain a first feature image output by the i-th first unit includes: inputting the first stitched image into the first convolution sub-unit included in the i-th first unit for convolution operation to obtain a fourth feature image; inputting the fourth feature image into the spatial aggregation sub-unit for spatial feature aggregation to obtain a fifth feature image; and inputting the fifth feature image into the channel aggregation sub-unit for channel feature aggregation to obtain a first feature image output by the i-th first unit.

[0110] Understandably, each of the N first units in the depth estimation model includes a first convolutional subunit, a spatial aggregation subunit, and a channel aggregation subunit. The following description, using the i-th first unit as an example, illustrates the process by which each first unit outputs a first feature image based on the input first stitched image. After obtaining the first stitched image, the electronic device can first input it into the first convolutional subunit for convolution processing to obtain the fourth feature image output by the first convolutional subunit. Then, the electronic device can input the fourth feature image into the spatial aggregation subunit to aggregate the features in the fourth feature image along the spatial dimension, obtaining the fifth feature image output by the spatial aggregation subunit. After obtaining the fifth feature image, the electronic device can input it into the channel aggregation subunit to aggregate the features along the channel dimension. Based on this, the electronic device can obtain the first feature image output by the channel aggregation subunit. At this point, the electronic device can use the first feature image output by the channel aggregation subunit as the first feature image output by the i-th first unit.

[0111] In some embodiments, the electronic device may perform multiple spatial feature aggregations on the fourth feature image through a spatial aggregation subunit.

[0112] In some embodiments, spatial feature aggregation can be the aggregation of spatial features in a fourth feature image.

[0113] In some embodiments, the spatial features of an image include, but are not limited to: local features, global features, multi-scale features (such as pyramid features, dilated convolution features), structural features, and high-level semantic features.

[0114] For example, local features include, but are not limited to, the edges, corners, and textures of an image.

[0115] For example, global features include, but are not limited to, the shape and spatial distribution of various objects in the image.

[0116] For example, multi-scale features include, but are not limited to, pyramid features, dilated convolution features, etc.

[0117] In some embodiments, channel feature aggregation can be the aggregation of features in the fifth feature image output from each channel.

[0118] In some embodiments, the number of first convolution sub-units, the number of spatial aggregation sub-units, and the number of channel aggregation sub-units included in each first unit can be determined as needed, and this disclosure does not limit this.

[0119] For example, refer to Figure 5 , Figure 5 This is a schematic diagram illustrating the structure of a first unit according to an exemplary embodiment. For example... Figure 5 As shown, the first unit may include a first convolution subunit 501, a spatial aggregation subunit 502, and a channel aggregation subunit 503.

[0120] In some embodiments, the convolution kernel used by the first convolution sub-unit and the step size used during convolution can be set as needed, and the embodiments disclosed herein do not limit this.

[0121] For example, the step size used by the first convolution sub-unit when performing convolution can be 2, and the size of the convolution kernel used by the first convolution sub-unit 501 can be 3×3, 4×4, 5×5, etc.

[0122] In some embodiments, the number of spatial aggregation sub-units included in each first unit may also be determined as needed, and this disclosure does not limit this.

[0123] In some embodiments, the spatial aggregation subunit may include: convolutional layers, batch normalization layers, activation layers, and skip connections, etc.

[0124] For example, refer to 6. Figure 6 This is a schematic diagram illustrating the structure of a spatially aggregated subunit according to an exemplary embodiment. For example... Figure 6As shown, the spatial aggregation subunit includes a first convolutional layer 601, a first batch normalization layer 602, a second convolutional layer 603, a first activation layer 604, a third convolutional layer 605, and a first skip connection 606. The first convolutional layer 601 is a 3×3 dilated convolutional layer, the second convolutional layer 603 and the third convolutional layer 605 are both 1×1 convolutional layers, and the activation function used by the first activation layer 604 can be the ReLU activation function. Figure 6 The spatial aggregation subunit is shown in the diagram. After obtaining the fourth feature image output by the first convolutional subunit in the i-th first unit, the electronic device can input the fourth feature image into the spatial aggregation subunit of the i-th first unit to process the fourth feature image sequentially through the first convolutional layer 601, the first batch normalization layer 602, the second convolutional layer 603, the first activation layer 604, and the third convolutional layer 605. At this time, the electronic device can obtain the eleventh feature image. After obtaining the eleventh feature image, the electronic device can perform an addition operation on the fourth feature image and the eleventh feature image through the first skip connection 606 to obtain the fifth feature image.

[0125] In some embodiments, the channel aggregation subunit may include: pooling layer, convolutional layer, batch normalization layer, activation layer, etc.

[0126] For example, Figure 7 This is a schematic diagram illustrating the structure of a channel aggregation subunit according to an exemplary embodiment. For example... Figure 7 As shown, the channel aggregation subunit includes a pooling layer 701, a fourth convolutional layer 702, a second activation layer 703, a fifth convolutional layer 704, a third activation layer 705, a second batch normalization layer 706, a sixth convolutional layer 707, a fourth activation layer 708, an eighth convolutional layer 709, a second skip connection 710, and a third skip connection 711. Pooling layer 701 performs global pooling on the input fifth feature image. The fourth convolutional layer 702, the fifth convolutional layer 704, the sixth convolutional layer 707, and the eighth convolutional layer 709 are all 1×1 convolutional layers. The activation function used by the second activation layer 703 and the fourth activation layer 708 can be the ReLU activation function. The activation function used by the third activation layer can be the Sigmoid activation function.

[0127] based on Figure 7The diagram shows a channel aggregation subunit. After obtaining the fifth feature image output by the spatial aggregation subunit in the i-th first unit, the electronic device can input the fifth feature image into the channel aggregation subunit of the i-th first unit for processing sequentially through pooling layer 701, fourth convolutional layer 702, second activation layer 703, fifth convolutional layer 704, and third activation layer 705. At this point, the electronic device can obtain the twelfth feature image. After obtaining the twelfth feature image, the electronic device can perform an addition operation on the twelfth feature image and the fifth feature image through the second skip connection 710 to obtain the thirteenth feature image. Then, the electronic device can sequentially process the thirteenth feature image through the second batch normalization layer 706, sixth convolutional layer 707, fourth activation layer 708, and eighth convolutional layer 709 to obtain the fourteenth feature image. Then, the electronic device can perform an addition operation on the fourteenth feature image and the fifth feature image through the third skip connection 711. At this point, the electronic device can obtain the first feature image and use the obtained first feature image as the first feature image output by the i-th first unit.

[0128] In this embodiment, the combination of spatial aggregation subunits and channel aggregation subunits effectively extracts depth information from the image, thereby improving the accuracy of the extracted depth information of the first image. Furthermore, aggregating features in the fifth feature image along the channel dimension allows the depth estimation model to learn the dependencies between the fifth feature images output by each channel, thus enhancing the features in important channels.

[0129] In some embodiments, the second module includes a third submodule and a fourth submodule. Inputting the first feature image into the second module for depth feature extraction to obtain a third image includes: inputting the first feature image into the third submodule for depth feature extraction to obtain a sixth feature image; and inputting the sixth feature image into the fourth submodule for image generation to obtain the third image output by the fourth submodule.

[0130] Understandably, after obtaining the first feature image output by the first module, the electronic device can input the obtained first feature image into the third sub-module for depth feature extraction to obtain the sixth feature image output by the third sub-model. After obtaining the sixth feature image, the electronic device can input the sixth feature image into the fourth sub-module for image generation. At this point, the electronic device can obtain the third image output by the fourth sub-module.

[0131] In some embodiments, since the number of first feature images output by the first module may be multiple, after obtaining multiple first feature images, the electronic device may first perform upsampling processing on at least a portion of the first feature images, and then input the multiple first feature images into the third sub-module.

[0132] For example, the electronic device may upsample the first feature image output by the Nth first unit so that the size of the first feature image output by the Nth first unit is the same as the size of the first feature image output by the (N-1)th first unit.

[0133] In some embodiments, the fourth submodule may include a convolution unit, an upsampling unit, and an activation unit.

[0134] It should be noted that the number of convolutional units, upsampling units, and activation units included in the fourth sub-module, as well as the size of the convolutional kernel used by the convolutional unit, the upsampling algorithm used by the upsampling unit, and the activation function used by the activation unit, can all be set as needed, and this embodiment does not limit them.

[0135] For example, Figure 8 This is a schematic diagram illustrating the structure of a fourth submodule according to an exemplary embodiment. For example... Figure 8 As shown, the fourth submodule may include a third convolutional unit 801, a first upsampling unit 802, a fourth convolutional unit 803, a fifth convolutional unit 804, a second upsampling unit 805, and an activation unit 806. Based on this, after obtaining the sixth feature image, the electronic device can input the sixth feature image into the fourth submodule, where it is processed sequentially by the third convolutional unit 801, the first upsampling unit 802, the fourth convolutional unit 803, the fifth convolutional unit 804, the second upsampling unit 805, and the activation unit 806. Based on this, the electronic device can obtain the third image output by the fourth submodule.

[0136] In some embodiments, each convolutional unit in the fourth submodule may include at least one convolutional layer. For example, the third convolutional unit 801 and the fourth convolutional unit 803 in the fourth submodule may each include a convolutional layer with a kernel size of 3×3. The fifth convolutional unit may include at least one convolutional kernel with a kernel size of 1×1.

[0137] In some embodiments, the upsampling algorithm used by the upsampling unit in the fourth submodule includes, but is not limited to: nearest neighbor difference algorithm, bilinear difference algorithm, bicubic difference algorithm, etc.

[0138] In some embodiments, the activation function used by the activation unit 806 includes, but is not limited to, the Sigmoid function, the Softmax function, etc.

[0139] In this embodiment, the electronic device performs two convolution processes on the sixth feature image using a convolution kernel, which allows for the extraction of more features from the sixth feature image. Multiple upsampling processes on the sixth feature image using the first and second upsampling units gradually increase the size of the output image, reducing the probability of noticeable jagged edges or artifacts in the third image. Furthermore, the electronic device performs convolution processing on the feature image output by the fourth convolution unit using a 1×1 convolution kernel, which adjusts the number of output channels of the feature image, reduces redundant features, and improves the computational efficiency of the depth estimation model.

[0140] In some embodiments, the number of first feature images is multiple; the third submodule includes M second units, where M is a positive integer. Inputting the first feature images into the third submodule for depth feature extraction to obtain the sixth feature image includes: stitching the seventh feature image and the first feature image of the same size as the seventh feature image along the channel dimension to obtain a second stitched image; inputting the second stitched image into the j-th second unit for depth feature extraction to obtain the eighth feature image output by the j-th second unit; and determining the eighth feature image output by the M-th second unit as the sixth feature image.

[0141] In some embodiments, the seventh feature image is a feature image obtained by upsampling the last first feature image output by the first module or the eighth feature image output by the (j-1)th second unit.

[0142] In some embodiments, j takes the value of an integer from 1 to M.

[0143] Understandably, the first module outputs multiple first feature images. The third sub-module in the depth estimation model may include M second units, where M is a positive integer. Based on this, for the j-th second unit among the M second units, if j is 1, after obtaining the last first feature image output by the first module, the electronic device upsamples the last first feature image to obtain a seventh feature image with the same size as the penultimate first feature image output by the first module. Then, the electronic device can stitch the seventh feature image and the first feature image with the same size as the seventh feature image along the channel dimension to obtain a second stitched image. After obtaining the second stitched image, the electronic device can input it into the first second unit for depth feature extraction to obtain the eighth feature image output by the first second unit. If j is greater than 1, the electronic device can obtain the eighth feature image output by the (j-1)-th second unit and determine the first feature image with the same size as the eighth feature image output by the (j-1)-th second unit from the multiple first feature images output by the first module. Subsequently, the electronic device can stitch the eighth feature image output by the (j-1)th second unit and a first feature image of the same size as the eighth feature image output by the (j-1)th second unit along the channel dimension to obtain a second stitched image. After obtaining the second stitched image, the electronic device can input the second stitched image into the j-th second unit for depth feature extraction to obtain the eighth feature image output by the j-th second unit. Based on the above steps, the electronic device can obtain the eighth feature image output by each of the M second units. At this point, the electronic device can input the eighth feature image output by the M-th second unit as the sixth feature image into the fourth submodule.

[0144] In some embodiments, the number of second units included in the third submodule can be determined as needed, and this disclosure does not limit this.

[0145] For example, the number of second units included in the third submodule can be any integer from 2 to 10. For instance, the number of second units included in the third submodule can be 2, 3, 4, 5, 7, 9, 10, etc.

[0146] For example, refer to Figure 2 , Figure 2 The first module includes a third submodule 208 and a fourth submodule 209. The third submodule 208 includes two second units (see reference). Figure 2 (Units d and e in the text). Based on Figure 2In the second module described above, after obtaining the first feature image output by unit c, the electronic device can upsample the first feature image output by unit c to obtain a seventh feature image of the same size as the first feature image output by unit b. Then, the electronic device can stitch the seventh feature image and the first feature image output by unit b along the channel dimension to obtain a first second stitched image. Next, the electronic device can input the first second stitched image into unit d for depth feature extraction to obtain an eighth feature image output by unit d. After obtaining the eighth feature image output by unit d, the electronic device can stitch the eighth feature image output by unit d into the first feature image output by unit a along the channel dimension (the size of the eighth feature image output by unit d is the same as the size of the first feature image output by unit a) to obtain a second second stitched image. Then, the electronic device can input the second second stitched image into unit e for depth feature extraction to obtain an eighth feature image output by unit e. Since unit e is the last second unit in the third submodule, the electronic device can input the eighth feature image output by unit e as the sixth feature image into the fourth submodule 209 to obtain the depth image 205 output by the fourth submodule 209.

[0147] In some embodiments, each second unit in the third submodule includes, but is not limited to, a convolution subunit and an upsampling subunit.

[0148] It should be noted that the number of convolutional sub-units and the number of upsampling sub-units included in each second unit can be determined as needed, and this embodiment does not limit this.

[0149] For example, Figure 9 This is a schematic diagram illustrating the structure of a second unit according to an exemplary embodiment. For example... Figure 9 As shown, the second unit may include a second convolution subunit 901, an upsampling subunit 902, and a third convolution subunit 903.

[0150] In some embodiments, based on Figure 9 The second unit is shown in the diagram. After the electronic device inputs the second stitched image into the j-th second unit, it can process it sequentially through the second convolution subunit 901, the upsampling subunit 902, and the third convolution subunit 903. At this time, the electronic device can obtain the eighth feature image output by the j-th second unit.

[0151] In some embodiments, both the second convolutional subunit 901 and the third convolutional subunit 903 may include at least one convolutional layer.

[0152] In some embodiments, the size of the convolution kernel used by the convolutional layers in the second convolutional subunit 901 and the third convolutional subunit 903 can be set as needed, and this disclosure does not limit this.

[0153] For example, the size of the convolution kernel used in the convolutional layers of the second convolutional subunit 901 and the third convolutional subunit 903 can be 2×2, 3×3, 5×5, etc.

[0154] In some embodiments, the first image is a frame from a monocular video stream.

[0155] In some embodiments, the first image may be the most recently acquired frame in a monocular video stream acquired by an image acquisition device.

[0156] In some embodiments, a monocular video stream can be understood as a video stream captured by a single camera.

[0157] In this embodiment, since the number of parameters in the depth estimation model is less than the parameter threshold, the depth estimation model provided in this application can be deployed on a terminal device, thereby enabling timely processing of each frame of the video stream. Testing shows that image processing using the image processing method provided in the above embodiments takes approximately 10 milliseconds to process one image. Furthermore, because the number of parameters in the depth estimation model is less than the parameter threshold, this depth estimation model also has the advantage of low power consumption.

[0158] refer to Figure 10 , Figure 10 This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment. Figure 10 As shown, the image processing apparatus 1000 includes:

[0159] The acquisition module 1001 is configured to acquire a first image, wherein the first image is a monocular image; the processing module 1002 is configured to perform at least one downsampling process on the first image to obtain at least one second image; wherein the at least one second image and the first image constitute an image pyramid; the input module 1003 is configured to input the image pyramid into a depth estimation model to obtain a third image, wherein the third image is a depth image of the first image.

[0160] In some embodiments, the depth estimation model includes a first module and a second module; wherein the input module 1003 is configured to: input an image pyramid into the first module for structural feature extraction to obtain a first feature image; wherein the number of first feature images is the same as the number of second images; and input the first feature images into the second module for depth feature extraction to obtain a third image.

[0161] In some embodiments, the first module includes a first submodule and a second submodule; wherein the input module 1003 is configured to: input a first image into the first submodule for structural feature extraction to obtain a second feature image; and input the second feature image and at least one second image into the second submodule for structural feature extraction to obtain a first feature image.

[0162] In some embodiments, the second submodule includes N first units, and the number of second images is N, where N is a positive integer; wherein, the input module 1003 is configured to: stitch the third feature image and the second image with the same size as the three feature images in the channel dimension to obtain a first stitched image, wherein the different second images have different sizes, and the third feature image is either the first feature image or the second feature image output by the (i-1)th first unit; input the first stitched image into the i-th first unit for structural feature extraction to obtain the first feature image output by the i-th first unit, where i is an integer from 1 to N.

[0163] In some embodiments, each of the N first units includes a first convolution sub-unit, a spatial aggregation sub-unit, and a channel aggregation sub-unit; wherein, the input module 1003 is configured to: input the first stitched image into the first convolution sub-unit included in the i-th first unit for convolution processing to obtain a fourth feature image; input the fourth feature image into the spatial aggregation sub-unit included in the i-th first unit for spatial feature aggregation to obtain a fifth feature image; and input the fifth feature image into the channel aggregation sub-unit included in the i-th first unit for channel feature aggregation to obtain a first feature image output by the i-th first unit.

[0164] In some embodiments, the second module includes a third submodule and a fourth submodule; wherein the input module is configured to: input the first feature image into the third submodule for depth feature extraction to obtain a sixth feature image; and input the sixth feature image into the fourth submodule for image generation to obtain a third image.

[0165] In some embodiments, the number of first feature images is multiple; the third submodule includes M second units, where M is a positive integer; wherein, the input module 1003 is configured to: stitch the seventh feature image and the first feature image with the same size as the seventh feature image in the channel dimension to obtain a second stitched image, wherein the seventh feature image is a feature image obtained by upsampling the last first feature image output by the first module or the eighth feature image output by the (j-1)th second unit; input the second stitched image into the jth second unit for depth feature extraction to obtain the eighth feature image output by the jth second unit, wherein j is an integer from 1 to M; and determine the eighth feature image output by the Mth second unit as the sixth feature image.

[0166] In some embodiments, the first image includes at least one frame of a monocular image from a monocular video stream.

[0167] Figure 11 This is a structural block diagram illustrating a terminal device 1100 according to an exemplary embodiment. For example, the terminal device 1100 may be a mobile phone, tablet computer, smartwatch, in-vehicle device, or other communication device.

[0168] Reference Figure 11 The terminal device 1100 may include one or more of the following components: processing component 1102, memory 1104, power supply component 1106, multimedia component 1108, audio component 1110, input / output (I / O) interface 1112, sensor component 1114, and communication component 1116.

[0169] Processing component 1102 typically controls the overall operation of terminal device 1100, such as operations associated with at least one of display, telephone call, data communication, camera operation, and recording operation. Processing component 1102 may include one or more processors 1120 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1102 may include one or more modules to facilitate interaction between processing component 1102 and other components. For example, processing component 1102 may include a multimedia module to facilitate interaction between multimedia component 1108 and processing component 1102.

[0170] Memory 1104 is configured to store various types of data to support operation on terminal device 1100. Examples of such data include at least one of the following: instructions for any application or method operating on terminal device 1100, contact data, phonebook data, messages, pictures, and videos. Memory 1104 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0171] Power supply component 1106 provides power to various components of terminal device 1100. Power supply component 1106 may include at least one of the following: a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to terminal device 1100.

[0172] Multimedia component 1108 includes a screen that provides an output interface between terminal device 1100 and the user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen may be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1108 includes a front-facing camera and / or a rear-facing camera. When terminal device 1100 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0173] Audio component 1110 is configured to output and / or input audio signals. For example, audio component 1110 includes a microphone (MIC) configured to receive external audio signals when terminal device 1100 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1104 or transmitted via communication component 1116. In some embodiments, audio component 1110 also includes a speaker for outputting audio signals.

[0174] I / O interface 1112 provides an interface between processing component 1102 and peripheral interface modules, such as keyboards, click wheels, and buttons. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0175] Sensor assembly 1114 includes one or more sensors for providing status assessments of various aspects of terminal device 1100. For example, sensor assembly 1114 may detect the on / off state of terminal device 1100, the relative positioning of components such as the display and keypad of terminal device 1100, changes in position of terminal device 1100 or one of its components, the presence or absence of user contact with terminal device 1100, orientation or acceleration / deceleration of terminal device 1100, and temperature changes of terminal device 1100. Sensor assembly 1114 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1114 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 1114 may also include, but is not limited to, at least one of the following: an accelerometer, a gyroscope, a magnetometer, a pressure sensor, and a temperature sensor.

[0176] Communication component 1116 is configured to facilitate wired or wireless communication between terminal device 1100 and other devices. Terminal device 1100 can access wireless networks based on communication standards, such as Wi-Fi, 4G, 5G, or combinations thereof. In one exemplary embodiment, communication component 1116 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1116 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), and other technologies.

[0177] In an exemplary embodiment, the terminal device 1100 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0178] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1104 including executable instructions or a computer program, which can be executed by the processor 1120 of the terminal device 1100 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0179] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a terminal device, enables the terminal device to perform any of the image processing methods described above in the embodiments of this disclosure. For example, the method includes:

[0180] A first image is acquired, wherein the first image is a monocular image; the first image is downsampled at least once to obtain at least one second image; wherein the at least one second image and the first image constitute an image pyramid; the image pyramid is input into a depth estimation model to obtain a third image, wherein the third image is a depth image of the first image.

[0181] This disclosure provides a computer program product comprising a computer program or executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or executable instructions from the computer-readable storage medium and executes the computer program or executable instructions, causing the computer device to perform any of the image processing methods described above in this disclosure.

[0182] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0183] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, The method includes: Acquire a first image, wherein the first image is a monocular image; The first image is downsampled at least once to obtain at least one second image; wherein the at least one second image and the first image constitute an image pyramid; The image pyramid is input into the depth estimation model to obtain a third image, wherein the third image is the depth image of the first image.

2. The method according to claim 1, characterized in that, The depth estimation model includes a first module and a second module; The step of inputting the image pyramid into the depth estimation model to obtain the third image includes: The image pyramid is input into the first module for structural feature extraction to obtain a first feature image; wherein the number of the first feature images is the same as the number of the second images; The first feature image is input into the second module for depth feature extraction to obtain the third image.

3. The method according to claim 2, characterized in that, The first module includes a first sub-module and a second sub-module; The step of inputting the image pyramid into the first module for structural feature extraction to obtain a first feature image includes: The first image is input into the first submodule for structural feature extraction to obtain the second feature image; The second feature image and the at least one second image are input into the second submodule for structural feature extraction to obtain the first feature image.

4. The method according to claim 3, characterized in that, The second submodule includes N first units, and the number of the second images is N, where N is a positive integer; The step of inputting the second feature image and the at least one second image into the second submodule for row structure feature extraction to obtain the first feature image includes: The third feature image and the second image with the same size as the third feature image are stitched together in the channel dimension to obtain the first stitched image, wherein the different second images have different sizes, and the third feature image is either the first feature image output by the (i-1)th first unit or the second feature image; The first stitched image is input into the i-th first unit for structural feature extraction to obtain the first feature image output by the i-th first unit, where i is an integer from 1 to N.

5. The method according to claim 4, characterized in that, Each of the N first units includes a first convolution subunit, a spatial aggregation subunit, and a channel aggregation subunit; The step of inputting the first stitched image into the i-th first unit for structural feature extraction to obtain the first feature image output by the i-th first unit includes: The first stitched image is input into the first convolution sub-unit included in the i-th first unit for convolution processing to obtain the fourth feature image; The fourth feature image is input into the spatial aggregation subunit included in the i-th first unit for spatial feature aggregation to obtain the fifth feature image; The fifth feature image is input into the channel aggregation subunit included in the i-th first unit for channel feature aggregation to obtain the first feature image output by the i-th first unit.

6. The method according to claim 2, characterized in that, The second module includes a third sub-module and a fourth sub-module; The step of inputting the first feature image into the second module for depth feature extraction to obtain the third image includes: The first feature image is input into the third sub-module for depth feature extraction to obtain the sixth feature image; The sixth feature image is input into the fourth submodule for image generation to obtain the third image.

7. The method according to claim 6, characterized in that, The number of the first feature images is multiple; the third submodule includes M second units, where M is a positive integer; The step of inputting the first feature image into the third submodule for depth feature extraction to obtain the sixth feature image includes: The seventh feature image and the first feature image of the same size as the seventh feature image are stitched together in the channel dimension to obtain the second stitched image. The seventh feature image is the feature image obtained by upsampling the last first feature image output by the first module or the eighth feature image output by the (j-1)th second unit. The second stitched image is input into the j-th second unit for depth feature extraction to obtain the eighth feature image output by the j-th second unit, where j is an integer from 1 to M. The eighth feature image output by the Mth second unit is determined as the sixth feature image.

8. The method according to claim 1, characterized in that, The first image includes at least one frame of a monocular image from a monocular video stream.

9. An image processing apparatus, characterized in that, The device includes: The acquisition module is configured to acquire a first image, wherein the first image is a monocular image; The processing module is configured to perform at least one downsampling process on the first image to obtain at least one second image; wherein the at least one second image and the first image constitute an image pyramid; The input module is configured to input the image pyramid into a depth estimation model to obtain a third image, wherein the third image is a depth image of the first image.

10. A terminal device, characterized in that, include: processor; Memory used to store computer programs or instructions; The processor executes the computer program or instructions to implement the steps of the method according to any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium storing a computer program or instructions, characterized in that, When the computer program or instructions in the storage medium are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

Images