Monocular real-time dense depth estimation method and system

Abstract

The application provides a monocular real-time dense depth estimation method and system, wherein the method comprises the following steps: in the process of controlling the continuous zoom of the focal length of the liquid lens by the integrated sensor and computer vision, multiple clear object edge images are collected; the focal length corresponding to the clear object edge in the field of view is determined during the image collection process; the depth information corresponding to the clear object edge is determined according to the focal length corresponding to the clear object edge in the field of view, the distance between the integrated sensor and the liquid lens, and the camera imaging principle; the clear object edge is subjected to color assignment processing according to the depth information, and the multiple clear object edge images are subjected to image fusion processing to generate a sparse depth estimation image; the clear object edge in the sparse depth estimation image is subjected to edge filling processing and depth filling processing to determine a dense depth estimation image. Through the application, efficient and accurate depth estimation is realized in an environment with limited computing resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a monocular real-time dense depth estimation method and system. BACKGROUND

[0002] Depth estimation is a key step in the fields of three-dimensional reconstruction, autonomous driving, and visual SLAM (Simultaneous Localization and Mapping), and has always been a hot topic in computer vision research. Among them, monocular depth estimation through unsupervised learning has attracted widespread attention due to its convenience in deployment and lower computational cost. However, monocular vision can only obtain two-dimensional image information from a single perspective, lacking depth cues such as stereo disparity, which leads to the problem of scale ambiguity in depth estimation. Scale ambiguity refers to the fact that in a monocular image, the absolute scale and distance of an object cannot be determined from a single frame of image. Due to the lack of depth cues, large objects in the distance and small objects in the near distance can appear the same size in the image, making it difficult to distinguish the true size and distance of the objects. This uncertainty makes it difficult for monocular depth estimation to accurately recover the true depth information of the scene. To solve the problem of scale ambiguity, some methods introduce prior auxiliary information such as object motion, semantic segmentation, and camera intrinsic matrix to provide additional depth cues and improve the accuracy of depth estimation. However, the introduction of these auxiliary information increases the data cost and memory consumption, complicating the network structure and making it difficult to deploy on embedded devices with limited memory and space. SUMMARY

[0003] In view of the defects in the prior art, the purpose of the present application is to provide a monocular real-time dense depth estimation method and system.

[0004] To achieve the above-mentioned purpose, according to one aspect of the present application, a monocular real-time dense depth estimation method is provided, comprising:

[0005] In the process of controlling the continuous zoom of the liquid lens by the sensor-computer-integrated sensor, multiple clear object edge images are collected by the sensor-computer-integrated sensor;

[0006] The clear object edge images are preprocessed during the image collection process to determine the focal length corresponding to the clear object edge in the field of view during the zooming process of the liquid lens;

[0007] According to the focal length corresponding to the clear object edge in the field of view, the distance between the sensor-computer-integrated sensor and the liquid lens, and the camera imaging principle, the depth information corresponding to the clear object edge in the field of view is determined;

[0008] According to the depth information corresponding to the clear object edge in the field of view, the clear object edge in the field of view is subjected to color assignment processing, and a plurality of clear object edge images are subjected to image fusion processing to generate a sparse depth estimation image;

[0009] The clear object edge with depth information in the sparse depth estimation image is subjected to edge filling processing and depth filling processing to determine a dense depth estimation image.

[0010] Optionally, in the process of controlling the liquid lens to continuously zoom by the sensor-computer-integrated vision sensor, a plurality of clear object edge images are collected by the sensor-computer-integrated sensor, which comprises:

[0011] The sensor-computer-integrated sensor sends a control instruction to the liquid lens;

[0012] According to the control instruction, the liquid lens is adjusted from the minimum focal length to the maximum focal length and then from the maximum focal length to the minimum focal length, and at each focal length, an image of a clear object in the field of view at the focal length is collected to determine the plurality of clear object edge images.

[0013] Optionally, the preprocessing comprises edge detection processing and noise reduction processing.

[0014] Optionally, in the image collection process, the clear object edge image is preprocessed to determine the focal length corresponding to the clear object edge in the field of view during the zooming process of the liquid lens, which comprises:

[0015] An edge detection algorithm is used to perform edge detection processing on the clear object edge image to determine an object edge image subjected to edge detection processing;

[0016] According to a preset filtering threshold, a high-pass filter is used to process the object edge image subjected to edge detection processing to determine the focal length corresponding to the clear object edge in the field of view during the zooming process of the liquid lens.

[0017] Optionally, according to the depth information corresponding to the clear object edge, the clear object edge is subjected to color assignment processing, and a plurality of clear object edge images are subjected to image fusion processing to generate a sparse depth estimation image, which comprises:

[0018] According to the depth information corresponding to the clear object edge in each clear object edge image, the clear object edge is subjected to color assignment processing to determine an object edge image subjected to color assignment processing;

[0019] A plurality of object edge images subjected to color assignment processing are subjected to image fusion processing to generate the sparse depth estimation image.

[0020] Optionally, the edge filling processing and the depth filling processing are performed on the clear object edges with depth information in the sparse depth estimation image to determine a dense depth estimation image, including:

[0021] The edge filling processing is performed on each clear object edge in the sparse depth estimation image by using an expansion algorithm, an edge interpolation algorithm, an edge detection algorithm, and an edge endpoint detection algorithm to determine a depth estimation image after edge filling processing;

[0022] The depth estimation image after edge filling processing is subjected to image erosion processing to determine a depth estimation image after image erosion processing;

[0023] The depth filling processing is performed on the inside of each clear object edge in the depth estimation image after image erosion processing according to the depth information corresponding to the clear object edge to determine the dense depth estimation image.

[0024] According to a second aspect of the present application, a monocular real-time dense depth estimation system is provided, including:

[0025] An image acquisition module is configured to acquire a plurality of clear object edge images by using a sensor integrated with storage and calculation during continuous zooming of a liquid lens controlled by a sensor integrated with storage and calculation;

[0026] An image preprocessing module is configured to preprocess the clear object edge images during image acquisition to determine the focal length corresponding to the clear object edge in the field of view during zooming of the liquid lens;

[0027] A depth information estimation module is configured to determine the depth information corresponding to the clear object edge in the field of view according to the focal length corresponding to the clear object edge in the field of view, the distance between the sensor integrated with storage and calculation and the liquid lens, and the camera imaging principle;

[0028] A sparse depth estimation module is configured to perform color assignment processing on the clear object edge in the field of view according to the depth information corresponding to the clear object edge in the field of view, and perform image fusion processing on the plurality of clear object edge images to generate a sparse depth estimation image;

[0029] A dense depth estimation module is configured to perform edge filling processing and depth filling processing on the clear object edges with depth information in the sparse depth estimation image to determine a dense depth estimation image.

[0030] According to a third aspect of the present application, a monocular real-time dense depth estimation device is provided, including a camera and a terminal, wherein the camera includes a sensor integrated with storage and calculation and a liquid lens.

[0031] The sensor-storage-computation integrated sensor is used for controlling the liquid lens to continuously zoom and collecting multiple clear object edge images; is also used for carrying out edge detection processing and noise reduction processing on the collected multiple clear object edge images, storing the object edge images, recording the focal length corresponding to the clear object edge in each field of view, calculating and storing the depth information corresponding to the clear object edge;

[0032] The liquid lens is used for being adjusted from the minimum focal length to the maximum focal length and then from the maximum focal length to the minimum focal length under the control of the sensor-storage-computation integrated sensor, and scanning the object;

[0033] The terminal is used for carrying out color assignment processing on the depth information corresponding to the clear object edge stored by the sensor-storage-computation integrated sensor, carrying out image fusion processing on the clear object edge images collected and stored, generating a sparse depth estimation image, and converting the sparse depth estimation image into a dense depth estimation image.

[0034] According to a fourth aspect of the present application, a non-transitory computer readable storage medium is provided, having stored thereon a computer program, wherein the program, when executed by a processor, implements the steps of any of the methods provided by the first aspect of the present application.

[0035] According to a fifth aspect of the present application, an electronic device is provided, comprising:

[0036] a memory having stored thereon a computer program;

[0037] a processor configured to execute the computer program stored in the memory to implement the steps of any of the methods provided by the first aspect of the present application.

[0038] Compared with the prior art, the embodiments of the present application have at least one of the following beneficial effects:

[0039] Through the above technical solution, the clear object edge image is collected by the sensor-storage-computation integrated sensor, and additional depth information is obtained without relying on external auxiliary information. Moreover, the sensor-storage-computation integrated sensor integrates the functions of sensing, computation and storage, effectively reduces the energy consumption of data transmission, improves the computation efficiency, through the preprocessing process of edge detection processing and noise reduction processing, the focal length corresponding to the clear object edge in the field of view is obtained, and based on the focal length corresponding to the clear object edge in the field of view and the camera imaging principle, the depth information corresponding to the clear object edge in the field of view is determined, a sparse depth estimation image is generated, and the sparse depth estimation image is converted into a dense depth estimation image. Additional depth information can be obtained in a resource-limited environment to realize efficient and accurate depth estimation. BRIEF DESCRIPTION OF DRAWINGS

[0040] Other features, objects, and advantages of the application will become more apparent from the following detailed description when read in conjunction with the accompanying drawings:

[0041] Figure 1 is a flow chart of a monocular real-time dense depth estimation method according to an exemplary embodiment.

[0042] Figure 2 is a block diagram of a monocular real-time dense depth estimation system according to an exemplary embodiment.

[0043] Figure 3 is a structural schematic diagram of a real-time dense depth estimation device according to an exemplary embodiment. DETAILED DESCRIPTION

[0044] The application will be described in detail below with specific embodiments. The following embodiments will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be noted that for those skilled in the art, without departing from the concept of the application, a number of modifications and improvements can be made. These all belong to the protection scope of the application.

[0045] Figure 1 is a flow chart of a monocular real-time dense depth estimation method according to an exemplary embodiment.

[0046] As shown in Figure 1 , the application provides a monocular real-time dense depth estimation method, comprising S11 to S15.

[0047] S11, in the process of controlling the continuous zoom of the liquid lens by the sensor-integrated vision sensor, a plurality of clear object edge images are collected by the sensor-integrated vision sensor.

[0048] The sensor-integrated vision sensor integrates the functions of perception, storage and calculation, and has the feature of in-sensor calculation.

[0049] The integrated sensing and computing sensor adopted in the application is a typical focal-plane sensor-processor (FPSP) device, which can capture, store and process visual data at the pixel level. The core of the typical focal-plane sensor-processor is a pixel processing array (PPA) composed of processing elements, which is good at extracting and directly outputting key information from images without the need for traditional recording, digital-to-analog conversion and transmission to an external computer for calculation, avoiding the dependence on external computing resources or professional software. The typical focal-plane sensor-processor adopted in the application can perform image processing at a speed of up to 10000 frames / s, with a running power consumption of about 1.5W and a computing energy efficiency of 1TOPS / W.

[0050] The liquid lens adopts a V-Sweep fast zoom scanning technology. The liquid lens is filled with a liquid light transmission medium, which is stored in a polymer inner membrane and a peripheral reservoir of the liquid lens. By controlling the liquid in and out between the peripheral reservoir and the polymer inner membrane, the curvature of the polymer inner membrane is changed, and the focal length of the liquid lens is adjusted, so that the change of the optical power of the liquid lens is consistent with the periodic V-shaped wave, and a large number of images are captured and processed in a short time to generate a single depth image frame.

[0051] In the application, the voltage applied to the liquid lens can be changed in real time, so as to control the liquid in and out between the peripheral reservoir and the polymer inner membrane, adjust the focal length of the liquid lens, and in the zooming process, the integrated sensing and computing sensor controls the liquid lens to scan the objects in the field of view, and the integrated sensing and computing sensor collects images corresponding to different focal lengths as object edge images, which can provide additional depth information for a monocular camera and relieve the scale ambiguity problem without relying on external auxiliary information.

[0052] S12, pre-processing the object edge image during image acquisition to determine the focal length corresponding to the clear object edge in the field of view during the zooming process of the liquid lens.

[0053] Pre-processing includes edge detection processing and noise reduction processing.

[0054] The object edge image is pre-processed simultaneously with the adjustment of the zooming process of the liquid lens. The integrated sensing and computing sensor can be used to pre-process the object edge image.

[0055] In this step, the Sand Castle summation method can also be used to measure the focal point corresponding to the clear edge. The Sand Castle summation method is used for grouping summation operation in the same level, parallelizes the calculation task, decomposes the calculation task of each level to multiple processors or calculation units, avoids the overloading of a single processor and the idling of other processors, improves the overall parallel computing efficiency, and realizes the parallel focal point measurement in the sensor-integrated sensor, which needs 0.32 ms.

[0056] S13, according to the focal length corresponding to the clear object edge in the field of view, the distance between the sensor-integrated sensor and the liquid lens, and the camera imaging principle, the depth information corresponding to the clear object edge in the field of view is determined.

[0057] The camera imaging principle represents the balance relationship between the distance between the object and the lens, the distance between the sensor and the lens, and the focal length of the lens.

[0058] In the present application, the sensor uses a sensor-integrated sensor, and the lens uses a liquid lens.

[0059] In a possible embodiment, the camera imaging principle is:

[0060]

[0061] Wherein, u represents the distance between the object and the liquid lens, v represents the distance between the sensor-integrated sensor and the liquid lens, and f represents the focal length corresponding to the clear object edge in the field of view.

[0062] Based on the camera imaging principle, the present application can determine the depth information corresponding to the edge according to the focal length corresponding to the clear object edge in the field of view. The depth information corresponding to the edge includes the distance between the object and the liquid lens, i.e. the real object distance corresponding to the edge.

[0063] S14, according to the depth information corresponding to the clear object edge in the field of view, color assignment processing is performed on the clear object edge in the field of view, and image fusion processing is performed on multiple clear object edge images to generate a sparse depth estimation image.

[0064] S15, the clear object edge with depth information in the sparse depth estimation image is subjected to edge filling processing and depth filling processing to determine a dense depth estimation image.

[0065] In the present application, steps S14 to S15 can be run in the terminal to generate a sparse depth estimation image by image processing on the depth image, and convert the sparse depth estimation image into a dense depth estimation image.

[0066] By the technical scheme, the clear object edge image is collected by the sensor integrated with sensing, storage and calculation, additional depth information is obtained without relying on external auxiliary information, the sensor integrated with sensing, storage and calculation integrates the functions of sensing, calculation and storage, energy consumption of data transmission is effectively reduced, calculation efficiency is improved, the focal length corresponding to the clear object edge in the field of view is obtained through the preprocessing process of edge detection processing and noise reduction processing, the depth information corresponding to the clear object edge in the field of view is determined based on the focal length corresponding to the clear object edge in the field of view and the camera imaging principle, the sparse depth estimation image is generated, and the sparse depth estimation image is converted into the dense depth estimation image, so that additional depth information can be obtained in a resource-limited environment to realize efficient and accurate depth estimation.

[0067] In a possible embodiment, S11 can include S111 to S112.

[0068] S111, sending a control instruction to the liquid lens by the sensor integrated with sensing, storage and calculation.

[0069] In the sensor integrated with sensing, storage and calculation, a focal plane sensing processor (FPSP) is used as a core calculation unit, which communicates with the liquid lens through a serial communication protocol, and the focal plane sensing processor (FPSP) sends a control instruction to the liquid lens through the serial communication protocol.

[0070] S112, adjusting the focal length of the liquid lens from the minimum focal length to the maximum focal length and then from the maximum focal length to the minimum focal length according to the control instruction, collecting the image of the clear object in the field of view at each focal length when the focal length of the liquid lens is adjusted to each focal length, and determining multiple clear object edge images.

[0071] The control instruction is used to control the liquid lens to perform zoom operation, and the optical power of the liquid lens is adjusted from the maximum to the minimum and then from the minimum to the maximum according to the control instruction, the focal length of the liquid lens is adjusted from the minimum focal length to the maximum focal length and then from the maximum focal length to the minimum focal length, and the focal point gradually increases from the minimum value and then gradually decreases from the maximum value.

[0072] In the zoom operation of the liquid lens, the area in the object edge image that matches the focal length position, that is, the object edge, will become clear, the sensor integrated with sensing, storage and calculation determines the sharpness of the image by using the sharpness algorithm, and the liquid lens performs scanning imaging on the object in the field of view, so as to obtain the object edge image with clear object edge.

[0073] In a possible embodiment, S12 can include S121 to S122.

[0074] S121, performing edge detection processing on the clear object edge image by using an edge detection algorithm to determine the object edge image after edge detection processing.

[0075] The edge detection algorithm can adopt Laplacian and Soble algorithm, and the edge detection algorithm is realized by simulating registers through parallel operation on a focal plane sensor processor (FPSP) to achieve high-speed parallelization, and one efficient edge information extraction is completed in 0.013 ms.

[0076] The edge detection algorithm is adopted to determine all edges of all objects scanned by the liquid lens in one zooming period, extract edge information, and determine an object edge image containing the edge information, i.e., an object edge image processed by edge detection.

[0077] In S122, the object edge image processed by edge detection is subjected to high-pass filtering according to a preset filtering threshold, and a clear object edge image collected by the liquid lens in the zooming process and a focal length corresponding to the clear object edge in the field of view are determined.

[0078] The noise reduction processing can adopt a high-pass filtering algorithm to enhance the clear object edge and realize image noise reduction.

[0079] In this step, the low-pixel density part and the noise part in each object edge image collected by the liquid lens in one zooming period are filtered by setting the filtering threshold and adopting high-pass filtering, the clear object edge in each object edge image is retained, and the focal length corresponding to each clear object edge is recorded.

[0080] In one possible embodiment, S14 can include S141 to S142.

[0081] In S141, each clear object edge is subjected to color assignment processing according to the depth information corresponding to the clear object edge in each object edge image, and an object edge image subjected to color assignment processing is determined.

[0082] Different colors are adopted to reflect different object distances, and each clear object edge is subjected to color assignment processing. Specifically, the same color can be assigned to edges having the same depth information, i.e., the same color is assigned to edges having the same distance between the object and the liquid lens; different colors are assigned to edges having different depth information, i.e., different colors are assigned to edges having different distances between the object and the liquid lens.

[0083] As an example, the edges of the objects can be assigned colors according to the field of view from near to far, the edges of the objects closest to the field of view can be assigned red, the edges of the objects farther from the field of view can be assigned green, and the edges of the objects farthest from the field of view can be assigned blue, so that the depth information of the edges is reflected by the gradient colors.

[0084] S142, image fusion processing is performed on the plurality of object edge images subjected to the color assignment processing to generate a sparse depth estimation image.

[0085] In step S142, image alignment processing can also be performed on the depth image subjected to the color assignment processing, and the depth image subjected to the image alignment processing is subjected to image fusion processing to generate a sparse depth estimation image, and different colors are used to reflect the depth at which the clear object edge is located.

[0086] In a possible embodiment, the object edge image subjected to the preprocessing is stored in a register of the sensor-integrated storage sensor, the focal length corresponding to each clear object edge is recorded, and then the depth image is transmitted to a terminal. After the terminal receives the object edge image, the object edge image is subjected to grayscale detection, the pixel points greater than a preset grayscale threshold value are retained, and the edges in the object edge image are subjected to color assignment processing based on the depth information corresponding to the determined edges. The depth image subjected to the color assignment processing is subjected to image alignment processing and image fusion processing to generate a sparse depth estimation image.

[0087] As an example, the terminal uses an image processing algorithm to perform grayscale detection on the object edge image, retains the edges greater than a preset grayscale threshold value in the object edge image, and filters out the edges less than the preset grayscale threshold value. The retained edges are drawn on a black canvas with the same size as the camera output picture, and the edges on the black canvas are binarized. According to the depth level of the depth information of the edges of the object edge image in one zooming period, a color threshold value corresponding to each depth information is assigned, and then the edges with different depth information are assigned colors according to the color threshold value corresponding to each depth. Finally, all the edges in the same zooming period are fused on the same picture, and the depth of the object is reflected by the color to generate a depth image, and the sparse depth estimation of the objects in the field of view is completed.

[0088] In a possible embodiment, S15 can include S151 to S153.

[0089] S151, an edge filling processing is performed on each clear object edge in the sparse depth estimation image by using an expansion algorithm, an edge interpolation algorithm, an edge detection algorithm, and an edge endpoint detection algorithm, and a depth estimation image subjected to the edge filling processing is determined.

[0090] The edge filling processing includes filling holes and cracks in the image edge and connecting and closing the edges of the same object.

[0091] S152, image erosion processing is performed on the depth estimation image subjected to the edge filling processing to determine a depth estimation image subjected to the image erosion processing.

[0092] Image erosion processing can remove noise from depth estimation images that have undergone edge filling, refine object edges, and separate adhered object edges, thereby enhancing the edge features of each object in the image and enabling further noise reduction processing of the depth estimation image.

[0093] S153, based on the depth information corresponding to the edges of sharp objects, perform depth filling processing on the interior of each sharp object edge in the depth estimation image after image erosion processing to determine a dense depth estimation image.

[0094] In this model, the edges of sharp objects are the same color as the interior of the sharp object edges, and the missing local depth information in the coefficient-type depth estimation image is compensated by depth filling processing.

[0095] Figure 2 This is a block diagram illustrating a monocular real-time intensive depth estimation system according to an exemplary embodiment.

[0096] Based on the same concept, the present invention also provides a monocular real-time dense depth estimation system 100, such as... Figure 2 As shown, it includes: an image acquisition module 110, an edge detection module 120, a depth information determination module 130, a sparse depth estimation module 140, and a dense depth estimation module 150.

[0097] Image acquisition module 110 is used to acquire multiple clear object edge images using an integrated sensor during the continuous zooming of a liquid lens controlled by an integrated sensor-memory-computing vision sensor.

[0098] The image preprocessing module 120 is used to preprocess the image of the edge of the clear object during the image acquisition process to determine the focal length corresponding to the edge of the clear object in the field of view of the liquid lens during the zoom process.

[0099] The depth information determination module 130 is used to determine the depth information corresponding to the edge of the clear object in the field of view based on the focal length corresponding to the edge of the clear object in the field of view, the distance between the integrated sensor and the liquid lens, and the camera imaging principle.

[0100] The sparse depth estimation module 140 is used to assign color to the edges of clear objects in the field of view based on the depth information corresponding to the edges of clear objects in the field of view, and to perform image fusion processing on multiple images of the edges of clear objects to generate a sparse depth estimation image.

[0101] The dense depth estimation module 150 is used to perform edge filling and depth filling processing on the edges of clear objects with depth information in the sparse depth estimation image to determine the dense depth estimation image.

[0102] By the technical solution, the clear object edge image is collected by the sensing-storage-computing integrated sensor, additional depth information is obtained without relying on external auxiliary information, the sensing-storage-computing integrated sensor integrates the functions of sensing, computing and storage, energy consumption of data transmission is effectively reduced, computing efficiency is improved, the focal length corresponding to the clear object edge in the field of view is obtained through the preprocessing process of edge detection processing and noise reduction processing, the depth information corresponding to the clear object edge in the field of view is determined based on the focal length corresponding to the clear object edge in the field of view and the camera imaging principle, the sparse depth estimation image is generated, and the sparse depth estimation image is converted into the dense depth estimation image, so that additional depth information can be obtained in a resource-limited environment to realize efficient and accurate depth estimation.

[0103] As to the embodiment of the system, the specific manner in which the various modules perform operations has been described in detail in the embodiment of the method, and will not be described in detail here.

[0104] Figure 3 Fig. 1 is a structural schematic diagram of real-time dense depth estimation according to an example embodiment.

[0105] As Figure 3 shown, the application also provides a real-time dense depth estimation device 200, comprising a camera 210 and a terminal 220.

[0106] The camera 210 comprises a sensing-storage-computing integrated sensor 211 and a liquid lens 212.

[0107] The sensing-storage-computing integrated sensor 211 is used to control the continuous zooming of the liquid lens 212 and collect multiple clear object edge images, and is also used to perform edge detection processing and noise reduction processing on the collected multiple clear object edge images, store the object edge images, record the focal length corresponding to the clear object edge in each field of view, and calculate and store the depth information corresponding to the clear object edge.

[0108] The voltage applied to the liquid transparent medium in the liquid lens 212 is regularly changed by the sensing-storage-computing integrated sensor 211, so as to control the in-and-out of the liquid between the peripheral reservoir and the polymer inner film, adjust the focal length of the liquid lens 212, realize the conversion of the liquid lens 212 from the minimum focal length to the maximum focal length and then from the maximum focal length to the minimum focal length, and in the zooming process, the liquid lens 212 controlled by the sensing-storage-computing integrated sensor 211 scans the objects in the field of view, and the sensing-storage-computing integrated sensor 211 collects multiple clear object edge images.

[0109] The edge detection algorithm, such as the Laplacian and Soble algorithm, is used to perform parallel edge detection on each object in the object edge image, and high-pass filtering is used to filter the low pixel density and noise part in each object edge image. The object edge image is stored, the focal length corresponding to the clear object edge in each field of view is recorded, and the depth information corresponding to the clear object edge is calculated and stored.

[0110] The liquid lens 212 is used to adjust from the minimum focal length to the maximum focal length and then from the maximum focal length to the minimum focal length under the control of the sensor 211, and scan the object.

[0111] The terminal 220 is used to perform color assignment processing on the depth information corresponding to the object edge stored by the sensor 211, perform image fusion processing on the clear object edge image collected and stored, generate a sparse depth estimation image, and convert the sparse depth estimation image into a dense depth estimation image.

[0112] The terminal 220 converts the sparse depth estimation image into a dense depth estimation image by performing edge filling processing, image erosion processing, and depth filling processing on the sparse depth estimation image.

[0113] Thus, the liquid lens and the sensor based on the integration of sensing, storage, and calculation can be combined in the variable-focus monocular camera, and the terminal can be used to obtain additional depth information. The liquid lens and the sensor based on the integration of sensing, storage, and calculation have a compact structure and are suitable for embedded devices and other space-limited carriers, and can realize efficient depth estimation in a resource-limited environment.

[0114] Based on the same idea, in another embodiment of the present application, an electronic device is also provided, which includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The processor is used to execute the monocular real-time dense depth estimation method when executing the program.

[0115] Optionally, the memory is configured to store a program; the memory can include volatile memory (e.g., random access memory (RAM), such as static random access memory (SRAM), Double Data Rate SDRAM (DDR SDRAM), etc.), and / or non-volatile memory (e.g., flash memory). The memory is configured to store computer programs (e.g., application programs, functional modules, etc. for implementing the above-described methods), computer instructions, etc. The computer programs, computer instructions, etc. described above can be stored in one or more memories in a distributed manner. Moreover, the computer programs, computer instructions, data, etc. described above can be invoked by the processor.

[0116] The computer programs, computer instructions, etc. described above can be stored in one or more memories in a distributed manner. Moreover, the computer programs, computer instructions, data, etc. described above can be invoked by the processor.

[0117] The processor is configured to execute the computer programs stored in the memory to implement each step in the methods described above in the embodiments. Details can be referred to the related descriptions in the method embodiments above.

[0118] The processor and the memory can be independent structures, or can be integrated into an integrated structure. When the processor and the memory are independent structures, the memory and the processor can be coupled and connected through a bus.

[0119] In the embodiments of the present application, a non-transitory computer readable storage medium is also provided, which stores a computer program. The computer program is executed by a processor to implement the steps of the monocular real-time dense depth estimation method in any of the above-described embodiments.

[0120] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer-usable program code.

[0121] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0122] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0123] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0124] While preferred embodiments of the application have been described, modifications and variations can be apparent to those skilled in the art once aware of the general underlying concepts. Accordingly, the appended claims intend to embrace all such modifications and variations as fall within the scope of the application.

[0125] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A monocular real-time dense depth estimation method, characterized in that, The application relates to a method for controlling continuous zooming of a liquid lens by using a sensor integrated with storage and calculation. In the process of controlling continuous zooming of a liquid lens by using a sensor integrated with storage and calculation, multiple clear object edge images are collected by using the sensor integrated with storage and calculation; In the process of image collection, the clear object edge images are preprocessed to determine the focal length corresponding to the clear object edge in the field of view during zooming of the liquid lens; According to the focal length corresponding to the clear object edge in the field of view, the distance between the sensor integrated with storage and calculation and the liquid lens, and the camera imaging principle, the depth information corresponding to the clear object edge in the field of view is determined; According to the depth information corresponding to the clear object edge in the field of view, color assignment processing is performed on the clear object edge, and the multiple clear object edge images are subjected to image fusion processing to generate a sparse depth estimation image; The clear object edge with depth information in the sparse depth estimation image is subjected to edge filling processing and depth filling processing to determine a dense depth estimation image.

2. The method of claim 1, wherein, In the process of controlling continuous zooming of a liquid lens by using a sensor integrated with storage and calculation, multiple clear object edge images are collected by using the sensor integrated with storage and calculation, which comprises the following steps: The sensor integrated with storage and calculation sends a control instruction to the liquid lens; According to the control instruction, the liquid lens is adjusted from the minimum focal length to the maximum focal length and then from the maximum focal length to the minimum focal length, and when the lens is adjusted to each focal length, the image of a clear object in the field of view at the focal length is collected to determine the multiple clear object edge images.

3. The method of claim 2, wherein, The preprocessing comprises edge detection processing and noise reduction processing; In the process of image collection, the clear object edge images are preprocessed to determine the focal length corresponding to the clear object edge in the field of view during zooming of the liquid lens, which comprises the following steps: An edge detection algorithm is used to perform edge detection processing on the clear object edge images to determine the object edge images subjected to edge detection processing; According to a preset filter threshold, high-pass filtering is performed on the object edge images subjected to edge detection processing to determine the focal length corresponding to the clear object edge in the field of view during zooming of the liquid lens.

4. The method of claim 1, wherein, The camera imaging principle comprises the following formula: Wherein, u represents the distance between an object and the liquid lens, v represents the distance between the sensor integrated with storage and calculation and the liquid lens, and f represents the focal length corresponding to the clear object edge in the field of view.

5. The method of claim 1, wherein, According to the depth information corresponding to the clear object edge, color assignment processing is performed on the clear object edge, and the multiple clear object edge images are subjected to image fusion processing to generate a sparse depth estimation image, which comprises the following steps: According to the depth information corresponding to the clear object edge in each clear object edge image, color assignment processing is performed on each clear object edge to determine the object edge images subjected to color assignment processing; The multiple object edge images subjected to color assignment processing are subjected to image fusion processing to generate the sparse depth estimation image.

6. The method of claim 1, wherein, The edge filling processing and the depth filling processing on the clear object edge with depth information in the sparse depth estimation image are performed to determine a dense depth estimation image, including: An edge filling processing is performed on each clear object edge in the sparse depth estimation image by using an expansion algorithm, an edge interpolation algorithm, an edge detection algorithm and an edge endpoint detection algorithm to determine a depth estimation image after the edge filling processing; An image erosion processing is performed on the depth estimation image after the edge filling processing to determine a depth estimation image after the image erosion processing; A depth filling processing is performed on the inside of each clear object edge in the depth estimation image after the image erosion processing according to the depth information corresponding to the clear object edge to determine the dense depth estimation image.

7. A monocular real-time dense depth estimation system, characterized in that, It includes: An image acquisition module is configured to acquire a plurality of clear object edge images by using a sensor in a process of controlling a liquid lens continuous zooming by a sensor-computer-integrated vision sensor; An image preprocessing module is configured to preprocess the clear object edge images in the image acquisition process to determine a focal length corresponding to a clear object edge in a field of view in a zooming process of the liquid lens; A depth information estimation module is configured to determine depth information corresponding to the clear object edge in the field of view according to the focal length corresponding to the clear object edge in the field of view, a distance between the sensor-computer-integrated sensor and the liquid lens and a camera imaging principle; A sparse depth estimation module is configured to perform color assignment processing on the clear object edge in the field of view according to the depth information corresponding to the clear object edge and perform image fusion processing on the plurality of clear object edge images to generate a sparse depth estimation image; A dense depth estimation module is configured to perform edge filling processing and depth filling processing on the clear object edge with depth information in the sparse depth estimation image to determine a dense depth estimation image.

8. A monocular real-time dense depth estimation apparatus, characterized by, It includes a camera and a terminal, and the camera includes a sensor-computer-integrated sensor and a liquid lens; The sensor-computer-integrated sensor is configured to control the liquid lens to continuously zoom and acquire a plurality of clear object edge images, and further configured to perform edge detection processing and noise reduction processing on the acquired plurality of clear object edge images, store the object edge images, record a focal length corresponding to a clear object edge in each field of view, and calculate and store depth information corresponding to the clear object edge; The liquid lens is configured to be adjusted from a minimum focal length to a maximum focal length and then from the maximum focal length to the minimum focal length under the control of the sensor-computer-integrated sensor, and scan an object; The terminal is configured to perform color assignment processing on the depth information corresponding to the clear object edge stored by the sensor-computer-integrated sensor, perform image fusion processing on the acquired and stored clear object edge images to generate a sparse depth estimation image, and convert the sparse depth estimation image into a dense depth estimation image.

9. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by a processor to implement the steps of the method of any one of claims 1-7.

10. An electronic device, comprising: It includes: A memory having a computer program stored thereon; a processor for executing the computer program in the memory to implement the steps of the method of any of claims 1-7.

Images