Monocular real-time intensive depth estimation method and system

Abstract

The invention provides a monocular real-time intensive depth estimation method and system, and the method comprises the steps: collecting a plurality of clear object edge images in a process that a sensing-storage-calculation integrated visual sensor controls the continuous zooming of the focal length of a liquid lens; determining a focal length corresponding to the edge of a clear object in the view in the image acquisition process; determining depth information corresponding to the edge of the clear object according to the focal length corresponding to the edge of the clear object in the visual field, the distance between the sensing-storage-calculation integrated sensor and the liquid lens and a camera imaging principle; according to the depth information, carrying out color endowing processing on the edge of the clear object, and carrying out image fusion processing on a plurality of clear object edge images to generate a sparse depth estimation image; and carrying out edge filling processing and depth filling processing on the clear object edge in the sparse depth estimation image, and determining an intensive depth estimation image. According to the invention, efficient and accurate depth estimation is realized in an environment with limited computing resources.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a monocular real-time intensive depth estimation method and system. Background Art

[0002] Depth estimation, as a key link in the fields of 3D reconstruction, autonomous driving and visual SLAM (Simultaneous Localization and Mapping), has always been a hot topic in computer vision research. Among them, unsupervised learning monocular depth estimation technology has attracted widespread attention due to its advantages of easy deployment and low computational cost. However, monocular vision can only obtain two-dimensional image information from a single perspective and lacks depth cues such as stereo disparity, resulting in scale ambiguity in depth estimation. The scale ambiguity problem refers to the inability to determine the absolute scale and distance of an object from a single frame in a monocular image. Due to the lack of depth cues, large objects in the distance and small objects in the near distance can appear the same size in the image, making it difficult to distinguish the true size and distance of the object. This uncertainty makes it difficult for monocular depth estimation to accurately restore the true depth information of the scene. To solve the scale ambiguity problem, some methods introduce prior auxiliary information such as object motion, semantic segmentation, and camera intrinsic parameter matrix to provide additional depth cues and improve the accuracy of depth estimation. However, the introduction of these auxiliary information will increase data costs and video memory consumption, complicate the network structure, and make it difficult to deploy on embedded devices with limited memory and space. Summary of the invention

[0003] In view of the defects in the prior art, an object of the present invention is to provide a monocular real-time intensive depth estimation method and system.

[0004] To achieve the above object, according to one aspect of the present invention, a monocular real-time intensive depth estimation method is provided, comprising:

[0005] In the process of controlling the continuous zoom of the liquid lens by the integrated sensing, storage and computing sensor, the integrated sensing, storage and computing sensor is used to collect multiple clear edge images of objects;

[0006] Preprocessing the edge image of the clear object during image acquisition to determine the focal length corresponding to the edge of the clear object in the field of view of the liquid lens during zooming;

[0007] Determine the depth information corresponding to the edge of the clear object in the field of view according to the focal length corresponding to the edge of the clear object in the field of view, the distance between the sensing-storage-computing integrated sensor and the liquid lens, and the camera imaging principle;

[0008] According to the depth information corresponding to the edge of the clear object in the field of view, color is assigned to the edge of the clear object in the field of view, and multiple images of the edge of the clear object are fused to generate a sparse depth estimation image;

[0009] An edge filling process and a depth filling process are performed on the edges of clear objects with depth information in the sparse depth estimation image to determine a dense depth estimation image.

[0010] Optionally, in the process of controlling the liquid lens to continuously zoom using the sensing, storage and computing integrated visual sensor, collecting a plurality of clear object edge images using the sensing, storage and computing integrated sensor, including:

[0011] Using the sensing, storage and computing integrated sensor to send a control instruction to the liquid lens;

[0012] According to the control instruction, the liquid lens is controlled to adjust from the minimum focal length to the maximum focal length and then from the maximum focal length to the minimum focal length. When the lens is adjusted to each focal length, images of clear objects in the field of view at each focal length are collected to determine the multiple clear object edge images.

[0013] Optionally, the preprocessing includes edge detection processing and noise reduction processing.

[0014] Optionally, preprocessing the clear object edge image during the image acquisition process to determine the focal length corresponding to the clear object edge in the field of view of the liquid lens during the zooming process includes:

[0015] Performing edge detection processing on the clear object edge image using an edge detection algorithm to determine the object edge image after edge detection processing;

[0016] According to a preset filtering threshold, high-pass filtering is performed on the object edge image after edge detection processing to determine the focal length corresponding to the edge of the clear object in the field of view during the zooming process of the liquid lens.

[0017] Optionally, the color assignment processing is performed on the clear object edge according to the depth information corresponding to the clear object edge, and image fusion processing is performed on multiple clear object edge images to generate a sparse depth estimation image, including:

[0018] According to the depth information corresponding to the clear object edge in each of the clear object edge images, color assignment processing is performed on each clear object edge to determine the object edge image after color assignment processing;

[0019] The plurality of object edge images that have been subjected to color assignment processing are subjected to image fusion processing to generate the sparse depth estimation image.

[0020] Optionally, performing edge filling processing and depth filling processing on the edges of clear objects having depth information in the sparse depth estimation image to determine a dense depth estimation image includes:

[0021] Performing edge filling processing on each clear object edge in the sparse depth estimation image by using a dilation algorithm, an edge interpolation algorithm, an edge detection algorithm, and an edge endpoint detection algorithm to determine a depth estimation image after the edge filling processing;

[0022] Performing image erosion processing on the depth estimation image that has undergone edge filling processing to determine the depth estimation image that has undergone image erosion processing;

[0023] According to the depth information corresponding to the clear object edge, depth filling processing is performed on the inside of each clear object edge in the depth estimation image that has undergone image erosion processing to determine the dense depth estimation image.

[0024] According to a second aspect of the present invention, a monocular real-time intensive depth estimation system is provided, comprising:

[0025] An image acquisition module is used to acquire multiple clear edge images of objects using the integrated sensing, storage and computing sensor during the process of the integrated sensing, storage and computing visual sensor controlling the continuous zooming of the liquid lens;

[0026] An image preprocessing module, used to preprocess the edge image of the clear object during image acquisition, and determine the focal length corresponding to the edge of the clear object in the field of view of the liquid lens during zooming;

[0027] A depth information estimation module, used to determine the depth information corresponding to the edge of the clear object in the field of view according to the focal length corresponding to the edge of the clear object in the field of view, the distance between the sensing, storage and computing integrated sensor and the liquid lens, and the camera imaging principle;

[0028] A sparse depth estimation module, configured to perform color assignment processing on the edge of a clear object in the field of view according to the depth information corresponding to the edge of the clear object in the field of view, and perform image fusion processing on a plurality of images of the edge of the clear object to generate a sparse depth estimation image;

[0029] The dense depth estimation module is used to perform edge filling processing and depth filling processing on the edges of clear objects with depth information in the sparse depth estimation image to determine a dense depth estimation image.

[0030] According to a third aspect of the present invention, there is provided a monocular real-time intensive depth estimation device, comprising: a camera and a terminal, wherein the camera comprises a sensing, storage and computing integrated sensor and a liquid lens;

[0031] The sensing, storage and computing integrated sensor is used to control the continuous zooming of the liquid lens and collect multiple clear object edge images; it is also used to perform edge detection and noise reduction processing on the collected multiple clear object edge images, store the object edge images, record the focal length corresponding to the edge of each clear object in the field of view, and calculate and store the depth information corresponding to the edge of the clear object;

[0032] The liquid lens is used to adjust from the minimum focal length to the maximum focal length and then from the maximum focal length to the minimum focal length under the control of the sensing, storage and computing integrated sensor, and scan the object;

[0033] The terminal is used to perform color assignment processing on the depth information corresponding to the clear object edge stored in the integrated sensing, storage and computing sensor, perform image fusion processing on the collected and stored clear object edge image, generate a sparse depth estimation image, and convert the sparse depth estimation image into a dense depth estimation image.

[0034] According to a fourth aspect of the present invention, there is provided a non-temporary computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the program implements the steps of any one of the methods provided in the first aspect of the present invention.

[0035] According to a fifth aspect of the present invention, there is provided an electronic device, comprising:

[0036] a memory having a computer program stored thereon;

[0037] A processor is used to execute the computer program in the memory to implement the steps of any one of the methods provided in the first aspect of the present disclosure.

[0038] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0039] Through the above technical solution, a sensing, storage and computing integrated sensor is used to collect edge images of clear objects and obtain additional depth information without relying on external auxiliary information. In addition, the sensing, storage and computing integrated sensor integrates perception, computing and storage functions, effectively reducing the energy consumption of data transmission and improving computing efficiency. Through the preprocessing process of edge detection processing and noise reduction processing, the focal length corresponding to the edge of the clear object in the field of view is obtained, and based on the focal length corresponding to the edge of the clear object in the field of view and the camera imaging principle, the depth information corresponding to the edge of the clear object in the field of view is determined, a sparse depth estimation image is generated, and the sparse depth estimation image is converted into a dense depth estimation image, which can obtain additional depth information in a resource-constrained environment to achieve efficient and accurate depth estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0041] Figure 1 The present invention is a flowchart of a monocular real-time intensive depth estimation method according to an exemplary embodiment.

[0042] Figure 2 The invention is a block diagram of a monocular real-time intensive depth estimation system according to an exemplary embodiment.

[0043] Figure 3 It is a schematic structural diagram of a real-time intensive depth estimation device according to an exemplary embodiment. DETAILED DESCRIPTION

[0044] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0045] Figure 1 The present invention is a flowchart of a monocular real-time intensive depth estimation method according to an exemplary embodiment.

[0046] like Figure 1 As shown, the present invention provides a monocular real-time intensive depth estimation method, including S11 to S15.

[0047] S11, in the process of controlling the continuous zoom of the liquid lens by the sensing-storage-computing integrated visual sensor, a plurality of clear edge images of the object are collected by using the sensing-storage-computing integrated sensor.

[0048] Among them, the integrated sensing, storage and computing sensor integrates perception, storage and computing functions, and has the characteristics of in-storage computing.

[0049] The sensor integrated with sensing, storage and computing used in the present invention is a typical focal plane sensor processor (Focal-Plane Sensor-Processor, FPSP) device, which can capture, store and process visual data at the pixel level. The core of a typical focal plane sensor processor is a pixel processor array (Pixel Process Array, PPA) composed of processing elements, which is good at extracting and directly outputting key information from images, without the need for traditional recording, digital-to-analog conversion and transmission to an external computer for calculation, avoiding dependence on external computing resources or professional software. The typical focal plane sensor processor used in the present invention can process images at a speed of up to 10,000 frames per second, with an operating power consumption of approximately 1.5W and a computing energy efficiency of 1TOPS / W.

[0050] The liquid lens adopts V-Sweep fast zoom scanning technology. The interior of the liquid lens is filled with liquid light-transmitting medium, which is stored in the polymer inner membrane and the outer reservoir of the liquid lens. By controlling the inflow and outflow of liquid between the outer reservoir and the polymer inner membrane, the curvature of the polymer inner membrane is changed, and then the focal length of the liquid lens is adjusted so that the change of its optical power is consistent with the periodic V-shaped wave, and a large number of images are captured and processed in a short time to generate a single depth image frame.

[0051] In the present invention, the voltage applied to the liquid lens can be changed in real time to control the inflow and outflow of liquid between the peripheral reservoir and the polymer inner membrane and adjust the focal length of the liquid lens. During the zooming process, the sensing, storage and computing integrated sensor controls the liquid lens to scan objects in the field of view. The sensing, storage and computing integrated sensor collects images corresponding to different focal lengths as the edge images of the object, which can provide additional depth information for the monocular camera and alleviate the scale blur problem without relying on external auxiliary information.

[0052] S12, pre-processing the edge image of the object during the image acquisition process to determine the focal length corresponding to the edge of the clear object in the field of view of the liquid lens during the zooming process.

[0053] Among them, the preprocessing includes edge detection processing and noise reduction processing.

[0054] The invention performs preprocessing of the edge image of the object and the process of adjusting the zoom of the liquid lens simultaneously. The edge image of the object can be preprocessed by using a sensor integrating sensing, storage and computing.

[0055] In this step, the Sand Castle summation method can also be used to measure the focus corresponding to the clear edge, and the Sand Castle summation method can be used to perform group summation operations within the same level, parallelizing the computing tasks, and decomposing the computing tasks of each level to multiple processors or computing units, avoiding excessive load on a single processor and idleness of other processors, improving the overall parallel computing efficiency, and realizing parallel focus measurement in the integrated sensing, storage and computing sensor, which takes 0.32ms.

[0056] S13, determining the depth information corresponding to the edge of the clear object in the field of view according to the focal length corresponding to the edge of the clear object in the field of view, the distance between the integrated sensing, storage and computing sensor and the liquid lens, and the camera imaging principle.

[0057] The camera imaging principle represents the balance between the distance between the object and the lens, the distance between the sensor and the lens, and the focal length of the lens.

[0058] In the present invention, the sensor adopts a sensing, storage and computing integrated sensor, and the lens adopts a liquid lens.

[0059] In a possible embodiment, the camera imaging principle is:

[0060]

[0061] Among them, u represents the distance between the object and the liquid lens, v represents the distance between the integrated sensing, storage and computing sensor and the liquid lens, and f represents the focal length corresponding to the edge of the clear object in the field of view.

[0062] Based on the camera imaging principle, the present invention can determine the depth information corresponding to the edge according to the focal length corresponding to the edge of the clear object in the field of view. The depth information corresponding to the edge includes the distance between the object and the liquid lens, that is, the real object distance corresponding to the edge.

[0063] S14, performing color assignment processing on the edge of the clear object in the field of view according to the depth information corresponding to the edge of the clear object in the field of view, and performing image fusion processing on multiple clear object edge images to generate a sparse depth estimation image.

[0064] S15, performing edge filling processing and depth filling processing on the edges of clear objects with depth information in the sparse depth estimation image to determine a dense depth estimation image.

[0065] In the present invention, step S14 to step S15 may be executed in a terminal, and a sparse depth estimation image is generated by performing image processing on the depth image, and the sparse depth estimation image is converted into a dense depth estimation image.

[0066] Through the above technical solution, a sensing, storage and computing integrated sensor is used to collect edge images of clear objects and obtain additional depth information without relying on external auxiliary information. In addition, the sensing, storage and computing integrated sensor integrates perception, computing and storage functions, effectively reducing the energy consumption of data transmission and improving computing efficiency. Through the preprocessing process of edge detection processing and noise reduction processing, the focal length corresponding to the edge of the clear object in the field of view is obtained, and based on the focal length corresponding to the edge of the clear object in the field of view and the camera imaging principle, the depth information corresponding to the edge of the clear object in the field of view is determined, a sparse depth estimation image is generated, and the sparse depth estimation image is converted into a dense depth estimation image, which can obtain additional depth information in a resource-constrained environment to achieve efficient and accurate depth estimation.

[0067] In a possible embodiment, S11 may include S111 to S112.

[0068] S111 uses a sensing, storage and computing integrated sensor to send control instructions to the liquid lens.

[0069] Among them, in the integrated sensing, storage and computing sensor of the present invention, a focal plane sensor processor (FPSP) is used as the core computing unit, which communicates with the liquid lens through a serial communication protocol, and the focal plane sensor processor (FPSP) sends control instructions to the liquid lens through the serial communication protocol.

[0070] S112, according to the control instruction, control the liquid lens to adjust from the minimum focal length to the maximum focal length and then from the maximum focal length to the minimum focal length, when the lens is adjusted to each focal length, collect images of clear objects in the field of view at each focal length, and determine multiple clear object edge images.

[0071] Among them, the control instruction is used to control the liquid lens to perform a zoom operation. According to the control instruction, the optical power of the liquid lens is controlled to be adjusted from maximum to minimum and then from minimum to maximum, the liquid lens is adjusted from minimum focal length to maximum focal length and then from maximum focal length to minimum focal length, and the focus gradually increases from the minimum value and then gradually decreases from the maximum value.

[0072] When the liquid lens performs a zoom operation, in the object edge image, the area that matches the focal length, that is, the object edge will become clear. The integrated sensing, storage and computing sensor uses a sharpness algorithm to determine the image clarity. The liquid lens scans and images the object in the field of view, thereby obtaining an object edge image with a clear object edge.

[0073] In a possible embodiment, S12 may include S121 to S122.

[0074] S121, performing edge detection processing on the clear object edge image using an edge detection algorithm to determine the object edge image after the edge detection processing.

[0075] Among them, the edge detection algorithm can use Laplacian and Soble algorithms. The edge detection algorithm realizes high-speed parallelization by parallel operation of analog registers on a focal plane sensor processor (FPSP), and completes an efficient edge information extraction in 0.013ms.

[0076] The present invention adopts an edge detection algorithm to determine all edges of all objects scanned by a liquid lens within a zoom cycle, extract edge information, and determine an object edge image containing edge information, that is, an object edge image processed by edge detection.

[0077] S122, performing high-pass filtering on the object edge image after edge detection processing according to a preset filtering threshold, and determining the focal length corresponding to the clear object edge image collected by the liquid lens during the zooming process and the clear object edge in the field of view.

[0078] Among them, the noise reduction processing can use a high-pass filtering algorithm to enhance the edges of clear objects and achieve image noise reduction.

[0079] In this step, by setting the filtering threshold, high-pass filtering is used to filter the low-pixel density and noise parts in each object edge image collected by the liquid lens within a zoom cycle, retaining the clear object edge in each object edge image, and recording the focal length corresponding to each clear object edge.

[0080] In a possible embodiment, S14 may include S141 to S142.

[0081] S141 , performing color assignment processing on each clear object edge according to depth information corresponding to the clear object edge in each object edge image, and determining an object edge image after the color assignment processing.

[0082] Different colors are used to reflect different object distances, and each clear object edge is given a color. Specifically, the same color can be given to edges with the same depth information, that is, the same color can be given to edges with the same distance between the object and the liquid lens; different colors can be given to edges with different depth information, that is, different colors can be given to edges with different distances between the object and the liquid lens.

[0083] As an example, the edges of objects can be assigned colors according to the field of view from near to far. The edges of the objects closest to the field of view can be assigned red, the edges of the objects farther away can be assigned green, and the edges of the objects farthest away can be assigned blue, so that the depth information of the edges is reflected by using gradient colors.

[0084] S142, performing image fusion processing on the plurality of object edge images that have been subjected to color assignment processing to generate a sparse depth estimation image.

[0085] Among them, in step S142, the depth image that has been processed by color assignment can also be subjected to image alignment processing, so as to further perform image fusion processing on the depth image that has been processed by image alignment to generate a sparse depth estimation image, and use different colors to reflect the depth of the edge of a clear object.

[0086] In a possible embodiment, a preprocessed object edge image is stored in a register of a sensing, storage and computing integrated sensor, the focal length corresponding to each clear object edge is recorded, and the depth image is then transmitted to a terminal. After receiving the object edge image, the terminal performs grayscale detection on the object edge image, retains pixels greater than a preset grayscale threshold, and based on the depth information corresponding to the determined edge, performs color assignment processing on the edge in the object edge image, performs image alignment processing and image fusion processing on the depth image that has been color assigned, and generates a sparse depth estimation image.

[0087] As an example, the terminal uses an image processing algorithm to perform grayscale detection on the edge image of the object, retains the edges in the object edge image that are greater than a preset grayscale threshold, and filters out the edges that are less than the preset grayscale threshold. The retained edges are drawn on a black canvas of the same size as the camera output image, and the edges on the black canvas are binarized. According to the depth level of the depth information of the edge of the object edge image within a zoom cycle, a color threshold corresponding to each depth information is assigned, and then colors are assigned to edges with different depth information according to the color threshold corresponding to each depth. Finally, all edges within the same zoom cycle are fused into the same picture, and the color is used to reflect the depth of the object to generate a depth image, thereby completing the sparse depth estimation of the object in the field of view.

[0088] In a possible embodiment, S15 may include S151 to S153.

[0089] S151, using a dilation algorithm, an edge interpolation algorithm, an edge detection algorithm, and an edge endpoint detection algorithm to perform edge filling processing on each clear object edge in the sparse depth estimation image, and determining a depth estimation image after the edge filling processing.

[0090] Among them, edge filling processing includes filling holes and cracks at the edges of the image and connecting and closing the edges of the same object.

[0091] S152, performing image erosion processing on the depth estimation image after the edge filling processing, to determine the depth estimation image after the image erosion processing.

[0092] Image erosion processing can remove noise in the depth estimation image after edge filling processing, refine the edges of objects and separate the edges of adhered objects, thereby enhancing the edge features of each object in the image and achieving further denoising of the depth estimation image.

[0093] S153 , performing depth filling processing on the inside of each clear object edge in the depth estimation image after the image erosion processing according to the depth information corresponding to the clear object edge, to determine a dense depth estimation image.

[0094] The edge of a clear object has the same color as the inside of the edge of the clear object, and the depth filling process is used to make up for the missing local depth information in the coefficient-based depth estimation image.

[0095] Figure 2 The invention is a block diagram of a monocular real-time intensive depth estimation system according to an exemplary embodiment.

[0096] Based on the same concept, the present invention also provides a monocular real-time intensive depth estimation system 100, such as Figure 2 As shown, it includes: an image acquisition module 110 , an edge detection module 120 , a depth information determination module 130 , a sparse depth estimation module 140 and a dense depth estimation module 150 .

[0097] The image acquisition module 110 is used to acquire multiple clear edge images of objects using the integrated sensing, storage and computing sensor during the process of the integrated sensing, storage and computing visual sensor controlling the continuous zooming of the liquid lens;

[0098] An image preprocessing module 120, used to preprocess the clear object edge image during image acquisition, and determine the focal length corresponding to the clear object edge in the field of view of the liquid lens during zooming;

[0099] A depth information determination module 130, configured to determine the depth information corresponding to the edge of the clear object in the field of view according to the focal length corresponding to the edge of the clear object in the field of view, the distance between the sensing-storage-computing integrated sensor and the liquid lens, and the camera imaging principle;

[0100] A sparse depth estimation module 140, configured to perform color assignment processing on the edge of the clear object in the field of view according to the depth information corresponding to the edge of the clear object in the field of view, and perform image fusion processing on a plurality of images of the edge of the clear object to generate a sparse depth estimation image;

[0101] The dense depth estimation module 150 is configured to perform edge filling processing and depth filling processing on the edges of clear objects with depth information in the sparse depth estimation image to determine a dense depth estimation image.

[0102] Through the above technical solution, a sensing, storage and computing integrated sensor is used to collect edge images of clear objects and obtain additional depth information without relying on external auxiliary information. In addition, the sensing, storage and computing integrated sensor integrates perception, computing and storage functions, effectively reducing the energy consumption of data transmission and improving computing efficiency. Through the preprocessing process of edge detection processing and noise reduction processing, the focal length corresponding to the edge of the clear object in the field of view is obtained, and based on the focal length corresponding to the edge of the clear object in the field of view and the camera imaging principle, the depth information corresponding to the edge of the clear object in the field of view is determined, a sparse depth estimation image is generated, and the sparse depth estimation image is converted into a dense depth estimation image, which can obtain additional depth information in a resource-constrained environment to achieve efficient and accurate depth estimation.

[0103] Regarding the embodiment of the above system, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0104] Figure 3 It is a structural diagram of a real-time intensive depth estimation according to an exemplary embodiment.

[0105] like Figure 3 As shown, the present invention also provides a real-time intensive depth estimation device 200, including a camera 210 and a terminal 220.

[0106] The camera 210 includes a sensing, storage and computing integrated sensor 211 and a liquid lens 212 .

[0107] The sensing, storage and computing integrated sensor 211 is used to control the continuous zooming of the liquid lens 212 and to collect multiple clear object edge images; it is also used to perform edge detection and noise reduction processing on the collected multiple clear object edge images, store the object edge images, record the focal length corresponding to the clear object edge in each field of view, and calculate and store the depth information corresponding to the clear object edge.

[0108] Among them, the voltage applied to the liquid light-transmitting medium in the liquid lens 212 is regularly changed by using the integrated sensing, storage and computing sensor 211, so as to control the inflow and outflow of the liquid between the outer reservoir and the polymer inner membrane, adjust the focal length of the liquid lens 212, and realize the conversion of the liquid lens 212 from the minimum focal length to the maximum focal length and then from the maximum focal length to the minimum focal length. During the zooming process, the integrated sensing, storage and computing sensor 211 controls the liquid lens 212 to scan the objects in the field of view, and collects multiple clear edge images of the objects.

[0109] The sensing, storage and computing integrated sensor 211 performs parallel edge detection on objects in each object edge image through edge detection algorithms, such as Laplacian and Soble algorithms, and uses high-pass filtering to filter out the low pixel density and noise parts in each object edge image, stores the object edge image, records the focal length corresponding to the edge of the clear object in each field of view, and calculates and stores the depth information corresponding to the edge of the clear object.

[0110] The liquid lens 212 is used to adjust from the minimum focal length to the maximum focal length and then from the maximum focal length to the minimum focal length under the control of the sensing, storage and computing integrated sensor 211, and scan the object.

[0111] The terminal 220 is used to perform color assignment processing on the depth information corresponding to the edge of the object stored in the integrated sensing, storage and computing sensor 211, perform image fusion processing on the collected and stored clear object edge image, generate a sparse depth estimation image, and convert the sparse depth estimation image into a dense depth estimation image.

[0112] The terminal 220 converts the sparse depth estimation image into a dense depth estimation image by performing edge filling processing, image corrosion processing and depth filling processing on the sparse depth estimation image.

[0113] Therefore, the liquid lens and the sensor based on integrated sensing, storage and computing can be combined in a variable-focus monocular camera and combined with a terminal to obtain additional depth information. The liquid lens and the sensor based on integrated sensing, storage and computing have a compact structure and are suitable for space-constrained carriers such as embedded devices, and can achieve efficient depth estimation in resource-constrained environments.

[0114] Based on the same concept as above, in another embodiment of the present invention, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor is used to execute a monocular real-time intensive depth estimation method when executing the program.

[0115] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0116] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories in partitions, and the above-mentioned computer programs, computer instructions, data, etc. may be called by a processor.

[0117] The processor is used to execute the computer program stored in the memory to implement the various steps in the method involved in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0118] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0119] In an embodiment of the present invention, a non-temporary computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of a monocular real-time dense depth estimation method in any of the above embodiments are implemented.

[0120] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0121] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0122] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0123] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0124] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0125] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A monocular real-time intensive depth estimation method, characterized in that: include: In the process of controlling the continuous zoom of the liquid lens by the integrated vision sensor for sensing, storage and computing, the integrated vision sensor for sensing, storage and computing is used to collect multiple clear edge images of objects; Preprocessing the edge image of the clear object during image acquisition to determine the focal length corresponding to the edge of the clear object in the field of view of the liquid lens during zooming; Determine the depth information corresponding to the edge of the clear object in the field of view according to the focal length corresponding to the edge of the clear object in the field of view, the distance between the sensing-storage-computing integrated sensor and the liquid lens, and the camera imaging principle; According to the depth information corresponding to the edge of the clear object in the field of view, color is assigned to the edge of the clear object in the field of view, and multiple images of the edge of the clear object are fused to generate a sparse depth estimation image; Perform edge filling processing and depth filling processing on the clear object edges with depth information in the sparse depth estimation image to determine a dense depth estimation image.

2. The method according to claim 1, characterized in that: In the process of controlling the continuous zoom of the liquid lens by the integrated sensing, storage and computing visual sensor, the integrated sensing, storage and computing sensor is used to collect multiple clear edge images of objects, including: Using the sensing, storage and computing integrated sensor to send a control instruction to the liquid lens; According to the control instruction, the liquid lens is controlled to adjust from the minimum focal length to the maximum focal length and then from the maximum focal length to the minimum focal length. When the lens is adjusted to each focal length, images of clear objects in the field of view at each focal length are collected to determine the multiple clear object edge images.

3. The method according to claim 2, characterized in that The preprocessing includes edge detection processing and noise reduction processing; The preprocessing of the clear object edge image during the image acquisition process to determine the focal length corresponding to the clear object edge in the field of view of the liquid lens during the zooming process includes: Performing edge detection processing on the clear object edge image using an edge detection algorithm to determine the object edge image after edge detection processing; According to a preset filtering threshold, high-pass filtering is performed on the object edge image after edge detection processing to determine the focal length corresponding to the edge of the clear object in the field of view during the zooming process of the liquid lens.

4. The method according to claim 1, characterized in that: The camera imaging principle includes: Among them, u represents the distance between the object and the liquid lens, v represents the distance between the integrated sensing, storage and computing sensor and the liquid lens, and f represents the focal length corresponding to the edge of the clear object in the field of view.

5. The method according to claim 1, characterized in that: The method of performing color assignment processing on the edge of the clear object according to the depth information corresponding to the edge of the clear object, and performing image fusion processing on a plurality of edge images of the clear object to generate a sparse depth estimation image includes: According to the depth information corresponding to the clear object edge in each of the clear object edge images, color assignment processing is performed on each clear object edge to determine the object edge image after color assignment processing; The plurality of object edge images that have been subjected to color assignment processing are subjected to image fusion processing to generate the sparse depth estimation image.

6. The method according to claim 1, characterized in that The step of performing edge filling processing and depth filling processing on the edges of clear objects having depth information in the sparse depth estimation image to determine a dense depth estimation image includes: Performing edge filling processing on each clear object edge in the sparse depth estimation image by using a dilation algorithm, an edge interpolation algorithm, an edge detection algorithm, and an edge endpoint detection algorithm to determine a depth estimation image after the edge filling processing; Performing image erosion processing on the depth estimation image that has undergone edge filling processing to determine the depth estimation image that has undergone image erosion processing; According to the depth information corresponding to the clear object edge, depth filling processing is performed on the inside of each clear object edge in the depth estimation image that has undergone image erosion processing to determine the dense depth estimation image.

7. A monocular real-time intensive depth estimation system, characterized in that: include: An image acquisition module is used to acquire multiple clear edge images of objects using the integrated sensing, storage and computing sensor during the process of the integrated sensing, storage and computing visual sensor controlling the continuous zooming of the liquid lens; An image preprocessing module, used to preprocess the edge image of the clear object during image acquisition, and determine the focal length corresponding to the edge of the clear object in the field of view of the liquid lens during zooming; A depth information estimation module, used to determine the depth information corresponding to the edge of the clear object in the field of view according to the focal length corresponding to the edge of the clear object in the field of view, the distance between the sensing, storage and computing integrated sensor and the liquid lens, and the camera imaging principle; A sparse depth estimation module, configured to perform color assignment processing on the edge of a clear object in the field of view according to the depth information corresponding to the edge of the clear object in the field of view, and perform image fusion processing on a plurality of images of the edge of the clear object to generate a sparse depth estimation image; The dense depth estimation module is used to perform edge filling processing and depth filling processing on the edges of clear objects with depth information in the sparse depth estimation image to determine a dense depth estimation image.

8. A monocular real-time intensive depth estimation device, characterized in that: It includes a camera and a terminal, wherein the camera includes a sensing, storage and computing integrated sensor and a liquid lens; The sensing, storage and computing integrated sensor is used to control the continuous zooming of the liquid lens and collect multiple clear object edge images; it is also used to perform edge detection and noise reduction processing on the collected multiple clear object edge images, store the object edge images, record the focal length corresponding to the edge of each clear object in the field of view, and calculate and store the depth information corresponding to the edge of the clear object; The liquid lens is used to adjust from the minimum focal length to the maximum focal length and then from the maximum focal length to the minimum focal length under the control of the sensing, storage and computing integrated sensor, and scan the object; The terminal is used to perform color assignment processing on the depth information corresponding to the clear object edge stored in the integrated sensing, storage and computing sensor, perform image fusion processing on the collected and stored clear object edge image, generate a sparse depth estimation image, and convert the sparse depth estimation image into a dense depth estimation image.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 7.

Images