Liquid zoom vision measurement method for multi-target depth perception
By building a liquid zoom vision system and a deep measurement neural network model, the power changes of liquid lenses are used to solve the problems of low accuracy and time-consuming bifocal imaging method, and fast and high-precision multi-objective depth measurement is achieved, which is suitable for High-performance deep perception of light and small unmanned platforms.
Patent Information
- Application Number
- CN202411235448.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-09-04
AI Technical Summary
Among the existing monocular visual depth measurement methods, the bifocal imaging method has low accuracy and long time, and the focal length value of the liquid zoom imaging system is easily affected by factors such as temperature and object distance, so it is impossible to achieve fast and high-precision depth measurement. .
Using a liquid zoom vision measurement method for multi-objective depth perception, a liquid zoom vision system with wide field of view and narrow field of view imaging state is built, and a deep measurement neural network model is used to combine the power changes of liquid lenses to quickly obtain the target. In-depth information. The method includes building a liquid zoom vision system, obtaining the power of the liquid lens, calculating the depth value of the target through the neural network model, and optimizing the structural parameters of the neural network using a genetic algorithm.
When the target information is unknown, the depth information of multiple objects in the imaging field is quickly obtained, which overcomes the problems of low accuracy and time-consuming of monocular bifocal imaging, improves the response speed and accuracy of the liquid zoom imaging system, and is suitable for High-performance deep perception of light and small unmanned platforms.
Smart Images

Figure CN118735983B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of computer vision, and in particular to a liquid zoom vision measurement method for multi-target depth perception. Background Art
[0002] Monocular vision depth perception is an important emerging branch of computer vision. It has good application prospects in the field of micro-unmanned detection due to its light weight and simple structure. At present, the monocular vision depth measurement methods used by domestic and foreign units are mainly divided into three categories: focusing method, defocusing method and dual-focus imaging method. Among them, the dual-focus imaging method is based on the premise of accurately calibrating the two different focal length states of the imaging system, and obtains the target image point position corresponding to the two focal length states based on image processing technology, and then calculates the target distance information from the focal length value and the image point vector length in combination with the imaging geometry.
[0003] From the current research situation, the dual-focus imaging method adopts a motor-driven displacement zoom system, that is, the motor drives multiple solid lenses to cause relative displacement to achieve the zoom and focusing function. Due to the limited response speed of the displacement optical zoom system, it is impossible to achieve fast zoom imaging, which will affect the real-time performance of monocular vision measurement. Liquid lens is a new type of optical element developed based on the concept of bionics. The focal length can be adjusted autonomously by changing the liquid interface profile. When it is applied to the design of optical zoom system, compared with all-solid lens, the system has significant advantages in response speed and structural volume. However, during the operation of this type of system, the focal length of the liquid lens is easily affected by factors such as temperature and object distance, which makes the focal length value of the system fluctuate to a certain extent. Since the current internal parameter calibration method is usually applicable to a single state of the system, it is impossible to accurately calibrate the dynamically changing system focal length, which limits the application of liquid zoom imaging system in depth measurement. Summary of the invention
[0004] In view of the deficiencies in the above-mentioned prior art, the present invention provides a liquid zoom vision measurement method for multi-target depth perception, which can quickly obtain the depth information of multiple object targets within the imaging field of view when the target information is unknown, and can overcome the shortcomings of the monocular bifocal imaging method such as low accuracy and long time consumption.
[0005] To achieve the above object, the present invention provides a liquid zoom vision measurement method for multi-target depth perception, comprising the following steps:
[0006] Step 1, building a liquid zoom vision system, wherein the liquid zoom vision system has at least two liquid lenses, and the liquid zoom vision system has a wide field of view imaging state and a narrow field of view imaging state;
[0007] Step 2, in the wide field imaging state, imaging a scene including a target based on the liquid zoom vision system, obtaining a first target image after automatic focusing, and defining the current focal power of each liquid lens as a first focal power;
[0008] Step 3, in the narrow field imaging state, imaging a scene including a target based on the liquid zoom vision system, obtaining a second target image after automatic focusing, and defining the current focal power of each liquid lens as the second focal power;
[0009] Step 4: Input the first optical focal length, the second optical focal length, and the area ratio of the target in the first target image and the second target image of each liquid lens into a depth measurement neural network model to obtain a depth value of the target.
[0010] In one embodiment, the depth measurement neural network model includes an input layer, a hidden layer and an output layer, wherein the node transfer function of the hidden layer is Tansig, and the node transfer function of the output layer is Purelin;
[0011] The processing process of the depth measurement neural network model is:
[0012] The input layer obtains the first optical focal length, the second optical focal length, and the area ratio of the target in the first target image and the second target image of each liquid lens, and then inputs them to the hidden layer;
[0013] In the hidden layer, the first optical power, the second optical power of each liquid lens, and the area ratio of the target in the first target image and the second target image are calculated by the node transfer function Tansig to obtain node parameters, which are then input to the output layer;
[0014] In the output layer, the node parameters are transferred through the node transfer function Purelin to obtain the target depth value, which is then output.
[0015] In one of the embodiments, during the training process of the depth measurement neural network model, a genetic algorithm is used to optimize the neural network structure parameters, that is, the global search capability of the genetic operator is used to generate the optimal solution for the weights and thresholds of the node transfer function Tansig and the node transfer function Purelin.
[0016] In one embodiment, the process of obtaining the area of the target in the image is:
[0017] After Gaussian smoothing of the image, the threshold corresponding to the maximum inter-class variance is automatically generated based on the Otsu algorithm, and the image is converted into a binary image according to the threshold;
[0018] After the binary image is processed by morphological filtering, the connected domain corresponding to the target in the binary image is extracted, and the position coordinates of the target contour in the image are obtained by tracing the boundaries of the connected domain to determine the target area in the image;
[0019] Perform strip segmentation on the target area in the image and calculate the area of the target in the image based on the segmentation results ,for:
[0020] ;
[0021] in, The number of horizontal lines after strip segmentation of the target area. is the number of horizontal lines in the target area that contain at least 2 pixels. For the The coordinate value of the left boundary point of the horizontal line in the u direction of the image coordinate system, For the The right boundary point of the horizontal line is in the image coordinate system Coordinate value in direction.
[0022] In one embodiment, the process of obtaining the first optical power and the second optical power is:
[0023] For a single liquid lens, change its input voltage , and read real-time temperature data through the temperature sensor , using a focal meter to measure the focal length , and obtain a series of discrete data ;
[0024] Based on all discrete data , the focal power function of the liquid lens is obtained by cubic spline interpolation fitting ;
[0025] Obtain the real-time input voltage and temperature data of the liquid lens when taking the first target image, and substitute them into the optical power function The first optical focal power can be obtained;
[0026] Obtain the real-time input voltage and temperature data of the liquid lens when taking the second target image, and substitute them into the optical power function The second optical power can be obtained.
[0027] In one embodiment, the wide field of view imaging state and the narrow field of view imaging state of the liquid zoom vision system are determined by a pre-calibration method, wherein the focal length of the liquid zoom vision system in the narrow field of view imaging state is more than 1.8 times the focal length in the wide field of view imaging state.
[0028] Compared with the prior art, the present invention has the following beneficial technical effects:
[0029] The present invention utilizes the adjustment characteristics of liquid lenses and neural network technology to quickly obtain depth information of multiple objects within the imaging field of view when the target information is unknown. It can overcome the shortcomings of monocular dual-focus imaging methods such as low precision and long time consumption, and can also be applied to light and small unmanned platforms, providing a new technical solution for realizing high-performance depth perception of monocular vision. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.
[0031] Figure 1 This is a flow chart of a liquid zoom vision measurement method for multi-target depth perception in an embodiment of the present invention;
[0032] Figure 2 A structural diagram of an implementation scheme of a liquid zoom vision system in an embodiment of the present invention;
[0033] Figure 3 It is a structural block diagram of the depth measurement neural network model in an embodiment of the present invention.
[0034] Figure numerals: solid lens group 1, photosensitive chip 2, three liquid lenses 3.
[0035] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0037] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.
[0038] In addition, in the present invention, descriptions such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0039] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.
[0040] like Figure 1 The figure shows a liquid zoom vision measurement method for multi-target depth perception disclosed in this embodiment, which mainly includes the following steps:
[0041] Step 1, building a liquid zoom vision system, wherein the liquid zoom vision system has at least two liquid lenses, and the liquid zoom vision system has a wide field of view imaging state and a narrow field of view imaging state;
[0042] Step 2, in a wide field imaging state, imaging a scene including a target based on a liquid zoom vision system, obtaining a first target image after automatic focusing, and defining the current focal power of each liquid lens as a first focal power;
[0043] Step 3, in a narrow field of view imaging state, imaging a scene including the target based on the liquid zoom vision system, obtaining a second target image after automatic focusing, and defining the current focal power of each liquid lens as the second focal power;
[0044] Step 4: Input the first optical focal length, the second optical focal length, and the area ratio of the target in the first target image and the second target image of each liquid lens into a depth measurement neural network model to obtain a depth value of the target.
[0045] refer to Figure 2 , is an implementation example of the liquid zoom vision system in this embodiment, which includes a solid lens group 1, a photosensitive chip 2 and three liquid lenses 3. The solid lens group 1 is used to undertake the system's optical focal length and aberration correction; the three liquid lenses 3 are connected to the host computer through a dedicated driver circuit board, and the host computer sends instructions to control the input voltage of the three liquid lenses to achieve the zoom and focus functions of the system; the photosensitive chip 2 is used to collect images and transmit the images to the host computer for display through a data line. Among them, the photosensitive chip can use CCD or CMOS.
[0046] In this embodiment, the wide field imaging state and the narrow field imaging state of the liquid zoom vision system are determined by a pre-calibration method, and the calibrated focal length of the liquid zoom vision system in the narrow field imaging state is more than 1.8 times the calibrated focal length in the wide field imaging state.
[0047] In the specific implementation process, the process of obtaining the area of the target in the image is:
[0048] Firstly, after Gaussian smoothing of the image, the threshold corresponding to the maximum inter-class variance is automatically generated based on the Otsu algorithm, and the image is converted into a binary image according to the threshold.
[0049] Among them, the binary image is subjected to morphological filtering, that is, an opening operation is performed first and a closing operation is performed later, which is used to eliminate noise in the image and refine the contour of the target in the image;
[0050] Then, the connected domain corresponding to the target in the binary image is extracted, and the position coordinates of the target contour in the image are obtained by tracing the boundaries of the connected domain. The image target contours under wide field of view and narrow field of view are encoded respectively in the form of 8-connected Freeman chain code to determine the target area in the image.
[0051] Finally, the target area in the image is segmented into strips, and the area of the target in the image is calculated based on the segmentation results. .
[0052] For example, suppose the target area can be divided into horizontal lines composed of pixels, where A horizontal line contains at least 2 points of the target area, If a horizontal line contains only one point in the target area, then the area of the target area in the image is (in pixels) can be calculated as:
[0053] ;
[0054] in, For the The coordinate value of the left boundary point of the horizontal line in the u direction of the image coordinate system, For the The coordinate value of the right boundary point of the horizontal line in the u direction of the image coordinate system. Traverse all chain code elements of the target contour and classify the target contour data according to the judgment principle shown in Table 1. In Table 1, L represents the left boundary point; R represents the right boundary point; LR represents the single pixel boundary (corresponding to the horizontal line composed of a single pixel point); O represents neither the left boundary point nor the right boundary point. Substituting the statistical results into the above formula can obtain the target area, and then calculate the ratio of the corresponding areas of the same object in the first target image and the second target image. .
[0055] Table 1 Principles for determining contour point categories based on chain codes
[0056]
[0057] The liquid lens in this embodiment adopts an electrowetting liquid lens, which can adjust the optical power by changing the input voltage. Since the optical power of the liquid lens will fluctuate due to temperature changes, real-time compensation of temperature factors must be considered when calibrating the parameters of this type of liquid lens. Specifically in this embodiment, the process of obtaining the first optical power and the second optical power is as follows:
[0058] For a single liquid lens, change its input voltage , and read real-time temperature data through the temperature sensor , using a focal meter to measure the focal length , and obtain a series of discrete data ;
[0059] Based on all discrete data , the focal power function of the liquid lens is obtained by cubic spline interpolation fitting ;
[0060] Obtain the real-time input voltage and temperature data of the liquid lens when taking the first target image, and substitute them into the optical power function The first optical power can be obtained;
[0061] Obtain the real-time input voltage and temperature data of the liquid lens when taking the second target image, and substitute them into the optical power function The second optical power can be obtained.
[0062] refer to Figure 3 The depth measurement neural network model in this embodiment includes an input layer, a hidden layer and an output layer, wherein the number of nodes in the input layer is 7; the number of nodes in the hidden layer is 9, and the node transfer function of the hidden layer is Tansig; the number of nodes in the output layer is 1, and the node transfer function of the output layer is Purelin. The data processing process of the depth measurement neural network model is:
[0063] First, the input layer obtains the first optical power, the second optical power of each liquid lens, and the area ratio of the target in the first target image and the second target image, and then inputs them to the hidden layer;
[0064] Secondly, the first optical power, the second optical power, and the area ratio of the target in the first target image and the second target image of each liquid lens in the hidden layer are calculated by the node transfer function Tansig to obtain the node parameters, which are then input to the output layer;
[0065] Finally, the output layer node parameters are transferred through the node Purelin function to obtain the target depth value and output it.
[0066] In this embodiment, a genetic algorithm is used to train and optimize the neural network structure parameters, that is, the global search capability of the genetic operator is used to generate the optimal solution of the weight and threshold of the node transfer function Tansig and the node transfer function Purelin. During the training process, the first optical power and the second optical power contained in the training data set are obtained by substituting the actual input voltage and the actual temperature data into the optical power function The area ratio of the target in the first target image and the second target image is obtained by calculating the area using the aforementioned formula, and the actual depth value of the target is obtained using a laser rangefinder.
[0067] During actual measurement, the liquid zoom vision system is first switched to a wide field imaging state to image the first target image, and then the liquid zoom vision system is switched to a narrow field imaging state to image the second target image. Since the liquid zoom vision system will perform automatic focusing during the imaging process, the actual focal length of the liquid zoom vision system will fluctuate at an upper limit compared to the calibrated focal length, and the actual focal length of the liquid zoom vision system is difficult to accurately obtain. Therefore, this embodiment utilizes the liquid lens adjustment characteristics and neural network technology to effectively avoid the influence of actual focal length fluctuations on the measurement results, and quickly obtains the depth information of multiple objects in the imaging field of view when the target information is unknown, thereby overcoming the shortcomings of the monocular dual-focus imaging method such as low accuracy and long time consumption.
[0068] It is worth noting that although this embodiment Figure 1 The steps in the process are shown in sequence as indicated by the arrows, but the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of the steps, and the steps can be executed in other orders. Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0069] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the inventive concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.
Claims
1. A liquid zoom vision measurement method for multi-target depth perception, characterized in that: The steps include: Step 1, building a liquid zoom vision system, wherein the liquid zoom vision system has at least two liquid lenses, and the liquid zoom vision system has a wide field of view imaging state and a narrow field of view imaging state; Step 2, in the wide field imaging state, imaging a scene including a target based on the liquid zoom vision system, obtaining a first target image after automatic focusing, and defining the current focal power of each liquid lens as a first focal power; Step 3, in the narrow field imaging state, imaging a scene including a target based on the liquid zoom vision system, obtaining a second target image after automatic focusing, and defining the current focal power of each liquid lens as the second focal power; Step 4, inputting the first optical focal length, the second optical focal length, and the area ratio of the target in the first target image and the second target image into a depth measurement neural network model to obtain a depth value of the target; The depth measurement neural network model includes an input layer, a hidden layer and an output layer, wherein the node transfer function of the hidden layer is Tansig, and the node transfer function of the output layer is Purelin; The processing process of the depth measurement neural network model is: The input layer obtains the first optical focal length, the second optical focal length, and the area ratio of the target in the first target image and the second target image of each liquid lens, and then inputs them to the hidden layer; In the hidden layer, the first optical power, the second optical power of each liquid lens, and the area ratio of the target in the first target image and the second target image are calculated by the node transfer function Tansig to obtain node parameters, which are then input to the output layer; In the output layer, the node parameters are transferred through the node transfer function Purelin to obtain the target depth value, which is then output.
2. The liquid zoom vision measurement method for multi-target depth perception according to claim 1 is characterized in that: During the training process of the depth measurement neural network model, a genetic algorithm is used to optimize the neural network structure parameters, that is, the global search capability of the genetic operator is used to generate the optimal solution of the weights and thresholds of the node transfer function Tansig and the node transfer function Purelin.
3. The liquid zoom vision measurement method for multi-target depth perception according to claim 1 or 2, characterized in that: The process of obtaining the area of the target in the image is: After Gaussian smoothing of the image, the threshold corresponding to the maximum inter-class variance is automatically generated based on the Otsu algorithm, and the image is converted into a binary image according to the threshold; After the binary image is processed by morphological filtering, the connected domain corresponding to the target in the binary image is extracted, and the position coordinates of the target contour in the image are obtained by tracing the boundaries of the connected domain to determine the target area in the image; Perform strip segmentation on the target area in the image and calculate the area of the target in the image based on the segmentation results ,for: in, The number of horizontal lines after strip segmentation of the target area. is the number of horizontal lines in the target area that contain at least 2 pixels. For the The coordinate value of the left boundary point of the horizontal line in the u direction of the image coordinate system, For the The coordinate value of the right boundary point of the horizontal line in the u direction of the image coordinate system.
4. The liquid zoom vision measurement method for multi-target depth perception according to claim 1 or 2, characterized in that: The process of obtaining the first optical power and the second optical power is as follows: For a single liquid lens, change its input voltage , and read real-time temperature data through the temperature sensor , using a focal meter to measure the focal length , and obtain a series of discrete data ; Based on all discrete data , the focal power function of the liquid lens is obtained by cubic spline interpolation fitting ; Obtain the real-time input voltage and temperature data of the liquid lens when taking the first target image, and substitute them into the optical power function The first optical focal power can be obtained; Obtain the real-time input voltage and temperature data of the liquid lens when taking the second target image, and substitute them into the optical power function The second optical power can be obtained.
5. The liquid zoom vision measurement method for multi-target depth perception according to claim 1 or 2, characterized in that: The wide field imaging state and the narrow field imaging state of the liquid zoom vision system are both determined by a pre-calibration method, wherein the focal length of the liquid zoom vision system in the narrow field imaging state is more than 1.8 times the focal length in the wide field imaging state.
