Real-time automatic focusing path tracking method and system, medium and equipment

Abstract

The invention provides a real-time automatic focusing path tracking method and system, a medium and equipment, and the method comprises the steps: carrying out the global search and optimal focal length initialization of a liquid lens, and determining an initial focal length position; if the current target is in the out-of-focus state, determining an optimal focus of the current target by adopting a hill-climbing search algorithm; determining pixel coordinates of a central point of the current target at the optimal focus position of the current target according to vertex coordinates of a bounding box of the current target; determining image coordinates of the central point of the current target according to the pixel coordinates of the central point of the current target; determining camera coordinates of the central point of the current target according to the image coordinates of the central point of the current target; and determining a path track of the current target according to the real-time camera coordinates of the current target. Through the method and the device, automatic focusing can be performed in a dynamic scene, path tracking is performed on a dynamic object, and the accuracy and the response speed of automatic focusing are improved on the premise of relatively low power consumption.

Description

Technical Field

[0001] The present disclosure relates to the technical field of optical imaging and image processing on a photosensitive plane, and in particular, to a real-time autofocus path tracking method, system, medium and device. Background Art

[0002] Vision chips are widely used in machine vision, consumer electronics, autonomous driving, security monitoring, medical imaging, logistics and other fields that require image capture, processing and analysis. These application scenarios rely on the accuracy of post-processed visual data. Autofocus technology can ensure that the images captured by the vision chip remain clear and stable at different distances, light and scene changes, providing high-quality data input for downstream visual processing algorithms, thereby improving the overall performance and reliability of the system.

[0003] Traditional autofocus systems usually rely on optical sensors, independent processing units, and mechanically movable lenses to achieve image acquisition and processing. This method has a slow system response and high energy consumption due to the data transmission bottleneck between the sensor and the processor, as well as the forward and backward movement of the lens control. In the autofocus process, traditional methods usually use phase detection or contrast detection technology, but these methods perform poorly when processing images in dynamic scenes or complex lighting conditions. Summary of the invention

[0004] In view of the defects in the prior art, the purpose of the present invention is to provide a real-time auto-focus path tracking method, system, medium and device.

[0005] According to one aspect of the present disclosure, a real-time auto-focus path tracking method is provided, comprising:

[0006] Performing a global search and initializing an optimal focal length on the liquid lens to determine an initial focal length position;

[0007] If the current target is out of focus, a hill climbing search algorithm is used to determine the optimal focus of the current target;

[0008] At the optimal focus position of the current target, determining the pixel coordinates of the center point of the current target according to the vertex coordinates of the rectangular bounding box of the current target;

[0009] Determining the image coordinates of the center point of the current target according to the pixel coordinates of the center point of the current target;

[0010] Determine the camera coordinates of the center point of the current target according to the image coordinates of the center point of the current target;

[0011] The path trajectory of the current target is determined according to the real-time camera coordinates of the center point of the current target.

[0012] Optionally, if the current target is in an out-of-focus state, using a hill climbing search algorithm to determine the optimal focus of the current target includes:

[0013] Using a preset first fixed step length to adjust the focal length of the liquid lens barrel from the initial focal length position to determine the moving direction of the current target;

[0014] In the moving direction of the current target, the preset first fixed step length is used to adjust the focal length of the liquid lens to determine the optimal focus corresponding to the current target.

[0015] Optionally, the adjusting the focal length of the liquid lens barrel from the initial focal length position by using a preset first fixed step length to determine the moving direction of the current target includes:

[0016] Using the preset first fixed step length to adjust the focal length of the liquid lens barrel from the initial focal length position in each direction, and determine a scanned image of the current target in each direction;

[0017] The moving direction of the current target is determined according to the focus value of the scanned image of the current target in each direction.

[0018] Optionally, performing a global search and initializing an optimal focal length on the liquid lens to determine an initial focal length position includes:

[0019] Using a sensing, storage and computing integrated sensor device to send control instructions to the liquid lens;

[0020] According to the control instruction, the liquid lens is controlled to traverse all focal lengths to determine the optimal focal length;

[0021] According to the optimal focal length, the optimal focal length of the liquid lens is initialized to determine the initial focal length position.

[0022] Optionally, determining the pixel coordinates of the center point of the current target according to the vertex coordinates of a rectangular bounding box of the current target at the optimal focus position of the current target includes:

[0023] At the optimal focus position of the current target, the pixel coordinates of the center point of the current target are determined according to the coordinates of the lower left vertex of the rectangular bounding box of the current target and the coordinates of the upper right vertex of the rectangular bounding box of the current target.

[0024] Optionally, determining the image coordinates of the center point of the current target according to the pixel coordinates of the center point of the current target includes:

[0025] The pixel coordinates of the center point of the current target are converted into an image coordinate system to determine the image coordinates of the center point of the current target.

[0026] Optionally, determining the camera coordinates of the center point of the current target according to the image coordinates of the center point of the current target includes:

[0027] The image coordinates of the center point of the current target are converted into a camera coordinate system to determine the camera coordinates of the center point of the current target.

[0028] Optionally, the method further comprises:

[0029] If the current target is not in an out-of-focus state, the pixel coordinates of the center point of the current target are determined according to the vertex coordinates of the rectangular bounding box of the current target at the best focal length position of the current target.

[0030] According to a second aspect of the present disclosure, a real-time auto-focus path tracking system is provided, comprising:

[0031] An initialization module, used for performing a global search and initializing the optimal focal length of the liquid lens to determine an initial focal length position;

[0032] An autofocus module, used to determine the optimal focus of the current target using a hill climbing search algorithm if the current target is out of focus;

[0033] A pixel coordinate determination module, used to determine the pixel coordinates of the center point of the current target according to the coordinates of the upper vertex of the boundary box of the current target at the optimal focus position of the current target;

[0034] An image coordinate determination module, used to determine the image coordinates of the center point of the current target according to the pixel coordinates of the center point of the current target;

[0035] A camera coordinate determination module, used to determine the camera coordinates of the center point of the current target according to the image coordinates of the center point of the current target;

[0036] The path tracking module is used to determine the path trajectory of the current target according to the real-time camera coordinates of the center point of the current target.

[0037] According to a third aspect of the present disclosure, a non-temporary computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods provided in the first aspect of the present disclosure are implemented.

[0038] According to a fourth aspect of the present disclosure, there is provided an electronic device, characterized in that it includes:

[0039] a memory having a computer program stored thereon;

[0040] A processor is used to execute the computer program in the memory to implement the steps of any one of the methods provided in the first aspect of the present disclosure.

[0041] Compared with the prior art, the embodiments of the present disclosure have at least one of the following beneficial effects:

[0042] Through the above technical solution, a hill climbing search algorithm is used to adjust the focal length of the liquid lens, optimize the focus path, and determine the optimal focus of the current target. There is no need to search the entire focusing range, and the optimal focus of the current target can be quickly determined with less focusing, thereby improving the efficiency of autofocus and reducing computing resource consumption. The pixel coordinates of the center point of the current target are determined at the optimal focus position, and then the two-dimensional pixel coordinates of the center point of the current target are converted to the three-dimensional camera coordinates of the center point of the current target to provide three-dimensional information for path tracking, effectively combining depth estimation to perform path tracking of dynamic objects, and improving the accuracy of path tracking.

[0043] The embodiments of the present disclosure use a sensing, storage and computing integrated sensor device and a liquid lens to acquire a scanned image, determine the optimal focal length, provide depth information support for autofocus, and enhance the robustness of path tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Other features, objects and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0045] Figure 1 The figure is a flowchart of a real-time auto-focus path tracking method according to an exemplary embodiment.

[0046] Figure 2 The figure is a schematic diagram showing the zoom principle of a liquid lens according to an exemplary embodiment.

[0047] Figure 3 It is a schematic diagram showing the principle of a hill climbing search algorithm according to an exemplary embodiment.

[0048] Figure 4 The figure is a schematic diagram showing the conversion between pixel coordinates and image coordinates according to an exemplary embodiment.

[0049] Figure 5 The figure is a schematic diagram showing the conversion between image coordinates and camera coordinates according to an exemplary embodiment.

[0050] Figure 6 The invention is a block diagram of a real-time auto-focus path tracking system according to an exemplary embodiment.

[0051] Figure 7 The diagram is a schematic diagram of the architecture of a real-time auto-focus path tracking system according to an exemplary embodiment. DETAILED DESCRIPTION

[0052] The present disclosure is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present disclosure, but are not intended to limit the present disclosure in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present disclosure. These all fall within the scope of protection of the present disclosure.

[0053] The present invention provides a real-time automatic focus path tracking method, which adjusts the curvature of the liquid lens by applying different voltages to the liquid lens to adjust the focal length, and uses a sensing, storage and computing integrated sensor device to calculate the image contrast in real time during the process of adjusting the focal length of the liquid lens, and tracks and obtains the optimal focal length image by using a short-distance hill climbing search algorithm.

[0054] Figure 1 The figure is a flowchart of a real-time auto-focus path tracking method according to an exemplary embodiment.

[0055] like Figure 1 As shown, the present disclosure provides a real-time auto-focus path tracking method, including S11 to S16.

[0056] S11, performing a global search and initializing the optimal focal length of the liquid lens to determine an initial focal length position.

[0057] Among them, global search means combining an integrated sensing, storage and computing sensor device with a liquid lens to traverse the entire focal length of the liquid lens and use parallel edge extraction technology to extract edge information.

[0058] Optimal focal length initialization means initializing the focal length of the liquid lens to the optimal focal length of the current target.

[0059] The sensor device of the present disclosure adopts a vision chip based on the focal plane sensor processor (FPSP), which has the ability to capture, store and process visual data at the pixel level. The core component of the vision chip based on the focal plane sensor processor is a pixel processor array (PPA) composed of processing elements. The vision chip based on the focal plane sensor processor can extract and directly output key information from the image without transferring all image information to an external computer for processing. It can perform image acquisition, digital-to-analog conversion and data transmission, and perform image processing operations at a speed of up to 1,000 frames per second, avoiding dependence on external computing resources and professional software.

[0060] The liquid lens disclosed in the present invention can change the charge distribution and conductivity inside the liquid lens by applying different external voltages or electric fields, thereby adjusting the shape and refractive index of the liquid lens to achieve focusing and zooming of light.

[0061] Figure 2 The figure is a schematic diagram showing the zoom principle of a liquid lens according to an exemplary embodiment.

[0062] like Figure 2 As shown, the liquid lens consists of two parallel transparent electrodes, and a liquid material with an electronic refractive index is filled between the two transparent electrodes.

[0063] When an electric field is applied to the liquid lens, the charge distribution inside the liquid material changes, and the electric field changes the shape of the liquid material, thereby changing the refractive index of the liquid. By adjusting the size and distribution of the electric field, the shape and refractive index of the liquid lens can be precisely controlled, the light can be focused and zoomed, and the focal length of the liquid lens can be adjusted.

[0064] S12, if the current target is out of focus, a hill climbing search algorithm is used to determine the optimal focus of the current target.

[0065] The current target may be a dynamic object. After the current target is displaced, its initial focal length position is no longer the optimal focal length of the current target, and the current target is in a defocused state.

[0066] Figure 3 It is a schematic diagram showing the principle of a hill climbing search algorithm according to an exemplary embodiment.

[0067] like Figure 3As shown in the figure, the basic principle of the hill climbing search algorithm is: during the focusing process, the focus value information corresponding to the scanned image collected by the front and back adjusted focal length is used to determine the slope peak position. In the process of using the hill climbing search algorithm to determine the optimal focus, the integrated sensing, storage and computing device and the liquid lens collect the scanned image corresponding to each focal length in real time.

[0068] Among them, starting from a preset fixed step length, when the focus value increases continuously, it is determined and recorded that the current state is a climbing state, and when the focus value decreases continuously, it is determined and recorded that the current state is a downhill state and has passed the slope peak position. When it is determined to be a downhill state, the fixed step length is adjusted in the opposite direction to be smaller than the fixed step length of the previous climbing, and the climbing is repeated again, until the maximum focus value, that is, the slope peak position, is determined as the optimal focal length of the current target, and the focus formed is the optimal focus.

[0069] The hill climbing search algorithm does not need to search the entire focusing range. It can quickly and accurately determine the position of the optimal focus with fewer focusing steps, improve the efficiency of determining the optimal focus, and reduce the consumption of computing resources.

[0070] S13, at the optimal focus position of the current target, determine the pixel coordinates of the center point of the current target according to the vertex coordinates of the rectangular bounding box of the current target.

[0071] The rectangular bounding box of the current target is the largest bounding box of the current target, and its vertex coordinates include the lower left vertex coordinates and the upper right vertex coordinates.

[0072] S14, determining the image coordinates of the center point of the current target according to the pixel coordinates of the center point of the current target.

[0073] S15, determining the camera coordinates of the center point of the current target according to the image coordinates of the center point of the current target.

[0074] The pixel coordinates of the center point of the current target are two-dimensional coordinate information, and the camera coordinates of the center point of the current target are three-dimensional coordinate information. The coordinate system conversion method in machine vision, such as trigonometric transformation and matrix calculation, is used to convert the pixel coordinates of the center point of the current target into the camera coordinates of the center point of the current target via image coordinates to provide three-dimensional information support for path tracking.

[0075] S16, determining the path trajectory of the current target according to the real-time camera coordinates of the center point of the current target.

[0076] Among them, the camera coordinates of the center point of the real-time current target can be used to draw several point sets in the three-dimensional coordinate system to generate the path trajectory of the current target, thereby achieving stable focus and path tracking in dynamic scenes.

[0077] Through the above technical solution, through the above technical solution, a hill climbing search algorithm is used to adjust the focal length of the liquid lens, optimize the focusing path, and determine the optimal focus of the current target. There is no need to search the entire focusing range, and the optimal focus of the current target is quickly determined with less focusing, thereby improving the efficiency of autofocus and reducing computing resource consumption. The pixel coordinates of the center point of the current target are determined at the optimal focus position, and then the two-dimensional pixel coordinates of the center point of the current target are converted to the three-dimensional camera coordinates of the center point of the current target to provide three-dimensional information for path tracking, effectively combining depth estimation to perform path tracking of dynamic objects, and improving the accuracy of path tracking.

[0078] In a possible embodiment, S11 may include S21 to S23.

[0079] S21, using a sensing, storage and computing integrated sensor device to send control instructions to the liquid lens.

[0080] Among them, the sensing, storage and computing integrated sensor device adopts a sensing, storage and computing integrated vision chip based on a focal plane sensor processor as the core computing unit, which communicates with the liquid lens through a communication protocol. The sensing, storage and computing integrated vision chip based on the focal plane sensor processor sends control signals, i.e., control instructions, to the liquid lens through the communication protocol.

[0081] S22, according to the control instruction, controlling the liquid lens to traverse all focal lengths and determining the optimal focal length.

[0082] Among them, traversing all focal lengths can control the focus of the liquid lens to change from the minimum value to the maximum value, or from the maximum value to the minimum value, and quickly collect scanned images through the pixel processor array of the sensing, storage and computing integrated sensor device, and collect scanned images corresponding to each focus; use edge detection algorithms, such as Sobel edge detection algorithm and Laplace edge detection algorithm, to perform parallel edge detection processing on the scanned images, extract edges in the scanned images, measure the image sharpness of the scanned images, and based on the Gaussian distribution characteristics of the image sharpness of the scanned images that change with the focus, determine that the focus corresponding to the maximum image sharpness value is the optimal focus.

[0083] Among them, the Sobel edge detection algorithm and the Laplace edge detection algorithm can be realized by high-speed parallelization through parallel operation of analog registers on the focal plane sensor processor integrated vision chip (Focal-Plane-Sensor-Processor, FPSP), and the extraction of edge information can be completed once within less than 13μm.

[0084] The integrated sensing, storage and computing sensor device and liquid lens can complete a focal length adjustment, scan image acquisition, parallel edge detection processing and image sharpness measurement process within ten microseconds.

[0085] S23, initializing the optimal focal length of the liquid lens according to the optimal focal length to determine the initial focal length position.

[0086] By adopting a sensing, storage and computing integrated sensor device and a liquid lens to acquire the scanned image, the optimal focal length is determined, depth information support is provided for autofocus, and the robustness of path tracking is enhanced.

[0087] In a possible embodiment, S12 may include S31 to S32.

[0088] S31, using a preset first fixed step length to adjust the focal length of the liquid lens barrel from an initial focal length position to determine a moving direction of the current target.

[0089] Among them, according to the principle of the above-mentioned hill climbing search algorithm, a short-distance hill climbing search is used to determine the moving direction of the current target.

[0090] In a possible embodiment, S31 may include S311 to S312.

[0091] S311, adjusting the focal length of the liquid lens barrel from the initial focal length position in each direction using a preset first fixed step length to determine a scanned image of the current target in each direction.

[0092] After adjusting the focal length of a preset first fixed step length in each direction once, a sensing, storage and computing integrated sensor device and a liquid lens are used to collect a scanned image of the current target.

[0093] S312, determining the moving direction of the current target according to the focus value of the scanned image of the current target in each direction.

[0094] Continuing with the above example, the focus values ​​of the scanned image of the current target in each direction are respectively subtracted from the focus values ​​of the scanned image of the current target collected at the initial focal length position, and the direction corresponding to the scanned image with the maximum focus value difference is used as the moving direction of the current target.

[0095] S32, in the moving direction of the current target, using a preset first fixed step length, adjusting the focal length of the liquid lens to determine the optimal focus corresponding to the current target.

[0096] According to the basic principle of the hill climbing search algorithm described above, the focal length of the liquid lens is adjusted.

[0097] As an example, in the moving direction of the current target, a preset first fixed step length is used to adjust the focal length of the liquid lens multiple times, and after each focal length adjustment, a sensing, storage and computing integrated sensor device and a liquid lens are used to collect a scanning image corresponding to each focal length.

[0098] If the focus value of the scanned image corresponding to the current focal length is greater than the focus value of the scanned image corresponding to the previous focal length, it indicates that the current state is in a climbing state. Continue to use the preset first fixed step size to climb until the focus value of the scanned image corresponding to the current focal length is no greater than the focus value of the scanned image corresponding to the previous focal length, indicating that the current state is in a downhill state and the peak of the slope has been passed.

[0099] Reduce the fixed step size, and use a second fixed step size that is smaller than the preset first fixed step size to climb again in the opposite direction of the moving direction of the current target, until the focus value of the scanned image corresponding to the current focal length is no greater than the focus value of the scanned image corresponding to the previous focal length, change the direction again and reduce the fixed step size, repeat the above steps until the maximum focus value is determined, and determine the focus corresponding to the focal length with the maximum focus value as the optimal focus.

[0100] In the process of automatic focusing using the hill climbing search algorithm, the contrast value of the focus can be used as an evaluation index. The higher the contrast value, the higher the focus value, which means the focus is clearer, so as to determine the optimal focus. The automatic focusing process takes less than 100ms.

[0101] In a possible embodiment, S13 may include:

[0102] At the optimal focus position of the current target, the pixel coordinates of the center point of the current target are determined according to the coordinates of the lower left vertex of the rectangular bounding box of the current target and the coordinates of the upper right vertex of the rectangular bounding box of the current target.

[0103] As an example, the average value of the horizontal coordinates of the lower left vertex coordinates and the upper right vertex coordinates of the rectangular bounding box of the current target is rounded as the horizontal coordinate of the pixel of the center point of the current target, and the average value of the vertical coordinates of the lower left vertex coordinates and the upper right vertex coordinates of the rectangular bounding box of the current target is rounded as the vertical coordinate of the pixel of the center point of the current target to determine the pixel coordinates of the center point of the current target.

[0104] In a possible embodiment, S14 may include S41.

[0105] S41, converting the pixel coordinates of the center point of the current target into an image coordinate system, and determining the image coordinates of the center point of the current target.

[0106] Among them, the pixel coordinate system and the image coordinate system are both on the imaging plane. The origin of the pixel coordinate system is the upper right corner of the output image, and the unit of the pixel coordinate system is pixel; the origin of the image coordinate system is the intersection of the camera optical axis and the imaging plane, usually the midpoint of the imaging plane, and the unit of the image coordinate system is millimeter mm.

[0107] The conversion relationship between the pixel coordinate system and the image coordinate system is 1pixel = dx mm. dx represents the millimeter of each column, and dy represents the millimeter of each row.

[0108] Figure 4 The figure is a schematic diagram showing the conversion between pixel coordinates and image coordinates according to an exemplary embodiment.

[0109] like Figure 4 As shown, as an example, O(u, v) represents the origin of the pixel coordinate system, u represents, v represents, o represents the origin of the image coordinate system, x represents the horizontal axis of the image coordinate system, and y represents the vertical axis of the image coordinate system. 0, v0) is converted to the coordinates p(x,y) in the image coordinate system. The specific process is as follows:

[0110]

[0111] In a possible embodiment, S15 may include S42.

[0112] S42, converting the image coordinates of the center point of the current target into a camera coordinate system, and determining the camera coordinates of the center point of the current target.

[0113] The image coordinate system is located on a two-dimensional plane, and the camera coordinate system is located in a three-dimensional space. Converting the image coordinates to the camera coordinate system is a perspective projection relationship from 2D to 3D. The transformation matrix can be obtained using the principle of similar triangles.

[0114] Figure 5 The figure is a schematic diagram showing the conversion between image coordinates and camera coordinates according to an exemplary embodiment.

[0115] like Figure 5 As shown, as an example, O c Indicates the origin of the camera coordinate system, X c Represents the horizontal axis of the camera coordinate system, Y c Represents the vertical axis of the camera coordinate system, Z c represents the depth axis of the camera coordinate system, o represents the origin of the image coordinate system, x represents the horizontal axis of the image coordinate system, y represents the vertical axis of the image coordinate system, p(x,y) represents the two-dimensional projection point in the image coordinate system, P(X c ,Y c ,Z c ) represents the three-dimensional coordinate point in the camera coordinate system, and converts the coordinate p(x,y) in the image coordinate system to the coordinate P(X c ,Y c ,Z c ), the specific process is as follows:

[0116] ΔABO C ~ΔoCO C

[0117] ΔPBO C ~ΔpCO C

[0118]

[0119] In a possible embodiment, a real-time automatic focusing path tracking method may further include S16.

[0120] S16, if the current target is not in a defocused state, at the best focal length position of the current target, the pixel coordinates of the center point of the current target are determined according to the vertex coordinates of the rectangular bounding box of the current target.

[0121] Among them, if the current target corresponds to the best focal length and is not in a defocused state, the pixel coordinates of the center point of the current target can be directly determined according to the initial focal length position of the current target or the maximum rectangular bounding box of the optimal focal position, that is, the vertex coordinates of the largest bounding box.

[0122] Step S16 of the present disclosure may be performed after step S11 or step S12.

[0123] Figure 6 The invention is a block diagram of a real-time auto-focus path tracking system according to an exemplary embodiment.

[0124] Based on the same concept, the present disclosure also provides a real-time automatic focus path tracking system 100, such as Figure 6 As shown, it includes: an initialization module 110 , an auto-focus module 120 , a pixel coordinate determination module 130 , an image coordinate determination module 140 , a camera coordinate determination module 150 , and a path tracking module 160 .

[0125] An initialization module 110 is used to perform a global search and optimal focal length initialization on the liquid lens to determine an initial focal length position;

[0126] The auto-focus module 120 is used to determine the optimal focus of the current target by using a hill climbing search algorithm if the current target is out of focus;

[0127] A pixel coordinate determination module 130, configured to determine the pixel coordinates of the center point of the current target according to the vertex coordinates of the boundary box of the current target at the optimal focus position of the current target;

[0128] An image coordinate determination module 140, configured to determine the image coordinates of the center point of the current target according to the pixel coordinates of the center point of the current target;

[0129] A camera coordinate determination module 150, configured to determine the camera coordinates of the center point of the current target according to the image coordinates of the center point of the current target;

[0130] The path tracking module 160 is used to determine the path trajectory of the current target according to the real-time camera coordinates of the center point of the current target.

[0131] Through the above technical solution, a hill climbing search algorithm is used to adjust the focal length of the liquid lens, optimize the focus path, and determine the optimal focus of the current target. There is no need to search the entire focusing range, and the optimal focus of the current target can be quickly determined with less focusing, thereby improving the efficiency of autofocus and reducing computing resource consumption. The pixel coordinates of the center point of the current target are determined at the optimal focus position, and then the two-dimensional pixel coordinates of the center point of the current target are converted to the three-dimensional camera coordinates of the center point of the current target to provide three-dimensional information for path tracking, effectively combining depth estimation to perform path tracking of dynamic objects, and improving the accuracy of path tracking.

[0132] Regarding the embodiment of the above system, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0133] Figure 7 The diagram is a schematic diagram of the architecture of a real-time auto-focus path tracking system according to an exemplary embodiment.

[0134] like Figure 7 As shown in the figure, the core of a real-time autofocus path tracking system is a sensor-storage-computing integrated vision chip, which consists of a series of processing elements, control and readout circuits. The camera integrates peripheral circuits, including bias, clock, analog / digital converter and MCU, including processor core, memory and interface. Each pixel of the visual sensor device contains a processing unit, including light sensor circuit (PIX) and a microprocessor-like data path, including local memory (REGISTERS), arithmetic logic unit (ALU), local communication (NEWS), input / output (I / O) circuit and local activity control (FLAG). The processing element executes software programs, performs image calculations on the sensor, and outputs high-level information to the rest of the system.

[0135] Based on the same concept as above, in another embodiment of the present disclosure, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor is used to execute a real-time autofocus path tracking method when executing the program.

[0136] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0137] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories in partitions, and the above-mentioned computer programs, computer instructions, data, etc. may be called by a processor.

[0138] The processor is used to execute the computer program stored in the memory to implement the various steps in the method involved in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0139] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0140] In an embodiment of the present disclosure, a non-temporary computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of a real-time automatic focus path tracking method in any of the above embodiments are implemented.

[0141] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0143] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0145] Although the preferred embodiments of the present disclosure have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present disclosure.

[0146] Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A real-time auto-focus path tracking method, characterized in that: include: Performing a global search and initializing an optimal focal length on the liquid lens to determine an initial focal length position; If the current target is out of focus, a hill climbing search algorithm is used to determine the optimal focus of the current target; At the optimal focus position of the current target, determining the pixel coordinates of the center point of the current target according to the vertex coordinates of the rectangular bounding box of the current target; Determine the image coordinates of the center point of the current target according to the pixel coordinates of the center point of the current target; Determine the camera coordinates of the center point of the current target according to the image coordinates of the center point of the current target; The path trajectory of the current target is determined according to the real-time camera coordinates of the center point of the current target.

2. The method according to claim 1, characterized in that If the current target is in an out-of-focus state, a hill climbing search algorithm is used to determine the optimal focus of the current target, including: Using a preset first fixed step length to adjust the focal length of the liquid lens barrel from the initial focal length position to determine the moving direction of the current target; In the moving direction of the current target, the preset first fixed step length is used to adjust the focal length of the liquid lens to determine the optimal focus corresponding to the current target.

3. The method according to claim 2, characterized in that The step of adjusting the focal length of the liquid lens barrel from the initial focal length position by using a preset first fixed step length to determine the moving direction of the current target includes: Using the preset first fixed step length to adjust the focal length of the liquid lens barrel from the initial focal length position in each direction to determine a scanned image of the current target in each direction; The moving direction of the current target is determined according to the focus value of the scanned image of the current target in each direction.

4. The method according to claim 1, characterized in that: The step of performing a global search and initializing an optimal focal length on the liquid lens to determine an initial focal length position includes: Using a sensing, storage and computing integrated sensor device to send control instructions to the liquid lens; According to the control instruction, the liquid lens is controlled to traverse all focal lengths to determine the optimal focal length; According to the optimal focal length, the optimal focal length of the liquid lens is initialized to determine the initial focal length position.

5. The method according to claim 1, characterized in that The step of determining the pixel coordinates of the center point of the current target at the optimal focus position of the current target according to the vertex coordinates of the rectangular bounding box of the current target includes: At the optimal focus position of the current target, the pixel coordinates of the center point of the current target are determined according to the coordinates of the lower left vertex of the rectangular bounding box of the current target and the coordinates of the upper right vertex of the rectangular bounding box of the current target.

6. The method according to claim 1, characterized in that Determining the image coordinates of the center point of the current target according to the pixel coordinates of the center point of the current target includes: Convert the pixel coordinates of the center point of the current target to an image coordinate system to determine the image coordinates of the center point of the current target; The step of determining the camera coordinates of the center point of the current target according to the image coordinates of the center point of the current target includes: The image coordinates of the center point of the current target are converted into a camera coordinate system to determine the camera coordinates of the center point of the current target.

7. The method according to claim 1, characterized in that The method further comprises: If the current target is not in an out-of-focus state, the pixel coordinates of the center point of the current target are determined according to the vertex coordinates of the rectangular bounding box of the current target at the best focal length position of the current target.

8. A real-time auto-focus path tracking system, characterized in that: include: An initialization module, used for performing a global search and initializing the optimal focal length of the liquid lens to determine an initial focal length position; An autofocus module, used to determine the optimal focus of the current target using a hill climbing search algorithm if the current target is out of focus; A pixel coordinate determination module, used to determine the pixel coordinates of the center point of the current target according to the vertex coordinates of the boundary box of the current target at the optimal focus position of the current target; An image coordinate determination module, used to determine the image coordinates of the center point of the current target according to the pixel coordinates of the center point of the current target; A camera coordinate determination module, used to determine the camera coordinates of the center point of the current target according to the image coordinates of the center point of the current target; The path tracking module is used to determine the path trajectory of the current target according to the real-time camera coordinates of the center point of the current target.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 7.

Images