Array camera automatic focusing method, device and medium

Abstract

The invention provides an array camera automatic focusing method and device and a medium, a micro stepping motor is arranged behind each camera module in an array camera, and the method comprises the following steps: the array camera and the micro stepping motors are respectively connected with a terminal device; controlling the micro stepping motor to rotate by alpha degrees from the initial position every time, obtaining a key frame of a video stream of the micro stepping motor, calculating a definition score of the key frame by adopting an image evaluation function, selecting a picture with the highest score in N pictures, recording a score value of the picture as a current highest score GK, and rotating the micro stepping motor to an angle corresponding to the GK as a current position; and then reducing the step, respectively taking one step from the left and right sides of the current position to calculate scores, taking the highest score point as the current position, continuing to reduce the step, taking the key frame in a pendulum or progressive manner according to the current position, and continuously adjusting the current optimal position until the key frame cannot rotate, thereby completing focusing. According to the invention, efficient, automatic and accurate focusing of the array camera can be realized, and combination of manual fine tuning is supported.

Description

Technical Field

[0001] The present invention relates to the field of array camera focusing technology, and in particular to an array camera automatic focusing method, device and medium. Background Art

[0002] An array camera is a camera composed of multiple cameras arranged in a geometric pattern (such as 3 rows and 6 columns, etc.). That is, multiple small lenses are used to replace the shooting effect of a large lens. Its principle is similar to that of an array astronomical telescope and the compound eyes of insects. Its design and implementation comprehensively consider the synchronization of cameras, image fusion algorithms, computing resources, and system stability. With the development of computational photography and computer vision technology, array cameras are increasingly used in multiple fields.

[0003] As an emerging device in the field of computational photography, array cameras provide a good solution to the demand for real-time monitoring of large scenes. The focus adjustment of the camera module affects the imaging clarity of a single video stream, which will directly affect the effect of video fusion and machine recognition.

[0004] Currently, the focus of array cameras is mainly achieved by manual focus. Most array cameras have fixed-focus lens modules, which are arranged compactly. Each lens module has a manual focus ring for operators to focus manually. The video of each lens module is directly connected to a computer via a network cable for operators to view. Relying solely on manual operation to focus the array camera will lead to the following problems:

[0005] 1. Easy to touch by mistake and inaccurate. After the camera is set up and the angle is adjusted, the construction workers need to view the video stream of a single camera module through the computer and turn the focus ring by hand to focus, so as to achieve the purpose of clearly adjusting the video of a single module to be clear. However, the internal space of the equipment is small, and the hands cannot fully penetrate the interior of the lens module to operate. In the process of manually adjusting the focus of a single module, it is inevitable to touch other modules, causing the focus rings of other modules to be accidentally touched, and it is easy to cause pendulum-like excessive rotation, which can never achieve the best effect.

[0006] 2. High construction difficulty. Since array cameras are usually suitable for monitoring large scenes, they are often installed at high places, such as rooftops, mountaintops, signal towers, iron towers, etc., which requires construction personnel with high-altitude work qualifications to operate, and debugging and focusing require the knowledge of computer-related professional technicians. If two people are divided into two people to carry out the construction, one is focusing at a high place, and the other is using a computer to check the video clarity at a low place, the two parties often have communication difficulties and communication delays, which affect the final effect. When the installation site is at a high altitude outdoors, direct sunlight will also make the laptop screen unclear. In addition, the long-term high temperature environment of the equipment may cause the damping of the focus ring to decrease, which greatly increases the difficulty of adjustment.

[0007] 3. Low efficiency. Manual focus requires adjusting the camera modules one by one, and also requires checking the videos one by one on the PC to determine whether the focus is accurate, which makes the entire adjustment process inefficient and time-consuming. Summary of the invention

[0008] The technical problem to be solved by the present invention is to provide an array camera automatic focusing method, device and medium to overcome the problems of existing array camera manual focusing operation such as great difficulty, low precision and low efficiency.

[0009] In a first aspect, the present invention provides an array camera autofocus method, wherein each camera module in the array camera is provided with a micro stepping motor, and the method comprises the following steps:

[0010] Step S1, the array camera and the micro stepping motor are respectively connected to the terminal device, and each micro stepping motor number is bound to the corresponding camera module IP in a one-to-one correspondence through the terminal device;

[0011] Step S2, adjust all micro-stepping motors to their initial positions, record the total rotation angle when the micro-stepping motor is used to adjust the focus ring from one end to the other end, and divide it into N equal parts, each of which is used as an initial step α;

[0012] Step S3: performing an auto-focus operation on each camera module until all camera modules complete auto-focusing. The auto-focus operation specifically includes:

[0013] Step S31: rotate the micro-stepping motor by α degrees from the initial position, use the terminal device to pull the stream to obtain a key frame, record it as Pi, and use the image evaluation function to calculate the clarity score Gi of Pi. Rotate it N-1 times in total to obtain N pictures with scores;

[0014] Step S32: Select the picture with the highest score among the N pictures, and record its score as the current highest score G. K, rotate the micro stepper motor to G K The corresponding angle is taken as the current position;

[0015] Step S33, setting the current step β to half of the original one, and determining whether the current step is less than the minimum step length of the stepping motor. If so, the focusing is successful, and the debugging of the camera module is terminated. Otherwise, executing step S34;

[0016] Step S34: Rotate β degrees to both sides of the current position, respectively pull a key frame, and calculate the evaluation value G of each image through the image evaluation function. K1 , G K2 , G K , G K1 and G K2 For comparison, if G K1 <=G K And G K2 <=G K , then return to step S33; otherwise, proceed to step S35;

[0017] Step S35: G with the maximum value K1 or G K2 As G K , rotate the micro stepper motor to the maximum G K1 or G K2 The corresponding angle is used as the current position, and the step direction is adjusted to the maximum G K1 or G K2 direction;

[0018] Step S36, reduce the current step β by half, and determine whether the current step is less than the minimum step length of the stepping motor. If so, the focus is successful and the debugging of the camera module is terminated. Otherwise, execute step S37;

[0019] Step S37: Rotate β degrees according to the adjusted step direction and take the key frame P M And the evaluation value G is calculated by the image evaluation function M , if G M >G K , then G M As the current highest rated G K , the stepping direction remains unchanged and returns to step S37; otherwise, the stepping direction is reversed and returns to step S36.

[0020] Furthermore, the value of N is 20.

[0021] Furthermore, the image evaluation function is specifically as follows: a gradient matrix is ​​obtained by convolving the Laplace operator with the grayscale value of each pixel of the image, and the sum of the squares of the gradients of each pixel is taken as the image evaluation function value.

[0022] Furthermore, before step S37, it also includes: judging whether the corresponding direction is rotatable; if the corresponding direction is rotatable but not enough to rotate β degrees, the maximum rotatable angle is used as the current β value, and step S37 is continued; if the corresponding direction is not rotatable, the focus is considered to be successful, and the debugging of the camera is ended.

[0023] Furthermore, the method also includes step S4: after each camera component completes the autofocus operation, the terminal device displays a multi-stream playback interface for the user to view and review.

[0024] Furthermore, the method also includes the user zooming in on the video screen of the video stream and performing manual remote focusing, and fine-tuning the current camera module through the "←" and "→" keys on the keyboard, each press representing the stepper motor rotating counterclockwise / clockwise by the corresponding step, and the corresponding step is fine-tuned through the "↑" and "↓" keys, each press representing the corresponding step increasing / decreasing by one time, and the minimum is the minimum rotation angle of the stepper motor.

[0025] Furthermore, a Bluetooth communication module is added to the micro stepper motor, the micro stepper motor is connected to the terminal device via Bluetooth and is controlled by the terminal device, the array camera is connected to the device terminal via a network cable, and the micro stepper motor is connected to the focus ring on the camera module via gears.

[0026] In a second aspect, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0027] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0028] One or more technical solutions provided by the present invention have at least the following technical effects or advantages:

[0029] 1. The final focus position is gradually approached through a pendulum-like calculation with half the step, and only the current data is retained for each calculation. The cycle of step-detection-comparison-step-detection-comparison is adopted, which can be widely applied to embedded camera devices and improve the focus accuracy.

[0030] 2. By adding a stepper motor to the array camera, construction workers can focus on construction operations. Debugging personnel can directly connect the array camera via a computer network cable and connect the stepper motor via Bluetooth to achieve viewing and focusing at the same time, and effectively avoid operating the focus ring in a narrow area, which may cause accidental touches between the camera modules.

[0031] 3. It can support viewing on the terminal device and support individual review and focusing of each sub-lens module to achieve the best focusing effect;

[0032] 4. It can realize full automatic focusing of multiple sub-lenses at the same time, which is highly efficient and can realize the extraction and calculation of real-time video streams. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0034] Figure 1 The present invention is a flowchart of an array camera auto-focusing method.

[0035] Figure 2 The figure is a schematic diagram of the auto-focusing process of each camera module of the present invention.

[0036] Figure 3 This is a hardware topology diagram of a single camera module of the present invention.

[0037] Figure 4 The figure is a schematic diagram of the overall process of the present invention combined with manual focusing.

[0038] Figure 5 It is a schematic diagram of the initial operation activities of the present invention.

[0039] Figure 6 The figure is a schematic diagram of the specific calculation control flow of the automatic focusing of the present invention.

[0040] Figure 7 is a schematic diagram of the structure of an electronic device in Embodiment 2 of the present invention;

[0041] Figure 8 A schematic diagram of the structure of the medium in the third embodiment of the present invention DETAILED DESCRIPTION

[0042] The embodiment of the present application provides an array camera autofocus method, device and medium to solve the problem that the existing array camera is difficult to focus and the focusing effect is not ideal. The technical solution in the embodiment of the present application has the following general idea: by building a micro stepper motor on each sub-lens in the array camera, the micro stepper motor is connected to the focus ring on the sub-lens, and the array camera and the micro stepper motor are respectively connected to the device terminal (such as a computer, a mobile phone, etc.) used for debugging, and the flow acquisition, analysis, and focusing are fully automated on the device terminal. Specifically, each sub-lens is controlled to perform autofocus operation separately. The autofocus operation adopts the first large step positioning, and then gradually shortens the step to perform a pendulum-like search, and continuously adjusts the current position until it can no longer rotate, which is the final focusing result. After all cameras are automatically focused, the user can simultaneously check and review the multiple streams on the computer. If there is an unsatisfactory situation, the video screen of the corresponding video stream can be enlarged and fine-tuned through the keyboard "←" and "→" keys. Each press represents the stepper motor to rotate the current step β degrees counterclockwise / clockwise. You can also use the "↑" and "↓" buttons to fine-tune β. Each press represents a doubling of β, with the minimum being the minimum rotation angle of the stepping motor. This allows for optimal focusing.

[0043] In order to make the present invention more clearly understood, a preferred embodiment is now described in detail with reference to the accompanying drawings.

[0044] Embodiment 1

[0045] This embodiment provides an array camera auto-focus method, wherein each camera module in the array camera is provided with a micro stepping motor. Figures 1 to 6 As shown, the method comprises the following steps:

[0046] Step S1, the array camera and the micro stepping motor are respectively connected to the terminal device, and each micro stepping motor number is bound to the corresponding camera module IP in a one-to-one correspondence through the terminal device;

[0047] Step S2, adjust all micro-stepping motors to the initial position (i.e., rotate the focus ring to one end), record the total rotation angle when the focus ring is adjusted from one end to the other end using the micro-stepping motor, and divide it into N equal parts, for example, N is 20, and each equal part is used as the initial step α;

[0048] Step S3: performing an auto-focus operation on each camera module until all camera modules complete auto-focusing. The auto-focus operation specifically includes:

[0049] Step S31: rotate the micro-stepping motor by α degrees from the initial position, use the terminal device to pull the stream to obtain a key frame, record it as Pi, and use the image evaluation function to calculate the clarity score Gi of Pi. Rotate it N-1 times in total to obtain N pictures with scores;

[0050] Step S32: Select the picture with the highest score among the N pictures, and record its score as the current highest score G. K , rotate the micro stepper motor to G K The corresponding angle is used as the current position; only the current information is recorded to avoid taking up too much storage space of the embedded device.

[0051] Step S33, set the current step β to half of the original one (the initial value is β=α / 2), and determine whether the current step is less than the minimum step length of the stepper motor. If so, the focus is successful and the debugging of the camera module is terminated. Otherwise, execute step S34;

[0052] Step S34: Rotate β degrees to both sides of the current position (first rotate β degrees clockwise, then rotate 2β degrees counterclockwise), pull a key frame respectively, and calculate the evaluation value G of each image through the image evaluation function. K1 , G K2 , G K , G K1 and G K2 For comparison, if G K1 <=G K And G K2 <=G K , then return to step S33; otherwise, if G K1 and G K2 There is a ratio G K When the value is larger (such as G K2 <=G K <=G K1 , G K1 <=G K <=G K2 ) proceed to step S35;

[0053] Step S35: G with the maximum value K1 or G K2 As G K , rotate the micro stepper motor to the maximum G K1 or G K2 The corresponding angle is used as the current position, and the step direction is adjusted to the maximum G K1 or G K2 Direction, that is, stepping in the direction of greater clarity;

[0054] Step S36, reduce the current step β by half, and determine whether the current step is less than the minimum step length of the stepping motor. If so, the focus is successful and the debugging of the camera module is terminated. Otherwise, execute step S37;

[0055] Step S37: Rotate β degrees according to the adjusted step direction and take the key frame P M And the evaluation value G is calculated by the image evaluation function M , if G M >G K , indicating that it is currently turning in a clear direction, then G M As the current highest rated G K , the stepping direction remains unchanged, and the process returns to step S37; otherwise, it indicates that excessive rotation has occurred, and the stepping direction is reversed, and the process returns to step S36. This process may recalculate some positions. Since the picture is dynamic, the early clarity is not necessarily the current clarity, so it needs to be recalculated to achieve real-time update.

[0056] Preferably, the image evaluation function is specifically as follows: a gradient matrix is ​​obtained by convolving the grayscale value of each pixel of the image with the Laplace operator, and the sum of the squares of the gradients of each pixel is taken as the image evaluation function value. In the image clarity evaluation, a clear image has sharper edges and a significant change in grayscale value than a blurred image, so it has a larger gradient value. When processing an image, we regard it as a two-dimensional discrete matrix, extract grayscale information through a gradient function, and thus evaluate the clarity of the image. In discrete signals, the gradient is usually expressed in differential form. Common gradient functions include energy gradient function (EOG), Roberts operator, Tenengrad operator, Brenner operator, variance (Variance) and Laplace operator, etc. Here, a gradient matrix is ​​obtained by convolving the Laplace operator with the grayscale value of each pixel of the image, and the sum of the squares of the gradients of each pixel is taken as the evaluation function. The whole process is simple to calculate and has high accuracy.

[0057] Preferably, the step S37 also includes: determining whether the corresponding direction is rotatable; if the corresponding direction is rotatable but not enough to rotate β degrees, taking the maximum rotatable angle as the current β value, and continuing with step S37; if the corresponding direction is not rotatable (i.e., the maximum rotatable angle at the end is less than the minimum step length of the stepper motor, making it impossible to rotate), it is considered that the focus is successful and the debugging of the camera is ended.

[0058] Preferably, the method further comprises step S4, after each camera component completes the autofocus operation, the terminal device displays a multi-stream playback interface for the user to view and review.

[0059] Preferably, the method also includes the user zooming in on the video screen of the video stream and performing manual remote focusing, and fine-tuning the current camera module through the "←" and "→" keys on the keyboard, each press representing the stepper motor rotating counterclockwise / clockwise by a corresponding step, and the corresponding step is fine-tuned through the "↑" and "↓" keys, each press representing the corresponding step increasing / decreasing by one time, and the minimum is the minimum rotation angle of the stepper motor.

[0060] Preferably, a Bluetooth communication module is added to the micro stepper motor, the micro stepper motor is connected to the terminal device via Bluetooth and is controlled by the terminal device, the array camera is connected to the device terminal via a network cable, and the micro stepper motor is connected to the focus ring on the camera module via gears, thereby realizing remote focusing.

[0061] Based on the same inventive concept, the present application provides an electronic device embodiment corresponding to the first embodiment, see the second embodiment for details.

[0062] Embodiment 2

[0063] This embodiment provides an electronic device, such as Figure 7 As shown, it includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any implementation method in the first embodiment can be implemented.

[0064] Since the electronic device introduced in this embodiment is a device used to implement the method in the first embodiment of the present application, based on the method introduced in the first embodiment of the present application, a person skilled in the art can understand the specific implementation of the electronic device of the present embodiment and its various variations, so how the electronic device implements the method in the embodiment of the present application is not described in detail here. As long as a person skilled in the art implements the device used by the method in the embodiment of the present application, it belongs to the scope of protection of the present application.

[0065] Based on the same inventive concept, the present application provides a storage medium corresponding to Example 1, see Example 3 for details.

[0066] Embodiment 3

[0067] This embodiment provides a computer-readable storage medium, such as Figure 8 As shown, a computer program is stored thereon, and when the computer program is executed by a processor, any implementation method in Example 1 can be implemented.

[0068] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages: by adding a stepper motor design to the array camera, construction workers can focus on construction operations, and debugging personnel can directly connect the array camera via a computer network cable at a low location and connect the stepper motor via Bluetooth to achieve viewing and focusing at the same time, by first locating the optimal position in the large step, and then calculating a more accurate focus position through continuous pendulum-like real-time search, the focus accuracy is improved, and remote manual focus operations on the computer are supported for individual sub-lenses with poor focus effects, thereby achieving the best focus effect. The entire process can achieve fully automatic focus with high efficiency.

[0069] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. An array camera auto-focus method, characterized in that: Each camera module in the array camera is provided with a micro stepping motor, and the method comprises the following steps: Step S1, the array camera and the micro stepping motor are respectively connected to the terminal device, and each micro stepping motor number is bound to the corresponding camera module IP in a one-to-one correspondence through the terminal device; Step S2, adjust all micro-stepping motors to their initial positions, record the total rotation angle when the micro-stepping motor is used to adjust the focus ring from one end to the other end, and divide it into N equal parts, each of which is used as an initial step α; Step S3: performing an auto-focus operation on each camera module until all camera modules complete auto-focusing. The auto-focus operation specifically includes: Step S31: rotate the micro-stepping motor by α degrees from the initial position, use the terminal device to pull the stream to obtain a key frame, record it as Pi, and use the image evaluation function to calculate the clarity score Gi of Pi. Rotate it N-1 times in total to obtain N pictures with scores; Step S32: Select the picture with the highest score among the N pictures, and record its score as the current highest score G. K , rotate the micro stepper motor to G K The corresponding angle is taken as the current position; Step S33, setting the current step β to half of the original one, and determining whether the current step is less than the minimum step length of the stepping motor. If so, the focusing is successful, and the debugging of the camera module is terminated. Otherwise, executing step S34; Step S34: Rotate β degrees to both sides of the current position, respectively pull a key frame, and calculate the evaluation value G of each image through the image evaluation function. K1 , G K2 , G K , G K1 and G K2 For comparison, if G K1 <=G K And G K2 <=G K , then return to step S33; Otherwise, proceed to step S35; Step S35: G with the maximum value K1 or G K2 As G K , rotate the micro stepper motor to the maximum G K1 or G K2 The corresponding angle is used as the current position, and the step direction is adjusted to the maximum G K1 or G K2 direction; Step S36, reduce the current step β by half, and determine whether the current step is less than the minimum step length of the stepping motor. If so, the focus is successful and the debugging of the camera module is terminated. Otherwise, execute step S37; Step S37: Rotate β degrees according to the adjusted step direction and take the key frame P M And the evaluation value G is calculated by the image evaluation function M , if G M >G K , then G M As the current highest rated G K , the stepping direction remains unchanged and returns to step S37; otherwise, the stepping direction is reversed and returns to step S36.

2. The automatic focusing method of an array camera according to claim 1, characterized in that: The value of N is 20.

3. The automatic focusing method of an array camera according to claim 1, characterized in that: The image evaluation function is specifically as follows: a gradient matrix is ​​obtained by convolving the Laplace operator with the grayscale value of each pixel of the image, and the square sum of the gradients of each pixel is taken as the image evaluation function value.

4. The automatic focusing method of an array camera according to claim 1, characterized in that: Before step S37, it also includes: judging whether the corresponding direction is rotatable; if the corresponding direction is rotatable but not enough to rotate β degrees, the maximum rotatable angle is used as the current β value, and step S37 is continued; if the corresponding direction is not rotatable, it is considered that the focus is successful, and the debugging of the camera is ended.

5. The automatic focusing method of an array camera according to claim 1, characterized in that: The method further includes step S4, after each camera component completes the autofocus operation, the terminal device displays a multi-stream playback interface for the user to view and review.

6. The automatic focusing method of an array camera according to claim 1, characterized in that: The method also includes the user zooming in on the video screen of the video stream and performing manual remote focusing, and fine-tuning the current camera module through the "←" and "→" keys on the keyboard, each press represents the stepper motor rotating counterclockwise / clockwise by the corresponding step, and the corresponding step is fine-tuned through the "↑" and "↓" keys, each press represents the corresponding step increasing / decreasing by one time, and the minimum is the minimum rotation angle of the stepper motor.

7. The automatic focusing method of an array camera according to claim 1, characterized in that: A Bluetooth communication module is added to the micro stepper motor. The micro stepper motor is connected to the terminal device via Bluetooth and is controlled by the terminal device. The array camera is connected to the device terminal via a network cable. The micro stepper motor is connected to the focus ring on the camera module via a gear.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images