A multi-lens focus tracking shooting method, device and computer readable storage medium

Abstract

The application discloses a multi-lens focus-pursuing shooting method, device and computer readable storage medium, wherein the method comprises the following steps: in the focus-pursuing shooting process, the first positioning information and the predicted trajectory information of a focus-pursuing object are transmitted to a long-focus lens of the device through a wide-angle lens of the device as auxiliary focus-pursuing information for the focus-pursuing shooting; after the long-focus lens acquires a current frame image and receives the auxiliary focus-pursuing information, the focus-pursuing object is subjected to sharpness calculation, and the motor position is calculated according to the calculated sharpness information; the motor of the device is driven to the motor position to perform focus-pursuing shooting, and the motor is driven to a next frame position in advance according to the predicted trajectory information. The application realizes a multi-lens focus-pursuing shooting scheme, greatly improves the success rate and imaging quality of focus-pursuing shooting, reduces the shooting burden of a user, and enhances the shooting experience of the user.

Description

Technical Field

[0001] This invention relates to the field of mobile communications, and more particularly to a multi-lens focus-tracking shooting method, device, and computer-readable storage medium. Background Technology

[0002] In the current technology, with the continuous development of smart terminal devices, users have increasingly higher demands for portable shooting devices with multiple cameras. When people use the telephoto lens of the device to track moving objects in the distance, due to the fast speed of the moving objects in the picture and the narrow angle of view of the lens, it is easy to fail to track the objects, resulting in low image quality.

[0003] Therefore, improving the success rate and image quality of autofocus tracking shooting has become an urgent technical problem to be solved. Summary of the Invention

[0004] To address the aforementioned technical deficiencies in the prior art, this invention proposes a multi-lens focusing shooting method, which includes:

[0005] During the focus tracking shooting process, the first positioning information and predicted trajectory information of the focus tracking object are transmitted to the telephoto lens of the device through the wide-angle lens of the device as auxiliary focus tracking information for focus tracking shooting, and the new frame of the focus tracking object and the second positioning information of the focus tracking object are received from the telephoto lens.

[0006] After the telephoto lens acquires the current frame image and receives the auxiliary focus tracking information, it performs a sharpness calculation on the focus tracking object and calculates the motor position based on the calculated sharpness information.

[0007] Drive the device's motor to perform focus tracking and shooting according to the motor position, and use the predicted trajectory information to drive the motor to the next frame position in advance;

[0008] After acquiring a new frame image, the new image information is transmitted to the wide-angle lens for information exchange with the tracking object to obtain new auxiliary tracking information. During the tracking shooting process, the telephoto lens is used again to perform sharpness calculation and motor drive operation.

[0009] Optionally, during the focus tracking shooting process, the first positioning information and predicted trajectory information of the focus tracking object are transmitted from the wide-angle lens of the device to the telephoto lens of the device as auxiliary focus tracking information for the focus tracking shooting, and new frame images and the second positioning information of the focus tracking object are received from the telephoto lens, before the following steps are taken:

[0010] The telephoto lens is used to detect moving objects within the captured image and to mark the candidate bounding rectangle of the moving objects.

[0011] After receiving a selection signal for any of the candidate circumscribed rectangles and taking the moving object corresponding to the selected candidate circumscribed rectangle as the focus tracking object, the focus tracking object is transmitted to the wide-angle lens for focus tracking shooting.

[0012] Optionally, during the focus tracking shooting process, the first positioning information and predicted trajectory information of the focus tracking object are transmitted from the wide-angle lens of the device to the telephoto lens of the device as auxiliary focus tracking information for the focus tracking shooting, and new frame images and the second positioning information of the focus tracking object are received from the telephoto lens. Prior to this, the process also includes:

[0013] After the wide-angle lens receives the object being tracked, it marks the selected outer rectangle of the object being tracked.

[0014] The coordinates of the tracking object in the captured image are calculated based on the selected circumscribed rectangle.

[0015] Optionally, during the focus tracking shooting process, the first positioning information and predicted trajectory information of the focus tracking object are transmitted from the wide-angle lens of the device to the telephoto lens of the device as auxiliary focus tracking information for the focus tracking shooting, and new frame images and the second positioning information of the focus tracking object are received from the telephoto lens, including:

[0016] The wide-angle lens is used to track the coordinates of the focus object in each frame to obtain a high-order curve function fitted by multiple coordinate points.

[0017] The first positioning information of the tracking object is calculated based on Kalman filtering and the higher-order curve function, and the predicted trajectory information is obtained.

[0018] Optionally, during the focus tracking shooting process, the first positioning information and predicted trajectory information of the focus tracking object are transmitted from the wide-angle lens of the device to the telephoto lens of the device as auxiliary focus tracking information for focus tracking shooting, and new frame images and the second positioning information of the focus tracking object are received from the telephoto lens, further comprising:

[0019] The wide-angle lens and the telephoto lens exchange the positioning information of the object being tracked;

[0020] When exchanging the positioning information, the positioning coordinates in the positioning information are converted according to the imaging parameters of the wide-angle lens and the telephoto lens.

[0021] Optionally, after the telephoto lens acquires the current frame image and receives the auxiliary focus tracking information, the step of calculating the sharpness of the focus tracking object and calculating the motor position from the calculated sharpness information includes:

[0022] The motor that drives the telephoto lens to move is identified;

[0023] The target orientation of the telephoto lens is calculated based on the resolution information, and the required motor position is determined based on the target orientation.

[0024] Optionally, the step of driving the device's motor to perform focus tracking and shooting according to the motor position, and using the predicted trajectory information to drive the motor to the next frame position in advance, includes:

[0025] After the motor is driven to the motor position, a new frame image of the next frame is acquired through the telephoto lens;

[0026] The new frame image is transmitted to the wide-angle lens.

[0027] Optionally, after acquiring a new frame image, transmitting the new image information to the wide-angle lens for information exchange with the tracking object to obtain new auxiliary tracking information, and again utilizing the telephoto lens to perform sharpness calculation and motor drive operation during the tracking shooting process, includes:

[0028] Repeat the information exchange operation of the tracking object to update the auxiliary tracking information;

[0029] Repeat the sharpness calculation and the motor drive operation to perform the new information exchange operation until the focus tracking shooting is completed.

[0030] The present invention also proposes a multi-lens focus tracking shooting device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the multi-lens focus tracking shooting method as described in any of the preceding claims.

[0031] The present invention also proposes a computer-readable storage medium storing a multi-lens focus tracking shooting program, which, when executed by a processor, implements the steps of the multi-lens focus tracking shooting method as described in any of the preceding claims.

[0032] The multi-lens tracking focus shooting method, device, and computer-readable storage medium of the present invention, during the tracking focus shooting process, transmits the first positioning information and predicted trajectory information of the tracking object to the telephoto lens of the device through the wide-angle lens of the device as auxiliary tracking focus information, and receives new frame images and second positioning information of the tracking object from the telephoto lens; after the telephoto lens acquires the current frame image and receives the auxiliary tracking focus information, it performs sharpness calculation on the tracking object, and calculates the motor position based on the calculated sharpness information; drives the motor of the device according to the motor position for tracking focus shooting, and uses the predicted trajectory information to drive the motor to the next frame position in advance; after acquiring a new frame image, the new image information is transmitted to the wide-angle lens for information exchange on the tracking object to obtain new auxiliary tracking focus information, and the telephoto lens is used again to perform sharpness calculation and motor drive operation during the tracking focus shooting process. This achieves a multi-lens tracking focus shooting scheme that greatly improves the success rate and image quality of tracking focus shooting, reduces the user's shooting burden, and enhances the user's shooting experience. Attached Figure Description

[0033] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0034] Figure 1 This is a schematic diagram of the hardware structure of a mobile terminal according to the present invention;

[0035] Figure 2 This is the first flowchart of the multi-lens tracking focus shooting method of the present invention;

[0036] Figure 3 This is the second flowchart of the multi-lens focusing shooting method of the present invention;

[0037] Figure 4 This is the third flowchart of the multi-lens tracking focus shooting method of the present invention;

[0038] Figure 5 This is the fourth flowchart of the multi-lens focusing shooting method of the present invention;

[0039] Figure 6 This is the fifth flowchart of the multi-lens focusing shooting method of the present invention;

[0040] Figure 7 This is the sixth flowchart of the multi-lens tracking focus shooting method of the present invention;

[0041] Figure 8 This is the seventh flowchart of the multi-lens tracking focus shooting method of the present invention;

[0042] Figure 9 This is the eighth flowchart of the multi-lens tracking focus shooting method of the present invention. Detailed Implementation

[0043] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0044] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0045] Terminals can be implemented in various forms. For example, the terminals described in this invention may include mobile terminals such as mobile phones, tablets, laptops, handheld computers, personal digital assistants (PDAs), portable media players (PMPs), navigation devices, wearable devices, smart bracelets, pedometers, etc., as well as fixed terminals such as digital TVs and desktop computers.

[0046] The following description will use a mobile terminal as an example. Those skilled in the art will understand that, apart from elements specifically designed for mobile purposes, the construction according to embodiments of the present invention can also be applied to fixed-type terminals.

[0047] Please see Figure 1 This is a schematic diagram of the hardware structure of a mobile terminal implementing various embodiments of the present invention. The mobile terminal 100 may include: an RF (Radio Frequency) unit 101, a WiFi module 102, an audio output unit 103, an A / V (Audio / Video) input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, a processor 110, and a power supply 111, etc. Those skilled in the art will understand that... Figure 1 The mobile terminal structure shown does not constitute a limitation on the mobile terminal. The mobile terminal may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0048] The following is combined with Figure 1 A detailed introduction to each component of the mobile terminal:

[0049] The radio frequency unit 101 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with the processor 110; additionally, it transmits uplink data to the base station. Typically, the radio frequency unit 101 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier, and a duplexer. Furthermore, the radio frequency unit 101 can also communicate wirelessly with networks and other devices. The aforementioned wireless communications may use any communication standard or protocol, including but not limited to GSM (Global System of Mobile communication), GPRS (General Packet Radio Service), CDMA2000 (Code Division Multiple Access 2000), WCDMA (Wideband Code Division Multiple Access), TD-SCDMA (Time Division-Synchronous Code Division Multiple Access), FDD-LTE (Frequency Division Duplexing-Long Term Evolution), and TDD-LTE (Time Division Duplexing-Long Term Evolution).

[0050] WiFi is a short-range wireless transmission technology. Mobile terminals, through the WiFi module 102, can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 1 WiFi module 102 is shown, but it is understood that it is not a necessary component of a mobile terminal and can be omitted as needed without changing the nature of the invention.

[0051] The audio output unit 103 can convert audio data received by the radio frequency unit 101 or the WiFi module 102 or stored in the memory 109 into audio signals and output them as sound when the mobile terminal 100 is in call signal receiving mode, call mode, recording mode, voice recognition mode, broadcast receiving mode, etc. Furthermore, the audio output unit 103 can also provide audio output related to specific functions performed by the mobile terminal 100 (e.g., call signal receiving sound, message receiving sound, etc.). The audio output unit 103 may include a speaker, a buzzer, etc.

[0052] The A / V input unit 104 is used to receive audio or video signals. The A / V input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos acquired by an image capture device (such as a camera) in video capture mode or image capture mode. The processed image frames can be displayed on the display unit 106. The image frames processed by the GPU 1041 can be stored in the memory 109 (or other storage medium) or transmitted via the radio frequency unit 101 or the WiFi module 102. The microphone 1042 can receive sound (audio data) in operating modes such as telephone call mode, recording mode, and voice recognition mode, and can process such sound into audio data. The processed audio (voice) data can be converted into a format that can be transmitted to a mobile communication base station via the radio frequency unit 101 in telephone call mode. The microphone 1042 can implement various types of noise cancellation (or suppression) algorithms to eliminate (or suppress) noise or interference generated during the reception and transmission of audio signals.

[0053] The mobile terminal 100 also includes at least one sensor 105, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 1061 according to the ambient light level, and the proximity sensor can turn off the display panel 1061 and / or backlight when the mobile terminal 100 is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, tapping), etc. Other sensors that may be configured in the phone, such as fingerprint sensors, pressure sensors, iris sensors, molecular sensors, gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0054] The display unit 106 is used to display information input by the user or information provided to the user. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0055] User input unit 107 can be used to receive input numerical or character information, and generate key signal inputs related to user settings and function control of the mobile terminal. Specifically, user input unit 107 may include touch panel 1071 and other input devices 1072. Touch panel 1071, also known as touch screen, can collect touch operations on or near the user (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near touch panel 1071), and drive corresponding connection devices according to a pre-set program. Touch panel 1071 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to processor 110, and can receive and execute commands from processor 110. In addition, touch panel 1071 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1071, the user input unit 107 may also include other input devices 1072. Specifically, other input devices 1072 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc., without being limited here.

[0056] Furthermore, the touch panel 1071 may cover the display panel 1061. When the touch panel 1071 detects a touch operation on or near it, it transmits the information to the processor 110 to determine the type of touch event. Subsequently, the processor 110 provides corresponding visual output on the display panel 1061 based on the type of touch event. Although in Figure 1 In this embodiment, the touch panel 1071 and the display panel 1061 are two independent components to realize the input and output functions of the mobile terminal. However, in some embodiments, the touch panel 1071 and the display panel 1061 can be integrated to realize the input and output functions of the mobile terminal. The specific implementation is not limited here.

[0057] Interface unit 108 serves as an interface through which at least one external device can connect to mobile terminal 100. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, and so on. Interface unit 108 may be used to receive input (e.g., data, power, etc.) from the external device and transmit the received input to one or more elements within mobile terminal 100, or it may be used to transmit data between mobile terminal 100 and the external device.

[0058] The memory 109 can be used to store software programs and various data. The memory 109 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback, image playback, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). Furthermore, the memory 109 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0059] The processor 110 is the control center of the mobile terminal. It connects various parts of the mobile terminal via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 109, and by calling data stored in the memory 109, it performs various functions and processes data of the mobile terminal, thereby providing overall monitoring of the mobile terminal. The processor 110 may include one or more processing units; preferably, the processor 110 may integrate an application processor and a modem processor. The application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 110.

[0060] The mobile terminal 100 may also include a power supply 111 (such as a battery) that supplies power to various components. Preferably, the power supply 111 can be logically connected to the processor 110 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0061] although Figure 1 As not shown, the mobile terminal 100 may also include a Bluetooth module, etc., which will not be described in detail here.

[0062] Based on the above-described mobile terminal hardware structure, various embodiments of the method of the present invention are proposed.

[0063] Figure 2 This is the first flowchart of the multi-lens focus tracking shooting method of the present invention. This embodiment proposes a multi-lens focus tracking shooting method, which includes:

[0064] S1. During the focus tracking shooting process, the first positioning information and predicted trajectory information of the focus tracking object are transmitted to the telephoto lens of the device through the wide-angle lens of the device as auxiliary focus tracking information for focus tracking shooting, and the new frame of the telephoto lens and the second positioning information of the focus tracking object are received.

[0065] S2. After the telephoto lens acquires the current frame image and receives the auxiliary focus tracking information, the sharpness of the focus tracking object is calculated, and the motor position is calculated from the calculated sharpness information.

[0066] S3. Drive the motor of the device to perform focus tracking and shooting according to the motor position, and use the predicted trajectory information to drive the motor to the next frame position in advance;

[0067] S4. After acquiring a new frame image, the new image information is transmitted to the wide-angle lens to exchange information with the tracking object in order to obtain new auxiliary tracking information. During the tracking shooting process, the telephoto lens is used again to perform sharpness calculation and motor drive operation.

[0068] In this embodiment, the wide-angle lens on a portable shooting device is used to assist the telephoto lens in tracking fast-moving objects at a distance. It is understood that a wide-angle lens captures more image information, and moving objects appear to move slower within the wide-angle frame, making them easier to track. Therefore, in this embodiment, addressing the challenge of tracking moving objects due to the narrower angle of view and the faster speed of objects within the telephoto frame, the positioning information of the moving object within the wide-angle frame is fully utilized to assist the telephoto lens in tracking, thereby effectively improving the difficulty of tracking moving objects with a telephoto lens.

[0069] Optionally, in this embodiment, in the image captured by the wide-angle lens, the positioning information of the moving object in multiple frames is used to simulate and predict the direction of the object's movement, and the prediction information is transmitted to the telephoto lens in real time, thereby greatly improving the telephoto lens's focusing speed on the moving object.

[0070] The beneficial effects of this embodiment are as follows: During the focus-tracking shooting process, the wide-angle lens of the device transmits the first positioning information and predicted trajectory information of the target object to the telephoto lens of the device as auxiliary focus-tracking information, and receives new frame images and the second positioning information of the target object from the telephoto lens; after the telephoto lens acquires the current frame image and receives the auxiliary focus-tracking information, it performs sharpness calculation on the target object and calculates the motor position based on the calculated sharpness information; the device's motor is driven according to the motor position for focus-tracking shooting, and the predicted trajectory information is used to drive the motor to the next frame position in advance; after acquiring a new frame image, the new image information is transmitted to the wide-angle lens for information exchange on the target object to obtain new auxiliary focus-tracking information, and the telephoto lens is used again to perform sharpness calculation and motor drive operation during the focus-tracking shooting process. This realizes a multi-lens focus-tracking shooting scheme, which greatly improves the success rate and image quality of focus-tracking shooting, reduces the user's shooting burden, and enhances the user's shooting experience.

[0071] Figure 3 This is a second flowchart of the multi-lens tracking focus shooting method of the present invention. Based on the above embodiment, during the tracking focus shooting process, the first positioning information and predicted trajectory information of the tracking object are transmitted to the telephoto lens of the device through the wide-angle lens of the device as auxiliary tracking focus information for tracking focus shooting, and the new frame and the second positioning information of the tracking object are received from the telephoto lens. Prior to this, the following steps are included:

[0072] S01. Detect moving objects within the captured image using the telephoto lens and mark the candidate bounding rectangle of the moving objects;

[0073] S02. After receiving the selection signal of any of the candidate circumscribed rectangles and taking the moving object corresponding to the selected candidate circumscribed rectangle as the focus tracking object, the focus tracking object is transmitted to the wide-angle lens for focus tracking shooting.

[0074] Optionally, in this embodiment, distortion correction is first performed on the wide-angle lens. Considering that wide-angle lenses can capture a wider field of view than conventional camera modules, but are generally prone to barrel distortion (where the captured image expands outwards in a circle centered on the lens), this embodiment addresses this distortion by calibrating the camera to obtain intrinsic parameters and then performing an indiscriminate transformation on the input image to restore its flatness. This effectively eliminates the distortion.

[0075] Optionally, in this embodiment, the target to be tracked is then located. Moving objects are detected by the telephoto lens within the frame, and the bounding rectangle of the moving object is marked. Simultaneously, the target to be tracked can be manually input via the touchscreen, triggered when the preview interface is clicked. The clicked position coordinates are used as the focus point and passed to the tracking algorithm. The bounding rectangle of the target to be tracked is marked. After the telephoto lens selects the target to be tracked, the target is transmitted in real-time to the wide-angle lens for tracking.

[0076] Figure 4 This is the third flowchart of the multi-lens tracking focus shooting method of the present invention. Based on the above embodiment, during the tracking focus shooting process, the first positioning information and predicted trajectory information of the tracking object are transmitted to the telephoto lens of the device through the wide-angle lens of the device as auxiliary tracking focus information for tracking focus shooting, and the new frame and the second positioning information of the tracking object are received from the telephoto lens. Prior to this, the method further includes:

[0077] S03. After the wide-angle lens receives the tracking object, mark the selected outer rectangle of the tracking object;

[0078] S04. Calculate the coordinates of the tracking object in the shooting frame based on the selected circumscribed rectangle.

[0079] Optionally, in this embodiment, the tracking object is located using a wide-angle lens. After receiving the tracking object selected by the telephoto lens, the wide-angle lens marks the bounding rectangle of the moving object and calculates the coordinates of the center point of the moving object in the image.

[0080] Figure 5 This is the fourth flowchart of the multi-lens tracking focus shooting method of the present invention. Based on the above embodiments, during the tracking focus shooting process, the first positioning information and predicted trajectory information of the tracking object are transmitted to the telephoto lens of the device through the wide-angle lens of the device as auxiliary tracking focus information for tracking focus shooting, and the new frame and the second positioning information of the tracking object are received from the telephoto lens, including:

[0081] S11. The focus object in each frame is tracked by the wide-angle lens to obtain a high-order curve function fitted by multiple coordinate points.

[0082] S12. Calculate the first positioning information of the tracking object based on Kalman filtering and the higher-order curve function, and calculate the predicted trajectory information.

[0083] Optionally, in this embodiment, target tracking and trajectory prediction are performed on the object being tracked. Specifically, firstly, the multi-frame positioning coordinates of the moving object are obtained. Tracking and predicting the trajectory of the object need to be done in a wide-angle lens. Moving objects are detected in each frame, and multiple frames of the moving object within the wide-angle lens are selected. The bounding rectangle of the marker is continuously tracked, and the coordinate positions of the moving object are calculated. Then, the object's trajectory is fitted. The coordinate points of the moving object in multiple frames are obtained from the previous step, and a high-order curve function is fitted from these coordinate points.

[0084] y = a0 + a1*x + a2*x 2 +a3*x 3 +a4*x 4 ;

[0085] Finally, the direction of the object's motion is predicted. This prediction is based on probability statistics; for example, in this embodiment, the trajectory is predicted using a Kalman filter combined with a fitted curve. The optimal estimate at a given moment is:

[0086]

[0087] From the optimal estimate at the previous moment, we can obtain the predicted value at this moment:

[0088]

[0089] in, z is the optimal estimate from the previous time step. t Let H be the current observation value, H be the specific relationship matrix between the current state and the sensor data, and K be the current observation value. t Let F be the Kalman gain, F be the state transition matrix, B be the state control matrix, and u be the Kalman gain. t-1 This is the state control vector.

[0090] In this embodiment, trajectory correction is performed after probability-based statistical prediction. Specifically, after obtaining a new frame, the coordinate information is updated by detection and localization, and the trajectory is predicted in real time using the new fitting information, and the trajectory information is updated and corrected.

[0091] Figure 6 This is the fifth flowchart of the multi-lens tracking focus shooting method of the present invention. Based on the above embodiments, during the tracking focus shooting process, the first positioning information and predicted trajectory information of the tracking object are transmitted to the telephoto lens of the device through the wide-angle lens of the device as auxiliary tracking focus information for tracking focus shooting, and the new frame and the second positioning information of the tracking object are received from the telephoto lens, further comprising:

[0092] S13. Exchange the positioning information of the tracking object with the wide-angle lens and the telephoto lens;

[0093] S14. When exchanging the positioning information, the positioning coordinates in the positioning information are converted according to the imaging parameters of the wide-angle lens and the telephoto lens.

[0094] Optionally, in this embodiment, the positioning and prediction information are transmitted and shared in real time after the trajectory information is updated and corrected. Specifically, the wide-angle lens transmits the positioning information of the object being tracked and the predicted trajectory information to the telephoto lens as auxiliary tracking information, and receives new frame images and the positioning information of the object being tracked from the telephoto lens.

[0095] Optionally, in this embodiment, a scheme for converting the positioning coordinate information between the wide-angle and telephoto lenses is provided. Specifically, when the wide-angle and telephoto lenses exchange the positioning information of the tracking object, the positioning coordinates need to be converted. The conversion process is as follows:

[0096] (X g Y g Z g )=T·(u g v g );

[0097] (X c Y c Z c )=H·(X g Y g Z g );

[0098] (u c v c )=Z·(X c Y c Z c );

[0099] Among them, (X) g Y g Z g (u) represents the wide-angle camera coordinates, and T represents the matrix relating the wide-angle camera coordinates to the wide-angle pixel coordinates. g v g (X) is a wide-angle pixel matrix. c Y c Z c Let H be the coordinates of the telephoto camera, and let H be the transformation matrix between wide-angle and telephoto camera coordinates. c v c ) represents the telephoto pixel coordinates, and Z is the matrix relating the telephoto camera coordinates and the telephoto pixel coordinates.

[0100] Figure 7This is the sixth flowchart of the multi-lens tracking focus shooting method of the present invention. Based on the above embodiments, the step of calculating the sharpness of the tracking object after the telephoto lens acquires the current frame image and receives the auxiliary tracking focus information, and calculating the motor position from the calculated sharpness information, includes:

[0101] S21. Determine the motor that drives the telephoto lens to move;

[0102] S22. Calculate the target orientation of the telephoto lens based on the sharpness information, and determine the desired motor position based on the target orientation.

[0103] Figure 8 This is the seventh flowchart of the multi-lens tracking focus shooting method of the present invention. Based on the above embodiments, the step of driving the motor of the device according to the motor position for tracking focus shooting, and using the predicted trajectory information to drive the motor to the next frame position in advance, includes:

[0104] S31. After the motor is driven to the motor position, a new frame image of the next frame is acquired through the telephoto lens;

[0105] S32. Transmit the new frame image to the wide-angle lens.

[0106] Figure 9 This is the eighth flowchart of the multi-lens tracking focus shooting method of the present invention. Based on the above embodiments, after acquiring a new frame image, the new image information is transmitted to the wide-angle lens for information exchange of the tracking object to obtain new auxiliary tracking focus information. Furthermore, during the tracking focus shooting process, the telephoto lens is used again to perform sharpness calculation and motor drive operations, including:

[0107] S41. Repeat the information exchange operation of the tracking object to update the auxiliary tracking information;

[0108] S42. Repeat the sharpness calculation and motor drive operation to perform the new information exchange operation until the focus tracking shooting is completed.

[0109] Optionally, in this embodiment, during the process of assisting focus tracking based on acquired information, the telephoto lens, after acquiring the current frame image and receiving the assisting focus tracking information, performs sharpness calculation on the object being tracked. In this embodiment, the calculated sharpness information is used to calculate the motor position, drive the motor to complete the focus tracking, and use predictive information to drive the motor to the next frame position in advance. After acquiring the new frame image, the new image information is transmitted to the wide-angle lens for information exchange, thereby obtaining new assisting focus tracking information and continuing to perform sharpness calculation.

[0110] Based on the above embodiments, the present invention also proposes a multi-lens tracking focus shooting device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the multi-lens tracking focus shooting method as described in any of the above embodiments.

[0111] It should be noted that the above-described device embodiments and method embodiments belong to the same concept. The specific implementation process can be found in the method embodiments, and the technical features in the method embodiments are also applicable to the device embodiments, which will not be repeated here.

[0112] Based on the above embodiments, the present invention also proposes a computer-readable storage medium storing a multi-lens focus tracking shooting program, which, when executed by a processor, implements the steps of the multi-lens focus tracking shooting method as described in any of the above embodiments.

[0113] It should be noted that the above-described medium embodiments and method embodiments belong to the same concept. The specific implementation process can be found in the method embodiments, and the technical features in the method embodiments are also applicable to the medium embodiments, which will not be repeated here.

[0114] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0115] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0116] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0117] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A multi-lens focus tracking shooting method, characterized in that, The method includes: During the focus tracking shooting process, the telephoto lens of the device detects moving objects in the shooting scene and marks the candidate circumscribed rectangle of the moving objects; After receiving a selection signal for any of the candidate bounding rectangles and taking the moving object corresponding to the selected candidate bounding rectangle as the focus tracking object, the focus tracking object is transmitted to the wide-angle lens of the device for focus tracking and shooting. After the wide-angle lens receives the tracking object, it marks the selected outer rectangle of the tracking object and calculates the coordinates of the tracking object in the shooting frame based on the selected outer rectangle; The wide-angle lens transmits the first positioning information and predicted trajectory information of the tracking object to the telephoto lens as auxiliary tracking information for tracking shooting. It also receives new frame images and the second positioning information of the tracking object from the telephoto lens. Specifically, the wide-angle lens performs coordinate tracking on the tracking object in each frame, obtaining a high-order curve function fitted by multiple coordinate points. Based on Kalman filtering and the high-order curve function, the first positioning information of the tracking object is calculated, and the predicted trajectory information is also calculated. The positioning information of the tracking object is exchanged between the wide-angle lens and the telephoto lens. During this exchange, the positioning coordinates in the positioning information are converted according to the imaging parameters of the wide-angle lens and the telephoto lens. After the telephoto lens acquires the current frame image and receives the auxiliary focus tracking information, the sharpness of the focus tracking object is calculated, and the motor position is calculated based on the calculated sharpness information. The motor that drives the telephoto lens to move is determined, the target orientation of the telephoto lens is calculated based on the sharpness information, and the motor position to be reached by the motor is determined based on the target orientation. The device's motor is driven to perform tracking shooting according to the motor position, and the motor is driven to the next frame position in advance using the predicted trajectory information. After the motor is driven to the motor position, a new frame image of the next frame is acquired through the telephoto lens and transmitted to the wide-angle lens. After acquiring a new frame image, the new image information is transmitted to the wide-angle lens to exchange information with the tracking object in order to obtain new auxiliary tracking information. During the tracking shooting process, the telephoto lens is used again to perform sharpness calculation and motor drive operation. Repeat the information exchange operation of the tracking object to update the auxiliary tracking information, repeat the sharpness calculation and the motor drive operation to perform a new information exchange operation, until the tracking shooting is completed.

2. A multi-lens autofocus shooting device, characterized in that, The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the multi-lens focus tracking shooting method as described in claim 1.

3. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a multi-lens focus tracking shooting program, which, when executed by a processor, implements the steps of the multi-lens focus tracking shooting method as described in claim 1.

Images