Focusing assistance device, method and system, and storage medium

Abstract

Embodiments of the present description provide a focusing assistance device, method and system, and a storage medium. The focusing assistance device is used for: acquiring a calibration image pair which is obtained by a camera apparatus and a distance measurement apparatus separately performing image acquisition on a calibration pattern; determining a mapping relationship between a lens of the camera apparatus and the distance measurement apparatus on the basis of a matched feature point pair in the calibration image pair, and storing the mapping relationship; and acquiring a scene image of a photography scene acquired by the camera apparatus and a depth image of the photography scene acquired by the distance measurement apparatus, determining depth information of an object in the scene image on the basis of the mapping relationship and the depth image, determining a current depth-of-field range of the lens on the basis of the movement of the lens fed back by a focus adjustment apparatus, and displaying the depth information and the depth-of-field range on a display interface. Thus, users can conveniently know the real-time focusing status and perform focus adjustment or focusing target setting on the basis of the information, thereby improving the focusing efficiency of the users.

Description

Auxiliary focusing device, method, system and storage medium Technical Field

[0001] This specification relates to the field of focusing technology, and in particular to an auxiliary focusing device, method, system and storage medium. Background Art

[0002] Currently, in some complex shooting scenes, a focus operator is required to adjust the lens focus position based on the clarity of each object in the image captured by the camera device. This adjustment method requires the focus operator to repeatedly practice and master the focus technique, and to manually focus for a long time at the shooting scene. The operation is complicated and has a certain threshold. Some solutions can also be equipped with an integrated device that integrates two hardware devices, a ranging device and a camera, on the camera device. The integrated device is used to determine the depth information of each object within the camera's field of view to assist the user in focusing. However, the integrated device needs to be equipped with two hardware devices, a camera and a ranging device, which is relatively expensive. In addition, due to the differences in the posture and viewing angle of the camera in the integrated device and the lens of the camera device, there will be certain errors when directly using the depth image captured by the integrated device for focusing. Therefore, it is necessary to provide a lower-cost and more accurate focusing solution.

[0003] Summary of the Invention

[0004] Based on this, the present specification provides an auxiliary focusing device, method, system and storage medium.

[0005] According to a first aspect of an embodiment of the present specification, there is provided an auxiliary focusing device, the auxiliary focusing device including a display interface, the auxiliary focusing device being communicatively connected to a camera device, a distance measuring device, and a focus adjustment device, respectively, the focus adjustment device being configured to drive a lens of the camera device to move so as to adjust a focus position of the lens;

[0006] The auxiliary focusing device is used to obtain a calibration image pair obtained by the camera device and the ranging device respectively capturing images of a calibration pattern; based on the matched feature point pairs in the calibration image pair, determine and store a mapping relationship between the lens and the ranging device, wherein the mapping relationship is a correspondence between pixel points of the image captured by the lens and pixel points of the image captured by the ranging device;

[0007] and for obtaining a scene image of the shooting scene captured by the camera device and a depth image of the shooting scene captured by the ranging device, determining depth information of objects in the scene image based on the mapping relationship and the depth image, determining the current depth of field range of the lens based on the movement of the lens fed back by the focus adjustment device, and displaying the depth information and the depth of field range on the display interface.

[0008] According to a second aspect of an embodiment of this specification, there is provided an auxiliary focusing method, applicable to an auxiliary focusing device, wherein the auxiliary focusing device includes a display interface, the auxiliary focusing device is communicatively connected to a camera device, a distance measuring device, and a focus adjustment device, respectively, wherein the focus adjustment device is configured to drive a lens of the camera device to move to adjust a focus position of the lens, the method comprising:

[0009] Obtaining a calibration image pair obtained by the camera device and the ranging device respectively capturing images of a calibration pattern; determining and storing a mapping relationship between the lens and the ranging device based on matching feature point pairs in the calibration image pair, wherein the mapping relationship is a correspondence between pixel points of the image captured by the lens and pixel points of the image captured by the ranging device;

[0010] And obtaining a scene image of the shooting scene captured by the camera device and a depth image of the shooting scene captured by the ranging device, determining depth information of objects in the scene image based on the mapping relationship and the depth image, and determining the current depth of field range of the lens based on the movement of the lens fed back by the focus adjustment device, and displaying the depth information and the depth of field range on the display interface.

[0011] According to a third aspect of the embodiments of this specification, an auxiliary focus system is provided, which includes the auxiliary focus device mentioned in the first aspect above, a camera device, a distance measuring device, and a focus adjustment device that are communicatively connected to the auxiliary focus device.

[0012] According to a fourth aspect of the embodiments of this specification, a computer storage medium is provided, wherein the computer-readable storage medium includes a computer program, and when the computer program is executed by a processor, the method mentioned in the first aspect is implemented.

[0013] By applying the solution of the embodiments of this specification, compared to additionally mounting an integrated device that integrates a camera and a ranging device on the camera device, the present application can directly mount a ranging device on the camera device, and can calibrate the mapping relationship between the ranging device and the lens of the camera device through an auxiliary focusing device that is communicatively connected to the camera device. Then, the scene image of the shooting scene captured by the lens of the camera device can be reused, and based on the scene image and the depth image captured by the ranging device, as well as the mapping relationship between the two, the depth information of one or more objects in the scene image captured by the lens of the camera device can be determined. At the same time, the auxiliary focusing device can also determine the current depth of field range of the lens based on the movement of the lens of the camera device fed back by the focus adjustment device, and display the depth information of the objects in the scene image and the depth of field range of the current lens to the user through an interactive interface, so that the user can intuitively see the real-time focusing status. In the manual focusing scene, the focus position can be adjusted based on the displayed content. In the automatic focusing scene, the focus target can also be set through the interactive interface based on the displayed content.

[0014] By directly reusing the scene image captured by the camera lens, there is no need to use an additional integrated device that integrates a distance measuring device and a camera. Only one distance measuring device is needed, which can save hardware costs. And because the present application can ultimately directly determine the depth information of each object under the viewing angle of the camera lens without error, it can also make it possible to focus based on the determined depth information without errors caused by different postures and viewing angles, and the focusing result is more accurate. And by determining the current depth of field range of the camera lens and the depth information of the objects in the scene in real time and displaying them to the user, it is convenient for the user to understand the real-time focusing situation, adjust the focus or set the focus target based on this information, and improve the user's focusing efficiency.

[0015] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0017] FIG. 1 is a schematic diagram of implementing manual focus in the related art.

[0018] FIG2 is a schematic diagram of using a depth image captured by a depth camera to assist a user in focusing in the related art.

[0019] FIG3 is a schematic diagram showing the working principle of the auxiliary focusing device according to an embodiment of the present disclosure.

[0020] FIG4 is a schematic diagram of a depth of field range of a lens according to an embodiment of this specification.

[0021] FIG5 is a schematic diagram of the working principle of an auxiliary focusing device according to an embodiment of this specification.

[0022] FIG6 is a schematic diagram of a mapping relationship between a calibrated lens focus position and a motor motion condition according to an embodiment of this specification.

[0023] FIG. 7 is a schematic diagram of a display interface of a focusing auxiliary device according to an embodiment of this specification.

[0024] FIG8 is a schematic diagram of a focus adjustment component according to an embodiment of the present specification.

[0025] FIG9 is a schematic diagram of the working principle of an auxiliary focusing device according to an embodiment of this specification.

[0026] FIG10 is a schematic diagram showing the working principle of an auxiliary focusing device according to an embodiment of this specification. DETAILED DESCRIPTION

[0027] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this specification. Rather, they are merely examples of apparatus and methods consistent with certain aspects of this specification, as detailed in the appended claims.

[0028] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. As used in this specification and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0029] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information without departing from the scope of this specification. Depending on the context, the term "if" as used herein may be interpreted as "when," "when," or "in response to determining."

[0030] In complex filming scenarios, such as those involving film and television, the large number of objects in a scene requires greater flexibility in lens switching. For example, the lens focus position may need to be frequently adjusted during filming, switching between different objects. Therefore, existing autofocus technology struggles to meet the demands of such scenarios. Currently, dedicated focus pullers are often required to manually adjust the focus position to ensure a clear image of the subject.

[0031] As shown in Figure 1, in manual focus scenarios, a focus adjustment device is typically used to adjust the lens focus of the camera device. The focus adjustment device includes gears and a motor for driving the gears. The lens is typically provided with a focus ring that meshes with the gears on the focus adjustment device. The focus adjustment device is typically connected to a manual focus follower (e.g., a focus follower handwheel, a wireless focus follower, etc.). During the focus follower process, the user can manually control the manual focus follower to issue a focus command. Based on the user's focus command, the manual focus follower can send motion control commands to the motor, controlling the motor's rotation to drive the gears, thereby driving the lens to move and adjust the focus position. In this scenario, the user typically needs to adjust the manual focus follower while observing the clarity of each object in the image captured by the camera device to determine whether the target object has been focused on. This method often requires the user to repeatedly practice and master the focus follower technique, and is relatively difficult to operate.

[0032] Some solutions, in order to assist users in focusing, can install some integrated devices that integrate both a rangefinder and a camera on the camera device. This device automatically determines the depth information of each object in the shooting scene and displays it on the screen of the camera device or a monitor connected to the camera device, so that the user can quickly adjust the focus position based on the depth information of each object. For example, as shown in Figure 2, the above-mentioned integrated device usually includes a rangefinder and a camera. The mapping relationship between the rangefinder and the camera (i.e., the correspondence between the pixels in the images captured by the two) is calibrated before the device leaves the factory and is fixed. Subsequently, the integrated device can determine the depth information of each object in the scene based on the depth information captured by the rangefinder and the image captured by the camera, as well as the mapping relationship between the two. The focus operator can then adjust the position of the lens focus based on the depth information of each object in the scene.

[0033] This solution facilitates user focus, but it presents several challenges. Firstly, it requires the installation of an integrated device on the camera that combines both the rangefinder and the camera. This requires the integration of two hardware devices, resulting in high costs. Secondly, due to the differences in viewing angles and poses between the camera in the integrated device and the lens of the camera, adjusting the focus position of the camera lens directly based on the depth image captured by the integrated device can introduce certain errors.

[0034] Based on this, an embodiment of the present application provides a focus tracking solution. Compared to additionally installing an integrated device with an integrated camera and a ranging device on the camera device, as shown in Figure 3, the present application can directly install a ranging device on the camera device. Since the ranging device is temporarily installed on the camera device, the mapping relationship between the ranging device and the image captured by the lens of the camera device can be calibrated during use. Then, the scene image of the shooting scene captured by the lens of the camera device can be reused. Based on the scene image and the depth image captured by the ranging device, as well as the mapping relationship between the two, the depth information of one or more objects in the scene image captured by the lens of the camera device can be determined. At the same time, the current depth of field range of the lens can also be determined based on the movement of the lens of the camera device fed back by the focus adjustment device. The depth information of the objects in the scene image and the depth of field range of the current lens are displayed to the user so that the user can intuitively understand the real-time focus situation. In the manual focus scenario, the focus position can be adjusted based on the displayed content. In the automatic focus scenario, the focus target can also be set through the interactive interface based on the displayed content.

[0035] Taking into account that in order to facilitate users to view the images captured by the camera device in real time, a device with a display screen is usually equipped to communicate with the camera device, obtain the images captured by the camera device in real time and display them to the user, in this application, the above-mentioned calibration mapping relationship, determination of depth information and depth of field range functions can be integrated into the device to obtain an auxiliary focusing device with auxiliary focusing function, so that users can view the images captured by the camera device through the auxiliary focusing device, and use the depth information and depth of field range displayed by the auxiliary focusing device to adjust the lens focus and set the focus target.

[0036] The solution provided by this application, by directly reusing the scene images captured by the camera lens, can eliminate the need for an additional integrated device that integrates both a distance measuring device and a camera. Instead, only one distance measuring device is required, which can save hardware costs. Furthermore, since this application can ultimately directly determine the depth information of each object under the viewing angle of the camera lens without error, it can also ensure that when focusing based on the determined depth information, there are no errors caused by different postures and viewing angles, and the focusing results are more accurate. Furthermore, by determining the current depth of field range of the camera lens and the depth information of the objects in the scene in real time and displaying them to the user, it is convenient for the user to adjust the focus based on this information, thereby improving the efficiency of the user's focusing.

[0037] Based on the above-mentioned inventive concept, an embodiment of the present application provides an auxiliary focus device, as shown in FIG3 . The auxiliary focus device is respectively connected to a camera device, a distance measuring device, and a focus adjustment device. The focus adjustment device can be used to drive the lens movement and thereby adjust the lens focus position. The auxiliary focus device can be connected to the camera device, the distance measuring device, and the focus adjustment device via a wired or wireless connection. In some scenarios, the auxiliary focus device can be connected to the camera device, the distance measuring device, and the focus adjustment device via a wired transmission interface. For example, the auxiliary focus device can include a communication interface such as SDI or HDMI, and connect to the camera device via this communication interface to obtain image data captured by the camera device. The auxiliary focus device can also include a USB interface for connecting to the distance measuring device to obtain the depth image captured by the distance measuring device. Of course, the auxiliary focus device can also be wirelessly connected to the camera device, the distance measuring device, and the focus adjustment device via wireless communication methods such as Bluetooth and WiFi. The specific communication method can be set based on actual needs and is not limited by the embodiments of the present application.

[0038] In some embodiments, the auxiliary focusing device can be connected to the camera device and the distance measuring device by wires, and connected to the focus adjustment device wirelessly.

[0039] Among them, the ranging device can be various devices that can measure distance, such as laser radar, millimeter wave radar, infrared rangefinder, etc.

[0040] To ensure that the FOV of the ranging device is as consistent as possible with the FOV of the camera lens, and to enable the determination of the depth information of most objects in the image captured by the camera lens based on the depth image captured by the ranging device, the ranging device can be mounted on the camera body. For example, in some scenarios, the camera body includes a hot shoe or a cold shoe, and thus the ranging device can be mounted on the hot shoe or cold shoe of the camera body. Of course, in scenarios where the ranging device is mounted on the hot shoe of the camera body, the ranging device can also communicate with the auxiliary focus device through the camera body, transmitting the captured depth image to the auxiliary focus device.

[0041] Of course, in some scenarios, the ranging device can also be installed on the external structure of the camera device or the peripheral kit, for example, it can be fixed on a rabbit cage, and this embodiment of the present application does not limit it.

[0042] The focus tracking solution of the present application includes two processes: one is the process of calibrating the mapping relationship between the distance measuring device and the lens of the camera device, and the other is the process of using the calibrated mapping relationship for subsequent focus tracking application. Among them, the mapping relationship refers to the correspondence between the pixel points in the image captured by the distance measuring device and the pixel points captured by the lens of the camera device. This mapping relationship is related to the relative posture relationship between the distance measuring device and the lens, as well as the intrinsic parameters of the distance measuring device and the lens (such as focal length, distortion coefficient, etc.).

[0043] Each time the ranging device is installed in the camera device, the mapping relationship between the camera lens and the ranging device can be recalibrated due to differences in the installation method, position, angle, lens type, etc. When calibrating the mapping relationship between the two, the user can place a pre-prepared calibration pattern within the field of view of the camera device and the ranging device, wherein the calibration pattern includes some easy-to-extract feature points, and the distances between these feature points are known, for example, it can be a checkerboard pattern. The calibration mode can then be turned on the auxiliary focus device. For example, in some embodiments, a calibration mode start button can be set on the auxiliary focus device, and the user can turn on the calibration mode when clicking the button. After the calibration mode is turned on, the auxiliary focus device can control the camera device and the ranging device to capture images of the calibration pattern to obtain calibration image pairs, wherein each set of calibration image pairs includes one frame of RGB image captured by the camera device and one frame of grayscale image captured by the ranging device. The auxiliary focusing device can extract feature points from each of the two frames in the calibration image pair, then match the feature points in the two frames to obtain matched feature point pairs. Based on these matched feature point pairs, the relative position relationship between the ranging device and the camera lens can be determined. Furthermore, the aforementioned mapping relationship can be determined based on the intrinsic parameters of the ranging device and the intrinsic parameters of the camera lens. The ranging device can capture a grayscale image and a depth image corresponding to the same scene each time. This mapping relationship represents both the pixel correspondence between the RGB image captured by the lens and the grayscale image captured by the ranging device, and the pixel correspondence between the RGB image captured by the lens and the depth image captured by the ranging device.

[0044] The step of calibrating the mapping relationship between the ranging device and the lens can be performed each time the ranging device is installed. As long as the mapping relationship between the two does not change, the mapping relationship can be used for the subsequent focus tracking process.

[0045] During the focus tracking process, the auxiliary focus device can obtain a scene image of the shooting scene captured by the camera device and a depth image of the shooting scene captured by the ranging device. Then, the depth information of objects in the scene image can be determined using the mapping relationship between the ranging device and the lens calibrated during the calibration process and the depth image. For example, based on the mapping relationship, the corresponding pixel point P2 in the scene image of pixel point P1 in the depth image can be determined, and the depth information of pixel point P1 can be used as the depth information of pixel point P2 to determine the depth information of one or more objects in the scene image.

[0046] As shown in Figure 4, objects typically need to be within a certain distance range before or after the lens's focal point to be clearly imaged. This distance range is the lens's depth of field. While focusing on the target object, the user can adjust the focal point so that the target object is within the lens's current depth of field. This allows for a clear image of the target object. Therefore, the auxiliary focus device can also determine the lens's current depth of field and display the depth information of objects in the scene image and the lens's current depth of field on a display interface to assist the user in focusing.

[0047] In order to allow the auxiliary focusing device to determine the current depth of field range of the lens, the focus adjustment device can feed back the movement of the lens to the auxiliary focusing device.

[0048] In some embodiments, the focus adjustment device includes a gear and a motor for driving the gear to rotate. The gear engages with a focus ring installed on the lens of the camera device. The motor drives the gear to rotate, which can drive the lens to move.

[0049] The motion condition may refer to various types of information that can be used to determine the current position of the lens focus, such as the current position of the lens, the direction and distance of lens movement, and in scenarios where the focus adjustment device includes a motor and gears, the motion condition may also be the rotation direction and number of rotations of the gears used to drive the lens movement, or the rotation direction and number of rotations of the motor used to drive the lens movement, etc. After driving the lens to move, the focus adjustment device may return the lens motion condition to the auxiliary focus device. The auxiliary focus device may determine the current focus position of the lens based on the lens motion condition, and then determine the depth of field range of the current lens in combination with parameters such as the aperture and focal length of the lens.

[0050] The depth information may be the depth information of all objects in the scene image, or the depth information of some objects in the scene image, and the specific setting may be based on actual needs. As shown in FIG5 , the objects in the scene may be represented in the form of a projection point distribution map. For example, a depth axis may be displayed on the display interface. The depth axis may identify the corresponding depth value. The three-dimensional point cloud of the object is then projected in the depth direction to obtain a projection point distribution map of each object, which is then displayed at the corresponding position of the depth axis. The depth range may also be identified on the depth axis. For example, a certain distance range may be selected on the depth axis (as shown in the shaded portion in FIG5 ) to identify the depth range. In order to facilitate the user to associate the depth information of each object with each object in the scene image, an association identifier may also be displayed on the display interface at the same time to facilitate the user to match the two. The association identifier may be a digital number or other characters. For example, the digital number in FIG5 may be used to identify the same character near each object in the scene image and near the projection distribution map of the object, so as to associate the two information, making it convenient for the user to associate the two types of information.

[0051] In some embodiments, as shown in FIG6 , in order to facilitate the user to use the auxiliary focusing device to adjust the focus of the lens, the auxiliary focusing device can also be used to obtain a focus instruction, generate a motion control instruction based on the focus instruction and send it to the focus adjustment device, so that the focus adjustment device controls the movement of the lens based on the motion control instruction to adjust the focus position of the lens. The focus instruction can be automatically generated by the auxiliary focusing device based on the depth information of the target object to be focused and the current depth of field range of the lens, or can be issued by the user. The auxiliary focusing device can generate a motion control instruction based on the focus instruction. The motion control instruction can be used to indicate the movement mode of the lens or the movement mode of the motor that drives the lens movement, such as which direction the lens should move and how far it should move, or which direction the motor should rotate and how many revolutions it should rotate, etc., so that the focus adjustment device can drive the lens movement based on the motion control instruction to achieve focusing on the target object to be focused.

[0052] In some embodiments, to facilitate the auxiliary focusing device in determining the depth of field range based on the movement of the lens and controlling the focus adjustment device, the mapping relationship between the focus position, lens position, motor rotation direction, and number of rotations can be pre-calibrated. For example, during the calibration process, the target object to be focused can be positioned at different distances from the lens, such as 1m, 2m, and 3m. The lens position can then be adjusted to achieve a clear image of the target object. The distance the lens moves from the extreme position to the position, as well as the motor rotation direction and number of rotations, can be recorded. This allows the mapping relationship between the focus position, lens position, motor rotation direction, and number of rotations to be obtained. For example, as shown in Figure 7, assume that the two extreme positions during the lens movement are O1 and O2, that is, the lens movement range is the distance between O1 and O2. If the lens is at position A and the image of the target object at a distance of 1m is clear, the focus position can be recorded as when the lens is at position A 1m from the lens and the motor rotates 2 rotations in the forward direction. Similarly, if the lens is at B and the image of the target object at a distance of 2m is clear, the focus position can be recorded as being 2m away from the lens, the lens is at position B, and the motor rotates 4 circles in the forward direction. In the subsequent focusing process, the direction in which the lens should move and how far it should move can be determined based on the depth information of the target object to be focused and the depth of field range of the current lens. To ensure that the lens moves in this way, the direction in which the motor should rotate and how many circles it should rotate, etc., thereby issuing motion control instructions to control the movement of the motor. Similarly, when the focus adjustment device feeds back the movement of the lens, for example, the lens is at position A, the auxiliary focus device can also determine that the focus position at this time is 1m based on the pre-calibrated mapping relationship mentioned above, and then determine the current depth of field range of the lens based on information such as the focus position, aperture, and focal length of the lens at this time.

[0053] In the solution of the present application, when focusing the lens of the camera device, two focus modes may be included: an automatic focus mode and a manual focus mode.

[0054] In some embodiments, when the lens is in manual focus mode, the above focus instruction can be issued by the user through the display interface of the auxiliary focus device.

[0055] In some embodiments, the focus adjustment device can also be connected to a manual focus device in communication. For example, the manual focus device can be a focus hand wheel, or other focus devices with similar functions. When the lens is in manual focus mode, the manual focus device can obtain the user's focus command and control the focus adjustment device to drive the lens movement based on the focus command. Taking the above-mentioned focus command issued by the user through the display interface of the auxiliary focus device as an example, in some embodiments, as shown in Figure 5, the display interface can also display a focus adjustment component, which can be used to adjust the focus position of the lens. The above-mentioned focus command can also be generated based on the interaction between the user and the focus adjustment component.

[0056] In some embodiments, the focus adjustment component can be a virtual focus wheel. Users can interact with the virtual focus wheel to control its rotation, thereby controlling the motor's rotation. A mapping between the number of rotations of the virtual focus wheel and the lens's range of motion can be pre-set. The direction and distance of lens movement can be determined based on the number of rotations of the virtual focus wheel controlled by the user. In some embodiments, to facilitate user adjustment of the lens's focus position, as shown in FIG8 , the focus adjustment component can also include a range element for displaying the lens's range of motion and an adjustment element. The user can drag the adjustment element within the range element to control lens movement. The range of motion represented by the range element corresponds to the actual range of motion of the lens, and each position within the range element corresponds to the actual position of the lens. When the adjustment element is dragged to a position within the range element, the motor can be controlled based on the actual lens position corresponding to that position, thereby driving the lens to the corresponding position.

[0057] In some embodiments, to facilitate users in viewing real-time focus adjustment results, as shown in FIG8 , while the user drags the adjustment element within the interval element, the auxiliary focus device can determine the current focus position of the lens based on the position of the adjustment element within the interval element in real time, and determine the current depth of field range of the lens based on the current focus position of the lens in real time, and display the re-determined depth of field range on the display interface. This allows the user to browse the current adjustment results and determine whether the target object to be focused on has been achieved.

[0058] In some embodiments, when the lens is in autofocus mode, the aforementioned focus instructions may also be automatically generated by an auxiliary focus device. For example, the auxiliary focus device may identify a target object to be focused on from a scene image, and then generate a focus instruction based on the depth information of the target object and the current depth of field range of the lens. The focus instruction is used to adjust the current focus position of the lens, and thereby adjust the current depth of field range of the lens, so that the target object is within the depth of field range, thereby achieving a clear image.

[0059] Among them, the target object to be focused can be pre-set by the user through the display interface. For example, the characters in the scene can be used as the target object, or the user can click or select one or more objects from the scene image displayed in the display interface as the target object. For example, the user selects an object as the target object in the first frame scene image captured by the auxiliary focus device. Later, in the focusing process, the auxiliary focus device can track the target object based on the captured multiple frames of scene images, determine the position of the target object in each frame of the scene image, and then generate a focus instruction to instruct to focus on the target object. For scenes where the focus instruction is issued by the auxiliary focus device, the auxiliary focus device can automatically determine the movement of the lens based on the focus instruction, redetermine the depth of field range after the lens movement in combination with the current depth of field range of the lens and the movement, and use the redetermined depth of field range to update the currently displayed depth of field range.

[0060] For scenarios where focus instructions are issued through a manual focus device, in order to facilitate the auxiliary focus device to know the current movement of the lens, so as to determine and display the focus position and depth of field range of the current lens, the focus adjustment device can feed back the movement of the lens indicated by the focus instruction (for example, the direction and distance of lens movement, or the direction and number of motor rotations, etc.) to the auxiliary focus device, so that the auxiliary focus device can redetermine the depth of field range after the lens movement based on the current depth of field range of the lens and the movement status, and use the redetermined depth of field range to update the currently displayed depth of field range.

[0061] Generally speaking, the lens of a camera device used for large-scale shooting scenes such as film and television shooting generally includes multiple focal lengths. Since the mapping relationship between the lens and the distance measuring device will also change when the focal length of the lens changes, during the mapping relationship calibration process, the mapping relationship between the lens and the distance measuring device when the lens is at different focal lengths can be calibrated and stored for subsequent use.

[0062] In some embodiments, when calibrating the mapping relationship between the lens and the distance measuring device when the lens is at different focal lengths, the lens can be adjusted to a specified focal length among the multiple focal lengths, and then the lens and the distance measuring device are used to respectively capture images of the calibration pattern to obtain a calibration image pair at the specified focal length. The auxiliary focusing device can then determine a matching feature point pair from the calibration image pair, and then determine the relative position relationship between the lens and the distance measuring device at the focal length based on the feature point pair, and then determine the mapping relationship between the two at the specified focal length based on the internal parameters of the lens such as the focal length. In addition, the auxiliary focusing device can control the lens of the camera device to be adjusted to the multiple focal lengths respectively, and capture a frame of image for the same scene at each focal length. The auxiliary focusing device can obtain multiple frames of images captured by the lens for the same scene at different focal lengths, and determine the scaling relationship between the images captured by the lens at other focal lengths among the multiple focal lengths except the specified focal length and the images captured by the lens at the specified focal length. Based on the scaling relationship and the mapping relationship between the lens and the distance measuring device at the specified focal length, the mapping relationship between the lens and the distance measuring device at other focal lengths is determined.

[0063] In some embodiments, if the camera lens includes multiple focal lengths, during the focus tracking process, the auxiliary focus device, when determining the depth information of objects in the scene image based on the mapping relationship between the rangefinder and the lens and the depth image, may first determine the current target focal length of the lens, then determine a target mapping relationship corresponding to the target focal length based on predetermined mapping relationships between the lens and the rangefinder when the lens is at different focal lengths, and then determine the depth information of the objects in the scene image based on the target mapping relationship and the depth image. Only by selecting a matching mapping relationship based on the current focal length of the lens can the depth information of objects in the scene image be accurately determined.

[0064] In some embodiments, the auxiliary focus device is further used to communicate with a focal length adjustment device, which can be used to adjust the focal length of the lens of the camera device. For example, the focal length adjustment device may include a gear that engages with the lens zoom ring, and a motor that drives the gear to rotate and thereby drives the lens to move. The auxiliary focus device can obtain the movement of the lens from the focal length adjustment device and determine the current target focal length of the lens based on the movement. For example, the correspondence between the rotation direction and number of revolutions of the motor and the focal length of the lens can be pre-calibrated, and then the rotation direction and number of revolutions of the motor can be obtained from the focal length adjustment device, and the current focal length can be determined based on this information.

[0065] Generally, the principle of distance measurement implemented by a ranging device is as follows: the ranging device emits a detection light beam, which is reflected back after being irradiated by the object to be measured. The ranging device can determine the distance between the object to be measured and the ranging device based on the emission time of the detection light beam, the time of receiving the return light, and the light beam. When the ranging device is measuring distance, when the frequency of the detection light beam emitted by the ranging device is high, the measurable distance is short and the measurement accuracy is high. When the frequency of the detection light beam emitted by the ranging device is low, the measurable distance is long and the measurement accuracy is low. Currently, due to the uncertainty of the distance between the object in the shooting scene and the camera device, in order to simultaneously ensure the measurement distance and measurement accuracy of the ranging device, two transmitting units with different frequencies are required to alternately emit detection light beams. This method will increase the power consumption of the ranging device, and since the two frames of data need to be combined to obtain the final depth image, the delay is high.

[0066] Considering that the present application can determine the depth information of objects within the camera's lens field of view, i.e., the distance between the objects in the captured scene and the camera can be determined, the frequency of the detection beam emitted by the ranging device can be determined based on the distance between the objects in the captured scene and the camera. For example, if the distance between the objects in the captured scene and the camera is generally close, a higher-frequency beam can be emitted; otherwise, a lower-frequency beam can be emitted. In this way, a dual-frequency mode is not required, and a single-frequency mode can be used instead, emitting a detection beam with a frequency that matches the depth information.

[0067] Therefore, in some embodiments, the ranging device includes multiple operating modes, and the frequencies of the detection beams emitted by the ranging device in different operating modes vary. The auxiliary focusing device can determine a target operating mode that matches the depth information of objects in the scene image from the multiple operating modes, and then control the ranging device to switch its current operating mode to the target operating mode. The target operating mode that matches the depth information means that the frequency of the detection beam emitted by the ranging device in the target operating mode ensures that all objects in the captured scene can be detected while also ensuring good measurement accuracy.

[0068] In some embodiments, the greater the maximum value of the depth information of an object in the scene image, the smaller the frequency of the probe beam emitted in the target operating mode. By flexibly switching the frequency of the probe beam, it is possible to achieve the desired detection distance while ensuring measurement accuracy and reducing latency.

[0069] Taking into account scenes such as film and television shooting, a wireless transmission system is usually configured to send the video images captured by the camera device to one or more devices, so that the director and other staff can receive and view the video images captured by the camera device to achieve remote monitoring and scheduling. The wireless transmission system usually includes a transmitting device and a receiving device. The transmitting device is used to connect to the camera device to obtain the video image captured by the camera device in real time. The receiving device is wirelessly connected to the transmitting device to obtain the video image from the transmitting device and display it to the user. In some embodiments, as shown in Figure 9, the auxiliary focus device can be a transmitting device in the wireless transmission system, and the auxiliary focus device is also used to communicate with one or more receiving devices in the wireless transmission system and send the captured image to the one or more receiving devices.

[0070] In some embodiments, as shown in FIG10 , the auxiliary focus device may also be a receiving device in a wireless transmission system. The auxiliary focus device is further configured to communicate with a transmitting device in the wireless transmission system and receive scene images captured by a camera device and depth images captured by a ranging device from the transmitting device, and to send motion control instructions to a focus adjustment device via the transmitting device. Of course, in some scenarios, the auxiliary focus device may also be simultaneously connected to a focus adjustment device. After generating a motion control instruction, the motion control instruction may be directly sent to the focus adjustment device (as shown by the dotted arrow in the figure) without being forwarded via the transmitting device.

[0071] The various technical features in the above embodiments can be arbitrarily combined as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, the arbitrary combination of the various technical features in the above embodiments also falls within the scope of disclosure of this specification.

[0072] Furthermore, an embodiment of the present application also provides an auxiliary focusing method, which is applicable to an auxiliary focusing device, wherein the auxiliary focusing device includes a display interface, and the auxiliary focusing device is respectively communicatively connected to a camera device, a distance measuring device, and a focus adjustment device, wherein the focus adjustment device is used to drive the lens of the camera device to move to adjust the focus position of the lens. The method includes:

[0073] Obtaining a calibration image pair obtained by the camera device and the ranging device respectively capturing images of a calibration pattern; determining and storing a mapping relationship between the lens and the ranging device based on matching feature point pairs in the calibration image pair, wherein the mapping relationship is a correspondence between pixel points of the image captured by the lens and pixel points of the image captured by the ranging device;

[0074] And obtaining a scene image of the shooting scene captured by the camera device and a depth image of the shooting scene captured by the ranging device, determining depth information of objects in the scene image based on the mapping relationship and the depth image, and determining the current depth of field range of the lens based on the movement of the lens fed back by the focus adjustment device, and displaying the depth information and the depth of field range on the display interface.

[0075] In some embodiments, the ranging device includes multiple operating modes, and the frequencies of the detection light beams emitted by the ranging device in different operating modes are different; the method further includes:

[0076] Determine a target operating mode that matches the depth information from a plurality of operating modes; and control the ranging device to switch the current operating mode to the target operating mode.

[0077] In some embodiments, the larger the maximum value in the depth information is, the smaller the frequency of the detection light beam emitted in the target working mode is.

[0078] In some embodiments, the lens includes multiple focal lengths, the mapping relationship includes a mapping relationship between the lens and the ranging device when the lens is at different focal lengths, and determining the depth information of the object in the scene image based on the mapping relationship between the ranging device and the lens and the depth image includes:

[0079] Determining a current target focal length of the lens;

[0080] determining a target mapping relationship corresponding to the target focal length based on a mapping relationship between the lens and the distance measuring device when the lens is at different focal lengths;

[0081] Depth information of an object in the scene image is determined based on the target mapping relationship and the depth image.

[0082] In some embodiments, the method includes: obtaining a focus instruction, generating a motion control instruction based on the focus instruction and sending the instruction to the focus adjustment device, so that the focus adjustment device drives the lens to move based on the motion control instruction to adjust the focus position of the lens.

[0083] In some embodiments, when the lens is in manual focus mode, the focus instruction is issued by the user through the display interface.

[0084] In some embodiments, a focus adjustment component is also displayed in the display interface, and the focus instruction is generated based on the interaction between the user and the focus adjustment component.

[0085] In some embodiments, the focus adjustment component includes an interval element for displaying the range of motion of the lens and an adjustment element, and a user can drag the adjustment element to move within the interval element to control the movement of the lens.

[0086] In some embodiments, when the user drags the adjustment element to move within the interval element, the auxiliary focusing device is also used to determine the current focus position of the lens based on the real-time position of the adjustment element within the interval element, determine the current depth of field range of the lens based on the current focus position of the lens, and display the determined depth of field range in real time in the display interface.

[0087] In some embodiments, when the lens is in autofocus mode, the focus instruction is generated based on the following method:

[0088] A target object to be focused on is identified from the scene image, and a focus instruction is generated based on depth information of the target object and a current depth of field range of the lens, wherein the focus instruction is used to adjust the current depth of field range of the lens so that the target object is within the depth of field range.

[0089] In some embodiments, a focus ring is provided on the lens, and the focus adjustment device includes a gear meshing with the focus ring, and a motor for driving the gear to rotate.

[0090] In some embodiments, the mapping relationship between the lens and the ranging device when the lens is at different focal lengths is determined based on the following method:

[0091] For a specified focal length among the multiple focal lengths, a mapping relationship between the lens and the ranging device when the lens is at the specified focal length is determined based on a calibration image pair captured at the specified focal length, images of the same scene captured when the lens is respectively at the multiple focal lengths are obtained, and a scaling relationship between images captured by the lens at focal lengths other than the specified focal length among the multiple focal lengths and the image captured by the lens at the specified focal length is determined based on the images; and based on the scaling relationship and the mapping relationship between the lens and the ranging device at the specified focal length, a mapping relationship between the lens and the ranging device at other focal lengths is determined.

[0092] In some embodiments, the auxiliary focusing device is in communication with a focal length adjustment device, and the focal length adjustment device is used to drive the lens to move to adjust the focal length of the lens; the method further includes:

[0093] The movement of the lens fed back by the focal length adjustment device is obtained, and the current target focal length of the lens is determined based on the movement.

[0094] Among them, the specific implementation details of the above-mentioned auxiliary focus method can be referred to the description of each embodiment of the above-mentioned auxiliary focus device, and will not be repeated here.

[0095] In addition, an embodiment of the present application also provides an auxiliary focus system, which includes the auxiliary focus device introduced in the above embodiment, a camera device, a distance measuring device, and a focus adjustment device that are communicatively connected to the auxiliary focus device.

[0096] Accordingly, an embodiment of the present application further provides a computer storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method in any of the above embodiments is implemented.

[0097] The embodiments of the present application may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0098] Those skilled in the art will readily recognize other implementations of the embodiments of the present invention after considering the specification and practicing the instructions disclosed herein. The embodiments of the present invention are intended to cover any variations, uses, or adaptations of the embodiments of the present invention that follow the general principles of the embodiments of the present invention and include common knowledge or customary techniques in the art not disclosed in the embodiments of the present invention. The description and examples are to be considered as exemplary only, and the true scope and spirit of the embodiments of the present invention are indicated by the following claims.

[0099] It should be understood that the embodiments of the present invention are not limited to the precise structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the embodiments of the present invention is limited only by the appended claims.

[0100] The above description is only a preferred embodiment of the embodiments of this specification and is not intended to limit the embodiments of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the embodiments of this specification should be included in the scope of protection of the embodiments of this specification.

Claims

1. A focusing auxiliary device, It is characterized in that The auxiliary focusing device comprises a display interface, and the auxiliary focusing device is respectively connected to the camera device, the distance measuring device, and the focus adjusting device, and the focus adjusting device is used to drive the lens of the camera device to move so as to adjust the focus position of the lens; The auxiliary focusing device is used to obtain a calibration image pair obtained by the camera device and the distance measuring device respectively performing image acquisition on the calibration pattern; based on the matched feature point pairs in the calibration image pair, a mapping relationship between the lens and the distance measuring device is determined and stored, wherein the mapping relationship is a correspondence relationship between pixel points of the image acquired by the lens and pixel points of the image acquired by the distance measuring device; and for obtaining a scene image of the shooting scene captured by the camera device and a depth image of the shooting scene captured by the ranging device, determining depth information of an object in the scene image based on the mapping relationship and the depth image, determining a current depth of field range of the lens based on the movement of the lens fed back by the focus adjustment device, and displaying the depth information and the depth of field range on the display interface.

2. The auxiliary focusing device according to claim 1, It is characterized in that The auxiliary focusing device is also used to obtain a focusing instruction, generate a motion control instruction based on the focusing instruction and send it to the focus adjustment device, so that the focus adjustment device drives the lens to move based on the motion control instruction to adjust the focus position of the lens.

3. The auxiliary focusing device according to claim 2, It is characterized in that When the lens is in manual focus mode, the focus instruction is issued by the user through the display interface.

4. The auxiliary focusing device according to claim 3, It is characterized in that The display interface also displays a focus adjustment component, and the focus instruction is generated based on the interaction between the user and the focus adjustment component.

5. The auxiliary focusing device according to claim 4, It is characterized in that The focus adjustment component includes an interval element for displaying the range of motion of the lens and an adjustment element. A user can drag the adjustment element to move within the interval element to control the movement of the lens.

6. The auxiliary focusing device according to claim 5, It is characterized in that When the user drags the adjustment element to move within the interval element, the auxiliary focusing device is also used to determine the current focal position of the lens based on the real-time position of the adjustment element within the interval element, determine the current depth of field range of the lens based on the current focal position of the lens, and display the determined depth of field range in real time in the display interface.

7. The auxiliary focusing device according to claim 1, It is characterized in that When the lens is in autofocus mode, the focus instruction is generated based on the following method: A target object to be focused is identified from the scene image, and the focus instruction is generated based on depth information of the target object and a current depth of field range of the lens, wherein the focus instruction is used to adjust the current depth of field range of the lens so that the target object is within the depth of field range.

8. The auxiliary focusing device according to claim 1, It is characterized in that The focus adjustment device is communicatively connected to the manual focus follow device. When the lens is in manual focus mode, the manual focus follow device is used to obtain a focus instruction from a user and control the focus adjustment device to drive the lens to move based on the focus instruction.

9. The auxiliary focusing device according to claim 1, It is characterized in that A focus ring is arranged on the lens, and the focus adjustment device comprises a gear meshed with the focus ring and a motor for driving the gear to rotate.

10. The auxiliary focusing device according to claim 1, It is characterized in that The lens includes a plurality of focal lengths, and the mapping relationship includes a mapping relationship between the lens and the distance measuring device when the lens is at different focal lengths.

11. The auxiliary focusing device according to claim 10, It is characterized in that The mapping relationship between the lens and the distance measuring device when the lens is at different focal lengths is determined based on the following method: For a specified focal length among the multiple focal lengths, determining, based on a calibration image pair acquired at the specified focal length, the mapping between the lens and the distance measuring device when the lens is at the specified focal length obtaining images captured for the same scene when the lens is respectively at the multiple focal lengths, and determining, based on the images, a scaling relationship between the images captured by the lens at other focal lengths among the multiple focal lengths except the designated focal length and the image captured by the lens at the designated focal length; Based on the scaling relationship and the mapping relationship between the lens and the distance measuring device at the specified focal length, the mapping relationship between the lens and the distance measuring device at other focal lengths is determined.

12. The auxiliary focusing device according to claim 10, It is characterized in that The auxiliary focusing device is used to determine the depth information of the object in the scene image based on the mapping relationship between the distance measuring device and the lens and the depth image, specifically for: Determining a current target focal length of the lens; Determining a target mapping relationship corresponding to the target focal length based on a mapping relationship between the lens and the distance measuring device when the lens is at different focal lengths; Depth information of an object in the scene image is determined based on the target mapping relationship and the depth image.

13. The auxiliary focusing device according to claim 1, It is characterized in that The auxiliary focusing device is in communication with a focal length adjustment device, and the focal length adjustment device is used to drive the lens to move so as to adjust the focal length of the lens; The auxiliary focusing device is used to obtain the movement of the lens fed back by the focal length adjustment device, and determine the current target focal length of the lens based on the movement.

14. The auxiliary focusing device according to claim 1, It is characterized in that The distance measuring device includes multiple working modes, and the frequencies of the detection light beams emitted by the distance measuring device in different working modes are different; The auxiliary focusing device is also used to determine a target working mode that matches the depth information from a plurality of working modes; and control the distance measuring device to switch the current working mode to the target working mode.

15. The auxiliary focusing device according to claim 14, It is characterized in that in, The larger the maximum value in the depth information is, the smaller the frequency of the detection light beam emitted in the target working mode is.

16. The auxiliary focusing device according to claim 1, It is characterized in that The auxiliary focus device is a transmitting device in the wireless transmission system, and the auxiliary focus device is also used to communicate with one or more receiving devices in the wireless transmission system and send the collected image to the one or more receiving devices.

17. The auxiliary focusing device according to claim 1, It is characterized in that The auxiliary focusing device is a receiving device in the wireless transmission system. The auxiliary focusing device is also used to communicate with the transmitting device in the wireless transmission system, and receive the scene image captured by the camera device and the depth image captured by the ranging device from the transmitting device, and send the motion control instruction to the focus adjustment device through the transmitting device.

18. The auxiliary focusing device according to claim 1, It is characterized in that The auxiliary focusing device is connected to the camera device by wire, connected to the distance measuring device by wire, and connected to the focus adjusting device by wireless.

19. The auxiliary focusing device according to claim 1, It is characterized in that The distance measuring device is installed on the hot shoe of the camera device.

20. A focusing assist method, It is characterized in that The method is applicable to an auxiliary focusing device, the auxiliary focusing device includes a display interface, the auxiliary focusing device is respectively connected to a camera device, a distance measuring device, and a focus adjustment device, the focus adjustment device is used to drive the lens of the camera device to move to adjust the focus position of the lens, and the method includes: Acquire a calibration image pair obtained by the camera device and the distance measuring device respectively performing image acquisition on the calibration pattern; determine and store a mapping relationship between the lens and the distance measuring device based on the matched feature point pairs in the calibration image pair, wherein the mapping relationship is a correspondence relationship between pixel points of the image acquired by the lens and pixel points of the image acquired by the distance measuring device; And obtaining a scene image of the shooting scene captured by the camera device and a depth image of the shooting scene captured by the ranging device, determining depth information of objects in the scene image based on the mapping relationship and the depth image, determining a current depth of field range of the lens based on the movement of the lens fed back by the focus adjustment device, and displaying the depth information and the depth of field range on the display interface.

21. The method according to claim 20, It is characterized in that The distance measuring device includes multiple working modes, and the frequencies of the detection light beams emitted by the distance measuring device in different working modes are different; the method also includes: Determine a target working mode that matches the depth information from a plurality of working modes; and control the distance measuring device to switch the current working mode to the target working mode.

22. The method according to claim 21, It is characterized in that in, The larger the maximum value in the depth information is, the smaller the frequency of the detection light beam emitted in the target working mode is.

23. The method according to claim 21, It is characterized in that The lens includes a plurality of focal lengths, the mapping relationship includes a mapping relationship between the lens and the ranging device when the lens is at different focal lengths, and the determining the depth information of the object in the scene image based on the mapping relationship between the ranging device and the lens and the depth image includes: Determining a current target focal length of the lens; Determining a target mapping relationship corresponding to the target focal length based on a mapping relationship between the lens and the distance measuring device when the lens is at different focal lengths; Depth information of an object in the scene image is determined based on the target mapping relationship and the depth image.

24. An auxiliary focusing system, It is characterized in that The auxiliary focus system includes the auxiliary focus device as described in any one of claims 1 to 19, and a camera device, a distance measuring device, and a focus adjustment device that are communicatively connected to the auxiliary focus device.

25. A computer-readable storage medium, It is characterized in that The computer-readable storage medium comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 20 to 23 is implemented.

Images