Camera synchronization method and device, and storage medium and computer program product

Abstract

The present application relates to the technical field of cameras. Disclosed are a camera synchronization method and device, and a storage medium and a computer program product. The method comprises: acquiring a first frame end time and a second frame end time, and determining the absolute value of a time difference between the first frame end time and the second frame end time, wherein the first frame end time indicates a frame end time of image frame capture by a first camera, the second frame end time indicates a frame end time of image frame capture by a second camera, and the first camera and the second camera are cameras to be synchronized; if the absolute value of the time difference is greater than a preset threshold value, adjusting the frame length of the first camera and / or the second camera, and returning to the step of acquiring the first frame end time and the second frame end time; and if the absolute value of the time difference is less than or equal to the preset threshold value, determining that camera synchronization between the first camera and the second camera is completed. The present application reduces the implementation cost of multi-camera synchronization.

Description

Camera synchronization method, device, storage medium and computer program product

[0001] The present application claims priority to the Chinese patent application No. 202411029182.5, filed on July 29, 2024, and entitled "Camera synchronization method, device, storage medium and computer program product", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of cameras, in particular to a camera synchronization method, device, storage medium and computer program product. BACKGROUND

[0003] At present, multi-cameras have been applied to various fields, such as unmanned aerial vehicle flight control, extended reality (XR), motion capture, robots, SLAM (Simultaneous Localization and Mapping), and self-driving, etc. In order to make multi-cameras work better and reduce calculation errors, each application scenario has strict requirements for the synchronization error of multi-cameras.

[0004] Generally, the synchronization of multi-cameras can be realized by sending a synchronization signal through a Field-Programmable Gate Array (FPGA) at the hardware level. However, the structure of FPGA is complex, and the implementation logic is complex, which leads to high implementation cost of multi-camera synchronization. Therefore, how to reduce the implementation cost of multi-camera synchronization is a problem to be solved at present.

[0005] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0006] The main purpose of the present application is to provide a camera synchronization method, device, storage medium and computer program product, which aims to solve the technical problem of how to reduce the implementation cost of multi-camera synchronization.

[0007] To achieve the above purpose, the present application provides a camera synchronization method, which comprises:

[0008] obtaining a first frame end time and a second frame end time, and determining a time difference absolute value between the first frame end time and the second frame end time, wherein the first frame end time indicates a frame end time of a first camera collecting an image frame, the second frame end time indicates a frame end time of a second camera collecting an image frame, and the first camera and the second camera are cameras to be synchronized;

[0009] if the absolute value of the time difference is greater than a preset threshold, adjusting a frame length of the first camera and / or the second camera, and returning to execute the step of acquiring the first frame end time and the second frame end time;

[0010] if the absolute value of the time difference is less than or equal to the preset threshold, determining that the first camera and the second camera complete camera synchronization.

[0011] In an embodiment, the step of adjusting the frame length of the first camera and / or the second camera comprises:

[0012] determining an adjustment step based on the absolute value of the time difference, wherein the adjustment step is positively correlated with the absolute value of the time difference;

[0013] if the first frame end time is less than the second frame end time, adjusting the frame length of the first camera and / or the second camera based on the adjustment step, wherein the adjusted frame length of the first camera is greater than the unadjusted frame length of the first camera, and the adjusted frame length of the second camera is less than the unadjusted frame length of the second camera;

[0014] if the first frame end time is greater than the second frame end time, adjusting the frame length of the first camera and / or the second camera based on the adjustment step, wherein the adjusted frame length of the first camera is less than the unadjusted frame length of the first camera, and the adjusted frame length of the second camera is greater than the unadjusted frame length of the second camera.

[0015] In an embodiment, the frame length comprises a field blanking time, and the step of adjusting the frame length of the first camera and / or the second camera based on the adjustment step comprises:

[0016] adjusting the field blanking time of the first camera and / or the second camera based on the adjustment step.

[0017] In an embodiment, the step of determining the adjustment step based on the absolute value of the time difference comprises:

[0018] taking a ratio between the absolute value of the time difference and a preset exposure line time as the adjustment step.

[0019] In an embodiment, the step of adjusting the frame length of the first camera and / or the second camera comprises:

[0020] if the first camera is a secondary camera, adjusting the frame length of the first camera;

[0021] if the second camera is a secondary camera, adjusting the frame length of the second camera.

[0022] In an embodiment, after the step of determining that the first camera and the second camera complete camera synchronization, the method further comprises:

[0023] If there is an adjustment to the frame length of the first camera, a first initial frame length of the first camera is obtained, and the first camera is controlled to perform image acquisition based on the first initial frame length, wherein the first initial frame length is the frame length of the image frame acquired before the step length of the first camera is adjusted.

[0024] If there is an adjustment to the frame length of the second camera, a second initial frame length of the second camera is obtained, and the second camera is controlled to perform image acquisition based on the second initial frame length, wherein the second initial frame length is the frame length of the image frame acquired before the step length of the second camera is adjusted.

[0025] In an embodiment, the first camera is an RGB camera, and the second camera is a 6DOF camera.

[0026] In addition, to achieve the above-mentioned purposes, the present application also provides an extended reality device, which comprises a first camera, a second camera and a processor, the processor is connected with the first camera and the second camera respectively, and the processor is used to execute the steps of the camera synchronization method as described above.

[0027] In addition, to achieve the above-mentioned purposes, the present application also provides a storage medium, which is a computer readable storage medium, and a computer program is stored on the storage medium, the computer program is executed by the processor to realize the steps of the camera synchronization method as described above.

[0028] In addition, to achieve the above-mentioned purposes, the present application also provides a computer program product, which comprises a computer program, and the computer program is executed by the processor to realize the steps of the camera synchronization method as described above.

[0029] The one or more technical solutions provided by the present application have at least the following technical effects:

[0030] The first frame end time and the second frame end time are acquired, and an absolute value of a time difference between the first frame end time and the second frame end time is determined, where the first frame end time indicates a frame end time of an image frame captured by a first camera, the second frame end time indicates a frame end time of an image frame captured by a second camera, and the first camera and the second camera are cameras to be synchronized, so that the frame end (EOF) times of the cameras to be synchronized (including the first camera and the second camera) are acquired to perform subsequent synchronization, without limiting the specific camera types of the first camera and the second camera, and any type of camera can be synchronized, especially different types of the first camera and the second camera can also be synchronized, improving the universality of the synchronization mode.

[0031] If the absolute value of the time difference is greater than a preset threshold, the frame length of the first camera and / or the second camera is adjusted, and the step of acquiring the first frame end time and the second frame end time is performed again. When the absolute value of the time difference between the first frame end time and the second frame end time is greater than the preset threshold, the frame length (frame_length) of the first camera and / or the second camera is adjusted to synchronize the first camera and the second camera. In this way, the first camera and the second camera are synchronized at the software level, without the need to add additional hardware structures, thereby reducing the implementation cost of multi-camera synchronization.

[0032] If the absolute value of the time difference is less than or equal to the preset threshold, it is determined that the first camera and the second camera complete camera synchronization. When the absolute value of the time difference between the first frame end time and the second frame end time is less than or equal to the preset threshold, it is determined that the first camera and the second camera complete camera synchronization, that is, the frame end times of the synchronized cameras are synchronized, rather than the frame start (SOF) times or the exposure start (Expstart) times. Since the cameras transmit the captured image frames after the frame end time, for different types of cameras, the frame start times or the exposure start times are synchronized, but the frame end times are not necessarily synchronized, so that if the frame start times or the exposure start times of the synchronized cameras are synchronized, the frame end times cannot be guaranteed to be synchronized, which will cause the transmission times of the image frames to be unsynchronized, and thus the upper layer cannot simultaneously receive the images captured by the first camera and the second camera for subsequent processing (such as depth measurement), resulting in low accuracy of the subsequent processing. The embodiment of the present application synchronizes the frame end times of the cameras, so that the transmission times of the image frames are synchronized, and thus the upper layer can simultaneously receive the images captured by the first camera and the second camera for subsequent processing (such as depth measurement), improving the accuracy of the subsequent processing.

[0033] Thus, the embodiment of the present application synchronizes the first camera and the second camera at the software level by adjusting the frame length of the first camera and / or the second camera, thereby reducing the implementation cost of multi-camera synchronization. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0035] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 is a flowchart of the camera synchronization method of this application in the first embodiment;

[0037] Figure 2 is a simplified flowchart of an embodiment of the camera synchronization method of this application;

[0038] Figure 3 is a schematic diagram of camera timestamps involved in an embodiment of the camera synchronization method of this application;

[0039] Figure 4 is a schematic diagram of the timestamp before camera synchronization in an embodiment of the camera synchronization method of this application;

[0040] Figure 5 is a schematic diagram of the camera timestamp after synchronizing the camera in Figure 4 using the camera synchronization method of this application;

[0041] Figure 6 is a schematic diagram of the device structure of the hardware operating environment involved in the camera synchronization method in this application embodiment.

[0042] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0043] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Triangulation is a measurement technique based on the principle of triangulation, which is used to determine the distance between an object and a measurement point. This method is commonly used in fields such as robot navigation, autonomous vehicles, XR, etc. Triangulation determines the distance of a target point by measuring the angle from two different positions to the target point. In SLAM, triangulation can be used to estimate the depth of feature points. For example, if there are two cameras on the same plane and the relative position between them is known, the depth of a feature point can be calculated by measuring the angle from each camera to the same feature point.

[0045] In the object ranging scene of the XR device, the distance of the object is usually measured by using a binocular 6DOF (Six Degrees of Freedom) camera, but because the focal length of the 6DOF camera is short, the field of view is large, and the distortion is large, the distance of the object at a long distance is not accurate; the distance can also be measured by using a monocular RGB (Red Green Blue) camera, which needs to estimate the camera motion between two frames (requires a translational motion), and then obtains the depth information of the object by triangulation, but the scale ambiguity and distance measurement inaccuracy problems are caused by the scale uncertainty of monocular vision.

[0046] There are algorithms that combine 6DOF cameras and RGB cameras to solve the problem of inaccurate distance measurement. Such algorithms use 6DOF camera image frames to estimate camera motion, and then use RGB camera image frames combined with 6DOF camera motion to obtain accurate distance by triangulation, but require accurate synchronization of RGB camera image frames and 6DOF camera image frames, usually requiring the timestamp error of the image frames to be less than 2ms.

[0047] Based on this, the main solution of the present application is: obtaining a first frame end time and a second frame end time, determining an absolute value of a time difference between the first frame end time and the second frame end time, wherein the first frame end time indicates a frame end time of a first camera collecting an image frame, the second frame end time indicates a frame end time of a second camera collecting an image frame, and the first camera and the second camera are cameras to be synchronized; if the absolute value of the time difference is greater than a preset threshold, adjusting the frame length of the first camera and / or the second camera, and returning to execute the step of obtaining the first frame end time and the second frame end time; if the absolute value of the time difference is less than or equal to the preset threshold, determining that the first camera and the second camera complete camera synchronization.

[0048] The present application adjusts the frame length of the first camera and / or the second camera to realize the synchronization of the first camera and the second camera at the software level, thereby reducing the implementation cost of multi-camera synchronization.

[0049] It should be noted that the execution subject of each embodiment of the camera synchronization method of the present application can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an XR device capable of realizing the above functions. The description and explanation of each embodiment of the camera synchronization method of the present application are performed with the XR device as the execution subject.

[0050] Based on this, the first embodiment of the camera synchronization method of the present application is proposed as follows. Referring to FIG. 1, FIG. 1 is a flowchart of the first embodiment of the camera synchronization method of the present application.

[0051] In this embodiment, the camera synchronization method comprises steps S10-S30:

[0052] In step S10, the first frame end time and the second frame end time are obtained, and the absolute value of the time difference between the first frame end time and the second frame end time is determined, wherein the first frame end time indicates the frame end time of the image frame collected by the first camera, the second frame end time indicates the frame end time of the image frame collected by the second camera, and the first camera and the second camera are cameras to be synchronized.

[0053] The first camera and the second camera are cameras to be synchronized. In a preferred embodiment, the camera types of the first camera and the second camera are different, for example, the first camera is an A camera, and the second camera is a camera other than A. In a specific embodiment, the first camera is an RGB camera, and the second camera is a 6DOF camera. Exemplarily, the description of each embodiment of the camera synchronization method of the present application is performed with the camera types of the first camera and the second camera being different.

[0054] Further, the first camera can be one or more, and the present embodiment does not limit this. When there are multiple first cameras, each first camera can be synchronized first. Specifically, the synchronization between cameras of the same type can be completed based on the existing method, or each first camera can be synchronized based on the method of the present embodiment. Specifically, one first camera can be selected as the alignment reference, and then other first cameras are sequentially synchronized with the first camera serving as the alignment reference until all first cameras are synchronized.

[0055] Similarly, the second camera can also be one or more, and the present embodiment does not limit this. When there are multiple second cameras, each second camera can be synchronized first.

[0056] After the plurality of first cameras are present and are synchronized with each other, and after the plurality of second cameras are present and are synchronized with each other, one first camera and one second camera are selected to be synchronized by the embodiment, and in the alignment of the two first cameras and the second camera, the same type of cameras are also synchronized, so that after the selected first camera and the second camera are synchronized, all the first cameras and all the second cameras are synchronized.

[0057] Exemplarily, assuming that the first camera is an RGB camera and there is only one, and the second camera is a 6DOF camera and there are two, which are 6DOF_Left camera and 6DOF_Right camera, then the 6DOF_Left camera and the 6DOF_Right camera can be synchronized first, and after the 6DOF_Left camera and the 6DOF_Right camera are synchronized, the RGB camera and the 6DOF_Left (6DOF_Right) camera can be synchronized, and in the synchronization process of the RGB camera and the 6DOF_Left (6DOF_Right) camera, the 6DOF_Right (6DOF_Left) camera is synchronized with the 6DOF_Left (6DOF_Right) camera, so that after the RGB camera and the 6DOF_Left (6DOF_Right) camera are synchronized, the RGB camera, the 6DOF_Left camera and the 6DOF_Right camera are synchronized.

[0058] It can be understood that the acquisition process of the camera image frame can refer to FIG. 3, the camera records light information from the exposure start time, prepares to acquire the image frame, the frame start time, the camera starts to acquire the current image frame, the frame end time, the camera completes the acquisition of the current image frame, waits for a field blanking time, and enters the exposure start time of the next image frame, so as to repeat the acquisition of the image frame. The frame length of a frame of image is the length of time between the exposure start time of the current image frame and the exposure start time of the next image frame, and the frame length is the length of time between the frame start time of the current image frame and the exposure start time of the next image frame. The field blanking time is used for processing data (such as transmitting the acquired image frame), preparing the next image frame exposure, etc. Thus, the EOF interrupt trigger time, that is, the EOF time, can be stored in the process of acquiring the image frame by the camera, so that the frame end time of the camera can be obtained from the storage position of the EOF time.

[0059] In a specific embodiment, assuming that the first camera is an RGB camera and the second camera is a 6DOF camera, in the CamSensorNode processing flow of the Camera HAL layer (Camera Hardware Abstraction Layer), it can be determined according to the CameraID attribute whether the current camera is an RGB camera or a 6DOF camera, and the EOF interrupt (frame end interrupt) trigger time is saved into the rgbEofTime variable in the processing flow of the RGB camera; the 6dof eof interrupt (frame end interrupt) trigger time is saved into the 6dofEofTime variable in the processing flow of the 6DOF camera, so that the first frame end time is obtained from the rgbEofTime variable and the second frame end time is obtained from the 6dofEofTime when camera synchronization is performed.

[0060] It should be noted that when the frame end time (including the first frame end time and the second frame end time) is initially obtained, the frame end time is specifically the frame end time of the i-th frame image collected by the first camera and the second camera. For example, referring to FIG. 3, the first camera is an RGB camera and the second camera is a 6DOF camera, the first frame end time can be the EOF time of the first frame image collected by the RGB camera, and the second frame end time can be the EOF time of the first frame image collected by the 6DOF camera; the first frame end time can be the EOF time of the second frame image collected by the RGB camera, and the second frame end time can be the EOF time of the second frame image collected by the 6DOF camera. It can be understood that the camera periodically collects images, and for different values of i, the time difference between the frame end times of the images collected by the first camera and the second camera is also constant, so that i is any integer greater than zero.

[0061] In step S20, if the absolute value of the time difference is greater than a preset threshold, the frame length of the first camera and / or the second camera is adjusted, and the step of obtaining the first frame end time and the second frame end time is returned to be executed.

[0062] The preset threshold can be any value set according to actual needs, such as 2 milliseconds, 5 milliseconds, etc., and the present embodiment does not make a specific limitation thereto.

[0063] It can be understood that referring to FIG. 3, the frame length is the time length between the frame start time of the current image frame and the exposure start time of the next image frame. By adjusting the frame length of the camera, the frame end time of the next image frame collected by the camera is changed, that is, the frame end time of the camera is indirectly adjusted by the frame length.

[0064] It should be noted that after the frame length of the first camera and / or the second camera is adjusted, the camera with the adjusted frame length acquires the next frame of image based on the adjusted frame length, and the camera without the adjusted frame length acquires the next frame of image based on the current frame length, so as to reacquire the frame end time of the first camera acquiring the next frame of image and the frame end time of the second camera acquiring the next frame of image, that is, reacquire the first frame end time and the second frame end time. It is determined whether the absolute value of the time difference is less than or equal to the preset threshold based on the reacquired first frame end time and second frame end time. If the absolute value of the time difference is greater than the preset threshold, the frame length of the first camera and / or the second camera is adjusted again until the preset end condition is met, and the camera synchronization process is ended.

[0065] The preset end condition can be whether the absolute value of the time difference is less than or equal to the preset threshold, or whether the cumulative adjustment times are greater than a preset number threshold. The embodiment does not make a specific limitation on this. It can be understood that the adjustment times are increased by one each time the frame length of the first camera and / or the second camera is adjusted, and the initial value of the adjustment times is zero, so that the cumulative value of the adjustment times, that is, the cumulative adjustment times, can be obtained after the adjustment of the frame length of the camera is completed.

[0066] It should be noted that in the underlying implementation logic, the frame length of the camera can be adjusted in the CamSensorNode processing flow of the Camera HAL layer. Specifically, the frame length of the first camera is adjusted in the CamSensorNode processing flow of the first camera, and the frame length of the second camera is adjusted in the CamSensorNode processing flow of the second camera.

[0067] Further, in a preferred embodiment, the frame length of the first camera is adjusted or the frame length of the second camera is adjusted, so that only the frame length of one camera needs to be adjusted, the number of cameras that need to be adjusted is reduced, and the synchronization efficiency of the camera can be improved.

[0068] In step S30, if the absolute value of the time difference is less than or equal to the preset threshold, it is determined that the first camera and the second camera complete camera synchronization.

[0069] If the absolute value of the time difference between the first frame end time and the second frame end time is less than or equal to the preset threshold, it is determined that the first camera and the second camera complete camera synchronization.

[0070] In addition, after the first camera and the second camera complete camera synchronization, image frames can be collected based on the synchronized first camera and the second camera, and subsequent processing can be performed based on the image frames collected by the first camera and the second camera, such as triangulation based on the image frames collected by the first camera and the second camera by using a SLAM algorithm or a 6DOF algorithm. In a specific embodiment, the first camera and the second camera can transmit the collected image frames to a Camera Service component, and perform triangulation by using a SLAM algorithm or a 6DOF algorithm provided by the Camera Service, thereby improving the ranging accuracy of the SLAM algorithm or the 6DOF algorithm.

[0071] The embodiment determines a time difference absolute value between a first frame end time and a second frame end time by obtaining the first frame end time and the second frame end time, where the first frame end time indicates a frame end time at which the first camera collects an image frame, and the second frame end time indicates a frame end time at which the second camera collects an image frame. The first camera and the second camera are cameras to be synchronized, so that the frame end (EOF) time of the cameras to be synchronized (including the first camera and the second camera) is obtained for subsequent synchronization, and the specific camera types of the first camera and the second camera are not limited. Any type of camera can be synchronized, and in particular, different types of first camera and second camera can also be synchronized, thereby improving the universality of the synchronization method.

[0072] If the time difference absolute value is greater than a preset threshold, the frame length of the first camera and / or the second camera is adjusted, and the step of obtaining the first frame end time and the second frame end time is performed again. When the time difference absolute value between the first frame end time and the second frame end time is greater than the preset threshold, the first camera and the second camera are synchronized by adjusting the frame length (frame_length) of the first camera and / or the second camera. In this way, the first camera and the second camera are synchronized at the software level without adding additional hardware structures, thereby reducing the implementation cost of multi-camera synchronization.

[0073] If the absolute value of the time difference is less than or equal to the preset threshold, it is determined that the first camera and the second camera complete camera synchronization. When the absolute value of the time difference between the first frame end time and the second frame end time is less than or equal to the preset threshold, it is determined that the first camera and the second camera complete camera synchronization, that is, the frame end times of the synchronized cameras are synchronized, but not the start of frame (SOF) times or exposure start (Expstart) times of the synchronized cameras. Since the cameras will transmit the collected image frames after the frame end time, for different types of cameras, the start of frame times or the exposure start times are synchronized, but the frame end times are not necessarily synchronized, so if the start of frame times or the exposure start times of the synchronized cameras are synchronized, the frame end times cannot be guaranteed to be synchronized, which will cause the transmission times of the image frames to be unsynchronized, and thus the upper layer cannot simultaneously receive the images collected by the first camera and the second camera for subsequent processing (such as depth measurement), resulting in low accuracy of subsequent processing. The embodiment synchronizes the frame end times of the cameras, so that the transmission times of the image frames are synchronized, and thus the upper layer can simultaneously receive the images collected by the first camera and the second camera for subsequent processing (such as depth measurement), thereby improving the accuracy of subsequent processing.

[0074] Therefore, the embodiment adjusts the frame length of the first camera and / or the second camera to achieve synchronization of the first camera and the second camera at the software level, thereby reducing the implementation cost of multi-camera synchronization.

[0075] To help understand the technical concept and technical principle of the camera synchronization method of the embodiment, a specific example of a camera synchronization method is proposed. In this specific example, the XR device includes an RGB camera, a 6DOF camera, and an ISP (Image Signal Processor), and the RGB camera and the 6DOF camera are connected to the ISP. The specific steps are as follows:

[0076] 1. The RGB camera and the 6DOF camera transmit the collected data to the ISP through MIPI (Mobile Industry Processor Interface), and the ISP sends control signals to control the operation of the RGB camera and the 6DOF camera through CSI (Camera Sensor Interface).

[0077] 2. The ISP performs IFE (Image Front-End) processing on the received data, and outputs YUV (luminance, red chroma, green chroma) format image data to the Camera HAL layer. Specifically, the IFE processing includes but is not limited to color space conversion, and can also include analog-to-digital conversion, noise removal from the image, color space conversion, white balance adjustment, image compression, etc.

[0078] 3. The Camera HAL layer implements synchronization of the RGB camera and the 6DOF camera in the CamSensorNode processing flow.

[0079] 4. After the RGB camera and the 6DOF camera are synchronized, the collected data is transmitted to the Camera HAL layer, and the Camera HAL layer transmits the data to the Camera Service component, and performs triangulation through the SLAM algorithm or the 6DOF algorithm provided by the Camera Service.

[0080] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the camera synchronization method of the present application. More forms of simple transformation based on this technical concept are within the protection scope of the present application.

[0081] Based on the first embodiment, a second embodiment of the camera synchronization method of the present application is proposed. In this embodiment, the same or similar contents as the first embodiment can be referred to the above description, and will not be repeated hereinafter. In this embodiment, the step of adjusting the frame length of the first camera and / or the second camera includes:

[0082] Step A10, determining an adjustment step based on the absolute value of the time difference, wherein the adjustment step is positively correlated with the absolute value of the time difference;

[0083] The adjustment step is positively correlated with the absolute value of the time difference. Thus, the greater the absolute value of the time difference, the greater the adjustment step, so as to adjust the frame length of the camera with a larger adjustment step, thereby effectively shortening the time difference between the end time of the first frame and the end time of the second frame.

[0084] Further, for the convenience of subsequent description and illustration, let the time difference be timeDelta, the absolute value of the time difference be |timeDelta|, and the adjustment step be adjust_frame_length.

[0085] Step A20, if the first frame end time is less than the second frame end time, adjusting the frame length of the first camera and / or the second camera based on the adjustment step length, wherein the adjusted frame length of the first camera is greater than the unadjusted frame length of the first camera, and the adjusted frame length of the second camera is less than the unadjusted frame length of the second camera.

[0086] If the first frame end time is less than the second frame end time, the frame length of the first camera and / or the second camera is adjusted based on the adjustment step length. Specifically, if only the frame length of the first camera is adjusted and the adjustment step length is greater than zero, the current frame length of the first camera is denoted as frame_length1_origin, then the frame length of the first camera can be adjusted as the adjustment step length plus the current frame length, i.e., the adjusted frame length is denoted as frame_length1_new, then frame_length1_new = frame_length1_origin + adjust_frame_length; if only the frame length of the second camera is adjusted and the adjustment step length is greater than zero, the current frame length of the second camera is denoted as frame_length2_origin, then the frame length of the second camera can be adjusted as the adjustment step length minus the current frame length, i.e., the adjusted frame length is denoted as frame_length2_new, then frame_length2_new = frame_length2_origin - adjust_frame_length; if the frame length of the first camera and the second camera is adjusted and the adjustment step length is greater than zero, then the frame length of the first camera can be adjusted as one half of the adjustment step length plus the current frame length, and the frame length of the second camera can be adjusted as one half of the adjustment step length minus the current frame length, i.e., frame_length1_new = frame_length1_origin + 1 / 2 adjust_frame_length, frame_length2_new = frame_length2_origin - 1 / 2 adjust_frame_length.

[0087] Step A30, if the first frame end time is greater than the second frame end time, adjusting the frame length of the first camera and / or the second camera based on the adjustment step length, wherein the adjusted frame length of the first camera is less than the unadjusted frame length of the first camera, and the adjusted frame length of the second camera is greater than the unadjusted frame length of the second camera.

[0088] If the second frame end time is greater than the first frame end time, adjust the frame length of the first camera and / or the second camera based on the adjustment step. Specifically, if only the frame length of the first camera is adjusted and the adjustment step is greater than zero, the frame length of the first camera can be adjusted as the adjustment step minus the current frame length, i.e., frame_length1_new = frame_length1_origin - adjust_frame_length; if only the frame length of the second camera is adjusted and the adjustment step is greater than zero, the frame length of the second camera can be adjusted as the adjustment step plus the current frame length, i.e., frame_length2_new = frame_length2_origin + adjust_frame_length; if the frame length of the first camera and the second camera are adjusted and the adjustment step is greater than zero, the frame length of the first camera can be adjusted as one-half of the adjustment step minus the current frame length, and the frame length of the second camera can be adjusted as one-half of the adjustment step plus the current frame length, i.e., frame_length1_new = frame_length1_origin - 1 / 2 adjust_frame_length, frame_length2_new = frame_length2_origin + 1 / 2 adjust_frame_length.

[0089] In the embodiment, the adjustment step is determined based on the absolute value of the time difference, and the frame length of the first camera and / or the second camera is adjusted based on the adjustment step. Specifically, for the camera with the earlier frame end time, the frame length of the camera is increased so that the frame end time of the camera is delayed, and for the camera with the later frame end time, the frame length of the camera is shortened so that the frame end time of the camera is advanced, thereby reducing the absolute value of the time difference and ensuring effective and fast alignment of the frame end times of the first camera and the second camera.

[0090] In a possible implementation, the frame length includes a field blanking time, and the step of adjusting the frame length of the first camera and / or the second camera based on the adjustment step includes:

[0091] Step B10, adjusting the field blanking time of the first camera and / or the second camera based on the adjustment step.

[0092] It can be understood that, referring to FIG. 3, the frame length includes the time between the frame start time and the frame end time and the field blanking time, and it can be understood that the frame start time and the frame end time of the camera are usually fixed and cannot be adjusted. Therefore, the field blanking time is adjusted in the embodiment, and the field blanking time is increased or decreased based on the adjustment step to achieve the purpose of adjusting the frame length.

[0093] In a possible implementation, the step of determining the adjustment step length based on the absolute value of the time difference comprises:

[0094] The step C10 comprises taking the ratio between the absolute value of the time difference and the preset exposure line time as the adjustment step length.

[0095] The preset exposure line time is denoted as line time, and line_time = line_length / pclk, where line_length is the length of a line, i.e., the number of pixel points included in a line of image, and line_length is N in an example if the resolution of the image captured by the camera is M*N, and pclk is the output clock of the pixel.

[0096] The adjustment step length can be specifically the ratio between the absolute value of the time difference and the preset exposure line time, which can be expressed by a formula as follows: adjust_frame_length = |timeDelta| / line_time.

[0097] It should be noted that if the resolutions of the images captured by the first camera and the second camera are different, the adjustment step lengths corresponding to the cameras are respectively calculated according to the resolutions of the cameras, that is, the preset exposure line times corresponding to the first camera and the second camera can be different, and when the frame length of the camera needs to be adjusted subsequently, the frame length of the camera is adjusted according to the adjustment step length corresponding to the camera. For example, it is assumed that the preset exposure line time corresponding to the first camera is line time 1, and the preset exposure line time corresponding to the second camera is line time 2, the adjustment step length for the first camera is adjust_frame_length1 = |timeDelta| / line_time1, the adjustment step length for the second camera is adjust_frame_length2 = |timeDelta| / line_time1, the frame length of the first camera is adjusted based on adjust_frame_length1, and the frame length of the second camera is adjusted based on adjust_frame_length1.

[0098] In this embodiment, the ratio between the absolute value of the time difference and the preset exposure line time is taken as the adjustment step length, the calculation of the adjustment step length is simple, the calculation amount is small, and the complexity of camera synchronization is reduced.

[0099] Based on the first and / or second embodiments, a third embodiment of the camera synchronization method is provided. In this embodiment, the same or similar contents as those in the first and second embodiments can be referred to the foregoing description, and will not be described hereinafter. The step of adjusting the frame length of the first camera and / or the second camera comprises:

[0100] Step D10, if the first camera is a secondary camera, adjusting the frame length of the first camera;

[0101] Step D20, if the second camera is a secondary camera, adjusting the frame length of the second camera.

[0102] It can be understood that the relevant personnel can set the camera roles of the cameras in the XR device in advance, including the primary camera and the secondary camera. The primary camera is the core of the camera system, providing the best shooting experience and image quality, while the secondary camera is used to supplement the functions of the primary camera, providing more shooting options and creative space.

[0103] Considering that the secondary camera is used to supplement the functions of the primary camera, the frame length of the secondary camera is adjusted without adjusting the frame length of the primary camera, so that the primary camera can work normally, and the problem of affecting the main functions of the XR device during or after the adjustment of the frame length of the primary camera can be avoided, thereby ensuring that the main functions of the XR device are not affected during the synchronization of the cameras, and improving the user experience.

[0104] In a possible implementation, after the step of determining that the first camera and the second camera complete camera synchronization, the method further includes:

[0105] Step E10, if the frame length of the first camera is adjusted, obtaining a first initial frame length of the first camera, and controlling the first camera to perform image acquisition based on the first initial frame length, wherein the first initial frame length is the frame length of the image frame acquired by the first camera before the step length is adjusted;

[0106] Step E20, if the frame length of the second camera is adjusted, obtaining a second initial frame length of the second camera, and controlling the second camera to perform image acquisition based on the second initial frame length, wherein the second initial frame length is the frame length of the image frame acquired by the second camera before the step length is adjusted.

[0107] It should be noted that the initial frame length (including the first initial frame length and the second initial frame length) is the frame length before the step length is adjusted. Specifically, the frame lengths of the first camera and the second camera can be obtained before the first camera and the second camera are synchronized, and the frame lengths are stored as the initial frame lengths of the cameras, so that the first initial frame length and the second initial frame length can be obtained from the storage position of the initial frame length after the first camera and the second camera are synchronized.

[0108] The camera performs image acquisition at the initial frame length, that is, the frame rate of image acquisition of the camera also returns to the frame rate before synchronization, so that the image acquisition frame rate of the camera remains unchanged, and the camera performs image acquisition at the initially set frame rate, so that the image acquisition frame rate meets the requirements.

[0109] To help understand the above embodiments of the camera synchronization method of the present application, an example combining the camera synchronization methods in the above embodiments is proposed. Referring to FIGS. 4-5, the first camera is an RGB camera and the second camera is a 6DOF camera. FIG. 4 shows the timestamps before the RGB camera and the 6DOF camera are synchronized, and FIG. 5 shows the timestamps after the RGB camera and the 6DOF camera are synchronized. The unit of the timestamps in FIGS. 4-5 is millisecond. The frame rate of the cameras is 30 fps (frames per second) in FIGS. 4-5, so the frame time of each image is 33 milliseconds. To make the images clearer and more intelligible, the exposure start time is marked as ES, the frame start time is marked as S, the frame end time is marked as E, and the blanking time is marked as BT in FIGS. 4-5. The time between the exposure start time and the frame end time of the RGB camera is the exposure time, the time between the exposure start time and the frame start time of the 6DOF camera is the exposure time, and the exposure time is denoted as ET. Based on this, the specific steps of camera synchronization are as follows:

[0110] The cameras on the XR device are 6DOF_Left, 6DOF_Right, and RGB camera, respectively. The RGB camera is the main camera, and the 6DOF_Left and 6DOF_Right are auxiliary cameras. The 6DOF_Left and 6DOF_Right have been synchronized.

[0111] Referring to FIG. 4, the 6DOF camera is an auxiliary camera. The frame length of the 6DOF camera is adjusted, specifically, the blanking time of the 6DOF camera is adjusted to align the frame end time of the RGB camera and the 6DOF camera. In the CamSensorNode processing flow of the Camera HAL layer, it is determined whether the current camera is an RGB camera or a 6DOF camera according to the CameraID attribute. In the processing flow of the RGB camera, the EOF interrupt (frame end interrupt) trigger time is saved to the rgbEofTime variable. In the processing flow of the 6DOF camera, the rgbEofTime of the RGB camera is obtained, and the EOF interrupt (frame end interrupt) trigger time of the 6DOF is saved to the 6dofEofTime variable. The time difference timeDelta of 6dofEofTime-rgbEofTime is calculated. If the absolute value of timeDelta is greater than a preset threshold (the preset threshold is adjusted according to the actual frame rate. For example, the preset threshold is set to 2 ms for 30 fps), the synchronization processing is performed. The absolute value of timeDelta is converted to adjust_frame_length using the following formula, where linetime is the preset exposure line time corresponding to the 6DOF camera.

[0112] adjust_frame_length = |timeDelta| / linetime

[0113] Then the frame_length of the 6DOF camera is adjusted according to the following formula, and the adjusted frame_length_new is written into the 6DOF sensor register.

[0114] When the absolute value of timeDelta corresponding to the next frame image is less than a preset threshold, the frame_length_origin of the 6DOF camera before the frame length is adjusted is written into the 6DOF sensor register, otherwise the frame_length of the 6DOF camera is continuously adjusted; After the above operation, the RGB camera and the 6DOF camera are soft-synchronized, that is, the synchronization of the RGB camera and the 6DOF camera is realized at the software level.

[0115] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the camera synchronization method of the present application. More forms of simple transformation based on this technical concept are within the protection scope of the present application.

[0116] The embodiment of the present application provides an extended reality device, which comprises a first camera, a second camera and a processor, wherein the first camera and the second camera are respectively connected with the processor, and the processor is used to execute the steps of the above-mentioned camera synchronization method.

[0117] The first camera includes but is not limited to an RGB camera, and the second camera includes but is not limited to a 6DOF camera.

[0118] Further, as shown in FIG. 6, the extended reality device can also include a processing apparatus 1001 (for example, a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read only memory (ROM) 1002 or a program loaded from a storage apparatus 1003 into a random access memory (RAM) 1004. In the RAM 1004, various programs and data required for the operation of the extended reality device are also stored. The processing apparatus 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input apparatus 1007 including, for example, a touch screen, a touch pad, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output apparatus 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage apparatus 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication apparatus 1009. The communication apparatus 1009 can allow the extended reality device to communicate wirelessly or by wire with other devices to exchange data. Although the extended reality device having various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0119] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication apparatus, or installed from the storage apparatus 1003, or installed from the ROM 1002. When the computer program is executed by the processing apparatus 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are executed.

[0120] Compared with the prior art, the extended reality device provided by the embodiments of the present application has the same beneficial effects as the camera synchronization method provided by the above-mentioned embodiments, and other technical features in the extended reality device are the same as those disclosed in the previous embodiment method, which will not be repeated here.

[0121] It should be understood that the parts disclosed in the embodiments of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0122] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0123] The embodiment of the present application provides a computer readable storage medium having computer readable program instructions (i.e. computer programs) stored thereon, the computer readable program instructions being used to execute the camera synchronization method in the above embodiment.

[0124] The computer readable storage medium provided by the embodiment of the present application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, system or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiment, the computer readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer readable storage medium can be transmitted by any appropriate medium, including but not limited to: electric wire, optical cable, RF (Radio Frequency: radio frequency), etc., or any suitable combination of the above.

[0125] The above computer readable storage medium can be contained in an extended reality device; or can exist separately without being assembled into an extended reality device.

[0126] The above computer readable storage medium carries one or more programs, when the one or more programs are executed by the extended reality device, the extended reality device executes the above functions defined in the method of the embodiments disclosed by the present application.

[0127] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0128] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a procedure, or a part of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or in the reverse order, depending on the functionality involved. It is also noted that each block of the block diagrams and / or flow diagrams and combinations of blocks in the block diagrams and / or flow diagrams can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or by combinations of dedicated hardware and computer instructions.

[0129] The modules involved in the embodiments of the present application can be implemented in software or hardware. In some cases, the names of the modules do not constitute a limitation on the modules themselves.

[0130] The readable storage medium provided by the embodiments of the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer programs) for executing the above-mentioned camera synchronization method. Compared with the prior art, the computer readable storage medium provided by the embodiments of the present application has the same beneficial effects as the camera synchronization method provided by the above-mentioned embodiments, which will not be described here.

[0131] The embodiment of the present application further provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the camera synchronization method as described above.

[0132] Compared with the prior art, the computer program product provided by the embodiment of the present application has the same beneficial effects as the camera synchronization method provided by the above-mentioned embodiment, which will not be repeated here.

[0133] The above only describes some embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structural transformation, direct / indirect application in other related technical fields, or the like made by using the content of the present application specification and drawings is included in the patent protection scope of the present application.

Claims

1. A camera synchronization method, characterized by, The camera synchronization method comprises the following steps: obtaining a first frame end time and a second frame end time, determining an absolute value of a time difference between the first frame end time and the second frame end time, wherein the first frame end time indicates a frame end time of an image frame captured by a first camera, the second frame end time indicates a frame end time of an image frame captured by a second camera, and the first camera and the second camera are cameras to be synchronized; if the absolute value of the time difference is greater than a preset threshold, adjusting a frame length of the first camera and / or the second camera, and returning to the step of obtaining the first frame end time and the second frame end time; if the absolute value of the time difference is less than or equal to the preset threshold, determining that the first camera and the second camera complete camera synchronization.

2. The method of claim 1, wherein, The step of adjusting the frame length of the first camera and / or the second camera comprises: determining an adjustment step based on the absolute value of the time difference, wherein the adjustment step is positively correlated with the absolute value of the time difference; if the first frame end time is less than the second frame end time, adjusting the frame length of the first camera and / or the second camera based on the adjustment step, wherein the adjusted frame length of the first camera is greater than the unadjusted frame length of the first camera, and the adjusted frame length of the second camera is less than the unadjusted frame length of the second camera; if the first frame end time is greater than the second frame end time, adjusting the frame length of the first camera and / or the second camera based on the adjustment step, wherein the adjusted frame length of the first camera is less than the unadjusted frame length of the first camera, and the adjusted frame length of the second camera is greater than the unadjusted frame length of the second camera.

3. The method of claim 2, wherein, The frame length comprises a field blanking time, and the step of adjusting the frame length of the first camera and / or the second camera based on the adjustment step comprises: adjusting the field blanking time of the first camera and / or the second camera based on the adjustment step.

4. The method of claim 2, wherein, The step of determining the adjustment step based on the absolute value of the time difference comprises: taking the ratio between the absolute value of the time difference and a preset exposure line time as the adjustment step.

5. The method of claim 1, wherein, The step of adjusting the frame length of the first camera and / or the second camera comprises: if the first camera is a secondary camera, adjusting the frame length of the first camera; if the second camera is a secondary camera, adjusting the frame length of the second camera.

6. The method according to any one of claims 1 to 5, characterized in that, After the step of determining that the first camera and the second camera complete camera synchronization, the method further comprises: if the frame length of the first camera is adjusted, obtaining a first initial frame length of the first camera, and controlling the first camera to capture images based on the first initial frame length, wherein the first initial frame length is the frame length of the image frame captured by the first camera before the adjustment step; if the frame length of the second camera is adjusted, obtaining a second initial frame length of the second camera, and controlling the second camera to capture images based on the second initial frame length, wherein the second initial frame length is the frame length of the image frame captured by the second camera before the adjustment step.

7. The method according to any one of claims 1 to 5, wherein The first camera is an RGB camera, and the second camera is a 6DOF camera.

8. An extended reality device, comprising: The device comprises a first camera, a second camera and a processor connected to the first camera and the second camera respectively, and the processor is configured to perform the steps of the camera synchronization method according to any one of claims 1 to 7.

9. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the camera synchronization method according to any one of claims 1 to 7.

10. A computer program product, characterised in that, The computer program product comprises a computer program, and the computer program is executed by the processor to implement the steps of the camera synchronization method according to any one of claims 1 to 7.

Images