Systems, methods, processing modules, and storage media for image capture

Abstract

The present disclosure relates to a system, a method, a processing module and a storage medium for image capturing. The system comprises a plurality of image capturing units. Each image capturing unit comprises a camera module and a processing module. The camera module is configured to capture an image of a target object and send a current exposure time at which the image is captured to the processing module. The processing module is configured to determine a current frame frequency according to a time difference between the current target exposure time and the acquired current exposure time, and send the current frame frequency to the camera module, so that the camera module captures an image of the target object at a next exposure time obtained based on the current frame frequency and the current exposure time, wherein a time difference between the next target exposure time and the next exposure time gradually converges within a predetermined range. Through the mutual influence between the frame frequency and the exposure time, high-precision synchronous exposure can be achieved while reducing the cost.

Description

[0001] This application is a divisional application of Chinese patent application No. 202510555455.8, filed on April 29, 2025, entitled "System, Method, Processing Module and Storage Medium for Image Capture". Technical Field

[0002] This disclosure relates to the field of imaging, and more specifically to systems, methods, processing modules, and storage media for image capture in the field of imaging. Background Technology

[0003] In an increasing number of fields such as reverse engineering, medical imaging, cultural heritage preservation, manufacturing, and gaming, there is a need to perform three-dimensional (3D) scanning of target objects to capture their spatial form and detailed information.

[0004] In traditional exposure synchronization methods, an exposure command is typically sent to the camera, which then performs the exposure based on the command. The transmission delay of the exposure command is not only uncontrollable but also affects exposure synchronization, causing synchronization accuracy to exceed, for example, 50 milliseconds.

[0005] A combination of one or more cameras and a turntable is typically used for high-precision data acquisition of stationary objects. Alternatively, an array of independent cameras can be used for this purpose. However, because independent camera arrays usually lack synchronization triggering devices, or if they do have synchronization mechanisms, the synchronization accuracy is greater than 0.1 seconds, they cannot capture images of moving people or other objects. For example, when capturing a hand making a gesture, the low synchronization accuracy makes it difficult for the different cameras in an independent camera array to synchronize their exposures, resulting in images of the gesture taken from different angles at different timings. These gesture images are clearly distinct, making 3D modeling difficult.

[0006] In addition, multiple industrial cameras with built-in synchronization triggers are typically used for high-precision data acquisition of moving objects. During synchronized exposure, exposure can be triggered by external control or via network protocols. Alternatively, multiple SLR cameras or Genlock cameras with built-in synchronization triggers can be used for high-precision data acquisition. During synchronized exposure, exposure can be triggered by hardware or via the Genlock protocol. However, to achieve satisfactory exposure synchronization accuracy, these devices require highly precise components, resulting in high costs and hindering widespread adoption, thus impeding the development of 3D scanning.

[0007] Therefore, it is desirable to provide a way to achieve high-precision synchronous exposure while reducing costs. Summary of the Invention

[0008] One aspect of this disclosure relates to a system for image capture. According to embodiments of this disclosure, the system may include multiple image capture units. Each image capture unit may include a camera module and a processing module. The camera module may be configured to capture an image of a target object and send the current exposure time at the time of image capture to the processing module. The processing module may be configured to determine a current frame rate based on the time difference between the current target exposure time and the current exposure time obtained from the camera module, and send the current frame rate to the camera module, such that the camera module captures an image of the target object at the next exposure time obtained based on the current frame rate and the current exposure time, wherein the time difference between the next target exposure time and the next exposure time gradually converges to a predetermined range.

[0009] Another aspect of this disclosure relates to a method for image capture. According to embodiments of this disclosure, the method may include: acquiring the current exposure time when a camera module captures an image of a target object; determining a current frame rate based on the time difference between the current target exposure time and the current exposure time; and sending the current frame rate to the camera module such that the camera module captures an image of the target object at the next exposure time obtained based on the current frame rate and the current exposure time, wherein the time difference between the next target exposure time and the next exposure time gradually converges to a predetermined range.

[0010] Another aspect of this disclosure relates to a processing module. According to embodiments of this disclosure, the processing module may include a processor and a memory. The memory may include computer program instructions. The memory and the computer program instructions may be configured to cause the processing module to perform the methods described above via the processor.

[0011] Another aspect of this disclosure relates to a computer-readable storage medium storing one or more computer program instructions. According to embodiments of this disclosure, the one or more computer program instructions can cause the processing device to perform the methods described above when executed by a processing device.

[0012] The above overview is provided to summarize some exemplary embodiments to provide a basic understanding of the aspects of the subject matter described herein. Therefore, the features described above are merely examples and should not be construed as narrowing the scope or spirit of the subject matter described herein in any way. Other features, aspects, and advantages of the subject matter described herein will become apparent from the following detailed description, taken in conjunction with the accompanying drawings. Attached Figure Description

[0013] A better understanding of this disclosure can be obtained by considering the following detailed description of the embodiments in conjunction with the accompanying drawings. The same or similar reference numerals are used in the drawings to denote the same or similar parts. The drawings, together with the following detailed description, are incorporated in and form a part of this specification to illustrate embodiments of the disclosure and explain the principles and advantages of the disclosure. Wherein:

[0014] Figure 1 This is an example of a configuration combining a camera and a turntable in related technologies.

[0015] Figure 2A This is an example of the configuration of an independent camera array in related technologies, and Figure 2B yes Figure 2A An example of an image sequence obtained by simultaneously exposing a hand making a gesture using an array of independent cameras.

[0016] Figure 3A This is an example of a synchronized industrial camera in related technologies. Figure 3B This is an example of the configuration of a SLR camera array in related technologies. Figure 3C This is an example of a scenario where a single-lens reflex camera array is working in related technologies. Figure 3D This is an example of the configuration of a Genlock camera array in related technologies, and Figure 3D yes Figure 3C A partial view of the Genlock camera array.

[0017] Figure 4 This is an example of the configuration of a system for image capture according to embodiments of the present disclosure.

[0018] Figure 5 It is the convergence effect of the time difference between the target exposure time and the actual exposure time according to the embodiments of this disclosure.

[0019] Figure 6 This is a schematic diagram showing the exposure times of each camera module without enabling exposure synchronization control, according to an embodiment of the present disclosure.

[0020] Figure 7 This is a schematic diagram showing the exposure times of each camera module when exposure synchronization control is enabled, according to an embodiment of the present disclosure.

[0021] Figure 8 This is a schematic diagram of PID (proportional-integral-derivative) control according to an embodiment of the present disclosure.

[0022] Figure 9 This is a schematic diagram of an example structure of an image capture unit according to an embodiment of the present disclosure.

[0023] Figures 10A to 10CThese are examples of specific forms of camera modules according to embodiments of this disclosure.

[0024] Figure 11 This is a flowchart of a method for image capture according to an embodiment of the present disclosure.

[0025] Figure 12 This is a schematic diagram of the architecture of a system for image capture according to an embodiment of the present disclosure.

[0026] Figure 13 This is a block diagram of an example structure of a processing apparatus that may be used in the embodiments of this disclosure.

[0027] While the embodiments described in this disclosure may be readily modified and alternatively implemented, specific embodiments thereof are shown by way of example in the accompanying drawings and are described in detail herein. However, it should be understood that the drawings and the detailed description thereof are not intended to limit the embodiments to the specific forms disclosed, but rather are intended to cover all modifications, equivalents, and alternatives that fall within the spirit and scope of the claims. Detailed Implementation

[0028] The following description illustrates representative applications of the devices and methods described herein. These examples are provided merely to provide context and aid in understanding the described embodiments. Therefore, it will be apparent to those skilled in the art that the embodiments described below can be practiced without some or all of the specific details provided. In other instances, well-known process steps have not been described in detail to avoid unnecessarily obscuring the described embodiments. Other applications are also possible, and the scope of this disclosure is not limited to these examples.

[0029] exist Figure 1 The image shows an example configuration of a camera and turntable combination in related technologies. This combination is typically used to photograph static objects. Because... Figure 1 In the example, there is only one camera, so there is no issue with exposure synchronization. Even if there are multiple cameras, there is no issue with synchronized exposure because the objects are stationary.

[0030] exist Figure 2A The image shows an example of a configuration of a standalone camera array in related technologies. Standalone camera arrays are typically used to photograph static objects. Figure 2A Multiple cameras in a system expose for a centrally located target object, and the images captured by these exposures can be combined to create 3D images. When the subject is a moving object (e.g., a hand making a gesture), the multiple cameras need to expose synchronously to capture images of the same gesture from different angles for synthesis. However, because the synchronization accuracy of an independent camera array is greater than 0.1 seconds, it is difficult to synchronize the exposures of multiple cameras. Figure 2BThe image shows a sequence of images obtained by simultaneously exposing a hand making a gesture using multiple cameras. Each image in the sequence represents an image captured by a camera at a corresponding location. Although these cameras expose simultaneously, poor synchronization accuracy results in differences in the actual exposure time of each camera, leading to time differences in the captured images. Figure 2B As can be seen, the gestures in each image are different, making it impossible to successfully perform 3D modeling.

[0031] exist Figure 3A An example of a synchronous industrial camera in related technologies is shown. Due to their superior performance, synchronous industrial cameras are commonly used to capture moving objects for 3D scanning. However, their high cost hinders widespread adoption. Figure 3B An example of a configuration of a SLR camera array in related technologies is shown. Figure 3C The image illustrates an example of a DSLR camera array in operation. A DSLR camera array can perform 3D scanning of static or dynamic objects located at its center. Besides the high cost of each DSLR camera itself, the synchronous triggering device required to achieve synchronized exposure further increases the cost. Figure 3D The image shows an example of a configuration of a Genlock camera array in the related art. Genlock cameras are also used for filming moving objects due to their excellent performance, but are expensive due to the need for synchronized exposure.

[0032] Because the exposure synchronization accuracy of 3D scanning in related technologies is not high, or the cost of achieving successful 3D scanning is too high, there is a need for improvement in these technologies. This disclosure provides a novel system for image capture that improves synchronization accuracy while reducing costs through a novel exposure synchronization control method. Figure 4 An example configuration of a system 400 for image capture according to an embodiment of this disclosure is shown.

[0033] like Figure 4 As shown, system 400 may include multiple image capture units 410-1 to 410-3. Where it is not necessary to distinguish between different image capture units, the suffix 410 is omitted hereinafter to indicate the image capture unit. These image capture units may be arranged around the center position of the target object. They may be fixed to a support member (e.g., a pole, column, etc.) or, for example, movable around the target object. Although in Figure 4 The diagram shows three image capture units, but the number of image capture units can also be two, four, or more. An image of the target object is acquired by simultaneously exposing the target object using at least two image capture units.

[0034] Ideally, each image capture unit can continuously and synchronously expose the target object at the same predetermined frame rate while maintaining alignment at the reference time. However, in practice, even if each image capture unit has an aligned reference time and the same predetermined frame rate, over time, due to various factors such as processing deviations in clock and exposure circuits, and differences in temperature effects, the individual image capture units may not be able to maintain synchronous exposure.

[0035] exist Figure 4 In system 400, each image capture unit 410 has the same reference time via a synchronization protocol, and their initial frame rate for shooting is the same. To enable the various image capture units to perform and maintain synchronized exposure, embodiments of this disclosure provide a control method for synchronized exposure. This control method utilizes the interaction between frame rate and actual exposure time, adjusting the frame rate to change the next actual exposure time, and adjusting the current frame rate based on the actual exposure time, thereby allowing the actual exposure time to gradually track the target exposure time, achieving synchronized exposure. The specific implementation of synchronized exposure will be described below.

[0036] like Figure 4 As shown, each image capture unit may include a camera module and a processing module. For example, image capture unit 410-1 may include camera module 420-1 and processing module 430-1, image capture unit 410-2 may include camera module 420-2 and processing module 430-2, and image capture unit 410-3 may include camera module 420-3 and processing module 430-3. Where it is not necessary to distinguish between camera modules and processing modules of different image capture units, the suffix is ​​omitted below, and 420 is used to indicate the camera module and 430 to indicate the processing module. Camera modules 420 and processing modules 430 included in the same image capture unit 410 are associated with each other. Camera module 420 can send its exposure time to the associated processing module 430, and processing module 430 can send the frame rate to the associated camera module 420. In addition to transmitting exposure time and frame rate, the camera module 420 can also send the captured images and imaging information to the processing module 430 according to actual needs. The processing module 430 can also send control information such as switch and imaging parameters to the camera module 420.

[0037] To achieve synchronized exposure, each processing module 430 in the image capture unit can have the same time base. For example, synchronization protocols such as NTP (Network Time Protocol), SNTP (Simple Network Time Protocol), and PTP (Precise Time Protocol) can be used to align the clocks of each processing module 430. Thus, when the same exposure time base and the same frame rate (frames per second, in fps) are set for each processing module 430, each processing module 430 can determine the same target exposure time. For example, when the time base is 0ms and the frame rate is 100fps, each processing module synchronized by the synchronization protocol clock determines the target exposure time as 0ms, 10ms, 20ms, 30ms, etc.

[0038] For the current exposure (assuming it's the Nth exposure, where N is greater than or equal to 1), camera module 420 exposes the target object, records the current actual exposure time, and sends this actual exposure time to the associated processing module 430. In this document, the actual exposure time can also be simply referred to as the exposure time. Processing module 430 compares the current target exposure time corresponding to the current exposure (the Nth exposure) with the received actual exposure time and adjusts the frame rate based on the time difference between them. Processing module 430 sends the adjusted frame rate to camera module 420, enabling camera module 420 to obtain the exposure time (called the next exposure time) for the next exposure (the N+1th exposure) based on the received frame rate and the current actual exposure time.

[0039] Next, during the next exposure (the (N+1)th exposure), the camera module 420 obtains the current actual exposure time for the current exposure (the (N+1)th exposure) based on the frame rate (referred to as the previous frame rate) obtained from the processing module 430 during the previous exposure (the Nth exposure) and the actual exposure time (referred to as the previous exposure time) recorded during the previous exposure (the Nth exposure), and captures an image of the target object at that time. The camera module 420 sends the current actual exposure time (the actual exposure time of the N+1th exposure) to the processing module 430, so that the processing module 430 compares the current target exposure time corresponding to the current exposure (the N+1th exposure) with the received actual exposure time to adjust the frame rate according to the time difference between them. The processing module 430 sends the adjusted frame rate to the camera module 420, so that the camera module 420 obtains the exposure time for the next exposure (the N+2th exposure) based on the received frame rate and the current actual exposure time.

[0040] This process continues in a loop. Due to the interaction between the actual exposure time and the frame rate, the frame rate can provide feedback control over the actual exposure time, allowing the time difference between the target exposure time and the corresponding actual exposure time to gradually converge to a predetermined range. This predetermined range can be set in advance or based on a certain proportion of the frame rate. When the time difference is within this predetermined range, it can be considered that the actual exposure time has successfully tracked the target exposure time. The time difference between the target exposure time and the actual exposure time within the predetermined range is within the acceptable range in the field of 3D scanning, reaching a level where successful 3D modeling is possible.

[0041] Specifically, camera module 420 continuously exposes the target object at a certain frame rate to capture the image of the target object. Let t represent the current exposure, t-1 represent the previous exposure, and t+1 represent the next exposure. The target exposure time is denoted by T, so T(t) represents the current target exposure time, T(t-1) represents the previous target exposure time, and T(t+1) represents the next target exposure time. The actual exposure time of camera module 420 is denoted by E, so E(t) represents the current actual exposure time, E(t-1) represents the previous actual exposure time, and T(t+1) represents the next actual exposure time. The frame rate determined by processing module 430 is denoted by F, where F(t) represents the current frame rate, F(t-1) represents the previous frame rate, and F(t+1) represents the next frame rate.

[0042] In response to a user's shooting command, each processing module instructs its associated camera module to begin exposure for image capture. When the camera module first captures an image, t=0, T(0) is the exposure time reference (e.g., it can be equal to 0 or any specific moment); E(0) is the moment when the camera module initiates the exposure operation, such as the moment when exposure begins immediately upon receiving the command (e.g., at or near 0), and the camera module sends E(0) to the processing module; F(0) is the frame rate obtained by adjusting the initial predetermined frame rate FF set in each processing module according to the difference between T(0) and E(0), and the processing module sends F(0) to the camera module. The direction of adjustment of FF (i.e., whether the frame rate increases or decreases) ensures that the next actual exposure time E(1) is close to the next target exposure time T(1).

[0043] When the camera module captures the image for the second time, t=1, T(1) is T(0)+1 / FF; E(1) is the value obtained based on E(0) and 1 / F(0) (for example, E(1) is ideally E(0)+1 / F(0), but due to the processing delay of the camera module or clock drift, E(1) is actually a value near E(0)+1 / F(0), and the camera module sends E(1) to the processing module; F(1) is the frame rate obtained by adjusting F(0) according to the difference between T(1) and E(1), and the processing module sends F(1) to the camera module. The direction of adjustment of F(0) (i.e., whether the frame rate increases or decreases) is to make the next actual exposure time E(2) close to the next target exposure time T(2).

[0044] When the camera module captures the image for the third time, t=2, T(2) is T(1)+1 / FF; E(2) is the value obtained based on E(1) and 1 / F(1) (for example, E(2) is ideally E(1)+1 / F(1), but due to the internal processing delay or clock drift of the camera module, E(2) is actually a value near E(1)+1 / F(1)), and the camera module sends E(2) to the processing module; F(2) is the frame rate obtained by adjusting F(1) according to the difference between T(2) and E(2), and the processing module sends F(2) to the camera module. The adjustment direction of F(1) makes the next actual exposure time E(3) close to the next target exposure time T(3).

[0045] Similarly, when the camera module captures an image for the Nth time, t=N-1, T(N-1) is T(N-2)+1 / FF; E(N-1) is a value obtained based on E(N-2) and 1 / F(N-2) (for example, ideally E(N-1) is E(N-2)+1 / F(N-2), but due to internal processing delays or clock drift in the camera module, E(N-1) is actually a value near E(N-2)+1 / F(N-2), and the camera module sends E(N-1) to the processing module; F(N-1) is the frame rate obtained by adjusting F(N-2) according to the difference between T(N-1) and E(N-1), and the processing module sends F(N-1) to the camera module. The adjustment direction of F(N-2) is such that the next actual exposure time E(N) can be close to the next target exposure time T(N).

[0046] At the next exposure, i.e., when the camera module captures the image for the (N+1)th time, t=N, T(N) is T(N-1)+1 / FF; E(N) is the value obtained based on E(N-1) and 1 / F(N-1), and the camera module sends E(N) to the processing module; F(N) is the frame rate obtained by adjusting F(N-1) according to the difference between T(N) and E(N), and the processing module sends F(N) to the camera module. The adjustment direction of F(N-1) is such that the next actual exposure time E(N+1) is close to the next target exposure time T(N+1).

[0047] This process iterates in a loop. Each time the camera module captures an image, it adjusts the frame rate based on the difference between the target exposure time and the actual exposure time (when the change in the previous frame rate is 0, it can also be considered as adjusting the frame rate, with an adjustment amount of 0). The adjusted frame rate then influences the next actual exposure time, allowing the camera module's actual exposure time to gradually approach the target exposure time, ultimately achieving tracking of the target exposure time. Furthermore, due to the use of feedback control, synchronization accuracy can be improved to the microsecond level.

[0048] exist Figure 5 An example of the convergence effect of the time difference between the target exposure time and the actual exposure time according to an embodiment of the present disclosure is shown. Figure 5 In the diagram, the horizontal axis represents the capture number of consecutive images, and the vertical axis represents the time difference in microseconds. From Figure 5 As can be seen, according to the exposure synchronization control method proposed in the embodiments of this disclosure, as image capture proceeds, the actual exposure time gradually converges to the target exposure time, the time difference gradually converges to around 0, and finally oscillates around 0.

[0049] Before converging to a predetermined range, such as around 0, after each iteration, the time difference between the next target exposure time and the next exposure time can be smaller than the time difference between the current target exposure time and the current exposure time; that is, the time difference is gradually decreasing. After converging to the predetermined range, the time difference between the next target exposure time and the next exposure time, as well as the time difference between the current target exposure time and the current exposure time, stabilize within this predetermined range, and there is no necessary relationship between their magnitudes.

[0050] According to the exposure synchronization control method provided in this disclosure, the frame rate (i.e., the image acquisition speed) used by the camera module can be modified based on the time difference between the target exposure time (which is the accurate exposure time) and the actual exposure time to gradually align the exposure times of all camera modules. For example, for a camera module using 10fps, time synchronization of each image capture unit can be achieved through PTP (Precise Quasi-Time Protocol) so that the camera module of each image capture unit performs an exposure every 100 milliseconds; this is the protocol alignment time. In the case of 10fps, Figure 6 The diagram shows the exposure times of each camera module without enabling exposure synchronization control, according to an embodiment of the present disclosure. Figure 7 The diagram shows the exposure times of each camera module when exposure synchronization control is enabled, according to an embodiment of the present disclosure. Although Figure 6 and Figure 7 Only two image capture units of the camera module (referred to as cameras), namely camera 1 and camera 2, are shown, but those skilled in the art will understand that more cameras are possible.

[0051] Without exposure synchronization control enabled, such as Figure 6 As shown, the actual exposure times of cameras 1 and 2 (represented by short vertical lines) are significantly different from the target exposure time (represented by long vertical lines), making it difficult for cameras 1 and 2 to synchronize their exposures. With exposure synchronization control enabled, as... Figure 7 As shown, the actual exposure time of camera 1 gradually approaches the target exposure time and locks within a very small range of the target exposure time, almost coinciding. The same applies to camera 2. This allows camera 1 and camera 2 to synchronize their exposures at the same target exposure time, which is beneficial for subsequent 3D modeling.

[0052] In the aforementioned exposure synchronization control, the clock synchronization processing modules can be initially configured to have the same exposure time base and the same predetermined frame rate. Then, the final exposure synchronization can be achieved through iterative processing provided in this embodiment. Because the iterative processing involves frame rate feedback control, the frame rate and the actual exposure time influence each other. Therefore, this feedback-based synchronization control method improves synchronization accuracy, raising it from the millisecond level in related technologies to the microsecond level, meeting the requirements of dynamic 3D object scanning. Furthermore, the synchronization control in each image capture unit is independent and does not require communication with each other, thus reducing potential communication delays and resulting in faster and more accurate synchronization. In addition, this algorithm-based method of achieving synchronized exposure has no special hardware requirements, significantly reducing hardware costs and promoting technological development.

[0053] In the embodiments of this disclosure, to ensure that the actual exposure time converges to the target exposure time as quickly as possible and to maintain good tracking, the frame rate can be adjusted each time the camera module captures an image; that is, the frame rate is continuously adjusted to continuously determine the current frame rate. Various methods can be employed when the processing module determines the frame rate based on the time difference between the target exposure time and the actual exposure time. Examples of some methods are given below. These methods can be used individually in different embodiments, or two or more can be combined in different embodiments. Furthermore, these methods are merely examples, and those skilled in the art will realize that other automated methods can be used to track the target quantity (here, the target exposure time).

[0054] In some examples, a pre-defined lookup table that associates time difference with frame rate can be used to determine the current frame rate based on the time difference between the current target exposure time and the current exposure time. For example, assuming at the current time t, a lookup table can be used to determine the frame rate F(t) based on the time difference between T(t) and E(t). The lookup table can be set according to predetermined rules. For example, a specific time difference can be associated with a specific frame rate, or a specific range of time differences can be associated with a specific frame rate.

[0055] For example, assuming a predetermined frame rate (FF) of 10fps and an initial exposure time base of 100s, the target exposure times at each processing module are 100s, 100.1s, 100.2s, 100.3s, etc. The current frame rate (F(t)) can be determined using a lookup table as shown in Table 1, based on the difference between the current target exposure time (T(t)) and the current actual exposure time (E(t)) at each exposure. This current frame rate is used by the camera module to determine the next actual exposure time.

[0056] Table 1

[0057] E(t)-T(t) F(t) ≥50ms 10.5fps Greater than or equal to 40ms and less than 50ms 10.3fps Greater than or equal to 30ms and less than 40ms 10.1fps …… …… Greater than or equal to -1ms and less than 0ms 9.998fps …… ……

[0058] Those skilled in the art will understand that Table 1 is merely an example, and the relationship between time difference and frame rate can be set in other ways so that when the time difference E(t)-T(t) is greater than or equal to 0, the larger the time difference, the larger the determined frame rate; and when the time difference E(t)-T(t) is less than 0, the larger the time difference, the smaller the determined frame rate. For example, when the time difference E(t)-T(t) is greater than 50ms, F(t) is determined to be 10.5fps; when the time difference E(t)-T(t) is greater than 10ms, F(t) is determined to be 10.1fps; when the time difference E(t)-T(t) is greater than 5ms, F(t) is determined to be 10.05fps; when the time difference E(t)-T(t) is greater than 1ms, F(t) is determined to be 10.001fps; when the time difference E(t)-T(t) is less than -1ms, F(t) is determined to be 9.999fps; and so on.

[0059] By using the lookup table in Table 1, the current frame rate F(t) can be obtained at each exposure based on the difference between the current target exposure time T(t) and the actual exposure time E(t), as shown in Table 2, where N is an integer greater than or equal to 1:

[0060] Table 2

[0061] Current time t Current target exposure time T(t) Current actual exposure time E(t) E(t)-T(t) Current frame rate F(t) N-1 100.000s 100.050s 50ms 10.5fps N 100.100s 100.140s 40ms 10.3fps N+1 100.200s 100.230s 30ms 10.1fps N+2 100.300s 100.320s 20ms 10.05fps N+3 100.400s 100.410s 10ms 10.02fps N+4 100.500s 100.505s 5ms 10.01fps N+5 100.600s 100.602s 2ms 10.005fps N+6 100.700s 100.699s -1ms 9.998fps N+7 100.800s 100.801s 1ms 10.001fps N+8 100.900s 100.900s 0ms 10fps N+9 101.000s 101.000s 0ms 10fps N+10 101.100s 101.101s 1ms 10.001fps N+11 101.200s 101.200s 0ms 10fps

[0062] The determined current frame rate F(t) is sent by the processing module 430 to the camera module 420, so that the camera module 420, in the next exposure, will expose at the actual exposure time E(t+1) obtained based on E(t) and the received F(t), and return time E(t+1) to the processing module 430. Thus, the processing module 430 can continue to determine F(t+1) based on the difference between T(t+1) and E(t+1). By using a lookup table for synchronization control, the following can be achieved: Figure 5 and Figure 7 That effect improves synchronization accuracy.

[0063] In other examples, PID control can be used to determine the current frame rate based on the time difference between the current target exposure time and the current exposure time. Figure 8The diagram illustrates a PID control according to an embodiment of this disclosure. The PID control can be located in the processing module 430, and its structure is similar to related technologies, differing mainly in the input and output settings. Specifically, the output y(t) of the PID control is the current frame rate determined based on the current target exposure time input to the positive input and the current exposure time determined based on the previous frame rate input to the negative input. The next exposure time, determined based on the current frame rate, is fed back from y(t) to the negative input, and is then calculated with the next target exposure time input to the positive input. The absolute value of the difference between the target exposure time and the actual exposure time is used as e(t) and input to the subsequent proportional (P) control section, integral (I) control section, and derivative (D) control section for processing. The results are then summed to obtain u(t). u(t) is converted to the frame rate y(t) according to a mapping process such as a preset rule. By applying PID control, the accuracy of synchronous exposure can be controlled within a range of less than 10 microseconds. The effect of PID control can be similar to that of using a lookup table; for example, it can achieve the control method shown in Table 2 and reach the desired results. Figure 5 and 7 That's the effect. The specific control methods below can also achieve similar results.

[0064] In some examples, adaptive control can be used. Adaptive control can automatically adjust system parameters to cope with dynamic changes in the system. Fuzzy control can also be used. Fuzzy control does not rely on a precise mathematical model and is suitable for complex nonlinear systems. In addition, model predictive control (MPC) can be used. Model predictive control can handle multiple variables and system constraints simultaneously. Furthermore, robust control can be used. Robust control has strong disturbance rejection capabilities and can cope with uncertainties. Sliding mode control can also be used. Sliding mode control has good robustness and high tolerance to uncertainties. The above-mentioned adaptive control, fuzzy control, model predictive control, robust control, and sliding mode control can be implemented using specific methods in related technologies. Those skilled in the art, based on the exposure synchronization control disclosed above, can achieve control by adjusting the frame rate to synchronize the exposure time with the target exposure time, which will not be elaborated further here.

[0065] In some other examples, phase-locked loop (PLL) technology can be used. PLL technology typically focuses on frequency and phase synchronization and has a simple structure. For example, when PLL technology is applied... Figure 4In the processing module 430 shown, one input of the phase detector can be the current target exposure time, which serves as a reference signal. The output of the voltage-controlled oscillator (VCO) can be the current frame rate. The current frame rate is sent to the camera module 420, which adjusts the current exposure time based on the current frame rate to obtain the next actual exposure time. The next actual exposure time is sent to the processing module 430 to be fed back to the other input of the phase detector. The phase detector compares the phase difference between the received target exposure time and the actual exposure time, which are related to the same time, and outputs an error signal related to the phase difference. The error signal is input to a loop filter, where high-frequency noise and interference components are filtered out. The filtered signal is then input to the VCO to output the frame rate. By applying phase-locked loop (PLL) technology, the actual exposure time can also be tracked to the target exposure time by adjusting the frame rate.

[0066] By employing the aforementioned exposure synchronization control, the camera modules within each image capture unit can perform synchronized exposures with high synchronization accuracy, thereby obtaining multiple images corresponding to the same target exposure time. These camera modules can output the acquired images to the application platform, enabling the platform to perform 3D modeling based on the received images. 3D modeling can include, but is not limited to, 3D reconstruction, free-viewpoint video rendering, 3D Gaussian reconstruction, neural radiation field training and rendering, etc.

[0067] For example, each camera module can add metadata corresponding to the target exposure time or the actual exposure time to the image captured at a certain moment, so that the synchronously captured image can be determined based on the metadata. The camera module sends the captured image along with the associated metadata to the application platform. The application platform can be a server, computer, cloud resource, etc., with 3D image processing capabilities. It can synthesize the images related to the same moment obtained from each camera module to obtain a 3D model of the target object. When the target object moves, by obtaining a series of 3D models over time, the 3D shape of the object can be dynamically captured, realizing 3D modeling of the moving object.

[0068] As mentioned above, in Figure 4 Three image capture units are shown, but the number of image capture units can be at least two. According to embodiments of this disclosure, the number of image capture units can be at least four. The inventors have discovered that when at least four image capture units are used, accurate 3D modeling can be achieved without requiring the image capture units to rotate around the target object (i.e., with the image capture units fixed). This minimum number limitation is not addressed in other technologies, and it eliminates repetitive work for those skilled in the art, thereby improving the efficiency of setting up image capture units.

[0069] exist Figure 9A schematic diagram of an example structure of an image capture unit according to an embodiment of the present disclosure is shown. Figure 9 An image capture unit is shown. Each image capture unit in system 400 can employ... Figure 9 As shown in the embodiments of this disclosure, the image capture unit may include a camera module 910, a cable 920, and a processing module 930, and may further include a communication module 940.

[0070] Figure 9 The camera module 910 in the middle can correspond to Figure 4 The camera module 420 can be any device with image capture capabilities. For example, the camera module could be... Figure 10A The sensor circuit board shown can also be Figure 10B The camera shown is a digital camera with independent photo and video recording capabilities. Figure 10C The time-of-flight (ToF) / LiDAR sensor shown is an example. One or more of the sensor circuit board, digital camera, time-of-flight sensor, or LiDAR sensor, which are specific forms of camera modules, can be implemented in different image capture units, or the same camera module form can be used in each image capture unit of the same system 400. The specific form of the camera module used does not constitute a limitation of this disclosure.

[0071] Figure 9 Cable 920 can be any line capable of transmitting data. Wired connections allow for high-speed data transmission between the processing module and the camera module, further reducing data transmission latency. The processing module can be connected to the camera module via the cable, enabling it to send the current frame rate to the camera module and vice versa. The processing module can be detached from the camera module when the cable is detachable from at least one of the two. In other examples, if the cable is not detachable, or if the processing module is connected to the camera module via a hard connection (e.g., a circuit connection on a circuit board), the camera module and processing module can be integrated, for example, by integrating the processing module onto the image sensor to reside on the same chip or circuit board as the image sensor.

[0072] Figure 9 The processing module 930 in the middle can correspond to Figure 4The processing module 430 can be any device with information processing capabilities, such as an information processing device with a memory for storing instructions and a processor for processing instructions. According to embodiments of this disclosure, the processing module can be a card-type computer. A card-type computer is, for example, a development board such as an FPGA or a microcontroller, such as a commercial development board from a Raspberry Pi. Card-type computers are inexpensive, typically costing around several hundred yuan (e.g., around 400 yuan), while the camera module can use a common image sensor, also costing around several hundred yuan (e.g., around 300 yuan), making the cost of a single image capture unit approximately 700 yuan.

[0073] When high-precision synchronous exposure is required, existing equipment used for accurate 3D scanning of target objects, such as synchronous industrial cameras or Genlock cameras, is extremely expensive. Typically, a camera with a synchronization trigger can cost tens of thousands of yuan. This high cost is a significant obstacle to the development of 3D scanning. In contrast, when using the image capture unit according to embodiments of this disclosure, while achieving high-precision synchronous exposure, the cost of a single image capture unit is drastically reduced from tens of thousands of yuan to around 700 yuan. Therefore, through the aforementioned exposure synchronization control, embodiments of this disclosure can achieve similar effects to synchronous industrial cameras or Genlock cameras at a much lower cost, representing a significant technological advancement and arguably a qualitative leap in 3D scanning.

[0074] Figure 9 The communication module 940 is used for communication between the image capture unit and the outside world. The communication module 940 can be connected to at least one of the camera module 910 and the processing module 930. Although Figure 9 The diagram shows the communication module 940 connected to both the camera module 910 and the processing module 930; however, the communication module 940 may be connected to only one of the camera module 910 and the processing module 930. The communication module 940 can receive images captured by the camera module 910 directly from the camera module 910 or via the processing module 930, and send the received images to the application platform. The communication module 940 can also facilitate communication between processing modules in different image capture units and / or forward control information to the processing module, such as for time alignment between these processing modules.

[0075] exist Figure 11 A flowchart of a method 1100 for image capture according to an embodiment of the present disclosure is shown. Method 1100 can be executed by processing module 430.

[0076] In S1110, the current exposure time when the camera module captures the image of the target object is obtained. In S1120, the current frame rate is determined based on the time difference between the current target exposure time and the current exposure time. In S1130, the current frame rate is sent to the camera module, so that the camera module captures the image of the target object at the next exposure time obtained based on the current frame rate and the current exposure time. As the iterative loop executes, the time difference between the next target exposure time and the next exposure time gradually converges to a predetermined range. The above operations and details of method 1100 can be found in the section above regarding... Figures 4 to 10C To avoid repetition, the description will not be repeated here.

[0077] exist Figure 12 The diagram shows a schematic of the architecture of a system 1200 for image capture according to an embodiment of the present disclosure.

[0078] System 1200 can be similar to System 400 described above. System 1200 can include M image capture units, where M can be an integer greater than or equal to 2. Each row in System 1200 can correspond to one image capture unit. Each image capture unit includes a corresponding processing module and a camera module.

[0079] Specifically, system 1200, based on processing modules 1 to M and incorporating exposure feedback control technology, enables camera modules 1 to M to operate synchronously. Each of the processing modules 1 to M can be in the form of a card-type computer, resulting in a relatively low cost of around several hundred yuan. Processing modules 1 to M are time-aligned using a network time synchronization protocol to achieve the same target exposure time. Furthermore, each of them employs the aforementioned exposure synchronization control method for exposure time feedback compensation. In exposure time feedback compensation, the frame rate is adjusted based on the time difference between the target exposure time and the actual exposure time. This adjusted frame rate is then used to adjust the next actual exposure time, gradually reducing the difference between the actual exposure time and the corresponding target exposure time until the difference converges to a predetermined range. Thus, low-cost camera modules 1 to M can perform synchronous exposure to obtain corresponding synchronous images 1 to M. These synchronous images 1 to M can be transmitted to an application platform for, for example, 3D modeling using existing 3D modeling techniques.

[0080] In the image capture system provided according to embodiments of this disclosure, the frame rate is adjusted based on the time difference between the target exposure time and the actual exposure time to influence the next exposure time, and the frame rate is further adjusted using the next exposure time. This allows the actual exposure time to gradually approach and track the target exposure time through iterative loops, thanks to the mutual influence between the frame rate and the exposure time. Thus, the exposure synchronization of multiple cameras can be improved through iterative loops, achieving exposure accuracy at the microsecond level. Furthermore, while achieving microsecond-level synchronization accuracy, a single image capture unit in this disclosure can reduce costs from tens of thousands of yuan to a few hundred yuan, resulting in significant cost savings. In addition, the exposure synchronization control method in this disclosure does not require complex hardware triggering devices; synchronization can be achieved solely through software processing locally without complex interactions, improving synchronization efficiency and processing speed. Moreover, due to the very simple topology provided in this disclosure, the number of image capture units is theoretically unlimited, ranging from a dozen to hundreds, enabling exposure synchronization. The system has good scalability and can adapt to various 3D scanning needs.

[0081] The foregoing has described various exemplary devices and methods according to embodiments of this disclosure. It should be understood that the operation or function of these devices can be combined with each other to achieve more or fewer operations or functions than described. Similarly, the operational steps of the methods can be combined with each other in any suitable order to similarly achieve more or fewer operations than described.

[0082] It should be understood that the machine-executable instructions in a machine-readable storage medium or program product according to embodiments of this disclosure can be configured to perform operations corresponding to the above-described device and method embodiments. Embodiments of the machine-readable storage medium or program product will be clear to those skilled in the art when referring to the above-described device and method embodiments, and therefore will not be described again. Machine-readable storage media and program products used to carry or include the above-described machine-executable instructions also fall within the scope of this disclosure. Such storage media may include, but are not limited to, floppy disks, optical disks, magneto-optical disks, memory cards, memory sticks, etc.

[0083] Furthermore, it should be understood that the aforementioned series of processes and devices can also be implemented via software and / or firmware. In the case of software and / or firmware implementation, the data transfers from storage media or networks to processing devices with dedicated hardware architectures, such as… Figure 13 The processing device 1300 shown is equipped with programs that constitute the software, and when various programs are installed, the processing device is able to perform various functions, etc. Figure 13 This is a block diagram illustrating an example structure of a processing apparatus that may be employed in embodiments of this disclosure. Processing apparatus 1300 may be... Figure 4The processing module 430 can also be part of the processing device 1300. The processing device 1300 may include... Figure 13 All components shown may also be included, or only a portion of these components may be included, as long as they can perform corresponding operations according to the stored program.

[0084] exist Figure 13 In this system, the central processing unit (CPU) 1301 performs various processes based on the program stored in the read-only memory (ROM) 1302 or the program loaded into the random access memory (RAM) 1303 from the storage section 1308. The RAM 1303 also stores, as needed, the data required when the CPU 1301 performs various processes.

[0085] CPU 1301, ROM 1302 and RAM 1303 are connected to each other via bus 1304. Input / output interface 1305 is also connected to bus 1304.

[0086] The following components are connected to the input / output interface 1305: input section 1306, including a keyboard, mouse, etc.; output section 1307, including a display, such as a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; storage section 1308, including a hard disk, etc.; and communication section 1309, including a network interface card, such as a LAN card, modem, etc. The communication section 1309 performs communication processing via a network, such as the Internet.

[0087] As needed, drive 1310 is also connected to input / output interface 1305. Removable media 1311, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on drive 1310 as needed, so that computer programs read from them can be installed into storage section 1308 as needed.

[0088] When the above series of processes are implemented by software, the program constituting the software is installed from a network such as the Internet or a storage medium such as removable media 1311.

[0089] Those skilled in the art will understand that such storage media are not limited to Figure 13 The illustrated removable medium 1311 stores a program and is distributed separately from the device to provide the program to the user. Examples of removable media 1311 include magnetic disks (including floppy disks (registered trademark)), optical disks (including optical disc read-only memory (CD-ROM) and digital versatile disks (DVD)), magneto-optical disks (including mini-disk (MD) (registered trademark)), and semiconductor memory. Alternatively, the storage medium may be ROM 1302, a hard disk included in storage section 1308, etc., containing programs and distributed to the user along with the device containing them.

[0090] Exemplary embodiments of the present disclosure have been described above with reference to the accompanying drawings; however, the present disclosure is by no means limited to the examples described above. Various changes and modifications can be made by those skilled in the art within the scope of the appended claims, and it should be understood that such changes and modifications naturally fall within the technical scope of the present disclosure.

[0091] For example, the multiple functions included in one unit in the above embodiments can be implemented by separate devices. Alternatively, the multiple functions implemented by multiple units in the above embodiments can be implemented by separate devices respectively. In addition, one of the above functions can be implemented by multiple units. Needless to say, such a configuration is included within the scope of the present disclosure.

[0092] In this specification, the steps described in the flowchart include not only processes executed sequentially in the stated order, but also processes executed in parallel or individually, rather than necessarily sequentially. Furthermore, even within the steps of sequential processing, needless to say, the order can be appropriately altered.

[0093] While this disclosure and its advantages have been described in detail, it should be understood that various changes, substitutions, and modifications can be made without departing from the spirit and scope of this disclosure as defined by the appended claims. Furthermore, the terms "comprising," "including," or any other variations thereof used in embodiments of this disclosure are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0094] As will be appreciated from the description herein, embodiments of this disclosure can be configured as follows:

[0095] 1. A system for image capture, comprising multiple image capture units, each image capture unit including a camera module and a processing module, wherein:

[0096] The camera module is configured to capture an image of the target object and send the current exposure time at the time of image capture to the processing module; and

[0097] The processing module is configured to determine the current frame rate based on the time difference between the current target exposure time and the current exposure time obtained from the camera module, and send the current frame rate to the camera module, so that the camera module captures the image of the target object at the next exposure time obtained based on the current frame rate and the current exposure time, wherein the time difference between the next target exposure time and the next exposure time gradually converges to a predetermined range.

[0098] 2. The system according to Clause 1, wherein the camera module is further configured to take the moment when the exposure operation is initiated as the current exposure moment when the image of the target object is captured for the first time, and to obtain the current exposure moment based on the previous frame rate obtained from the processing module and the previous exposure moment recorded by the camera module when the image of the target object is captured for the second and subsequent times.

[0099] 3. The system according to Clause 1, wherein, before convergence to the predetermined range, the time difference between the next target exposure time and the next exposure time is less than the time difference between the current target exposure time and the current exposure time.

[0100] 4. The system according to Clause 1, wherein the processing module is further configured to determine the current frame rate in response to the camera module capturing an image of the target object at the current exposure time.

[0101] 5. The system according to Clause 1, wherein the processing module is further configured to maintain a consistent target exposure time for the plurality of image capture units based on a synchronization protocol.

[0102] 6. The system according to Clause 1, wherein the processing module is further configured to determine the current frame rate based on the time difference between the current target exposure time and the current exposure time in one of the following ways: a preset lookup table that associates the time difference with the frame rate, PID control, adaptive control, fuzzy control, model predictive control, robust control, sliding mode control, or phase-locked loop technology.

[0103] 7. The system according to Clause 1, wherein the camera module is further configured to output captured images to an application platform, so that the application platform performs 3D modeling based on images obtained from the camera modules of each image capture unit corresponding to the same target exposure time.

[0104] 8. The system according to Clause 1, wherein the number of the plurality of image capture units is at least four.

[0105] 9. The system according to Clause 1, wherein the processing module is separable from the camera module.

[0106] 10. The system according to Clause 1, wherein the processing module is connected to the camera module via a cable, such that the processing module sends the current frame rate to the camera module via the cable, and the camera module sends the current exposure time to the processing module via the cable.

[0107] 11. The system according to Clause 1, wherein the camera module and the processing module are integrated together.

[0108] 12. The system according to Clause 1, wherein the processing module includes a card-type computer.

[0109] 13. The system according to Clause 1, wherein the camera module comprises at least one of the following: a sensor circuit board, a digital camera, a time-of-flight sensor, or a lidar sensor.

[0110] 14. A method for image capture, comprising:

[0111] Obtain the current exposure time when the camera module captures an image of the target object;

[0112] The current frame rate is determined based on the time difference between the current target exposure time and the current exposure time; and

[0113] The current frame rate is sent to the camera module, so that the camera module can capture the image of the target object at the next exposure time obtained based on the current frame rate and the current exposure time, wherein the time difference between the next target exposure time and the next exposure time gradually converges to a predetermined range.

[0114] 15. The method according to Clause 14, wherein, when the camera module first captures an image of the target object, the current exposure time is the time when the camera module initiates the exposure operation, and

[0115] When the camera module captures an image of the target object for the second and subsequent times, the current exposure time is obtained based on the previous frame rate and the previous exposure time.

[0116] 16. The method according to Clause 14, wherein, before convergence to the predetermined range, the time difference between the next target exposure time and the next exposure time is less than the time difference between the current target exposure time and the current exposure time.

[0117] 17. The method described in Clause 14 further includes:

[0118] In response to the camera module capturing an image of the target object at the current exposure time, the current frame rate is determined.

[0119] 18. The method described in accordance with Clause 14 further includes:

[0120] Based on the synchronization protocol, the target exposure time is kept consistent for multiple camera modules.

[0121] 19. The method described in accordance with Clause 14 further includes:

[0122] The current frame rate is determined by one of the following methods based on the time difference between the current target exposure time and the current exposure time: a preset lookup table that associates the time difference with the frame rate, PID control, adaptive control, fuzzy control, model predictive control, robust control, sliding mode control, or phase-locked loop technology.

[0123] 20. The method according to Clause 14, wherein the captured image is output to an application platform to enable the application platform to perform 3D modeling based on images obtained from multiple camera modules corresponding to the same target exposure time.

[0124] 21. A processing module, comprising:

[0125] Processor; and

[0126] The memory includes computer program instructions, wherein the memory and computer program instructions are configured to cause the processing module to perform the method described in any one of clauses 14 to 20 via the processor.

[0127] 22. The processing module according to Clause 21, wherein the processing module is connected to a camera module to send the current frame rate to the camera module and receive the current exposure time from the camera module.

[0128] 23. A computer-readable storage medium having stored thereon computer program instructions that, when executed by a processing device, cause the processing device to perform the method described in any one of claims 14 to 20.

Claims

1. A system for image capture, comprising a plurality of processing modules corresponding to a plurality of camera modules respectively, each processing module being configured to: Obtain the actual exposure time of the camera module; and The frame rate is adjusted based on the actual exposure time of the camera module and the target exposure time determined based on the predetermined frame rate, so that after one or more adjustments to the frame rate, the difference between the actual exposure time and the target exposure time of the camera module is controlled within a predetermined range.

2. The system according to claim 1, wherein, The predetermined range is 10 microseconds.

3. The system according to claim 1, wherein, Adjusting the frame rate based on the actual exposure time of the camera module and the target exposure time determined based on a predetermined frame rate includes: When the actual exposure time is less than the target exposure time, the frame rate is reduced to make subsequent actual exposure times closer to subsequent target exposure times; and When the actual exposure time is greater than the target exposure time, the frame rate is increased to make the subsequent actual exposure time closer to the subsequent target exposure time.

4. The system according to claim 3, wherein, The smaller the actual exposure time is compared to the target exposure time, the greater the reduction in frame rate; and The greater the difference between the actual exposure time and the target exposure time, the greater the increase in frame rate.

5. The system according to claim 1, further comprising: The plurality of camera modules are each configured to determine the subsequent actual exposure time based on the frame rate adjusted by the corresponding processing module.

6. The system according to claim 5, wherein, Each camera module is further configured to output an image captured at the actual exposure moment to the application platform, so that the application platform can perform 3D modeling based on the images obtained from the multiple camera modules corresponding to the same target exposure moment.

7. The system according to claim 1, wherein, The multiple processing modules are clock-synchronized in advance via a synchronization protocol and have the same initial frame frequency as the predetermined frame frequency.

8. A method for image capture, comprising: Obtain the actual exposure time of the camera module; as well as The frame rate is adjusted based on the actual exposure time of the camera module and the target exposure time determined based on the predetermined frame rate, so that after one or more adjustments to the frame rate, the difference between the actual exposure time and the target exposure time of the camera module is controlled within a predetermined range.

9. A method for performing 3D modeling, comprising: Acquire multiple captured images corresponding to the same target exposure time from multiple camera modules; as well as 3D modeling is performed based on the multiple captured images.

10. A computer-readable storage medium having stored thereon computer program instructions that, when executed by a processing device, cause the processing device to perform the method according to claim 8 or 9.

Images