Image acquisition method and device, equipment, storage medium and program product

Abstract

The invention relates to an image acquisition method and device, equipment, a storage medium and a program product, and the method comprises the steps: obtaining the actual frame rate of each camera in a multi-view camera, obtaining a frame synchronization signal according to the actual frame rate of each camera and a preset target demand frame rate, and then transmitting the frame synchronization signal to each camera, and indicating each camera to output a collected image according to the frame synchronization signal. By adopting the method, the frame synchronization effect of the acquired images output by the cameras can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technologies, and particularly to an image acquisition method, apparatus, device, storage medium, and program product. Background Art

[0002] With the increasing demand for cameras in the market, traditional monocular cameras have gradually been unable to meet the market's needs, and multi-camera cameras have emerged as the times require.

[0003] In related technologies, for the image acquisition method of multi-camera cameras, the acquisition images of each camera are obtained separately, and then the images of each obtained camera are frame-synchronized to implement the functions of the multi-camera camera.

[0004] However, in related technologies, there is a problem of poor frame synchronization effect in the images collected by each camera of the multi-camera camera. Summary of the Invention

[0005] Based on this, in view of the above technical problems, it is necessary to provide an image acquisition method, apparatus, device, storage medium, and program product, which improves the frame synchronization effect of the images collected by each camera in the multi-camera camera.

[0006] In a first aspect, this application provides an image acquisition method, including:

[0007] Obtain the actual frame rates of each camera in the multi-camera camera;

[0008] Obtain a frame synchronization signal according to the actual frame rates of each camera and a preset target required frame rate;

[0009] Send the frame synchronization signal to each camera to instruct each camera to output the acquired image according to the frame synchronization signal.

[0010] In one embodiment, obtaining the actual frame rates of each camera in the multi-camera camera includes:

[0011] Generate an initial frame synchronization signal according to the target required frame rate;

[0012] Send the initial frame synchronization signal to each camera according to the output mode of each camera to instruct each camera to output an initial acquired image according to the initial frame synchronization signal;

[0013] Determine the actual frame rates of each camera according to each initial acquired image.

[0014] In one embodiment, obtaining a frame synchronization signal according to the actual frame rates of each camera and a preset target required frame rate includes:

[0015] For any camera, obtain the frame rate difference between the actual frame rate and the target required frame rate;

[0016] Analyze the frame rate difference through a preset frame rate adjustment algorithm to determine the control frame rate;

[0017] Obtain a frame synchronization signal according to the control frame rate.

[0018] In one embodiment, the method further includes:

[0019] In response to a frame change request of a multi-camera, use the frame rate to be updated as the target required frame rate.

[0020] In one embodiment, the method further includes:

[0021] Collect the captured images output by each camera according to the frame synchronization signal;

[0022] For any camera, obtain the capture time interval between the captured image output by the camera according to the frame synchronization signal and the previous captured image;

[0023] If the capture time interval is inconsistent with the preset frame rate interval time, discard the captured image of the camera.

[0024] In one embodiment, collecting the captured images output by each camera according to the frame synchronization signal includes:

[0025] When the capture frame rates of each camera are the same, if there is no phase difference between the frame synchronization signals of each camera, alternately collect the captured images output by each camera;

[0026] If there is a phase difference between the frame synchronization signals of each camera, sequentially collect the captured images output by each camera according to the frame synchronization signal.

[0027] In one embodiment, collecting the captured images output by each camera according to the frame synchronization signal includes:

[0028] When the capture frame rates of each camera are different, determine the capture ratio between each camera according to the capture frame rate of each camera;

[0029] Determine the actual required frame rate of each camera according to the capture ratio and the target required frame rate;

[0030] Determine the capture order of each camera according to the actual required frame rate of each camera;

[0031] Collect the captured images output by each camera according to the frame synchronization signal in the capture order.

[0032] In one embodiment, determining the actual required frame rate of each camera according to the capture ratio and the target required frame rate includes:

[0033] Determine the candidate required frame rates of each camera according to the acquisition ratio and the target required frame rate;

[0034] For any camera, if the candidate required frame rate of the camera is greater than the acquisition frame rate, then determine the acquisition frame rate as the actual required frame rate of the camera;

[0035] If the candidate required frame rate of the camera is less than or equal to the acquisition frame rate, then determine the candidate required frame rate as the actual required frame rate of the camera.

[0036] In one embodiment, determine the acquisition order of each camera according to the actual required frame rate of each camera, including:

[0037] Determine the acquisition weight of each camera according to the actual required frame rate of each camera;

[0038] Determine the acquisition order of each camera according to the acquisition weight of each camera.

[0039] In a second aspect, the present application also provides an image acquisition device, including:

[0040] A frame rate acquisition module, configured to acquire the actual frame rate of each camera in a multi-camera;

[0041] A signal acquisition module, configured to acquire a frame synchronization signal according to the actual frame rate of each camera and a preset target required frame rate;

[0042] A signal distribution module, configured to distribute the frame synchronization signal to each camera to instruct each camera to output an acquired image according to the frame synchronization signal.

[0043] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps of the method provided in any one of the first aspects are implemented.

[0044] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method provided in any one of the first aspects are implemented.

[0045] In a fifth aspect, an embodiment of the present application further provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the method provided in any one of the first aspects are implemented.

[0046] The above image acquisition method, device, equipment, storage medium and program product obtain the actual frame rates of the cameras in the multi-camera, and obtain a frame synchronization signal according to the actual frame rates of the cameras and a preset target required frame rate, and then send the frame synchronization signal to each camera to instruct each camera to output the acquired image according to the frame synchronization signal. In this method, by comparing and analyzing the actual frame rates output by each camera with the target required frame rate of the multi-camera, the frame synchronization signal is determined, so that the accuracy of the frame synchronization signal sent by the multi-camera to each camera is such that the actual frame rates of the acquired images output by each camera are as close as possible to the target required frame rate, thereby enabling the acquired images output by each camera to achieve frame synchronization. Description of the Drawings

[0047] To more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0048] Figure 1a Schematic structural diagram of a multi-camera in one embodiment;

[0049] Figure 1b Schematic structural diagram of a multi-camera in another embodiment;

[0050] Figure 1c Schematic structural diagram of a multi-camera in another embodiment;

[0051] Figure 1d Schematic structural diagram of a multi-camera in another embodiment;

[0052] Figure 1e Schematic structural diagram of a multi-camera in another embodiment;

[0053] Figure 1f Schematic diagram of the deviation between each camera and the frame synchronization signal in one embodiment;

[0054] Figure 2 Schematic flowchart of an image acquisition method in one embodiment;

[0055] Figure 3 Schematic flowchart of an image acquisition method in another embodiment;

[0056] Figure 4 Schematic diagram of the acquisition method of a binocular camera in one embodiment;

[0057] Figure 5 Schematic diagram of the acquisition method of a binocular camera in another embodiment;

[0058] Figure 6 It is a schematic flow chart of an image acquisition method in another embodiment;

[0059] Figure 7 It is a schematic structural diagram of a frame synchronization controller in one embodiment;

[0060] Figure 8 It is a schematic structural diagram of a frame rate generator in one embodiment;

[0061] Figure 9 It is a schematic flow chart of an image acquisition method in another embodiment;

[0062] Figure 10 It is a schematic flow chart of an image acquisition method in another embodiment;

[0063] Figure 11 It is a schematic flow chart of an image acquisition method in another embodiment;

[0064] Figure 12 It is a schematic diagram of the acquisition method of a three - camera in one embodiment;

[0065] Figure 13 It is a schematic diagram of the acquisition method of a three - camera in another embodiment;

[0066] Figure 14 It is a schematic flow chart of an image acquisition method in another embodiment;

[0067] Figure 15 It is a schematic diagram of the acquisition method of a two - camera in another embodiment;

[0068] Figure 16 It is a schematic flow chart of an image acquisition method in another embodiment;

[0069] Figure 17 It is a schematic flow chart of an image acquisition method in another embodiment;

[0070] Figure 18 It is a schematic flow chart of an image acquisition method in another embodiment;

[0071] Figure 19 It is a block diagram of the structure of an image acquisition device in one embodiment;

[0072] Figure 20 It is an internal structure diagram of a computer device in one embodiment. Specific embodiments

[0073] In order to make the objectives, technical solutions and advantages of the present application more clear and understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0074] The image acquisition method provided by the embodiment of the present application can be applied to an application environment as Figure 1a shown. Among them, the multi-camera 100 includes a System-on-a-Chip (SOC) 101. An Image Signal Processing (ISP) controller 102 is included on the SOC chip 101. The ISP controller 102 is communicatively connected to multiple cameras 103 respectively. It should be noted that in FIG. 1, a three-camera is used as an example for illustration.

[0075] In the related art, in order to ensure synchronous acquisition of data on multiple cameras, the hardware design solution of the multi-camera is to use multiple SOC chips on one multi-camera, connect one camera respectively, and then synchronize the acquired image data of each channel to realize the multi-camera function; or, implement multiple ISP controllers on a single SOC chip, and each ISP controller respectively acquires the image data output by one camera, and then performs image data synchronization processing.

[0076] Some problems will occur: First, using multiple SOC chips on the same multi-camera to connect multiple cameras or implementing multiple ISP controllers on a single SOC chip to connect multiple cameras will significantly increase the hardware cost and the workload of product development; second, for the solution of using multiple SOC chips or a single SOC chip to implement multiple ISP controllers to connect multiple cameras, it is also very difficult to perform frame synchronization on the acquired image data of each channel to ensure stable image frame rate and smooth preview; finally, it is also a very difficult thing to implement the frame rate dynamic frame control and ratio control of multiple cameras to meet the application requirements in different scenarios.

[0077] The image acquisition method provided by the embodiment of the present application is applied to the multi-camera provided by the embodiment of the present application. Therefore, when explaining the process of the image acquisition method, the hardware structure of the multi-camera provided by the embodiment of the present application will be explained first.

[0078] In the multi-camera of the embodiment of the present application, a single SOC chip and a single ISP controller share multiple Mobile Industry Processor Interface (MIPI) channels time-division multiplexing or build a MIPI swich module synchronization scheme peripherally to receive the image data output by multiple cameras. Therefore, the multi-camera in the embodiment of the present application uses a single ISP controller to time-division multiplex and collect the image data output by multiple cameras connected through MIPI interfaces to implement the function of the multi-camera.

[0079] The working modes of the camera support the master mode, the slave mode, and the master-slave mode. When the camera is in the master mode, it outputs image data to the ISP controller of the SOC chip according to its own frame synchronization signal. When the camera is in the slave mode, it outputs image data to the ISP controller of the SOC chip according to the frame synchronization signal (frame sync control, fsync) provided by the SOC chip. Therefore, the connection methods between the camera and the ISP controller include: the master-master mode, the slave-slave mode, and the master-slave mode. The following will introduce the hardware design structures of several modes taking the binocular and trinocular cameras as examples.

[0080] As Figure 1b shown, Figure 1b FIG. 10 is a schematic structural diagram of a slave-slave mode in a binocular camera; the ISP controller includes more than two MIPI interfaces, and two cameras are respectively connected to the ISP controller through the MIPI interfaces on the ISP controller, and the ISP controller can respectively send frame synchronization signals to each camera.

[0081] However, the slave-mode frame synchronization characteristics of different types of cameras vary greatly. For example, for a certain model of camera, in the slave mode, even without the frame synchronization signal provided by the ISP controller of the external SOC chip, it will continuously output image data. When the ISP controller of the SOC chip provides the frame synchronization signal, at the beginning, the synchronization deviation between the image data output by the camera and the frame synchronization signal is relatively large. As the camera continuously outputs, the frame synchronization deviation gradually converges, and the convergence time is relatively long. For another model of camera, in the slave mode, when the ISP controller of the SOC chip does not provide the frame synchronization signal to the camera, the camera will not output image data. If the frame synchronization signal is given to the camera, image data will be output quickly, and the deviation between the output image data and the frame synchronization signal is very small. However, the first frame of image data collected after synchronization is a bad frame and needs to be discarded.

[0082] It is also possible to control the frame synchronization of multiple cameras using a frame synchronization signal of the SOC chip. In this case, it is not possible to adjust the frame synchronization phases of two cameras, and only fixed-frame-rate acquisition can be performed. Taking a binocular camera as an example, as Figure 1c shown, Figure 1c FIG. 4 is a schematic structural diagram of another slave mode in a binocular camera. The ISP controller includes more than two MIPI interfaces. The two cameras are respectively connected to the ISP controller through the MIPI interfaces on the ISP controller. The ISP controller simultaneously sends a frame synchronization signal to each camera.

[0083] If a multi-camera uses the master-master mode, generally, the ISP controller of the SOC chip can hardly control the frame synchronization acquisition of the image data output by the camera. Therefore, it is difficult to synchronously acquire the image data of multiple cameras. However, the frame synchronization output of the image data by the camera can be controlled through the power control (Power Down, pwdn) pin of the camera. After the ISP controller of the SOC chip synchronously controls the pwdn pin with the frame synchronization signal, the camera immediately synchronously outputs the image data. However, the first frame of image data output is bad data and needs to be discarded. Therefore, after the first startup or after switching the frame rate, the ISP controller of the SOC chip actively synchronizes the pwdn pin of the camera through the frame synchronization signal, and the frame synchronization output of two cameras can be achieved. Since the frame synchronization deviation oscillation will still occur in the camera after running for a period of time, it is necessary to perform frame synchronization with the camera regularly. As Figure 1d shown, Figure 1d FIG. 5 is a schematic structural diagram of the master-master mode in a binocular camera.

[0084] In the embodiments of the present application, a multi-camera uses a single SOC chip and a single ISP controller to time-division multiplex multiple MIPI channels or externally build a MIPI swich module synchronization scheme to receive the image data output by multiple cameras. As Figure 1e shown, taking a three-camera as an example, on one of the MIPI channels, a MIPI switch chip is added. The channel switching of the MIPI switch chip is controlled through the general-purpose input / output (GPIO) port of the SOC chip to time-division collect the image data output by camera 1 or camera 2, so as to implement the multi-camera hardware solution of the three-camera. Specifically, through the control of the high and low levels output by a GPIO pin of the SOC chip, the image data output by the two connected cameras selectively passes through the MIPI swicth chip to reach the acquisition end of the ISP controller.

[0085] Based on the above hardware design scheme of the multi-camera, due to the significant differences in frame synchronization performance among cameras of different models. For example, in the slave mode output of one camera, the camera will output image data only after receiving the frame synchronization signal output by the ISP controller of the SOC chip, and the time deviation δt between the rising edge of the frame synchronization signal and the first row of valid data output by the camera is fixed at about 530 us. For example, when the frame rate configuration of the camera is switched from 30 fps to 10 fps, the frame synchronization deviation δt between the rising edge of the frame synchronization signal and the first row of valid data output by the camera is about 500 ms.

[0086] For another camera, there is a relatively large deviation in frame synchronization of the camera. When the camera switches from the daytime mode to the nighttime mode, the output frame rate of the camera decreases from 30 fps to 10 fps, and the camera stably outputs 10 fps. It takes 5 seconds for the frame synchronization deviation to converge. When the camera switches from the nighttime mode to the daytime mode, the output frame rate of the camera changes from 10 fps to 30 fps, and the stable frame rate of the camera is 15 fps. It takes 10 seconds for the frame synchronization deviation to converge.

[0087] In the main mode of one camera, after the first start of transmission or frame rate switching, the synchronization situation between the frame synchronization signal of the SOC chip and the data frame output by the camera is as follows: When the frame synchronization signal is sent to the pwdn pin of the camera, the first row of data output by the camera is basically synchronized with the frame synchronization signal of the SOC chip. When the camera changes the frame rate, there will be a situation of four black frames. After releasing pwdn, the data lane0 on the MIPI line will be pulled high again after 120 microseconds, and the interval δt between the frame start signal output by the camera and the falling edge of pwdn is 300 microseconds. The synchronization deviation δt between the released pwdn and the output data of the camera is fixed. The pwdn hold time is relatively short (120 microseconds), and there are no bad frames after release. When the pwdn hold time is relatively long, for example: 33 milliseconds, the first frame output after release is a bad frame. The synchronization deviation δt timing between the frame synchronization signal of the external SOC chip and the first row of MIPI data output by the camera is as Figure 1f shown.

[0088] Since the frame synchronization deviation δt of cameras of different models varies greatly, and the synchronization deviation after frame change needs to be considered. Frame change includes frame change from low frame rate to high frame rate, or from high frame rate to low frame rate. Also, during the normal operation of the camera, there will randomly be oscillations in the output synchronization deviation δt. Therefore, a more ingenious frame synchronization method needs to be implemented to solve the problem of frame synchronization timing deviation of the camera output. The camera itself will also automatically converge the synchronization deviation, but for some cameras, the convergence time is in units of rows and the convergence time is relatively long.

[0089] Based on the above, if the camera is in the master mode, it will be very difficult for the ISP controller of the SOC chip to synchronously collect the image data of multiple cameras and implement exposure control, resulting in frame loss and unstable frame rate, which will affect the smoothness of image preview. If the camera is in the slave mode, the synchronous data acquisition and frame rate control of multiple cameras have relatively strict requirements, because the frame synchronization deviation range of the image data output by different types of cameras is relatively large and it is difficult to be compatible; in addition, when the camera switches from a high frame rate mode to a low frame rate mode, or from a low frame rate mode to a high frame rate mode, it will cause the frame synchronization deviation of the data output by the camera to oscillate, resulting in frame loss, unstable frame rate, and abnormal phenomena such as image screen distortion when the camera changes the frame rate; moreover, the acquisition frame rate ratios of multiple cameras are also different in different application scenarios, and frame rate ratio control needs to be implemented.

[0090] Therefore, based on the traditional monocular camera solution, how to design a low-cost and fast-implemented multi-camera function solution fundamentally still needs to solve the frame synchronization problem of the multiple cameras' output in the multi-camera, as well as the frame rate control and frame rate ratio control in different application scenarios. This application aims to solve these problems, realizing synchronous acquisition of multiple cameras, dynamic frame rate control of multiple cameras, and frame rate ratio control on the basis of the monocular camera solution, meeting the application requirements in different scenarios, and realizing the function of a cost-effective multi-camera.

[0091] Based on this, the embodiments of this application provide an image acquisition method, which improves the frame synchronization effect of the images collected by each camera of the multi-camera on the basis of low cost.

[0092] In an exemplary embodiment, as Figure 2 shown, an image acquisition method is provided, including the following steps:

[0093] S201, obtain the actual frame rates of the cameras in the multi-camera.

[0094] Among them, the frame rate of the camera can represent the number of image frames that the camera can capture and transmit per second; the actual frame rate of the camera can represent the image frame rate actually output by the camera to the ISP controller.

[0095] Therefore, the ISP controller can determine the actual frame rates of the cameras in the multi-camera according to the output images of the cameras in the multi-camera.

[0096] For any camera, the ISP controller can count the number of image frames captured by the camera within a preset time period from the current moment, and then determine the image frame rate according to the number of image frames.

[0097] S202. Obtain a frame synchronization signal according to the actual frame rate of each camera and the preset target required frame rate.

[0098] The target required frame rate can be the frame rate at which each camera needs to be acquired by the ISP controller; the frame synchronization signal can ensure that the camera is synchronized with the target required frame rate; for example, if the actual frame rate of the camera is higher than the target required frame rate, the frame rate can be reduced by dropping frames or adjusting the sampling interval; if the actual frame rate is lower than the target required frame rate, the frame rate can be increased by inserting repeated frames or adjusting the acquisition interval.

[0099] Therefore, according to the actual frame rate of each camera and the target required frame rate, the difference between the actual frame rate of each camera and the target required frame rate can be calculated to obtain the frame rate difference; according to the frame rate difference, the actual frame rate and the target required frame rate, determine the target output frame rate, and then determine the frame synchronization signal according to the target output frame rate; the frame synchronization signal is used to indicate the time when the camera outputs an image.

[0100] Among them, the idea of determining the frame synchronization signal according to the actual frame rate and the target required frame rate can be that if the actual frame rate of the camera represents that an image is output every 50 milliseconds, and the target required frame rate represents that an image is output every 40 milliseconds, then the start time of the output image can be advanced by adjusting the frame synchronization signal so that the frame rate at which the camera outputs the acquired image can be synchronized with the target required frame rate.

[0101] S203. Send the frame synchronization signal to each camera to instruct each camera to output the acquired image according to the frame synchronization signal.

[0102] After obtaining the frame synchronization signal, the ISP controller can send the frame synchronization signal to each camera to instruct each camera to output the acquired image according to the frame synchronization signal.

[0103] In the image acquisition method provided by the embodiments of the present application, the actual frame rate of each camera in the multi-camera is obtained, and a frame synchronization signal is obtained according to the actual frame rate of each camera and the preset target required frame rate, and then the frame synchronization signal is sent to each camera to instruct each camera to output the acquired image according to the frame synchronization signal. In this method, by comparing and analyzing the actual frame rate output by each camera with the target required frame rate of the multi-camera, the frame synchronization signal is determined, so that the accuracy of the frame synchronization signal sent by the multi-camera to each camera is such that the actual frame rate at which each camera outputs the acquired image is as close as possible to the target required frame rate, so that the acquired images output by each camera are frame-synchronized.

[0104] In an exemplary embodiment, as Figure 3 shown, obtaining the actual frame rate of each camera in the multi-camera includes:

[0105] S301. Generate an initial frame synchronization signal according to the target required frame rate.

[0106] The target required frame rate represents the frame rate at which the ISP controller needs to capture images of the camera. The initial frame synchronization signal is used to control the time when the camera outputs the captured images. Therefore, the initial frame synchronization signal can be determined according to the target required frame rate.

[0107] Among them, the initial frame synchronization signal can be generated according to a preset frame rate generator. Specifically, input the target required frame rate into the frame rate generator, and through the analysis of the target required frame rate by the frame rate generator, output the initial frame synchronization signal.

[0108] S302. Send the initial frame synchronization signal to each camera according to the output mode of each camera, so as to instruct each camera to output the initial captured images according to the initial frame synchronization signal.

[0109] Among them, the output mode represents the way each camera outputs the captured images, including co-directional output and cross output.

[0110] Taking a multi-camera as a binocular camera as an example for illustration, as Figure 4 shown, Figure 4 shows a schematic diagram of each camera synchronously outputting the captured images, that is, two cameras simultaneously output the captured image data according to the received frame synchronization signal. In this output mode, the ISP controller simultaneously sends the initial frame synchronization signal to each camera.

[0111] As Figure 5 shown, Figure 5 shows a schematic diagram of each camera cross-outputting the captured images, that is, two cameras alternately output the captured images according to the received frame synchronization signal. In this output mode, the ISP controller can send the initial frame synchronization signal to each camera in turn with a phase difference of 50%.

[0112] S303. Determine the actual frame rate of each camera according to each initial captured image.

[0113] The ISP controller collects the initial captured images output by each camera according to the preset collection method, and then determines the actual frame rate of each camera according to the received initial captured images.

[0114] For any camera, determine the actual frame rate of the camera according to the time interval of the initial captured images output by the camera; the initial captured image can be any frame of the captured images output by the camera.

[0115] Optionally, a frame rate statistic measurement unit can be added to the ISP controller. The camera includes a sensor execution unit. The frame rate statistic measurement unit obtains the real-time actual frame rate through the frame synchronization signal output in real time by the sensor execution unit.

[0116] In the image acquisition method provided by the embodiment of the present application, an initial frame synchronization signal is generated according to the target required frame rate, and the initial frame synchronization signal is sent to each camera according to the output mode of each camera to instruct each camera to output an initial acquired image according to the initial frame synchronization signal, and then the actual frame rate of each camera is determined according to each initial acquired image. In this method, the initial frame synchronization signal is generated through the target required frame rate of the multi-camera, so that the initial frame synchronization signal can represent the target required frame rate, improving the frame synchronization effect of the acquired images of each camera.

[0117] In an exemplary embodiment, as Figure 6 shown, obtaining the frame synchronization signal according to the actual frame rate of each camera and the preset target required frame rate includes the following steps:

[0118] S601, for any camera, obtain the frame rate difference between the actual frame rate and the target required frame rate.

[0119] Determine the difference between the target required frame rate and the actual frame rate as the frame rate difference.

[0120] S602, analyze the frame rate difference through a preset frame rate adjustment algorithm to determine the control frame rate.

[0121] The frame rate adjustment algorithm can be a proportional, integral, derivative (PID) algorithm. As shown in formula (1).

[0122]

[0123] Among them, u(t) represents the control frame rate, K p 、K i and K d are an adjustment parameter, K p represents the proportional gain, K i represents the integral gain, K d represents the derivative gain, e represents the frame rate difference, and t represents time.

[0124] Specifically, K p The proportional control considers the current error, the error value and a positive constant K p(Indicating the ratio) Multiply. By establishing a linear function relationship between the deviation (the target required frame rate minus the actual frame rate) and the "adjustment strength" of the adjustment device, the most basic "proportional" control can be achieved. K p The larger the value of K, the more radical the adjustment effect. When K p is adjusted smaller, the adjustment effect will be more conservative.

[0125] K d Differential control takes into account future errors, calculates the first derivative of the error, and multiplies it by a positive constant K d . With the proportional effect, it is not difficult to find that the entire system is not particularly stable and always "jitters". Because when it is relatively close to the target required frame rate, the proportional control effect is relatively small. The closer it is to the target required frame rate, the gentler the proportional effect. There are many internal or external factors that cause small fluctuations in the control quantity. K d The larger the parameter, the larger the granularity of the stabilizing effect on the jitter.

[0126] K i Integral control takes into account past errors, multiplies the sum of the error values over a period of time by a positive constant K i . Set an integral value. As long as the deviation exists, continuously integrate and accumulate the deviation and reflect it in the adjustment strength. After reaching the target required frame rate, assuming there is no fluctuation in the deviation, the integral value will no longer change and remain stable. The larger the coefficient multiplied during integration, the more obvious the integration effect. Therefore, the role of integration is to reduce the error in the static situation and make the controlled physical quantity as close as possible to the target required frame rate.

[0127] Directly input the frame rate difference into the PID algorithm to obtain the control frame rate; among them, K p 、K i and K d can all determine the most suitable adjustment parameters through experimental analysis.

[0128] S603. Obtain the frame synchronization signal according to the control frame rate.

[0129] According to the signal generation algorithm, input the control frame rate into the signal generation algorithm to obtain the frame synchronization signal.

[0130] The ISP controller includes a frame synchronization controller. As Figure 7 shown, the frame synchronization controller includes a frame rate statistical measurement unit, a PLL clock latch, a frame rate generator, a frame rate adjustment module, and a frame rate controller.

[0131] Compare the actual frame rate output by the frame rate statistic measurement unit with the target required frame rate output by the frame rate generator. If there is a deviation between the target required frame rate and the actual frame rate, start the frame synchronization control frame rate adjustment module to perform PID proportional integral derivative adjustment on the frame synchronization signal output by the frame rate generator, and adjust the frame rate controller to near the target required frame rate. After several consecutive frames of PID adjustment, the synchronization between the actual frame rate output by the camera and the target required frame rate can be achieved quickly.

[0132] Specifically, the implementation principle of the frame synchronization controller includes: providing the unit of clock counting through the PLL clock latch: clock counting, providing the target required frame rate output by the frame rate generator, calculating the clock unit provided by the PLL, and sending out the high and low pulse widths of each frame synchronization signal as the target required frame rate synchronization control signal for the camera output. If there is a deviation between the time of collecting image data in the first row of each frame output by the camera and the input frame synchronization time, it is necessary to adjust through the frame synchronization controller. Through proportional, integral, and derivative adjustments, an output close to the target required frame rate is sent to each camera through the frame rate controller. The camera will gradually converge the synchronization deviation according to the received frame synchronization signal. After several rounds of adjustment, the camera can basically achieve input and output frame synchronization.

[0133] Let the frame synchronization control deviation of the camera (the target required frame rate minus the actual frame rate of the currently captured image) be related to the "adjustment strength" K of the adjustment device p , establish a linear function relationship. By analyzing the synchronization convergence time of a large number of cameras, take the time for transmitting one row of images as the basic unit of the PID adjustment strength, and obtain the best adjustment ratio parameter K through experimental analysis p , and the most basic frame synchronization "proportional" control can be achieved.

[0134] When the synchronization deviation δt between the frame synchronization signal output by the ISP controller of the SOC chip and the first row of mipidata of the currently captured frame output by the camera oscillates, calculate the first derivative of the deviation and multiply it by a constant K d . And obtain the best adjustment differential parameter K through experimental statistical analysis d , and the most basic frame synchronization "differential" control can be achieved.

[0135] When the synchronization deviation δt between the frame synchronization signal output by the SOC chip ISP controller and the first row of mipi data of the currently captured frame output by the camera persists, multiply the past period of the deviation value by a constant K i . Set an integral quantity. As long as the deviation exists, continuously integrate and accumulate the deviation and reflect it in the adjustment strength, and the most basic frame synchronization "integral" control can be achieved, and obtain the best adjustment integral parameter K through experimental analysisi .

[0136] Through the above method, the problem that the frame synchronization deviation δt between the first line of image data mipi data of each frame output by cameras of different models in slave mode and the frame synchronization signal output by the SOC chip is very different, and the camera convergence synchronization deviation δt speed is also different; the frame synchronization deviation δt exhibited by cameras of different models when the frame rate changes is also different; the frame synchronization deviation δt oscillations generated by cameras of different models during normal operation are also different.

[0137] In the image acquisition method provided in the embodiment of the present application, for any camera, the frame rate difference between the actual frame rate and the target required frame rate is obtained, and the frame rate difference is analyzed by a preset frame rate adjustment algorithm to determine the control frame rate, and then a frame synchronization signal is obtained according to the control frame rate. In this method, the deviation between the captured image output by each camera and the frame synchronization signal output by the ISP controller, as well as the oscillation deviation of the frame synchronization signal and the synchronization deviation of the dynamic frame change when the camera is running are accurately and dynamically adjusted through the frame rate adjustment algorithm.

[0138] When the camera changes the frame rate in the slave mode, there will be synchronization problems. For example, when switching from a high frame rate during the day to a low frame rate at night, or from a low frame rate at night to a high frame rate during the day, the camera will have a frame synchronization deviation δt, causing frame loss, screen distortion and other problems, affecting the image preview effect. Due to the different exposures during the day and at night, in order to increase the brightness of the night image, extend the exposure time, and reduce the image noise of night vision, the camera generally uses a low frame rate configuration at night and a high frame rate configuration during the day. For example, the camera outputs 30 frames during the day and 15 frames at night. Therefore, there is a need for dynamic switching of the frame rate configuration for multi-eye cameras. However, due to different synchronization deviations when switching the frame rates of different cameras, the synchronization deviation δt output by the camera after switching is relatively different, which will cause image frame loss, or output several consecutive bad frames after switching, resulting in a black screen or screen distortion. The main reason is that there is an oscillation in the synchronization deviation δt between the data output by the camera and the frame synchronization signal, and the convergence speed of the deviation of different cameras is also different. In addition, when the ISP controller outputs a frame synchronization signal that changes frames, or when the frame synchronization signal changes from the same direction output to the reverse direction, a synchronization deviation oscillation will be generated in the SOC chip.

[0139] Since multi-camera has the requirement of dynamic frame rate change and different frame rates are needed in different application scenarios, the frame rate controller needs to dynamically adjust the frame rate, and the ISP controller of the SOC chip needs to dynamically output the frame synchronization signal. In the traditional frame rate change method, the frame synchronization signal output needs to be turned off first, then the frame rate configuration is adjusted to generate the target required frame rate, and finally the frame synchronization signal output is turned on again to output the frame synchronization control signal of the target required frame rate. This process is time-consuming and directly affects the frame synchronization output of the camera.

[0140] In an exemplary embodiment, the embodiment includes: in response to a frame rate change request of a multi-camera, using the frame rate to be updated as the target required frame rate.

[0141] In this embodiment, when the multi-camera needs to adjust the target required frame rate, the target required frame rate in the frame rate generator is directly changed to the frame rate to be updated, without the need to turn off the output of the original frame synchronization signal. Only after detecting that the current frame synchronization signal output is completed at the original target required frame rate, the frame rate to be updated is immediately used as the target required frame rate.

[0142] Moreover, after the target required frame rate is updated, the frame synchronization signal can be obtained according to the updated target required frame rate and the actual frame rate output by the camera, and the frame synchronization signal is sent to each camera.

[0143] Optionally, as Figure 8 shown, a frame rate generation buffer, a frame rate refresh controller, a frame rate positive pulse counter, and a frame rate negative pulse counter can be added to the frame rate generator. When the ISP controller needs to change the target required frame rate, only the frame rate positive pulse controller, the frame rate negative pulse counter, and the frame rate positive pulse counter in the frame rate generator need to be directly changed to generate the new target required frame rate (the frame rate to be updated), without the need to turn off the output of the original frame synchronization signal. Then the frame rate refresh controller is started. Then, after detecting that the current frame synchronization signal output is completed at the original frame rate, the frame rate generation buffer immediately updates the frame rate to be updated configured in the buffer, that is, at most waiting for the time interval of one frame of the previous frame rate, the ISP controller can output to the camera according to the new target required frame rate (the frame rate to be updated), and there will be no problem of frame synchronization output oscillation. Then, the frame synchronization process between the frame synchronization signal and the mipi data output by the camera is restarted once. Since the PID adjustment is added in this process, the convergence speed of the frame synchronization deviation δt of the camera is significantly improved, that is, the synchronization of multi-camera dynamic frame rate change can be achieved in at most one to two frames of time.

[0144] In this embodiment, the PID algorithm is adopted to follow the deviation change of the oscillation, and the output time of the frame synchronization signal is adjusted in a timely manner, so as to adjust the synchronization deviation δt of the camera output at the fastest speed, accelerate the convergence speed of the camera, and achieve frame synchronization. In this way, it is easy to solve the problem of frame synchronization deviation oscillation that occurs when the frame rate of the frame synchronization signal output by the ISP controller of the SOC chip changes frames, or when it changes from in-phase to cross acquisition, as well as the problem of slow frame synchronization convergence speed of the camera.

[0145] In the image acquisition method provided by the embodiment of the present application, in response to the frame change request of the multi-camera, the to-be-updated frame rate is used as the target required frame rate. In this method, when the target required frame rate of the ISP controller needs to change, the to-be-updated frame rate is directly adjusted to the target required frame rate, solving the synchronization deviation problem caused by the untimely frame change of the fixed frame rate of the traditional SOC chip frame synchronization signal and the deviation convergence problem caused by the out-of-sync output of the camera.

[0146] In an exemplary embodiment, as Figure 9 shown, this embodiment includes the following steps:

[0147] S901, collect the acquisition images output by each camera according to the frame synchronization signal.

[0148] The ISP controller can collect the acquisition images output by each camera according to the frame synchronization signal according to the preset acquisition method.

[0149] In an exemplary embodiment, as Figure 10 shown, collecting the acquisition images output by each camera according to the frame synchronization signal includes the following steps:

[0150] S1001, when the acquisition frame rates of each camera are the same, if there is no phase difference between the frame synchronization signals of each camera, alternately collect the acquisition images output by each camera.

[0151] For a multi-camera, a single ISP controller needs to synchronously collect data from multiple cameras. For example, when the image sensor of each camera outputs 30 acquisition images per second. Since there is only 1 ISP controller for collection and its acquisition performance is also 30 frames per second, in order to simultaneously collect the acquisition image data of two cameras at the same resolution and keep the ISP controller collecting 30 acquisition image data per second, it is necessary to alternately collect the two cameras in turn, collect each camera at an interval of one frame, that is, only collect 15 frames of each camera, and add up to collect 30 acquisition image data of two cameras per second. The most important thing is that the frame synchronization signal of the ISP controller of the SOC chip synchronously outputs to control the frame synchronization of the two cameras.

[0152] Therefore, if there is no phase difference in the frame synchronization signals of each camera, that is, the frame synchronization signals of each camera are the same, it can be indicated that each camera captures image data in the same direction. In this way, the captured images output by each camera can be alternately captured.

[0153] Please continue to refer to Figure 4 , Figure 4 which gives a same-direction capture method. The ISP controller sequentially captures the captured images output by two cameras, and the captured images output by the un-captured cameras are discarded.

[0154] As Figure 11 shown, taking a multi-camera including two camera sensors sensor0 and sensor1 as an example, an image capture process is given, including the following steps:

[0155] S1101, the ISP controller enters the single-frame capture mode.

[0156] S1102, the frame rate generator in the ISP controller issues frame synchronization signals to sensor0 and sensor1 according to the target required frame rate, and the phase difference of the frame synchronization signals of sensor0 and sensor1 is 0%.

[0157] S1103, sensor0 and sensor1 output captured images according to the frame synchronization signals.

[0158] S1104, the ISP controller allocates memory (DMA) capture memory to the first sensor0 and starts the capture.

[0159] S1105, after the captured image output by sensor0 is completed, the ISP controller notifies the ISP controller to take away the captured image of the current frame.

[0160] Among them, the ISP controller generates an interrupt every 20 lines of data of the captured image output by sensor0, marks a timestamp, and generates a frame end interrupt after capturing a complete frame of captured image.

[0161] S1106, the ISP controller allocates direct memory access (Direct Memory Access, DMA) capture memory to the first sensor1 and starts the capture.

[0162] S1107, after the captured image output by sensor1 is completed, the ISP controller notifies the ISP controller to take away the captured image of the current frame and continues to execute step S1104.

[0163] Among them, the ISP controller generates an interruption after collecting 20 lines of data of the captured image output by sensor1, stamps a timestamp, and generates a frame end interruption after capturing a complete captured image.

[0164] S1108, the frame rate statistic counts the arrival time of the first line of the captured image, obtains the actual frame rate, compares the actual frame rate with the target required frame rate, calculates the frame rate difference, and adjusts the frame synchronization signal through the frame rate adjustment module to gradually converge the synchronization deviation.

[0165] S1002, if there is a phase difference between the frame synchronization signals of each camera, the captured images output by each camera according to the frame synchronization signal are collected in sequence.

[0166] For a multi-camera, a single ISP controller needs to synchronously collect data from multiple cameras. For example: when each camera outputs 15 frames per second. Since there is only 1 ISP controller for collection and its maximum collection performance is 30 frames per second, in order to achieve a frame rate of 30 frames, it is necessary to cross-collect the captured image data output by two cameras within the frame synchronization range of each frame, that is, the phase difference between the frame synchronization signals of the two cameras is 50%. Since each camera captures 15 frames, the ISP controller of the two cameras collects a total of 30 frames per second. The most important thing is that the phase of the frame synchronization signals output by the ISP controller of the SOC chip to the two cameras is opposite, and the phase difference is 50%. At the SOC chip end, the frame synchronization signal 0 of the ISP controller controls the frame synchronization of sensor0, and the frame synchronization signal 1 controls the frame synchronization of sensor1. By adjusting the phase difference between the frame synchronization signals of the two cameras, the ISP control phase difference of the external SOC chip can be adjusted to 0% and 50%, just enabling co-directional and reverse cross-collection.

[0167] Please continue to refer to Figure 5 , Figure 5 A reverse cross-collection method is given. The ISP controller sequentially collects the captured images output by two cameras; among them, the collection process is basically the same as that of co-directional collection, and reference can be made to Figure 4 ; among them, the difference between the collection processes of co-directional collection and reverse cross-collection is that in co-directional collection, the captured images output by each camera are collected every other frame, while in reverse cross-collection, the captured images output by the next camera are collected through a 50% deviation of the frame synchronization signals of the two cameras.

[0168] S902, for any camera, obtain the collection time interval between the captured image output by the camera according to the frame synchronization signal and the previous captured image.

[0169] After the ISP controller acquires the captured image output by the camera, it can determine the capture time interval based on the time of the currently captured image and the time of the previous captured image; the difference between the two times is determined as the time interval.

[0170] S903, if the capture time interval is inconsistent with the preset frame rate interval time, discard the captured image of the camera.

[0171] If the capture time interval is inconsistent with the preset frame rate interval time, it means that the captured image output by the camera acquired by the ISP controller does not meet the user's frame interval timestamp requirement, then discard the captured image output by the obtained camera according to the frame synchronization signal.

[0172] In the image acquisition method provided by the embodiments of the present application, capture the captured images output by each camera according to the frame synchronization signal; for any camera, obtain the capture time interval between the captured image output by the camera according to the frame synchronization signal and the previous captured image. If the capture time interval is inconsistent with the preset frame rate interval time, discard the captured image of the camera. In this method, discarding the images with inconsistent frame rate time intervals can ensure that the captured images meet the frame rate requirements of the ISP controller, thereby improving the frame synchronization effect of the captured images of each camera.

[0173] In an exemplary embodiment, taking a multi-camera as a three-camera as an example for illustration, please continue to refer to Figure 1e , by outputting high and low levels through the GPIO pins of the SOC chip to control the channel switching of the MIPI switch chip, the image data of camera 1 or camera 2 can be selected for capture. The capture mode is still divided into high frame rate forward capture and low frame rate reverse cross capture of the camera. For the capture of a multi-camera, frame synchronization control still needs to be performed, and the dynamic frame rate synchronization control adjustment of PID is added, which can well achieve frame synchronization control.

[0174] As Figure 12 shown, Figure 12 is a schematic diagram of the synchronous capture process of a three-camera. The first frame image captured by the ISP controller is the captured image output by camera 0, the second frame image captured by the ISP controller is the captured image output by camera 1, the third frame image captured by the ISP controller is the captured image output by camera 2, and the fourth frame image captured by the ISP controller is the captured image output by camera 0.

[0175] As Figure 13 shown, Figure 13Schematic diagram of the reverse cross-acquisition process for a three-eye camera. The first frame of image collected by the ISP controller is the acquisition image output by camera 0, the second frame of image collected by the ISP controller is the acquisition image output by camera 1, the third frame of image collected by the ISP controller is the acquisition image output by camera 0, and the fourth frame of image collected by the ISP controller is the acquisition image output by camera 2.

[0176] In an exemplary embodiment, as Figure 14 shown, the steps for collecting the acquisition images output by each camera according to the frame synchronization signal include the following:

[0177] S1401. When the acquisition frame rates of each camera are inconsistent, determine the acquisition ratio between each camera according to the acquisition frame rates of each camera.

[0178] Determine the ratio of the acquisition frame rates of each camera as the acquisition ratio between each camera. The image sensor of the camera can be a CMOS image sensor, where CMOS can represent Complementary Metal Oxide Semiconductor.

[0179] Taking the binocular dynamic frame change as an example, first, the frame rate change control of the camera needs to be completed to achieve the frame rate change of the camera and the ISP control. Finally, configure the frame rate ratio of each camera according to the acquisition frame rate requirements of the ISP controller. For example, if camera 0 requires 15 fps and camera 1 requires 10 fps, the driver layer realizes the acquisition according to an alternating ratio of 3:2 (3 frames of camera 0 and 2 frames of camera 1). It is required that the frame rate interval of the driver layer when allocating the acquisition of 2 cameras is as uniform as possible to meet the human eye's sense of smoothness. If the frame rate ratio of the two cameras for acquisition is set as m:n, as Figure 15 shown, Figure 15 it is the frame rate ratio control of the binocular camera.

[0180] Taking a three-eye camera as an example, when the desired acquisition frame rates of the three cameras are configured by the driver according to the frame rate requirements for the proportion of frame rate acquisition. First, the frame rate change control of the camera needs to be completed to achieve the frame rate change of the camera and the ISP controller. Finally, the proportion of frame rate acquisition for each camera is configured according to the frame rate requirements. For example, camera 0 requires 15 fps (assumed to be directly input through the MIPI0 interface), camera 1 requires 10 fps (assumed to be input through the MIPI1 interface via the MIPI switch chip), and camera 2 requires 10 fps (assumed to be input through the MIPI1 interface via the MIPI switch chip). The driver layer realizes the acquisition according to the alternating ratio of 3:2 (3 frames of camera 0, 2 frames of camera 1 or camera 2 or 1 frame of camera 1 and 1 frame of camera 2). It is required that the frame rate interval of the driver layer during the allocation of two-channel acquisition is as uniform as possible to conform to the human eye's sensory perception. Support for the setting of the frame rate ratio of three-channel CIS acquisition is as Figure 16 shown Figure 16 That is, the acquisition ratio of the three cameras is m:n:0 or m:0:n; it should be noted that the acquisition ratios of camera 1 and camera 2 in the figure can be interchanged, that is, camera 1:camera 2 can be n:0 or 0:n.

[0181] Optionally, when the driver layer acquires images according to the alternating ratio of m:n, the acquisition ratio of the three cameras can also be m:k:(n-k), that is, the acquisition ratio of camera 1 and camera 2 can be k:(n-k).

[0182] S1402. Determine the actual required frame rates of each camera according to the acquisition ratio and the target required frame rate.

[0183] Among them, the actual required frame rate can represent the frame rate that each camera actually needs to acquire.

[0184] In an exemplary embodiment, as Figure 17 shown, determining the actual required frame rates of each camera according to the acquisition ratio and the target required frame rate includes the following steps:

[0185] S1701. Determine the candidate required frame rates of each camera according to the acquisition ratio and the target required frame rate.

[0186] The candidate required frame rates of each camera can be determined by performing equal-ratio calculation according to the acquisition ratio between each camera and the target required frame rate of the multi-eye camera.

[0187] Taking two cameras as an example, if the acquisition ratio of camera 0 and camera 1 is 3:1 and the target required frame rate of the binocular camera is to acquire 4 frames of acquisition images per second, then the candidate required frame rate of camera 0 can be determined to be 3 frames per second, and the candidate required frame rate of camera 1 can be determined to be 1 frame per second.

[0188] S1702. For any camera, if the candidate required frame rate of the camera is greater than the acquisition frame rate, then determine the acquisition frame rate as the actual required frame rate of the camera.

[0189] If the candidate required frame rate of the camera is greater than the acquisition frame rate, it means that the frame rate requirement applied for by the multi-camera exceeds the effective frame rate range currently output by this camera. Then replace the acquisition frame rate with the actual required frame rate of the camera, that is, determine the acquisition frame rate as the actual acquisition frame rate of the camera.

[0190] S1703. If the candidate required frame rate of the camera is less than or equal to the acquisition frame rate, then determine the candidate required frame rate as the actual required frame rate of the camera.

[0191] If the candidate required frame rate of the camera is less than or equal to the acquisition frame rate of this camera, it means that the number of frames collected by the ISP controller per second at the actual acquisition frame rate applied for by the user is more than the frame rate requirement applied for by the user. Then collect images at the actual acquisition frame rate of the camera. Additionally, if the frame data collected by the ISP controller does not meet the user's frame interval timestamp requirement, discard it.

[0192] In this embodiment, according to the acquisition ratio and the target required frame rate, determine the candidate required frame rate of each camera; for any camera, if the candidate required frame rate of the camera is greater than the acquisition frame rate, then determine the acquisition frame rate as the actual required frame rate of the camera; if the candidate required frame rate of the camera is less than or equal to the acquisition frame rate, then determine the candidate required frame rate as the actual required frame rate of the camera. In this method, by comparing the magnitude relationship between the candidate required frame rate of the camera and the actual acquisition frame rate, determine the actual required frame rate of the camera, thereby making the actual required frame rate of the camera more accurate.

[0193] S1403. According to the actual required frame rate of each camera, determine the acquisition order of each camera.

[0194] Calculate according to the actual required frame rate of each camera through a preset weighted algorithm to determine the acquisition order of each camera.

[0195] Specifically, in one embodiment, determining the acquisition order of each camera according to the actual required frame rate of each camera includes: determining the acquisition weight of each camera according to the actual required frame rate of each camera; determining the acquisition order of each camera according to the acquisition weight of each camera.

[0196] It is possible to determine the actual acquisition ratio of each camera according to the actual required frame rate of each camera, then use the actual acquisition ratio of each camera as the acquisition weight of each camera, and then determine the acquisition order of each camera according to the relationship between the acquisition weights of each camera.

[0197] Taking a binocular camera as an example, the actual acquisition ratio of camera 0 and camera 1 is m:n = 7:9. Then the acquisition weight of camera 0 is 7, and the acquisition weight of camera 1 is 9. The acquisition period of the binocular camera is 16 frames. Through the weight algorithm process, the acquisition period ratio process can be as shown in Table 1.

[0198] Table 1

[0199]

[0200]

[0201] After a collection cycle is completed, it repeats and enters the next sampling cycle, still following 7:9. However, for each frame of data sent to the upper layer, it must be uploaded according to the actual required frame rate interval time of the channel corresponding to the frame rate requested by the upper layer user. For example: the frame rate of camera 0 channel is 7 frames, and the frame rate of camera 1 channel is 9 frames. Then, the frame interval time of camera 0 channel is 142 ms, and the frame interval time of camera 1 channel is 111 ms. If the current frame timestamp is less than the channel interval time requirement, the data collected this time will be discarded. Then it keeps collecting cyclically like this, and the frame rate ratio control and visual fluency are ensured according to the weight ratio of the weighted algorithm.

[0202] Optionally, taking a binocular camera as an example, if the acquisition weight of camera 0 is K1 and the acquisition weight of camera 1 is K2, and the initial default value of the global variable is 0, the ISP controller determines which camera's output acquisition image to collect by comparing the relationship between the global variable and the acquisition weight K1.

[0203] Specifically, in the process of determining the acquisition order, the global variable can be compared with K1. When the global variable is less than K1, it is determined that camera 0 is the camera for collecting the image, and the sum of the global variable and K2 is used as the new global variable; when the global variable is greater than or equal to K1, it is determined that camera 1 is the camera for collecting the image, and the difference between the global variable and K1 is used as the new global variable; according to this principle, the acquisition order of each camera can be obtained.

[0204] Taking the first 3 frames in a collection cycle as an example for illustration, the collection process is as follows:

[0205] S1, when the initial global variable 0 is less than K1, it is determined that the first frame collected is the image of camera 0, and then the global variable is updated (the sum of the initial global variable 0 and the acquisition weight K2 of camera 1 is used as the new global variable).

[0206] S2. When the global variable is greater than or equal to K1, determine Camera 1 as the camera for the second acquisition, and then update the global variable (the difference between the global variable and the acquisition weight of Camera 0 being K1 is used as the new global variable). When the global variable is less than K1, determine Camera 0 as the camera for the second acquisition, and then update the global variable (the sum of the global variable and the acquisition weight of Camera 1 being K2 is used as the new global variable).

[0207] S3. When the global variable is greater than or equal to K1, determine Camera 1 as the camera for the third acquisition, and then update the global variable (the difference between the global variable and the acquisition weight of Camera 0 being K1 is used as the new global variable). When the global variable is less than K1, determine Camera 0 as the camera for the third acquisition, and then update the global variable (the sum of the global variable and the acquisition weight of Camera 1 being K2 is used as the new global variable).

[0208] In this embodiment, a weighted algorithm is adopted for the dynamic frame rate ratio control of the multi-camera. Since the multi-camera will configure arbitrary frame rate ratios for multiple cameras according to the application scenario requirements, in order to make the frame rate intervals collected by the ISP controller as uniform as possible, which conforms to the smooth feeling of the human eye.

[0209] S1404. Collect the acquisition images output by each camera according to the frame synchronization signal in the acquisition order.

[0210] Based on the obtained acquisition order above, collect the acquisition images output by each camera according to the frame synchronization signal in the acquisition order.

[0211] In the image acquisition method provided by the embodiment of the present application, when the acquisition frame rates of each camera are inconsistent, according to the acquisition frame rates of each camera, determine the acquisition ratio between each camera, and according to the acquisition ratio and the target required frame rate, determine the actual required frame rate of each camera. Then, according to the actual required frame rate of each camera, determine the acquisition order of each camera, and finally collect the acquisition images output by each camera according to the frame synchronization signal in the acquisition order. In this method, according to the acquisition ratio between each camera, determine the actual required frame rate of each camera, and further determine the acquisition order of each camera according to the actual required frame rate of each camera, which realizes the different frame rate ratio control requirements of each camera in the multi-camera, meets the application requirements in different scenarios, and solves the frame rate variable frame control and frame rate ratio control of the multi-camera.

[0212] Based on the variable frame rate ratio control of the above multi-camera, first, when there are frame rate change requests for multiple cameras, they are converted into actual frame rate ratio requirements at the acquisition end of the ISP controller. Taking a binocular camera as an example, for instance: the user requests that the frame rate of camera 0 is 5fps and the frame rate of camera 1 is 15fps. Then the frame rate ratio of the two cameras is 5fps / 15fps = 1 / 3.

[0213] Secondly, the ISP controller in the driver layer acquires data according to the actual acquisition frame rate ratio of each camera. For example: the frame rate ratio of camera 0 and camera 1 is 5fps / 15fps = 1 / 3. Then the driver layer will achieve frame synchronization and alternate acquisition according to a ratio of 1:3 (1 frame of camera 0, 3 frames of camera 1). After the acquisition is completed, it is distributed to different channels according to the actual frame rate required by the user; finally, according to the actual frame rate of each camera channel, the redundant frame rate data is discarded and passed to the upper-layer user.

[0214] For the synchronous acquisition method, taking a binocular camera as an example, when the maximum frame rate output by camera 0 and camera 1 is 30 frames, the maximum frame rate acquired by the ISP controller is 30 frames, and the maximum frame rate required for each acquisition can only be 30 frames. For example: if the multi-channel frame rate configuration requirements input by the user layer are that camera 0 acquires 1 frame and camera 1 acquires 29 frames, then the frame rate ratio of camera 0 and camera 1 is 1:29. That is to say, after camera 0 finishes acquiring 1 frame, camera 1 then continuously acquires 29 frames, and then this acquisition process is repeated periodically. After each channel is completed, the acquired image data will be passed to the user layer according to the actual frame rate requirements of the user layer.

[0215] For the reverse cross-acquisition method, taking a binocular camera as an example, when the maximum frame rate output by camera 0 and camera 1 is 15 frames, the maximum frame rate that the ISP controller can acquire is 30 frames, and the maximum frame rate for each camera to acquire can only be 15 frames. For example: if the multi-channel frame rate configuration requirements input by the user layer are that camera 0 acquires 5 frames and camera 1 acquires 15 frames, then the frame rate ratio of camera 0 and camera 1 is 5:15 = 1:3. That is to say, after camera 0 finishes acquiring 1 frame, camera 1 then continuously acquires 3 frames, and then this acquisition process is repeated periodically. After each channel is completed, the acquired image data will be passed to the user according to the actual frame rate requirements of the user layer.

[0216] In this application, a low-cost solution that can quickly implement the multi-camera function is achieved. That is, on the basis of a traditional single-camera, by increasing the number of MIPI channels of the camera or building a MIPI switch solution peripherally, a single ISP controller is used to multiplex and acquire the acquired image data of the cameras on multiple MIPI channels, realizing the multi-camera function, reducing the cost and workload of implementing the multi-camera product, and reducing the power consumption of the product.

[0217] The image acquisition method in the embodiment of the present application is compatible with cameras of multiple brands and multiple models, supports the slave mode output by the camera and the master mode controlled by pwdn input of the camera, basically realizes the synchronous acquisition of image data of multiple cameras of a multi-camera, realizes the product functions of a multi-camera, is compatible with the product requirements of cameras of multiple models and the output modes of multiple cameras, and can solve and be compatible with problems such as frame synchronization deviation and deviation convergence speed of different models of cameras in various scenarios.

[0218] The image acquisition method in the embodiment of the present application adopts a PID proportional integral derivative algorithm regulator and a frame rate statistical unit to accurately and dynamically adjust the deviation between the acquired image data output by the camera and the frame synchronization signal output by the ISP controller of the SOC chip, as well as the frame synchronization oscillation deviation and dynamic variable frame synchronization deviation during the operation of the camera, and adjusts the synchronization deviation convergence speed of the camera output to achieve the effect of fast synchronization, and realizes the stable frame rate of image data acquisition by the ISP controller, no frame loss, and the image preview process.

[0219] The image acquisition method in the embodiment of the present application adopts a newly designed frame rate generator, which adds a frame rate generation buffer and a frame rate refresh controller to the traditional fixed frame rate synchronization signal, and solves the synchronization deviation problem caused by the untimely variable frame of the fixed frame rate of the frame synchronization signal on the traditional SOC chip and the deviation convergence problem caused by the asynchronous output of the camera.

[0220] The image acquisition method in the embodiment of the present application adopts the latest frame rate generator to realize the requirement of dynamic frame rate variable frame control of a multi-camera, and designs a weighting algorithm to realize different frame rate ratio control requirements on different camera channels, meet the application requirements in different scenarios, and solve the frame rate variable frame control and frame rate ratio control of a multi-camera; the embodiment of the present application can solve problems such as multi-camera multi-camera extended frame synchronization and dynamic variable frame control and frame rate ratio control of a multi-camera in different scenarios.

[0221] In an exemplary embodiment, the embodiment of the present application provides an image acquisition method, as Figure 18 shown, the embodiment includes the following steps:

[0222] S1801, according to the target required frame rate, the frame rate generator generates an initial frame synchronization signal.

[0223] S1802, send the initial frame synchronization signal to each camera to instruct each camera to output an initial acquired image according to the initial frame synchronization signal.

[0224] S1803, collect the initial captured images output by each camera according to a preset collection method, and determine the actual frame rate of each camera based on each initial captured image.

[0225] Among them, the collection method includes the order of collecting the captured images output by each camera.

[0226] S1804, obtain a frame synchronization signal based on the actual frame rate of each camera and the target required frame rate.

[0227] S1805, send the frame synchronization signal to each camera to instruct each camera to output the captured images according to the frame synchronization signal.

[0228] It should be understood that although each step in the flowcharts involved in the above-described embodiments is shown in sequence according to the indication of the arrows, these steps do not necessarily need to be executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages do not necessarily need to be executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or steps in other steps.

[0229] Based on the same inventive concept, the embodiments of the present application also provide an image acquisition device for implementing the above-mentioned image acquisition method. The implementation solutions provided by this device to solve problems are similar to the implementation solutions described in the above method. Therefore, the specific limitations in one or more embodiments of the following image acquisition devices can refer to the limitations on the image acquisition method in the above text, and will not be repeated here.

[0230] In an exemplary embodiment, as Figure 19 shown, an image acquisition device 1900 is provided, including: a frame rate acquisition module 1901, a signal acquisition module 1902, and a signal sending module 1903, where:

[0231] The frame rate acquisition module 1901 is used to acquire the actual frame rate of each camera in the multi-camera.

[0232] The signal acquisition module 1902 is used to obtain a frame synchronization signal according to the actual frame rate of each camera and the preset target required frame rate.

[0233] The signal sending module 1903 is used to send the frame synchronization signal to each camera to instruct each camera to output the captured images according to the frame synchronization signal.

[0234] In one embodiment, the frame rate acquisition module 1901 includes:

[0235] A generation unit, configured to generate an initial frame synchronization signal according to a target required frame rate;

[0236] A signal distribution unit, configured to distribute the initial frame synchronization signal to each camera according to the output mode of each camera, so as to instruct each camera to output an initial captured image according to the initial frame synchronization signal;

[0237] A first determination unit, configured to determine the actual frame rate of each camera according to each initial captured image.

[0238] In one embodiment, the signal acquisition module 1902 includes:

[0239] An acquisition unit, configured to acquire a frame rate difference between the actual frame rate and the target required frame rate for any one camera;

[0240] An analysis unit, configured to analyze the frame rate difference through a preset frame rate adjustment algorithm to determine a control frame rate;

[0241] A obtaining unit, configured to obtain a frame synchronization signal according to the control frame rate.

[0242] In one embodiment, the apparatus 1900 further includes:

[0243] An update module, configured to use the frame rate to be updated as the target required frame rate in response to a frame rate change request of a multi-camera.

[0244] In one embodiment, the apparatus 1900 further includes:

[0245] An acquisition module, configured to acquire captured images output by each camera according to the frame synchronization signal;

[0246] A time interval acquisition module, configured to acquire an acquisition time interval between a captured image output by a camera according to the frame synchronization signal and the previous captured image for any one camera;

[0247] A discard module, configured to discard the captured image of the camera if the acquisition time interval is inconsistent with a preset frame rate interval time.

[0248] In one embodiment, the acquisition module includes:

[0249] A first acquisition unit, configured to alternately acquire captured images output by each camera if the acquisition frame rates of each camera are the same and there is no phase difference between the frame synchronization signals of each camera;

[0250] A second acquisition unit, configured to sequentially acquire captured images output by each camera according to the frame synchronization signal if there is a phase difference between the frame synchronization signals of each camera.

[0251] In one embodiment, the acquisition module includes:

[0252] A second determination unit, configured to determine an acquisition ratio between each camera according to the acquisition frame rates of each camera when the acquisition frame rates of each camera are inconsistent;

[0253] A third determination unit, configured to determine the actual required frame rate of each camera according to the acquisition ratio and the target required frame rate;

[0254] A fourth determination unit, configured to determine the acquisition order of each camera according to the actual required frame rate of each camera;

[0255] A third acquisition unit, configured to acquire the acquisition images output by each camera according to the frame synchronization signal in the acquisition order.

[0256] In one embodiment, the third determination unit includes:

[0257] A first determination subunit, configured to determine the candidate required frame rate of each camera according to the acquisition ratio and the target required frame rate;

[0258] A second determination subunit, configured to, for any camera, if the candidate required frame rate of the camera is greater than the acquisition frame rate, determine the acquisition frame rate as the actual required frame rate of the camera;

[0259] A third determination subunit, configured to, if the candidate required frame rate of the camera is less than or equal to the acquisition frame rate, determine the candidate required frame rate as the actual required frame rate of the camera.

[0260] In one embodiment, the fourth determination unit includes:

[0261] A fourth determination subunit, configured to determine the acquisition weight of each camera according to the actual required frame rate of each camera;

[0262] A fifth determination subunit, configured to determine the acquisition order of each camera according to the acquisition weight of each camera.

[0263] Each module in the above image acquisition device can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in the processor in the computer device in hardware form or be independent of it, or can be stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to each of the above modules.

[0264] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as Figure 20As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image acquisition data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through a network connection. The computer program, when executed by the processor, implements an image acquisition method.

[0265] Those skilled in the art can understand that Figure 20 the structure shown in is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0266] In one embodiment, a computer device is further provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0267] For each step implemented by the processor in this embodiment, its implementation principle and technical effect are similar to those of the above image acquisition method, and will not be elaborated here.

[0268] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0269] For each step implemented when the computer program in this embodiment is executed by the processor, its implementation principle and technical effect are similar to those of the above image acquisition method, and will not be elaborated here.

[0270] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0271] For each step implemented when the computer program in this embodiment is executed by the processor, its implementation principle and technical effect are similar to those of the above image acquisition method, and will not be elaborated here.

[0272] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0273] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., and are not limited thereto. The processors involved in the embodiments provided in this application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., and are not limited thereto.

[0274] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0275] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation to the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. An image acquisition method, characterized in that, The method includes: Obtaining the actual frame rate of each camera in the multi-camera; Obtaining a frame synchronization signal according to the actual frame rate of each camera and a preset target required frame rate; Sending the frame synchronization signal to each camera to instruct each camera to output a captured image according to the frame synchronization signal.

2. The method according to claim 1, wherein The obtaining the actual frame rate of each camera in the multi-camera includes: Generating an initial frame synchronization signal according to the target required frame rate; Sending the initial frame synchronization signal to each camera according to the output mode of each camera to instruct each camera to output an initial captured image according to the initial frame synchronization signal; Determining the actual frame rate of each camera according to each initial captured image.

3. The method according to claim 1 or 2, characterized in that, The obtaining the frame synchronization signal according to the actual frame rate of each camera and a preset target required frame rate includes: For any camera, obtaining the frame rate difference between the actual frame rate and the target required frame rate; Analyzing the frame rate difference through a preset frame rate adjustment algorithm to determine a control frame rate; Obtaining the frame synchronization signal according to the control frame rate.

4. The method according to claim 1 or 2, characterized in that, The method further includes: In response to a frame change request of the multi-camera, using the to-be-updated frame rate as the target required frame rate.

5. The method according to claim 1 or 2, characterized in that, The method further includes: Capturing the captured images output by each camera according to the frame synchronization signal; For any camera, obtaining the capture time interval between the captured image output by the camera according to the frame synchronization signal and the previous captured image; If the capture time interval is inconsistent with a preset frame rate interval time, discarding the captured image of the camera.

6. The method according to claim 5, wherein The capturing the captured images output by each camera according to the frame synchronization signal includes: When the capture frame rates of all cameras are the same, if there is no phase difference between the frame synchronization signals of all cameras, alternately capturing the captured images output by all cameras; If there is a phase difference between the frame synchronization signals of all cameras, sequentially capturing the captured images output by all cameras according to the frame synchronization signal.

7. The method according to claim 5, characterized in that, The capturing the captured images output by each camera according to the frame synchronization signal includes: When the capture frame rates of all cameras are different, determining the capture ratio between all cameras according to the capture frame rate of each camera; Determining the actual required frame rate of each camera according to the capture ratio and the target required frame rate; Determining the capture order of each camera according to the actual required frame rate of each camera; Capturing the captured images output by each camera according to the frame synchronization signal in accordance with the capture order.

8. The method according to claim 7, wherein The determining the actual required frame rate of each camera according to the capture ratio and the target required frame rate includes: Determining the candidate required frame rate of each camera according to the capture ratio and the target required frame rate; For any camera, if the candidate required frame rate of the camera is greater than the capture frame rate, determining the capture frame rate as the actual required frame rate of the camera; If the candidate required frame rate of the camera is less than or equal to the acquisition frame rate, the candidate required frame rate is determined as the actual required frame rate of the camera.

9. The method according to claim 7, wherein The determining the acquisition order of each camera according to the actual required frame rate of each camera includes: Determining the acquisition weight of each camera according to the actual required frame rate of each camera; Determining the acquisition order of each camera according to the acquisition weight of each camera.

10. An image acquisition device, characterized in that, The device includes: A frame rate acquisition module, configured to acquire the actual frame rate of each camera in a multi-camera; A signal acquisition module, configured to acquire a frame synchronization signal according to the actual frame rate of each camera and a preset target required frame rate; A signal sending module, configured to send the frame synchronization signal to each camera to instruct each camera to output an acquired image according to the frame synchronization signal.

11. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images