Low-power-consumption camera high-frame-rate small image rapid convergence method

Abstract

The invention provides a high-frame-rate small image rapid convergence method for a low-power-consumption camera. The method comprises the steps that S1, a main processor is started; s2, initializing and running the coprocessor, and running the starting of the system in the step S3 and the convergence processing of the coprocessor in the step S5 in parallel; s3, the main processor initializes the system to start other necessary resources; s4, the main processor sends a stop command to the coprocessor, and the step S6 is executed after the stop command is completed; and S5, convergence processing of the coprocessor, further comprising the following steps: S5.1, the coprocessor starts to operate; s5.2, a sensor is configured, and a high-frame-rate small image mode is used; s5.3, an AE module, an AWB module and a BLC module of the ISP are initialized; s5.4, AE and AWB convergence is carried out in a high-frame-rate small image mode; s5.5, receiving a stop command sent by the main processor, and stopping the ISP and the Sensor from working; s6, synchronizing a stop state, synchronizing a convergence result to the ISP, and converting the convergence result into a large graph parameter; s7, switching the Sensor to a normal frame rate large image mode; and S8, entering a large graph normal process. Hardware cost is not increased, and it is ensured that the first image effect does not shake.

Description

Technical Field

[0001] The invention belongs to the technical field of video image processing, and in particular relates to a method for fast convergence of small images with high frame rate for a low-power camera. Background Art

[0002] In the prior art, in order to reduce power consumption, the camera processor of low-power camera products is completely powered off, so each startup is a cold start. During the startup process, according to the principle of ISP, the image screen has a process of changing from dark to bright, which is called AE convergence process. If no special treatment is done, there will inevitably be such a convergence process, which is extremely bad for user experience. Therefore, low-power cameras have very high requirements for the first frame image effect.

[0003] In order to shorten this process, most practices rely on hardware photoresistors to feedback an initial ambient light state, and the ISP converges based on the value of the photoresistor as the AE convergence starting point.

[0004] However, the sensitivity range of photoresistors is usually small, and the angle of light exposure has a great influence on the resistance value. Therefore, in many scenarios, the feedback of ambient light is inaccurate, resulting in an inaccurate AE convergence starting point, resulting in the first frame image being too bright or too dark, resulting in a poor user experience. Figures 1 to 3 As shown in the figure, the conventional solution of the prior art does not introduce a high frame rate small picture, and only relies on the feedback of the photoresistor as its actual parameter. In the convergence process in certain scenarios, such as when the light brightness exceeds the photosensitivity range of the photosensitive component, the brightness of the first frame image will be inaccurate. Figure 1 As the first picture, Figure 2 For the second frame, Figure 3 This is the third frame.

[0005] In addition, common terminology explanations in the prior art include:

[0006] IPC: IP Camera, network camera.

[0007] Low-power camera: A camera that has requirements for power consumption, is generally battery-powered, and requires a quick startup function.

[0008] Fast startup: The entire system starts up within 500ms from power-on to acquiring the first image.

[0009] High frame rate small picture: The frame rate is higher than the normal frame rate, and the small picture is of small resolution. For example, the pixel and frame rate of a general camera are 1080P@15fps, while the high frame rate small picture can be 360P@120fps. ISP: Image Signal Processing, image signal processing unit.

[0010] AE: Automatic Exposure, automatic exposure.

[0011] AWB: Automatic white balance, automatic white balance.

[0012] BLC: Black Level Correction, black level correction.

[0013] Power off: Completely cut off the power supply.

[0014] Cold start: Cut off the power supply and restart.

[0015] First frame image: The first frame image that can be obtained after the system is cold started.

[0016] CMOS Sensor: Image sensor.

[0017] Binning: An image readout mode that adds the charges induced by adjacent pixels together and reads them out in a one-pixel mode. For example, 1080P is output as 360P by merging 4 pixels into 1 pixel, so that there is no loss in field of view.

[0018] FPS: Frames Per Second, the number of frames transmitted per second.

[0019] Coprocessor: A processor developed and applied to assist the central processing unit in completing processing tasks that the central processing unit cannot perform or performs inefficiently or ineffectively. Summary of the invention

[0020] In order to solve the above problems, the purpose of this application is to address the defects of the supporting solution based on photoresistors for low-power cameras. A new method based on the cooperation of CMOS Sensor and ISP, namely the high frame rate small picture method, is introduced. This method can solve the defects of photoresistors, and in actual use, it does not rely on photoresistors. Without increasing the hardware cost, it can achieve the first picture effect more accurately.

[0021] Specifically, the present invention proposes a method for fast convergence of a small image with a high frame rate of a low-power camera, the method comprising the following steps:

[0022] S1, main processor power on;

[0023] S2, initialize and run the coprocessor, and run the system startup of step S3 and the convergence processing of the coprocessor of step S5 in parallel;

[0024] S3, the main processor initializes other resources necessary for system startup, wherein the other resources include normal startup resources of the system, including kernel firmware loading, file system loading, kernel firmware decompression, file system decompression, kernel initialization, and driver registration;

[0025] S4, the main processor sends a stop command to the coprocessor, and after completion, goes to step S6;

[0026] S5, convergence processing of the coprocessor further includes:

[0027] S5.1, the coprocessor starts running;

[0028] S5.2, configure the sensor Sensor and use the high frame rate small picture mode. The high frame rate small picture mode requires the support of the sensor and the ISP. The specific implementation principle of the high frame rate small picture mode is to use the binning method of the CMOS Sensor, which is consistent with the method in step S5.4. Step S5.2 mainly describes the configuration of the sensor, and S5.4 mainly describes the processing of the ISP. The sensor first generates a high frame rate small picture, and then switches to a large picture with a normal frame rate.

[0029] S5.3, initialize AE, AWB, and BLC modules of ISP;

[0030] S5.4, AE and AWB convergence is performed in a high frame rate small image mode. The binning method of the CMOS Sensor is used. Assuming a resolution of 1080P, it can be binned to a small image of 640*360, so that there is no loss of field of view and the maximum frame rate can reach 120fps; under the condition of 120fps, 24 frames of images can be converged in 200ms, so that a converged and stable image can be obtained in 300ms; like the high frame rate small image method in step S5.2, the Sensor and ISP need to cooperate to realize the high frame rate small image method;

[0031] S5.5, receiving the stop command sent by the main processor, stopping the ISP and Sensor work;

[0032] S6, synchronization stop state, synchronize the convergence results to ISP, convert to large image parameters, when the actual product is displaying images, it is still assumed to be displayed in 1080P full size. Since the high frame rate uses small images including 640*360 for processing, after the processing is completed, it is necessary to convert from small images to full-size large images to continue displaying to ensure the stability of the image effect;

[0033] S7, the Sensor switches to the normal frame rate large image mode, that is, it switches from a small size resolution to a large resolution, and then continues to output images;

[0034] S8, enter the normal process of the large picture.

[0035] In step S4, a MessageBox register is provided for communication between the dual cores, i.e., the main processor and the coprocessor. The stop command is to write 0x1 into this register. When the small core reads the value of this register as 0x1, it is a stop command. When jumping to step S6, when the small core reads the stop command, it will write 0x0 into the messagebox register, thus informing the main CPU that it has stopped working normally.

[0036] The parameters of the sensor configured in step S5.2 are assumed to be the sensor configuration of model os02k10, including:

[0037] When the pixel and frame rate are 2M@15fps, set to:

[0038] .width=1920,

[0039] .height=1080,

[0040] .fps=15<<16|1,

[0041] .mbus_code=V4L2_MBUS_FMT_SBGGR10_1X10,

[0042] .colorspace=V4L2_COLORSPACE_SRGB,

[0043] .regs=os02k10_init_regs_1920_1080_15fps,

[0044] When the pixel and frame rate are VGA@120fps, set to:

[0045] .width=640,

[0046] .height=480,

[0047] .fps=120<<16|1,

[0048] .mbus_code=V4L2_MBUS_FMT_SBGGR12_1X12,

[0049] .colorspace=V4L2_COLORSPACE_SRGB,

[0050] .regs=os02k10_init_regs_640_480_120fps,

[0051] When the pixel and frame rate are 2M@60fps, set to:

[0052] .width=1920,

[0053] .height=1080,

[0054] .fps=60<<16|1,

[0055] .mbus_code=V4L2_MBUS_FMT_SBGGR10_1X10,

[0056] .colorspace=V4L2_COLORSPACE_SRGB,

[0057] .regs=os02k10_init_regs_1920_1080_60fps.

[0058] The step S5.4 includes:

[0059] Call AWB resources, represented by: riscv_tiziano_awb.c;

[0060] Call ISP code, including AE part, represented as: riscv_tiziano.c;

[0061] Call the AWB algorithm library, represented by: Tiziano_Awb_Ct_Detect.o;

[0062] Call the AWB algorithm library, expressed as: Tiziano_Awb_Ct_Detect.o.imem.

[0063] The step S6 further includes the process of switching between large and small images, and the switching of the kernel stage:

[0064] S6.1, the ISP driver is reconfigured according to the large image parameters;

[0065] S6.2, the Sensor driver does not need to reconfigure the MCLK, Reset, and PowerDown operations. It only needs to reconfigure the large image resolution directly through IIC and add the Sensor driver according to the process.

[0066] Code description of the driver:

[0067] In the kernel / drivers / media / platform / tx-isp-txx / tx-isp-txx-device.c file, there are several global variables:

[0068] RISC-V mode switch, represented by: unsigned long high_framerate_risc_v_mode_en = 0; in the tx-isp-txx / tx-isp-fast.c file, there are several callback functions:

[0069]

[0070]

[0071] in,

[0072] tx_isp_start_device: This function is called by isp initialization in non-high frame rate small image mode. This function is responsible for configuring the sensor and initializing the corresponding module, configuring the channel, buffer, output width and height and other parameters, and starting image output;

[0073] In the following function, the channel is opened, the scaling and cropping parameters are set, the buffer is configured, and DMA is started to save the video data:

[0074] static int frame_channel_fast_start(unsigned int addr);

[0075] tx_isp_riscv_hf_prepare: When the RISCV high frame rate small picture mode function is turned on, the ISP will call this function before initialization to obtain the final result after RISCV processing and stop RISCV work;

[0076] tx_isp_riscv_hf_resize: When the riscv high frame rate small picture mode function is turned on, after calling tx_isp_riscv_hf_prepare, this function will be called back. This function mainly completes the initialization of ISP and the switching of Sensor large picture, and configures the opening of ISP related channels, scaling and cropping, and buffer configuration, and starts DMA to save video data.

[0077] The kernel stage provides the client with a kernel-mode interface, including:

[0078] Set ISP day and night vision mode

[0079] int isp_core_tuning_switch_day_or_night(int day);

[0080] Get the EV information of ISP and return the final result after the coprocessor converges

[0081] int tx_isp_get_ev_start(int*value);

[0082] Set the ISP start EV value, AE starts to converge from this point

[0083] int tx_isp_set_ev_start(int value);

[0084] Get the brightness information of the image and return the final result after the coprocessor converges

[0085] int tx_isp_get_ae_luma(uint32_t*luma);

[0086] Get the sensor again information and return the final result of the coprocessor convergence int tx_isp_get_sensor_again(uint32_t*value);

[0087] Set the sensor again information

[0088] int tx_isp_set_sensor_again(uint32_t value);

[0089] Get the exposure information of the sensor and return the final result of the coprocessor convergence

[0090] int tx_isp_get_sensor_inttime(uint32_t*value);

[0091] Set the exposure information of the sensor

[0092] int tx_isp_set_sensor_inttime(uint32_t value);

[0093] Get ISP gain information and return the final result of coprocessor convergence

[0094] int tx_isp_get_isp_gain(uint32_t*value);

[0095] Set ISP gain information

[0096] int tx_isp_set_isp_gain(uint32_t value);

[0097] Get the starting AWB information and return the final result of the coprocessor convergence

[0098] int tx_isp_get_awb_start(awb_start_t*awb_start);

[0099] Set the starting AWB information

[0100] int tx_isp_set_awb_start(awb_start_t awb_start);

[0101] Stop the coprocessor

[0102] int tx_isp_stop_riscv(void);

[0103] Check if the coprocessor has ended

[0104] int tx_isp_riscv_is_stop(void).

[0105] The step S6 further includes a parameter transfer method:

[0106] 1) AE and AWB statistics:

[0107] AE statistics data is 24KB, AWB statistics data is 16KB, the data will be assigned to the end of DDR's rmem space; because rmem is a reserved memory, it is used for ISP images and encoding, and the Linux system will not directly access it, so the memory data is written after statistics during the coprocessor; in the Linux stage, when the ISP is initialized again, there is no need to obtain AE parameters again and reconfigure them, and the data after small core statistics is directly used as the starting data; if you want to print out the corresponding EV value for debugging, use the following API to obtain the starting AE and AWB parameters:

[0108] int tx_isp_get_ev_start(int*value);

[0109] int tx_isp_get_awb_start(awb_start_t*awb_start);

[0110] If you want to fine-tune this parameter, reset the starting AE parameter through the following API: int tx_isp_set_ev_start(int value);

[0111] int tx_isp_set_awb_start(awb_start_t awb_start);

[0112] 2) Sensor exposure and gain parameters:

[0113] When AE converges, it adjusts the brightness by adjusting the exposure and gain of the sensor. Therefore, when switching to the large image, the values ​​of the exposure and gain registers need to be reconfigured due to parameter changes. Therefore, it is necessary to retain the convergence result of the small image and set the first frame of the large image after recalculation.

[0114] Get the final exposure and gain parameters of the sensor thumbnail through the following API: int tx_isp_get_sensor_analog_gain(int*value);

[0115] int tx_isp_get_sensor_integration_time(int*value).

[0116] The step S6 also includes the switching process step in the coprocessor:

[0117] A, when the coprocessor needs to be shut down;

[0118] B, cutting resolution: expressed as:

[0119]

[0120] The daytime and night vision parameters are different, so they are processed separately. The parameters of the small image are converted to the large image:

[0121]

[0122]

[0123] Perform parameter conversion, EV-->exposure, gain conversion; expressed as:

[0124] tiziano_ae_ev_to_gain(sw_ev,&switch_intt,&switch_again,&switch_dgain); update parameters: set_new_intt(switch_intt);

[0125] set_new_again(switch_again);

[0126] set_new_dgain(switch_dgain);

[0127] Assign values ​​and let the main CPU get the results:

[0128] mips_para_t->_ae_result[0]=switch_intt;

[0129] mips_para_t->_ae_result[1]=switch_again;

[0130] mips_para_t->_ae_result[2]=switch_dgain;

[0131] Re-issue the configuration of the large image:

[0132] actual_sensor_setting(get_switch_setting());

[0133] C, conversion completed, indicated by: stop_complete = 1;

[0134] E, hang, represented by: __asm("wfi").

[0135] The system includes a Linux system.

[0136] Therefore, the advantage of this application is that after the high frame rate small image converges, it directly switches to the large image mode, converts the small image parameters to the large image parameters, and the ISP large image can continue to converge based on this starting point to ensure that the first image effect is not jittery. BRIEF DESCRIPTION OF THE DRAWINGS

[0137] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0138] Figure 1 It is a schematic diagram of the first image in the convergence process in certain scenarios in the prior art.

[0139] Figure 2 It is a schematic diagram of the second frame image in the convergence process in certain scenarios in the prior art.

[0140] Figure 3 It is a schematic diagram of the third frame image in the convergence process in certain scenarios in the prior art.

[0141] Figure 4 It is a flowchart of the method using a small image with a high frame rate.

[0142] Figure 5 It is a schematic diagram of the Sensor configuration parameters of os02k10 in a specific embodiment of the method.

[0143] Figure 6 It is a directory diagram of the coprocessor software resource description in this method.

[0144] Figure 7 is a schematic diagram of the switching process in the coprocessor in this method.

[0145] Figure 8 This is a schematic diagram of the first figure in the actual measured effect of this method.

[0146] Fig. 9 It is a schematic diagram of the second frame of the measured effect of this method.

[0147] Fig.10 It is a schematic diagram of the third frame of the actual measured effect of this method. DETAILED DESCRIPTION

[0148] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.

[0149] 1) Introduction to the implementation method of the present invention

[0150] Since low-power cameras have requirements for startup speed, most of them require that the time from cold startup to taking the first picture should be within 300ms. Therefore, the image effect needs to be converged before 300ms.

[0151] According to the convergence principle of ISP, it usually takes 6 to 7 frames for ISP to converge. If the brightness of the ambient light is too different from the target brightness, the convergence frame number will be longer. Low-power cameras usually use 15FPS CMOS Sensor. If it converges 6 frames, it also takes 66ms*6=366ms, which exceeds the required time and cannot meet the user's usage requirements.

[0152] The specific implementation principle of high frame rate small pictures is to use the binning method of CMOS Sensor. For example, a 1080P sensor can be binned to a small picture of 640*360, so that there is no loss of field of view and the maximum frame rate can reach 120fps. Under the condition of 120fps, 24 frames of images can be converged in 200ms, so that a converged and stable image can be obtained in 300ms.

[0153] Since system startup also takes time, parallel convergence during the system startup process will affect each other, so the SOC internal coprocessor is introduced to perform convergence in parallel, and the main CPU is responsible for system startup. This way, each other is not directly affected, which can ensure convergence effect and startup time. At the same time, resources are relatively independent and easy to manage.

[0154] The present invention proposes a method for fast convergence of small images with high frame rate of low power consumption camera. The implementation flow chart of the present invention is as follows: Figure 4 As shown:

[0155] S1, main processor power on;

[0156] S2, initialize and run the coprocessor, and run the system startup of step S3 and the convergence processing of the coprocessor of step S5 in parallel;

[0157] S3, the main processor initializes other resources necessary for system startup, the other resources include normal startup resources of the Linux system, including kernel firmware loading, file system loading, kernel firmware decompression, file system decompression, Linux kernel initialization, and driver registration;

[0158] S4, the main processor sends a stop command to the coprocessor, and after completion, goes to step S6; for the convenience of describing the communication method, it can be understood that inside the chip, such as the T31, T41 and other chips of Beijing Ingenic Integrated Circuit Co., Ltd. (abbreviated as: Ingenic), there is a MessageBox register for communication between the dual cores, i.e., the main processor and the coprocessor. The stop command is to write 0x1 into this register. When the small core reads the value of this register as 0x1, it is a stop command; when the small core reads the stop command, it will write 0x0 into the messagebox register, thus informing the main CPU that it has stopped working normally;

[0159] S5, convergence processing of the coprocessor further includes:

[0160] S5.1, the coprocessor starts running;

[0161] S5.2, configure the sensor Sensor and use the high frame rate small image mode; the method is consistent with step S5.4. Step S5.2 mainly describes the configuration of the sensor, and S5.4 mainly describes the processing of the ISP. The sensor first generates a high frame rate small image, and then switches to a large image with a normal frame rate;

[0162] S5.3, initialize AE, AWB, and BLC modules of ISP;

[0163] S5.4, AE and AWB convergence in a high frame rate small image mode; the specific implementation principle of the high frame rate small image mode is to use the binning method of the CMOS Sensor. Assuming a resolution of 1080P, it can be binned to a small image of 640*360, so that there is no loss in the field of view, and the maximum frame rate can reach 120fps. Under the condition of 120fps, 24 frames of images can be converged in 200ms, so that a converged and stable image can be obtained in 300ms; like the high frame rate small image mode in step S5.2, the Sensor and ISP need to cooperate to implement the high frame rate small image method;

[0164] S5.5, receiving the stop command sent by the main processor, stopping the ISP and Sensor work;

[0165] S6, synchronization stop state, synchronize the convergence results to ISP, and convert them to large-size parameters; when the actual product displays images, it still displays them in 1080P full-size mode. Since the high frame rate uses small images such as 640*360 for processing, when the processing is completed, it is necessary to convert the small image to the full-size large image to continue displaying to ensure the stability of the image effect;

[0166] S7, the Sensor switches to the normal frame rate large image mode; switching means switching from a small size resolution to a large resolution, and then continuing to output images;

[0167] S8, enter the normal process of the large picture.

[0168] 2) Specific implementation requirements and software implementation of the present invention

[0169] a) The sensor needs to support multiple sets of parameters, one set of parameters for full-size resolution and one set of parameters for small images with high frame rate.

[0170] The exposure parameters corresponding to each set of resolutions are also different. Figure 5 The following is an example of the configuration of the sensor model os02k10, involving step S5.2:

[0171] When the pixel and frame rate are 2M@15fps, set to:

[0172] .width=1920,

[0173] .height=1080,

[0174] .fps=15<<16|1,

[0175] .mbus_code=V4L2_MBUS_FMT_SBGGR10_1X10,

[0176] .colorspace=V4L2_COLORSPACE_SRGB,

[0177] .regs=os02k10_init_regs_1920_1080_15fps,

[0178] When the pixel and frame rate are VGA@120fps, set to:

[0179] .width=640,

[0180] .height=480,

[0181] .fps=120<<16|1,

[0182] .mbus_code=V4L2_MBUS_FMT_SBGGR12_1X12,

[0183] .colorspace=V4L2_COLORSPACE_SRGB,

[0184] .regs=os02k10_init_regs_640_480_120fps,

[0185] When the pixel and frame rate are 2M@60fps, set to:

[0186] .width=1920,

[0187] .height=1080,

[0188] .fps=60<<16|1,

[0189] .mbus_code=V4L2_MBUS_FMT_SBGGR10_1X10,

[0190] .colorspace=V4L2_COLORSPACE_SRGB,

[0191] .regs=os02k10_init_regs_1920_1080_60fps.

[0192] b) Coprocessor software development requires porting ISP related resources, such as AE, AWB and other modules. Coprocessor software resource description, resource directory such as Figure 6 As shown, it involves the step S5.4: calling AWB resources, represented as: riscv_tiziano_awb.c;

[0193] Call ISP code, including AE part, represented as: riscv_tiziano.c;

[0194] Call the AWB algorithm library, represented by: Tiziano_Awb_Ct_Detect.o;

[0195] Call the AWB algorithm library, expressed as: Tiziano_Awb_Ct_Detect.o.imem.

[0196] c) The process of switching between large and small images and the method of transferring parameters involve step S6:

[0197] Kernel stage switching description, the main processing of the kernel stage:

[0198] S6.1. The ISP driver is reconfigured according to the large image parameters.

[0199] S6.2, Sensor driver does not need to reconfigure MCLK, Reset, PowerDown and other operations, only needs to reconfigure the large image resolution directly through IIC, and add sensor driver according to the process; the IIC bus, also known as the I2C bus (Inter-Integrated Circuit), is a serial communication bus standard used to connect integrated circuit chips. The IIC bus was launched by Philips (now NXP Semiconductors) in 1982 to simplify and reduce the number of connection lines between chips. The sensor adding process has no special processing, mainly including the initialization configuration of the sensor, exposure parameter information, exposure setting interface, and sensor communication interface.

[0200] Code description of the driver:

[0201] In the kernel / drivers / media / platform / tx-isp-txx / tx-isp-txx-device.c file, there are several global variables:

[0202] RISC-V mode switch, represented by: unsigned long high_framerate_risc_v_mode_en = 0; in the tx-isp-txx / tx-isp-fast.c file, there are several callback functions:

[0203]

[0204] in,

[0205] tx_isp_start_device: This function is called by isp initialization in non-high frame rate small image mode. This function is responsible for configuring the sensor and initializing the corresponding module, configuring the channel, buffer, output width and height and other parameters, and starting image output;

[0206] In the following function, the channel is opened, the scaling and cropping parameters are set, the buffer is configured, and DMA is started to save the video data:

[0207] static int frame_channel_fast_start(unsigned int addr);

[0208] tx_isp_riscv_hf_prepare: When the RISCV high frame rate small picture mode function is turned on, the ISP will call this function before initialization to obtain the final result after RISCV processing and stop RISCV work;

[0209] tx_isp_riscv_hf_resize: When the riscv high frame rate small picture mode function is turned on, after calling tx_isp_riscv_hf_prepare, this function will be called back. This function mainly completes the initialization of ISP and the switching of Sensor large picture, and configures the opening of ISP related channels, scaling and cropping, and buffer configuration, and starts DMA to save video data.

[0210] The kernel stage provides the client with a kernel-mode interface, including:

[0211] Set ISP day and night vision mode

[0212] int isp_core_tuning_switch_day_or_night(int day);

[0213] Get the EV information of ISP and return the final result after the coprocessor converges

[0214] int tx_isp_get_ev_start(int*value);

[0215] Set the ISP start EV value, AE starts to converge from this point

[0216] int tx_isp_set_ev_start(int value);

[0217] Get the brightness information of the image and return the final result after the coprocessor converges

[0218] int tx_isp_get_ae_luma(uint32_t*luma);

[0219] Get the sensor again information and return the final result of the coprocessor convergence int tx_isp_get_sensor_again(uint32_t*value);

[0220] Set the sensor again information

[0221] int tx_isp_set_sensor_again(uint32_t value);

[0222] Get the exposure information of the sensor and return the final result of the coprocessor convergence

[0223] int tx_isp_get_sensor_inttime(uint32_t*value);

[0224] Set the exposure information of the sensor

[0225] int tx_isp_set_sensor_inttime(uint32_t value);

[0226] Get ISP gain information and return the final result of coprocessor convergence

[0227] int tx_isp_get_isp_gain(uint32_t*value);

[0228] Set ISP gain information

[0229] int tx_isp_set_isp_gain(uint32_t value);

[0230] Get the starting AWB information and return the final result of the coprocessor convergence int tx_isp_get_awb_start(awb_start_t*awb_start);

[0231] Set the starting AWB information

[0232] int tx_isp_set_awb_start(awb_start_t awb_start);

[0233] Stop the coprocessor

[0234] int tx_isp_stop_riscv(void);

[0235] Check if the coprocessor has ended

[0236] int tx_isp_riscv_is_stop(void).

[0237] The kernel state interface provided by the present invention application to customers includes:

[0238] Set ISP day and night vision mode

[0239] int isp_core_tuning_switch_day_or_night(int day);

[0240] Get the EV information of ISP and return the final result after the coprocessor converges int tx_isp_get_ev_start(int*value);

[0241] Set the ISP starting EV value, AE starts to converge from this point int tx_isp_set_ev_start(int value);

[0242] Get the brightness information of the image and return the final result after the coprocessor converges int tx_isp_get_ae_luma(uint32_t*luma);

[0243] Get the sensor again information and return the final result of the coprocessor convergence int tx_isp_get_sensor_again(uint32_t*value);

[0244] Set the sensor again information

[0245] int tx_isp_set_sensor_again(uint32_t value);

[0246] Get the exposure information of the sensor and return the final result of the coprocessor convergence int tx_isp_get_sensor_inttime(uint32_t*value);

[0247] Set the exposure information of the sensor

[0248] int tx_isp_set_sensor_inttime(uint32_t value);

[0249] Get ISP gain information and return the final result of coprocessor convergence int tx_isp_get_isp_gain(uint32_t*value);

[0250] Set ISP gain information

[0251] int tx_isp_set_isp_gain(uint32_t value);

[0252] Get the starting AWB information and return the final result of the coprocessor convergence

[0253] int tx_isp_get_awb_start(awb_start_t*awb_start);

[0254] Set the starting AWB information

[0255] int tx_isp_set_awb_start(awb_start_t awb_start);

[0256] Stop the coprocessor

[0257] int tx_isp_stop_riscv(void);

[0258] Check if the coprocessor has ended

[0259] int tx_isp_riscv_is_stop(void).

[0260] The step S6 further includes a parameter transfer method:

[0261] 1) AE and AWB statistics:

[0262] The AE statistics data is 24KB and the AWB statistics data is 16KB, which will be assigned to the end of the DDR's rmem space. Since rmem is a reserved memory used for ISP images and encoding, the Linux system will not directly access it, so the memory data is written after statistics are performed during the coprocessor. In the Linux stage, when the ISP is initialized again, there is no need to obtain the AE parameters again and reconfigure them, and the data after statistics by the small core is directly used as the starting data.

[0263] If you want to print out the corresponding EV value for debugging, use the following API to obtain the starting AE and AWB parameters:

[0264] int tx_isp_get_ev_start(int*value);

[0265] int tx_isp_get_awb_start(awb_start_t*awb_start);

[0266] If you want to fine-tune this parameter, reset the starting AE parameter through the following API:

[0267] int tx_isp_set_ev_start(int value);

[0268] int tx_isp_set_awb_start(awb_start_t awb_start);

[0269] 2) Sensor exposure and gain parameters:

[0270] When AE converges, it adjusts the brightness by adjusting the exposure and gain of the sensor. Therefore, when switching to the large image, the values ​​of the exposure and gain registers need to be reconfigured due to parameter changes. Therefore, it is necessary to retain the convergence result of the small image and set the first frame of the large image after recalculation.

[0271] Use the following API to obtain the final exposure and gain parameters of the sensor thumbnail:

[0272] int tx_isp_get_sensor_analog_gain(int*value);

[0273] int tx_isp_get_sensor_integration_time(int*value).

[0274] like Figure 7 As shown, the step S6 also includes the switching process step in the coprocessor:

[0275] A, when the coprocessor needs to be shut down;

[0276] B, cutting resolution: expressed as:

[0277]

[0278] The daytime and night vision parameters are different, so they are processed separately. The parameters of the small image are converted to the large image:

[0279]

[0280] Perform parameter conversion, EV-->exposure, gain conversion; expressed as:

[0281] tiziano_ae_ev_to_gain(sw_ev,&switch_intt,&switch_again,&switch_dgain); update parameters: set_new_intt(switch_intt);

[0282] set_new_again(switch_again);

[0283] set_new_dgain(switch_dgain);

[0284] Assign values ​​and let the main CPU get the results:

[0285] mips_para_t->_ae_result[0]=switch_intt;

[0286] mips_para_t->_ae_result[1]=switch_again;

[0287] mips_para_t->_ae_result[2]=switch_dgain;

[0288] Re-issue the configuration of the large image:

[0289] actual_sensor_setting(get_switch_setting());

[0290] C, conversion completed, indicated by: stop_complete = 1;

[0291] D, hang, expressed as: __asm("wfi").

[0292] like Figures 8 to 10 As shown, it is based on the actual measured effect of the present invention. Figure 8 This is the first picture. Fig. 9 It's the second frame. Fig.10 This is a schematic diagram of the effect of the third frame.

[0293] In summary, the present invention describes the use of high frame rate small picture mode to make the first picture fast and stable, the use of coprocessors to improve the parallel processing speed, and the small picture to large picture switching method, all of which fall within the scope of protection of this patent application.

[0294] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for fast convergence of small images with high frame rate of low power consumption camera, characterized in that: The method comprises the following steps: S1, main processor power on; S2, initialize and run the coprocessor, and run the system startup of step S3 and the convergence processing of the coprocessor of step S5 in parallel; S3, the main processor initializes other resources necessary for system startup, wherein the other resources include normal startup resources of the system, including kernel firmware loading, file system loading, kernel firmware decompression, file system decompression, kernel initialization, and driver registration; S4, the main processor sends a stop command to the coprocessor, and after completion, goes to step S6; S5, convergence processing of the coprocessor further includes: S5.1, the coprocessor starts running; S5.2, configure the sensor Sensor and use the high frame rate small picture mode. The high frame rate small picture mode requires the support of the sensor and the ISP. The specific implementation principle of the high frame rate small picture mode is to use the binning method of the CMOS Sensor. The sensor first generates a high frame rate small picture and then switches to a large picture with a normal frame rate. S5.3, initialize AE, AWB, and BLC modules of ISP; S5.4, performing AE and AWB convergence in the high frame rate small image mode; S5.5, receiving the stop command sent by the main processor, stopping the ISP and Sensor work; S6, synchronization stop state, synchronize the convergence results to ISP, convert to large image parameters, when the actual product is displayed in the image, after the processing is completed, it is necessary to convert from the small image to the full-size large image to continue displaying to ensure the stability of the image effect; S7, the Sensor switches to the normal frame rate large image mode, that is, it switches from a small size resolution to a large resolution, and then continues to output images; S8, enter the normal process of the large picture.

2. According to claim 1, a method for fast convergence of small images with high frame rate of low power consumption camera is characterized in that: In step S4, a MessageBox register is provided for communication between the dual cores, i.e., the main processor and the coprocessor. The stop command is to write 0x1 into this register. When the small core reads the value of this register as 0x1, it is a stop command. When jumping to step S6, when the small core reads the stop command, it will write 0x0 into the messagebox register, thus informing the main CPU that it has stopped working normally.

3. According to claim 1, a method for fast convergence of small images with high frame rate of low power consumption camera is characterized in that: The parameters of the sensor configured in step S5.2 are assumed to be the sensor configuration of model os02k10, including: When the pixel and frame rate are 2M@15fps, set to: .width=1920, .height=1080, .fps=15<<16|1, .mbus_code=V4L2_MBUS_FMT_SBGGR10_1X10, .colorspace=V4L2_COLORSPACE_SRGB, .regs=os02k10_init_regs_1920_1080_15fps, When the pixel and frame rate are VGA@120fps, set to: .width=640, .height=480, .fps=120<<16|1, .mbus_code=V4L2_MBUS_FMT_SBGGR12_1X12, .colorspace=V4L2_COLORSPACE_SRGB, .regs=os02k10_init_regs_640_480_120fps, When the pixel and frame rate are 2M@60fps, set to: .width=1920, .height=1080, .fps=60<<16|1, .mbus_code=V4L2_MBUS_FMT_SBGGR10_1X10, .colorspace=V4L2_COLORSPACE_SRGB, .regs=os02k10_init_regs_1920_1080_60fps.

4. According to claim 1, a method for fast convergence of small images with high frame rate of low power consumption camera is characterized in that: The step S5.4 includes: Call AWB resources, represented by: riscv_tiziano_awb.c; Call ISP code, including AE part, represented as: riscv_tiziano.c; Call the AWB algorithm library, represented by: Tiziano_Awb_Ct_Detect.o; Call the AWB algorithm library, expressed as: Tiziano_Awb_Ct_Detect.o.imem.

5. According to claim 1, a method for fast convergence of small images with high frame rate of low power consumption camera is characterized in that: The step S6 further includes the process of switching between large and small images, and the switching of the kernel stage: S6.1, the ISP driver is reconfigured according to the large image parameters; S6.2, the Sensor driver does not need to reconfigure the MCLK, Reset, and PowerDown operations. It only needs to reconfigure the large image resolution directly through IIC and add the Sensor driver according to the process.

6. A method for fast convergence of small images with high frame rate of low power consumption camera according to claim 5, characterized in that: Code description of the driver: In the kernel / drivers / media / platform / tx-isp-txx / tx-isp-txx-device.c file, there are several global variables: RISC-V mode switch, represented by: unsigned long high_framerate_risc_v_mode_en = 0; in the tx-isp-txx / tx-isp-fast.c file, there are several callback functions: struct tx_isp_callback_ops g_tx_isp_callback_ops={ .tx_isp_start_device=tx_isp_start_device, .tx_isp_riscv_hf_prepare=tx_isp_riscv_hf_prepare, .tx_isp_riscv_hf_resize=tx_isp_riscv_hf_resize, }； in, tx_isp_start_device: This function is called by isp initialization in non-high frame rate small image mode. This function is responsible for configuring the sensor and initializing the corresponding module, configuring the channel, buffer, output width and height and other parameters, and starting image output; In the following function, the channel is opened, the scaling and cropping parameters are set, the buffer is configured, and DMA is started to save the video data: static int frame_channel_fast_start(unsigned int addr); tx_isp_riscv_hf_prepare: When the RISCV high frame rate small picture mode function is turned on, the ISP will call this function before initialization to obtain the final result after RISCV processing and stop RISCV work; tx_isp_riscv_hf_resize: When the riscv high frame rate small picture mode function is turned on, after calling tx_isp_riscv_hf_prepare, this function will be called back. This function mainly completes the initialization of ISP and the switching of Sensor large picture, and configures the opening of ISP related channels, scaling and cropping, and buffer configuration, and starts DMA to save video data.

7. The method for fast convergence of small images with high frame rate of a low-power camera according to claim 5, characterized in that: The kernel stage provides the client with a kernel-mode interface, including: Set ISP day and night vision mode int isp_core_tuning_switch_day_or_night(int day); Get the EV information of ISP and return the final result after the coprocessor converges int tx_isp_get_ev_start(int*value); Set the ISP starting EV value, AE starts to converge from this point int tx_isp_set_ev_start(int value); Get the brightness information of the image and return the final result after the coprocessor converges int tx_isp_get_ae_luma(uint32_t*luma); Get the sensor again information and return the final result of the coprocessor convergence int tx_isp_get_sensor_again(uint32_t*value); Set the sensor again information int tx_isp_set_sensor_again(uint32_t value); Get the exposure information of the sensor and return the final result of the coprocessor convergence int tx_isp_get_sensor_inttime(uint32_t*value); Set the exposure information of the sensor int tx_isp_set_sensor_inttime(uint32_t value); Get ISP gain information and return the final result of coprocessor convergence int tx_isp_get_isp_gain(uint32_t*value); Set ISP gain information int tx_isp_set_isp_gain(uint32_t value); Get the starting AWB information and return the final result of the coprocessor convergence int tx_isp_get_awb_start(awb_start_t*awb_start); Set the starting AWB information int tx_isp_set_awb_start(awb_start_t awb_start); Stop the coprocessor int tx_isp_stop_riscv(void); Check if the coprocessor has ended int tx_isp_riscv_is_stop(void).

8. The method for fast convergence of small images with high frame rate of a low power consumption camera according to claim 5, characterized in that: The step S6 further includes a parameter transfer method: 1) AE and AWB statistics: AE statistics data is 24KB, AWB statistics data is 16KB, and the data will be assigned to the end of DDR's rmem space. Since rmem is a reserved memory used for ISP images and encoding, the Linux system will not directly access it, so the memory data is written after statistics are performed during the coprocessor. In the Linux stage, when the ISP is initialized again, there is no need to obtain the AE parameters and reconfigure them. The data collected by the small core is directly used as the starting data. If you want to print out the corresponding EV value for debugging, use the following API to obtain the starting AE and AWB parameters: int tx_isp_get_ev_start(int*value); int tx_isp_get_awb_start(awb_start_t*awb_start); If you want to fine-tune this parameter, reset the starting AE parameter through the following API: int tx_isp_set_ev_start(int value); int tx_isp_set_awb_start(awb_start_t awb_start); 2) Sensor exposure and gain parameters: When AE converges, it adjusts the brightness by adjusting the exposure and gain of the sensor. Therefore, when switching to the large image, the values ​​of the exposure and gain registers need to be reconfigured due to parameter changes. Therefore, it is necessary to retain the convergence result of the small image and set the first frame of the large image after recalculation. Use the following API to obtain the final exposure and gain parameters of the sensor thumbnail: int tx_isp_get_sensor_analog_gain(int*value); int tx_isp_get_sensor_integration_time(int*value).

9. A method for fast convergence of small images with high frame rate of low power consumption camera according to claim 8, characterized in that: The step S6 also includes the switching process step in the coprocessor: A, when the coprocessor needs to be shut down; B, cutting resolution: expressed as: Perform parameter conversion, EV-->exposure, gain conversion; expressed as: tiziano_ae_ev_to_gain(sw_ev,&switch_intt,&switch_again,&switch_dgain); update parameters: set_new_intt(switch_intt); set_new_again(switch_again); set_new_dgain(switch_dgain); Assign values ​​and let the main CPU get the results: mips_para_t->_ae_result[0]=switch_intt; mips_para_t->_ae_result[1]=switch_again; mips_para_t->_ae_result[2]=switch_dgain; Re-issue the configuration of the large image: actual_sensor_setting(get_switch_setting()); C, conversion completed, indicated by: stop_complete = 1; D, hang, expressed as: __asm("wfi").

10. The method for fast convergence of small images with high frame rate of a low power camera according to claim 1, characterized in that: The system includes a Linux system.

Images