Soft photosensitive rapid convergence method for low-power-consumption camera

Abstract

The invention provides a soft photosensitive rapid convergence method for a low-power-consumption camera. The method comprises the following steps: S1, starting up a main processor; s2, initializing and operating a coprocessor, and parallelly operating the starting of the system in S3 and the convergence processing of the coprocessor in S5; s3, the main processor initializes the system to start other necessary resources; s4, the main processor sends a stop command to the coprocessor, and the step S6 is executed after the stop command is completed; s5, convergence processing of the coprocessor: S5.1, the coprocessor starts to operate; s5.2, configuring a sensor, and using a fixed exposure parameter; s5.3, initializing an ISP (Internet Service Provider); s5.4, obtaining statistical information of a brightness histogram; s5.5, looking up a table to configure Sensor parameters and sending the Sensor parameters to the main processor, and ending the coprocessor flow after receiving a stop command; s6, synchronizing a stop state, and synchronizing a table look-up result to the ISP; and S7, entering a normal process. Defects of the photoresistor can be solved, dependence on the photoresistor is avoided, and the first picture effect can be achieved without increasing hardware cost. The problem that a high-frame-rate small image depends on the Sensor can be solved, and common Sensors can use the method.

Description

Technical Field

[0001] The invention belongs to the technical field of video image processing, and in particular relates to a soft photosensitivity fast convergence method for a low-power camera. Background Art

[0002] In the prior art, in order to reduce power consumption, the camera processor of low-power camera products is completely powered off, so each startup is a cold start. During the startup process, according to the principle of ISP, the image screen has a process of changing from dark to bright, which is called AE convergence process. If no special treatment is done, there will inevitably be such a convergence process, which is extremely bad for user experience. Therefore, low-power cameras have very high requirements for the first frame image effect.

[0003] In order to shorten this process, most practices rely on hardware photoresistors to feedback an initial ambient light state, and the ISP converges based on the value of the photoresistor as the AE convergence starting point.

[0004] A more advanced approach is to use a high frame rate small image method to complete image convergence.

[0005] However, the photoresistor method usually has a small light-sensitive range, and the light angle has a great influence on the resistance value. Therefore, in many scenarios, the feedback of ambient light is inaccurate, resulting in an inaccurate AE convergence starting point, resulting in the first frame image being too bright or too dark, resulting in a poor user experience. Figures 1 to 3 Figure 2 shows the convergence process of the traditional photoresistor method in certain scenarios.

[0006] Due to the iteration of chip technology, the startup is actually getting faster and faster. From cold start to taking the first picture, it can be completed within 150ms. The high frame rate small picture method, on the one hand, has a certain dependence on the sensor. Some sensors cannot output high frame rate binning pictures. Some sensors take a long time to switch from small pictures to large pictures. In addition, after the chip startup speed is accelerated, there is not much time left for high frame rate small pictures, so another technology is needed, namely soft photosensitivity technology.

[0007] Soft photosensitivity simulates the principle of photoresistors by counting the image brightness of the sensor. The sensor has a wide range of self-sensitivity and is consistent with the real image information, thus making up for the defects of photoresistors. By looking up the table of sensor image brightness statistics, the target brightness parameters are obtained. No additional convergence time and switching time are required, and there is no strict dependence on sensor selection, so it also makes up for the shortcomings of the high frame rate small image method.

[0008] In addition, common terminology explanations in the prior art include:

[0009] IPC: IP Camera, network camera.

[0010] Low-power camera: A camera that has requirements for power consumption, is generally battery-powered, and requires a quick startup function.

[0011] Fast startup: The entire system starts up within 500ms from power-on to acquiring the first image.

[0012] High frame rate small picture: The frame rate is higher than the normal frame rate, and the small picture has a small resolution. For example, the pixel and frame rate of a general camera are 1080P@15fps, while the high frame rate small picture can be 360P@120fps.

[0013] ISP: Image Signal Processing, image signal processing unit.

[0014] AE: Automatic Exposure, automatic exposure.

[0015] AWB: Automatic white balance, automatic white balance.

[0016] BLC: Black Level Correction, black level correction.

[0017] Power off: Completely cut off the power supply.

[0018] Cold start: Cut off the power supply and restart.

[0019] First frame image: The first frame image that can be obtained after the system is cold started.

[0020] CMOS Sensor: Image sensor.

[0021] Binning: An image readout mode that adds the charges induced by adjacent pixels together and reads them out in a one-pixel mode. For example, 1080P is output as 360P by merging 4 pixels into 1 pixel, so that there is no loss in field of view.

[0022] FPS: Frames Per Second, the number of frames transmitted per second.

[0023] Coprocessor: A processor developed and applied to assist the central processing unit in completing processing tasks that the central processing unit cannot perform or performs inefficiently or ineffectively.

[0024] Soft photosensitivity: A method that directly relies on CMOS Sensor image information to replace hardware photosensors and can provide feedback on ambient light brightness.

[0025] Luma: Histogram statistics of image brightness. Summary of the invention

[0026] In order to solve the above problems, the purpose of this application is to address the defects of the supporting solution based on photoresistors for low-power cameras and introduce a new method based on the cooperation of CMOS Sensor and ISP, namely, the soft photosensitivity method.

[0027] Specifically, the present invention proposes a method for fast convergence of soft photosensitivity of a low-power camera, the method comprising the following steps:

[0028] S1, main processor power on;

[0029] S2, initialize and run the coprocessor, and run the system startup of step S3 and the convergence processing of the coprocessor of step S5 in parallel;

[0030] S3, the main processor initializes other resources necessary for system startup, the other resources include normal startup resources of the Linux system, including kernel firmware loading, file system loading, kernel firmware decompression, file system decompression, Linux kernel initialization, and driver registration;

[0031] S4, the main processor sends a stop command to the coprocessor. A MessageBox register is set for communication between the dual cores, i.e., the main processor and the coprocessor. The stop command is to write 0x1 into this register. When the small core reads the value of this register as 0x1, it is a stop command. After completion, go to step S6;

[0032] S5, soft photosensitivity convergence processing of the coprocessor. The soft photosensitivity operation is based on the quick start system. The sensor fixes a set of exposure parameters when it is powered on, starts and completes convergence under different ambient light conditions, and then obtains the initial average brightness information and the converged exposure parameters. The two parameters are collected under different lighting conditions to obtain a mapping table of the initial exposure parameters. While building the table, a light meter can be used to record the illumination of the current ambient light.

[0033] In actual use, the Sensor will obtain a starting average brightness value luma according to the fixed exposure parameters used when building the table in the actual scene. The corresponding EV value is found in the mapping table through the luma value, and then the exposure parameters can be reconfigured.

[0034] The day and night switching is also determined by the luma value. When the meter is built, the illumination information has been recorded and corresponds to the luma value one by one, so a luma value can be selected as the day and night switching threshold according to the illumination information; further including:

[0035] S5.1, the coprocessor starts running;

[0036] S5.2, configure the sensor Sensor, use fixed exposure parameters, the sensor exposure parameters include two parts, one is exposure time, the other is exposure gain, the actual image brightness output by the sensor is the product of exposure time and exposure gain;

[0037] S5.3, initializing ISP;

[0038] S5.4, obtaining brightness histogram statistical information, wherein the brightness histogram is information output by ISP statistics, and this information is stored in DDR after being output by ISP, and data can be directly taken out from DDR for calculation;

[0039] S5.5, configure the sensor parameters through table lookup and send them to the main processor. Since a new parameter is obtained after table lookup, the sensor needs to be reconfigured. On the other hand, the ISP of the main processor also needs to know the latest parameter to synchronize with the sensor parameters to prevent abnormal brightness caused by asynchrony. After receiving the stop command, the stop action is performed after the sensor parameters are configured, and the coprocessor process ends.

[0040] S6, synchronous stop state, synchronize the table lookup result to ISP. When the small core, i.e. the coprocessor, reads the stop command, it writes 0x0 to the messagebox register, thus informing the main processor that it has stopped working normally.

[0041] S7, enter the normal process.

[0042] The step S5 further comprises:

[0043] S5.1, in the coprocessor software running on the coprocessor, ISP related resources need to be transplanted, including AE and AWB modules;

[0044] Among them, AE and AWB statistics:

[0045] The AE statistics data is 24KB and the AWB statistics data is 16KB, which will be assigned to the end of the DDR's rmem space. Since rmem is a reserved memory used for ISP images and encoding, the Linux system will not directly access it, so the memory data is written after statistics during the coprocessor period. In the Linux stage, when the ISP is initialized again, there is no need to obtain AE parameters and reconfigure them. The small core, i.e., the coprocessor, can be used directly, and the statistical data is used as the starting data.

[0046] If you want to print out the corresponding EV value for debugging, you can use the following API to obtain the starting AE and AWB parameters:

[0047] int tx_isp_get_ev_start(int*value);

[0048] int tx_isp_get_awb_start(awb_start_t*awb_start);

[0049] If you want to fine-tune this parameter, you can reset the starting AE parameter through the following API: int tx_isp_set_ev_start(int value);

[0050] int tx_isp_set_awb_start(awb_start_t awb_start);

[0051] S5.2, configure the sensor and use fixed exposure parameters:

[0052] The operation flow of the coprocessor soft photosensitivity has been completed in the src / main.c file. Since the initial luma value statistics require 1 to 2 frames, statistics need to be performed based on fixed Sensor exposure parameters, so the Sensor settings need to be turned off to prevent the ISP from resetting the Sensor parameters.

[0053] You need to set the ISP's dgain to 1x;

[0054] Since the luma value will be affected by the exposure, Again, and ISP Dgain of the Sensor, in order to prevent the ISP Dgain from encoding in different scenarios and causing the luma value to change, the ISP Dgain is set to 1 times here. 1 times refers to the actual multiple. The corresponding setting parameter for log domain conversion is 1024; then the luma value is only related to the fixed parameters of the Sensor;

[0055] Soft photosensitivity operation requires closing the front sensor exposure parameter settings, which can be expressed as:

[0056] tx_isp_disable_sensor_expo();

[0057] Soft light-sensitive operation needs to turn off the dgain operation to prevent the influence of dgain. It is called before ISP initialization and dgain is set to 1x, which can be expressed as:

[0058] tx_isp_tuning_set_max_dgain(1024);

[0059] Sensor exposure and gain parameters:

[0060] Since only soft photosensitivity table lookup is performed in the small core, the initial fixed exposure parameters will be used to obtain the luma information table lookup to obtain new exposure and gain parameters. After the small core sets the sensor parameters, this parameter also needs to be synchronized to the kernel, that is, the ISP module of the Linux kernel, and is also passed to the kernel through a fixed memory address. The kernel can use the following API to obtain the final exposure and gain parameters in the case of the sensor small core:

[0061] int tx_isp_get_sensor_analog_gain(int*value);

[0062] int tx_isp_get_sensor_integration_time(int*value);

[0063] S5.3, initializing ISP;

[0064] Initialize ISP, which can be expressed as:

[0065] tx_isp_init();

[0066] Initialize the video in module: MIPI, VIC, which can be expressed as:

[0067] tx_isp_video_in_init().

[0068] The operation process of the soft photosensitivity in step S5 further includes:

[0069] Determine whether soft photosensitization has been done, expressed as:

[0070] if(!soft_photosensitive_done);

[0071] If the luma value is stable, then the EV search is performed, expressed as:

[0072] if(last_luma!=sp_result.luma)

[0073] last_luma=sp_result.lulma;

[0074] Prepare AE Table, expressed as:

[0075] get_ae_table_info();

[0076] The day and night state is determined by the threshold, expressed as:

[0077] g_is_night=check_is_day_or_night(sp_result.luma);

[0078] IRCUT IRLED processing, expressed as:

[0079] set-irled_and_ircut(g_is_night);

[0080] Enable sensor exposure parameter settings, and set the ISP statistical results to the Sensor via I2C, expressed as: tx_isp_enable_sensor_expo();

[0081] Remove the limit of dgain, which is currently set to 8 times. It needs to be set according to the actual situation of the Sensor, expressed as:

[0082] tx_ips_tuning_Set_max_dgain(8096);

[0083] Get the starting AE parameters, expressed as:

[0084] g_ev=get_ae_table_value(g_start_ae_table_p,sp_result.luma,g_is_night);

[0085] printf("%s luma:%d g_is_night:%d ev:%d\n",__func__,sp_result.lulma,

[0086] g_is_night,g_ev);

[0087] Set the Sensor start parameters, expressed as:

[0088] tiziano_ae_s_ev_sensor_start(g_ev);

[0089] IRCUT restores the idle state to prevent power loss and chip burn.

[0090] Due to kernel operation, it is not processed here and is expressed as:

[0091] soft_photosensitive_done=1;

[0092] Among them, the role of soft_photosensitive_done: After the soft photosensitivity processing is completed, if there is still time, the restrictions on sensor exposure and dgain can be released to allow ISP to continue to perform AE convergence based on soft photosensitivity.

[0093] In actual product applications, the soft light sensor is configured to process the sensor initialization sequence at the highest available frame rate for fast processing, which can improve the startup time; after the soft light sensor judgment is completed, the actual frame rate can be switched;

[0094] When counting the starting luma value, you need to stop the sensor exposure operation and set the ISP Dgain to 1x. After the statistics are completed, restore the sensor exposure settings and restore the ISP Dgain to the actual product requirement value.

[0095] After the IRCUT operation is completed, the IRCUT needs to be restored to the IDLE state to prevent power loss and the risk of burning the IRCUT.

[0096] The soft photosensitivity table building method comprises:

[0097] Take a uniform light source and put it in a dark room without any external light interference. Place the device so that the lens can be as close as possible. Then adjust the brightness of the light source. You can use a illuminance meter to record the corresponding illuminance value, and gradually adjust it from dark to bright. After adjustment, when the brightness is stable, turn on and off the device, and read the Luma and EV values ​​through the serial port and save them.

[0098] In the absence of uniform light sources, a table can be created by keeping a distance of 5 meters between the camera and the reference object in a dark room and changing the brightness of the room lights. Alternatively, this can be achieved in a real environment by writing a program script that records the brightness changes every 10 minutes throughout the day.

[0099] The method for fixing the initial exposure parameters includes:

[0100] Refer to the corresponding Sensor register manual and modify the exposure time and gain parameters appropriately. You can also directly view the separately provided Sensor driver. Assuming that the Sensor is a jxf35 sensor, find the exposure setting interface and see which registers need to be written to actually set the exposure. Find the registers and modify them accordingly. The gain parameter can also be obtained in the same way. After finding the register, write the exposure parameter into the initialization sequence of the following file.

[0101] tools / make_tag / sensor_init_setting / jxf35_t31_init.c

[0102] Then recompile the tag partition;

[0103] Selection of exposure value, which is the exposure parameter, including exposure time and exposure gain. You can take a value after convergence in an office daytime environment. You can add printing to the above Sensor driver file to feedback the value;

[0104] In addition, you need to use a illuminance meter. When recording the startup luma value, it can cover the day and night switching area and have a wide range of perceived brightness. The traditional photoresistor's sensitivity range is 0 to 100 lux, and the soft photosensitivity's sensitivity range is 0 to 6000 lux.

[0105] In the step S5.4, statistics are generated and a table is created: after obtaining the statistical values ​​in S5.4, these data are recorded and a table is created;

[0106] Using the printing method in the small core, print out the starting luma value of the src / main.c file, which can be expressed as:

[0107] Waiting for AE to process interrupt, expressed as: tx_isp_wait_ae_interrupt();

[0108] AE statistical processing, represented by: tx_isp_ae_process();

[0109] Get the statistical results, expressed as: tx_isp_get_ae_result(&sp_result);

[0110] Print Statistics

[0111] #if UART_DEBUG_EN||DEBUG_LOGREAD_EN

[0112] printf("num:%d intt:%d again:%d dgain:%d luma:%d ev:%d\n",\

[0113] sp_result.num,\

[0114] sp_result.intt,\

[0115] sp_result.again,\

[0116] sp_result.dgain,\

[0117] sp_result.luma,\

[0118] sp_result.ev);

[0119] #endif;

[0120] At the same time, turn on the kernel debug function. The second parameter is the number of frames to be printed, which should be at least 10 frames. After debug is turned on, you can see the statistics of each frame through dmesg information.

[0121] After the modification is completed, recompile the ISP driver through make clean&&make&&make release, and then recompile the kernel and burn it.

[0122] After the system is started up quickly, the corresponding statistical information can be seen through the dmesg information, and then the statistical information can be recorded in a table;

[0123] Get luma value information through riscvread, do not drop below 10 frames, and keep a stable value;

[0124] Obtain the stable EV value information;

[0125] Then start it at different brightness levels and record the corresponding values. The Luma value range is 0 to 255, so a maximum of 255 data points can be counted. The more data points, the more accurate the EV value setting.

[0126] Fill the created table into the list corresponding to the following file, expressed as:

[0127] tools / make_tag / sensor_start_ae_table / libz_jxf35.c

[0128] Daytime parameter list: struct vol_start_value_table table_day[]

[0129] Night vision parameter list: struct vol_start_value_table table_night[]

[0130] Then recompile the tag partition and burn it.

[0131] Therefore, the advantages of this application are: this method can solve the defects of photoresistors, and in actual use, it does not rely on photoresistors, and can achieve the first picture effect more accurately without increasing hardware costs. It can also solve the problem of high frame rate small pictures relying on sensors, and ordinary sensors can use this method. BRIEF DESCRIPTION OF THE DRAWINGS

[0132] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0133] Figure 1 It is a schematic diagram of the first picture during the convergence process in some scenarios.

[0134] Figure 2 is a schematic diagram of the second frame during the convergence process in some scenarios.

[0135] Figure 3 It is a schematic diagram of the third frame in the convergence process in some scenarios.

[0136] Figure 4 It is a schematic diagram of the process of rapid convergence using soft photosensitivity in this method.

[0137] Figure 5 It is a directory diagram of the coprocessor software resource description in this method.

[0138] Figure 6 This is a schematic diagram of the code for setting the sensor exposure parameters before the soft photosensitivity operation needs to be turned off in this method.

[0139] Figure 7 It is a schematic diagram of the code flow of the actual soft photosensitivity operation in this method.

[0140] Figure 8 It is a schematic diagram of the uniform table building method in this method.

[0141] Fig. 9 This is a code schematic diagram of modifying the exposure time using jxf35 as an example in the embodiment of this method.

[0142] Fig.10 This is a code schematic diagram of modifying the gain parameters in the embodiment of the method, taking jxf35 as an example.

[0143] Fig.11 This is a schematic diagram of the relevant code using the printing method in the small core in this method.

[0144] Fig.12 This is a schematic diagram of the code related to opening the kernel debug function in this method.

[0145] Fig.13 This is a schematic diagram of the relevant code for viewing the corresponding statistical information through dmesg information in this method.

[0146] Fig.14 This is a schematic diagram of the relevant code for obtaining luma value information through riscvread in this method.

[0147] Fig.15It is a schematic diagram of the stabilized EV value information obtained in this method.

[0148] Fig.16 This is a schematic diagram of the first figure in the actual measured effect of this method.

[0149] Fig.17 It is a schematic diagram of the second frame of the measured effect of this method.

[0150] Fig.18 It is a schematic diagram of the third frame of the measured effect of this method. DETAILED DESCRIPTION

[0151] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.

[0152] The implementation method of the present invention is introduced as follows:

[0153] Since low-power cameras have requirements for startup speed, chip performance is also constantly improving. For example, the T41 chip of Beijing Ingenic Integrated Circuit Co., Ltd. (Ingenic) only takes 150ms from cold startup to taking the first picture. Therefore, the image effect needs to be converged before 150ms.

[0154] The soft photosensitivity method uses the image brightness statistics of the sensor to feedback the current ambient light brightness. Usually only one frame of image is needed. After looking up the table based on the brightness statistics, the target brightness parameters are obtained. Generally, the exposure parameters of the sensor are effective every other frame, that is, the parameters are configured at the end of the first frame and are not really effective until the third frame. Therefore, the soft photosensitivity method actually only needs three frames to reach the target brightness, which greatly shortens the convergence time.

[0155] Since system startup also takes time, convergence will be performed during the system startup process, which will affect each other. Therefore, a coprocessor inside the SOC is introduced to perform soft photosensitivity convergence in parallel. The main CPU is responsible for system startup. This way, they are not affected by each other, ensuring convergence effect and startup time. At the same time, resources are relatively independent and easy to manage.

[0156] The implementation flow chart of the present invention is as follows Figure 4 As shown, a low-power camera soft photosensitivity fast convergence method is provided, comprising the following steps:

[0157] S1, main processor power on;

[0158] S2, initialize and run the coprocessor, and run the system startup of step S3 and the convergence processing of the coprocessor of step S5 in parallel;

[0159] S3, the main processor initializes other resources necessary for system startup; the other resources include normal startup resources of the Linux system, including kernel firmware loading, file system loading, kernel firmware decompression, file system decompression, Linux kernel initialization, driver registration, etc.;

[0160] S4, the main processor sends a stop command to the coprocessor, and after completion, it goes to step S6; for the convenience of describing the communication method, it can be understood that there is a MessageBox register inside the Ingenic chip for communication between the two cores. The stop command is to write 0x1 into this register. When the small core reads the value of this register as 0x1, it is a stop command;

[0161] S5, convergence processing of the coprocessor further includes:

[0162] S5.1, the coprocessor starts running;

[0163] S5.2, configure the sensor and use fixed exposure parameters; the sensor exposure parameters mainly include two parts, one is exposure time, the other is exposure gain. The actual image brightness output by the sensor is the product of exposure time and exposure gain;

[0164] S5.3, initializing ISP;

[0165] S5.4, obtain brightness histogram statistical information; the brightness histogram is the information output by ISP statistics. This information will be saved in DDR after being output by ISP, and data can be directly taken out from DDR for calculation;

[0166] S5.5, configure the sensor parameters by looking up the table and send them to the main processor. After receiving the stop command, the coprocessor process ends. Since a new parameter is obtained after looking up the table, the sensor needs to be reconfigured. On the other hand, the ISP of the main processor also needs to know the latest parameter to synchronize with the sensor parameter to prevent abnormal brightness caused by asynchrony.

[0167] S6, synchronous stop state, synchronize the table lookup result to ISP, when the small core reads the stop command, it will write 0x0 to the MessageBox register, thus informing the main CPU that the main processor has stopped working normally; the stop action will be performed after the Sensor parameters are configured;

[0168] S7, enter the normal process.

[0169] Specifically, the specific implementation requirements and application implementations of the present invention are as follows:

[0170] Soft photosensitivity is based on the quick start system implementation principle:

[0171] The sensor fixes a set of exposure parameters when it is powered on, starts and converges under different ambient light conditions, and then obtains the initial average brightness information and the converged exposure parameters. These two parameters are collected under different lighting conditions to obtain a mapping table of the initial exposure parameters. While building the table, you can use a illuminance meter to record the illuminance of the current ambient light.

[0172] In actual use, the Sensor will obtain a starting average brightness value luma according to the fixed exposure parameters used when building the table in the actual scene. The corresponding EV value is found in the mapping table through the luma value, and then the exposure parameters can be reconfigured.

[0173] The day and night switch is also determined by the luma value. When the meter is built, the illumination information has been recorded and corresponds to the luma value one by one, so a luma value can be selected as the day and night switch threshold based on the illumination information.

[0174] Coprocessor software development requires porting ISP related resources, such as AE, AWB and other modules. Coprocessor software resource description is as follows Figure 5 shown.

[0175] Parameter passing method:

[0176] AE, AWB statistics:

[0177] The AE statistics data is 24KB and the AWB statistics data is 16KB, which will be assigned to the end of the DDR's rmem space. Since rmem is a reserved memory used for ISP images and encoding, the Linux system will not directly access it, so the memory data is written after statistics are performed during the coprocessor. In the Linux stage, when the ISP is initialized again, there is no need to obtain the AE parameters again and reconfigure them. The data after statistics by the small core, i.e. the coprocessor, can be directly used as the starting data.

[0178] If you want to print out the corresponding EV value for debugging, you can use the following API to obtain the starting AE and AWB parameters:

[0179] int tx_isp_get_ev_start(int*value);

[0180] int tx_isp_get_awb_start(awb_start_t*awb_start);

[0181] If you want to fine-tune this parameter, you can reset the starting AE parameter through the following API: int tx_isp_set_ev_start(int value);

[0182] int tx_isp_set_awb_start(awb_start_t awb_start);

[0183] Sensor exposure and gain parameters:

[0184] Since only soft photosensitivity table lookup is performed in the small core, the initial fixed exposure parameters will be used to obtain the luma information table lookup to obtain new exposure and gain parameters. After the small core sets the sensor parameters, this parameter also needs to be synchronized to the kernel, that is, the ISP module of the Linux kernel (the core of the operating system, mainly referring to the software part), and is also passed to the kernel in the form of a fixed memory address. The kernel can use the following API to obtain the final exposure and gain parameters of the sensor small core:

[0185] int tx_isp_get_sensor_analog_gain(int*value);

[0186] int tx_isp_get_sensor_integration_time(int*value);

[0187] Soft photosensitivity processing method and process in coprocessor:

[0188] The coprocessor soft light-sensing operation process has been completed in the src / main.c file. Since the initial luma value statistics require 1 to 2 frames and need to be counted based on fixed Sensor exposure parameters, the Sensor settings need to be turned off to prevent the ISP from resetting the Sensor parameters.

[0189] You need to set the ISP's dgain to 1x;

[0190] Since the luma value is affected by the exposure, Again, and ISP Dgain of the Sensor, in order to prevent the ISP Dgain from encoding in different scenarios and causing the luma value to change, the ISP Dgain is set to 1 times (1024 times) here. 1 times refers to the actual multiple, and 1024 is the corresponding setting parameter, with log domain conversion;

[0191] Then the luma value affects only the fixed parameters of the Sensor, such as Figure 6 As shown:

[0192] Soft photosensitivity operation requires closing the front sensor exposure parameter settings, which can be expressed as:

[0193] tx_isp_disable_sensor_expo();

[0194] Soft light-sensitive operation needs to turn off the dgain operation to prevent the influence of dgain. It is called before ISP initialization and dgain is set to 1x, which can be expressed as:

[0195] tx_isp_tuning_set_max_dgain(1024);

[0196] Initialize ISP, which can be expressed as:

[0197] tx_isp_init();

[0198] Initialize the video in module: MIPI, VIC, which can be expressed as:

[0199] tx_isp_video_in_init().

[0200] The actual soft photosensitivity operation process, such as Figure 7 As shown:

[0201] Determine whether soft photosensitization has been done, expressed as:

[0202] if(!soft_photosensitive_done);

[0203] If the luma value is stable, then the EV search is performed, expressed as:

[0204] if(last_luma!=sp_result.luma)

[0205] last_luma=sp_result.lulma;

[0206] Prepare AE Table, expressed as:

[0207] get_ae_table_info();

[0208] The day and night state is determined by the threshold, expressed as:

[0209] g_is_night=check_is_day_or_night(sp_result.luma);

[0210] IRCUT IRLED processing, expressed as:

[0211] set-irled_and_ircut(g_is_night);

[0212] Enable sensor exposure parameter settings, and set the ISP statistical results to the Sensor via I2C, expressed as: tx_isp_enable_sensor_expo();

[0213] Remove the limit of dgain, which is currently set to 8 times. It needs to be set according to the actual situation of the Sensor, expressed as:

[0214] tx_ips_tuning_Set_max_dgain(8096);

[0215] Get the starting AE parameters, expressed as:

[0216] g_ev=get_ae_table_value(g_start_ae_table_p,sp_result.luma,g_is_night);

[0217] printf("%s luma:%d g_is_night:%d ev:%d\n",__func__,sp_result.lulma,

[0218] g_is_night,g_ev);

[0219] Set the Sensor start parameters, expressed as:

[0220] tiziano_ae_s_ev_sensor_start(g_ev);

[0221] IRCUT restores the idle state to prevent power loss and chip burn.

[0222] Due to kernel operation, it is not processed here and is expressed as:

[0223] soft_photosensitive_done=1;

[0224] Among them, the role of soft_photosensitive_done: After the soft photosensitivity processing is completed, if there is still time, the restrictions on sensor exposure and dgain can be released to allow ISP to continue to perform AE convergence based on soft photosensitivity.

[0225] In actual product use, in order to process soft photosensitivity quickly, try to configure the Sensor initialization sequence to the highest available frame rate to process, which can improve the startup time. After the soft photosensitivity judgment is completed, switch to the actual frame rate. In the process of counting the starting luma value, it is necessary to stop the Sensor exposure operation and set the ISP Dgain to 1x. After the statistics are completed, restore the Sensor exposure settings and restore the ISP Dgain to the actual product requirement value. After the IRCUT operation is completed, the IRCUT needs to be restored to the IDLE state to prevent power loss and the risk of burning the IRCUT.

[0226] Soft photosensitivity table creation method:

[0227] Take a uniform light source and place it in a dark room without any interference from external light. Figure 8 As shown, place the device so that the lens can be as close as possible. Then adjust the brightness of the light source. You can use a illuminance meter to record the corresponding illuminance value, and gradually adjust from dark to bright. After adjustment, when the brightness is stable, turn the device on and off, and read the Luma and EV values ​​through the serial port and save them.

[0228] In the absence of uniform light sources, you can create a table by keeping the camera and the reference object about 5 meters apart in a dark room and changing the brightness of the room lights. Or, in a real environment, write a small script to record the brightness changes every 10 minutes for a day.

[0229] Method for fixing the initial exposure parameters:

[0230] You can refer to the corresponding Sensor register manual to modify the exposure time and gain parameters appropriately. You can also directly view the separately provided Sensor driver. The separately provided sensor driver refers to a separate driver file. Taking jxf35 as an example, find the exposure setting interface and see which registers need to be written to actually set the exposure. Find the corresponding registers and modify them. Fig. 9 The gain parameter can also be used in the same way, such as Fig.10 shown.

[0231] After finding the register, write the exposure parameters into the initialization sequence of the following file: tools / make_tag / sensor_init_setting / jxf35_t31_init.c

[0232] Then recompile the tag partition.

[0233] The exposure value, i.e., the exposure parameters (mainly including exposure time and exposure gain), can be a value after convergence in an office daytime environment. Printing can be added to the above-mentioned Sensor driver file to feedback the value.

[0234] In addition, you need to use a illuminance meter. When recording the startup luma value, it can cover the day and night switching area, and the possible perceived brightness range can be wider. The traditional photoresistor's sensitivity range is approximately 0 to 100 lux, and the soft photosensitivity's sensitivity range is mostly 0 to 6000 lux.

[0235] After obtaining the statistical values ​​in S5.4, record these data and create a table:

[0236] Use the printing method in the small core to print out the starting luma value of the src / main.c file, such as Fig.11 As shown:

[0237] Waiting for AE to process interrupt, represented by: tx_isp_wait_ae_interrupt();

[0238] AE statistical processing, represented by: tx_isp_ae_process();

[0239] Get the statistical results, expressed as: tx_isp_get_ae_result(&sp_result);

[0240] Print Statistics

[0241] #if UART_DEBUG_EN||DEBUG_LOGREAD_EN

[0242] printf("num:%d intt:%d again:%d dgain:%d luma:%d ev:%d\n",\

[0243] sp_result.num,\

[0244] sp_result.intt,\

[0245] sp_result.again,\

[0246] sp_result.dgain,\

[0247] sp_result.luma,\

[0248] sp_result.ev);

[0249] #endif.

[0250] At the same time, turn on the kernel debug function. The second parameter is the number of frames to be printed. It is recommended to print more, such as 100 frames, and at least not less than 10 frames. 10 frames can basically show the convergence process and are generally stable. If there are problems, it is recommended to print a few more frames to facilitate problem analysis. After debugging is turned on, you can see the statistical information of each frame through the dmesg information, such as Fig.12 shown.

[0251] After the modification is completed, recompile the ISP driver through make clean&&make&&make release, and then recompile the kernel and burn it.

[0252] After the system is up, you can see the corresponding statistical information through the dmesg information, and then record the statistical information in a table, such as Fig.13 shown.

[0253] Get luma value information through riscvread. It is recommended to print more than 10 frames and keep stable values, such as Fig.14 shown.

[0254] like Fig.15 As shown, the stable EV value information is obtained.

[0255] Then start it at different brightness levels and record the corresponding values. The Luma value range is 0 to 255, so a maximum of 255 data points can be counted. The more data points there are, the more accurate the EV value setting will be.

[0256] Fill the created table into the list corresponding to the following file, expressed as:

[0257] tools / make_tag / sensor_start_ae_table / libz_jxf35.c

[0258] Daytime parameter list: struct vol_start_value_table table_day[]

[0259] Night vision parameter list: struct vol_start_value_table table_night[]

[0260] Then recompile the tag partition and burn it.

[0261] like Figures 16 to 18 As shown, it is the measured effect obtained based on the method of the present invention. Fig.16 This is the first picture. Fig.17 It's the second frame. Fig.18 This is a schematic diagram of the effect of the third frame.

[0262] In summary, the method described in the present invention of using soft photosensitivity to make the first image fast and stable, and the method of using a coprocessor to improve the parallel processing speed are both different from the methods of the prior art and fall within the protection scope of the present invention.

[0263] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for fast convergence of soft photosensitivity of a low-power camera, characterized in that: The method comprises the following steps: S1, main processor power on; S2, initialize and run the coprocessor, and run the system startup of step S3 and the convergence processing of the coprocessor of step S5 in parallel; S3, the main processor initializes other resources necessary for system startup, the other resources include normal startup resources of the Linux system, including kernel firmware loading, file system loading, kernel firmware decompression, file system decompression, Linux kernel initialization, and driver registration; S4, the main processor sends a stop command to the coprocessor. A MessageBox register is set for communication between the dual cores, i.e., the main processor and the coprocessor. The stop command is to write 0x1 into this register. When the small core reads the value of this register as 0x1, it is a stop command. After completion, go to step S6; S5, soft photosensitivity convergence processing of the coprocessor. The soft photosensitivity operation is based on the quick start system. The sensor fixes a set of exposure parameters when it is powered on, starts and completes convergence under different ambient light conditions, and then obtains the initial average brightness information and the converged exposure parameters. The two parameters are collected under different lighting conditions to obtain a mapping table of the initial exposure parameters. While building the table, a light meter can be used to record the illumination of the current ambient light. In actual use, the Sensor will obtain a starting average brightness value luma according to the fixed exposure parameters used when building the table in the actual scene. The corresponding EV value is found in the mapping table through the luma value, and then the exposure parameters can be reconfigured. The day and night switching is also determined by the luma value. When the meter is built, the illumination information has been recorded and corresponds to the luma value one by one, so a luma value can be selected as the day and night switching threshold according to the illumination information; further including: S5.1, the coprocessor starts running; S5.2, configure the sensor Sensor, use fixed exposure parameters, the sensor exposure parameters include two parts, one is exposure time, the other is exposure gain, the actual image brightness output by the sensor is the product of exposure time and exposure gain; S5.3, initializing ISP; S5.4, obtaining brightness histogram statistical information, wherein the brightness histogram is information output by ISP statistics, and this information is stored in DDR after being output by ISP, and data can be directly taken out from DDR for calculation; S5.5, configure the sensor parameters through table lookup and send them to the main processor. Since a new parameter is obtained after table lookup, the sensor needs to be reconfigured. On the other hand, the ISP of the main processor also needs to know the latest parameter to synchronize with the sensor parameters to prevent abnormal brightness caused by asynchrony. After receiving the stop command, the stop action is performed after the sensor parameters are configured, and the coprocessor process ends. S6, synchronous stop state, synchronize the table lookup result to ISP. When the small core, i.e. the coprocessor, reads the stop command, it writes 0x0 to the messagebox register, thus informing the main processor that it has stopped working normally. S7, enter the normal process.

2. A method for fast convergence of soft photosensitivity of a low-power camera according to claim 1, characterized in that: The step S5 further comprises: S5.1, in the coprocessor software running on the coprocessor, ISP related resources need to be transplanted, including AE and AWB modules; Among them, AE and AWB statistics: The AE statistics data is 24KB and the AWB statistics data is 16KB, which will be assigned to the end of the DDR's rmem space. Since rmem is a reserved memory used for ISP images and encoding, the Linux system will not directly access it, so the memory data is written after statistics during the coprocessor period. In the Linux stage, when the ISP is initialized again, there is no need to obtain AE parameters and reconfigure them. The small core, i.e., the coprocessor, can be used directly, and the statistical data is used as the starting data. If you want to print out the corresponding EV value for debugging, you can use the following API to obtain the starting AE and AWB parameters: int tx_isp_get_ev_start(int*value); int tx_isp_get_awb_start(awb_start_t*awb_start); If you want to fine-tune this parameter, you can reset the starting AE parameter through the following API: int tx_isp_set_ev_start(int value); int tx_isp_set_awb_start(awb_start_t awb_start); S5.2, configure the sensor and use fixed exposure parameters: The operation flow of the coprocessor soft photosensitivity has been completed in the src / main.c file. Since the initial luma value statistics require 1 to 2 frames, statistics need to be performed based on fixed Sensor exposure parameters, so the Sensor settings need to be turned off to prevent the ISP from resetting the Sensor parameters. You need to set the ISP's dgain to 1x; Since the luma value will be affected by the exposure, Again, and ISP Dgain of the Sensor, in order to prevent the ISP Dgain from encoding in different scenarios and causing the luma value to change, the ISP Dgain is set to 1 times here. 1 times refers to the actual multiple. The corresponding setting parameter for log domain conversion is 1024; then the luma value is only related to the fixed parameters of the Sensor; Soft photosensitivity operation requires closing the front sensor exposure parameter settings, which can be expressed as: tx_isp_disable_sensor_expo(); Soft light-sensitive operation needs to turn off the dgain operation to prevent the influence of dgain. It is called before ISP initialization and dgain is set to 1x, which can be expressed as: tx_isp_tuning_set_max_dgain(1024); Sensor exposure and gain parameters: Since only soft photosensitivity table lookup is performed in the small core, the initial fixed exposure parameters will be used to obtain the luma information table lookup to obtain new exposure and gain parameters. After the small core sets the sensor parameters, this parameter also needs to be synchronized to the kernel, that is, the ISP module of the Linux kernel, and is also passed to the kernel through a fixed memory address. The kernel can use the following API to obtain the final exposure and gain parameters in the case of the sensor small core: int tx_isp_get_sensor_analog_gain(int*value); int tx_isp_get_sensor_integration_time(int*value); S5.3, initializing ISP; Initialize ISP, which can be expressed as: tx_isp_init(); Initialize the video in module: MIPI, VIC, which can be expressed as: tx_isp_video_in_init().

3. A method for fast convergence of soft photosensitivity of a low-power camera according to claim 2, characterized in that: The operation process of the soft photosensitivity in step S5 further includes: Determine whether soft photosensitization has been done, expressed as: if(!soft_photosensitive_done); If the luma value is stable, then the EV search is performed, expressed as: if(last_luma!=sp_result.luma) last_luma=sp_result.lulma; Prepare AE Table, expressed as: get_ae_table_info(); The day and night state is determined by the threshold, which is expressed as: g_is_night=check_is_day_or_night(sp_result.luma); IRCUT IRLED processing, expressed as: set-irled_and_ircut(g_is_night); Enable sensor exposure parameter settings, and set the ISP statistical results to the Sensor via I2C, expressed as: tx_isp_enable_sensor_expo(); Remove the limit of dgain, which is currently set to 8 times. It needs to be set according to the actual situation of the Sensor, expressed as: tx_ips_tuning_Set_max_dgain(8096); Get the starting AE parameters, expressed as: g_ev=get_ae_table_value(g_start_ae_table_p,sp_result.luma,g_is_night); printf("%s luma:%d g_is_night:%d ev:%d\n",__func__,sp_result.lulma,g_is_night,g_ev); Set the Sensor starting parameters, expressed as: tiziano_ae_s_ev_sensor_start(g_ev); IRCUT restores the idle state to prevent power loss and chip burn. Due to kernel operation, it is not processed here and is expressed as: soft_photosensitive_done=1; Among them, the role of soft_photosensitive_done: After the soft photosensitivity processing is completed, if there is still time, the restrictions on sensor exposure and dgain can be released to allow ISP to continue to perform AE convergence based on soft photosensitivity.

4. A method for fast convergence of soft photosensitivity of a low-power camera according to claim 3, characterized in that: In actual product applications, for fast processing of soft light sensitivity, the sensor initialization sequence is configured to be processed at the highest available frame rate, which can improve the startup time; After the soft photosensitivity judgment is completed, switch to the actual frame rate; When counting the starting luma value, you need to stop the sensor exposure operation and set the ISP Dgain to 1x. After the statistics are completed, restore the sensor exposure settings and restore the ISP Dgain to the actual product requirement value. After the IRCUT operation is completed, the IRCUT needs to be restored to the IDLE state to prevent power loss and the risk of burning the IRCUT.

5. A method for fast convergence of soft photosensitivity of a low-power camera according to claim 4, characterized in that: The soft photosensitivity table building method comprises: Take a uniform light source and put it in a dark room without any external light interference. Place the device so that the lens can be as close as possible. Then adjust the brightness of the light source. You can use a illuminance meter to record the corresponding illuminance value, and gradually adjust it from dark to bright. After adjustment, when the brightness is stable, turn on and off the device, and read the Luma and EV values ​​through the serial port and save them. In the absence of uniform light sources, a table can be created by keeping a distance of 5 meters between the camera and the reference object in a dark room and changing the brightness of the room lights. Alternatively, this can be achieved in a real environment by writing a program script that records the brightness changes every 10 minutes throughout the day.

6. A method for fast convergence of soft photosensitivity of a low-power camera according to claim 1, characterized in that: The method for fixing the initial exposure parameters includes: Refer to the corresponding Sensor register manual and modify the exposure time and gain parameters appropriately. You can also directly view the separately provided Sensor driver. Assuming that the Sensor is a jxf35 sensor, find the exposure setting interface and see which registers need to be written to actually set the exposure. Find the registers and modify them accordingly. The gain parameter can also be obtained in the same way. After finding the register, write the exposure parameter into the initialization sequence of the following file. tools / make_tag / sensor_init_setting / jxf35_t31_init.c Then recompile the tag partition; Selection of exposure value, which is the exposure parameter, including exposure time and exposure gain. You can take a value after convergence in an office daytime environment. You can add printing to the above Sensor driver file to feedback the value; In addition, you need to use a illuminance meter. When recording the startup luma value, it can cover the day and night switching area and have a wide range of perceived brightness. The traditional photoresistor's sensitivity range is 0 to 100 lux, and the soft photosensitivity's sensitivity range is 0 to 6000 lux.

7. A method for fast convergence of soft photosensitivity of a low-power camera according to claim 6, characterized in that: In the step S5.4, statistics are generated and a table is created: after obtaining the statistical values ​​in S5.4, these data are recorded and a table is created; Using the printing method in the small core, print out the starting luma value of the src / main.c file, which can be expressed as: Waiting for AE to process interrupt, expressed as: tx_isp_wait_ae_interrupt(); AE statistical processing, represented by: tx_isp_ae_process(); Get the statistical results, expressed as: tx_isp_get_ae_result(&sp_result); Print Statistics #if UART_DEBUG_EN||DEBUG_LOGREAD_EN printf("num:%d intt:%d again:%d dgain:%d luma:%d ev:%d\n",\ sp_result.num,\ sp_result.intt,\ sp_result.again,\ sp_result.dgain,\ sp_result.luma,\ sp_result.ev); #endif; At the same time, turn on the kernel debug function. The second parameter is the number of frames to be printed, which should be at least 10 frames. After debug is turned on, you can see the statistics of each frame through dmesg information. After the modification is completed, recompile the ISP driver through make clean&&make&&make release, and then recompile the kernel and burn it.

8. A method for fast convergence of soft photosensitivity of a low-power camera according to claim 7, characterized in that: After the system is started up quickly, the corresponding statistical information can be seen through the dmesg information, and then the statistical information can be recorded in a table; Get luma value information through riscvread, do not drop below 10 frames, and keep a stable value; Obtain the stable EV value information; Then start it at different brightness levels and record the corresponding values. The Luma value range is 0 to 255, so a maximum of 255 data points can be counted. The more data points, the more accurate the EV value setting. Fill the created table into the list corresponding to the following file, expressed as: tools / make_tag / sensor_start_ae_table / libz_jxf35.c Daytime parameter list: struct vol_start_value_table table_day[] Night vision parameter list: struct vol_start_value_table table_night[] Then recompile the tag partition and burn it.

Images