Method for optimizing quick-start effect of low-power-consumption electronic product

Abstract

The invention provides a quick-start effect optimization method for a low-power-consumption electronic product. The quick-start effect optimization method comprises the following steps: S1, starting: waking up equipment by any event; s2, the device reads the AD value of the current environment and looks up the AD-EV table according to the AD value to obtain the EV value of the current AD in the laboratory environment; s3, the coprocessor RISCV is started at the boot starting stage, an EV value is obtained, an ISP module is started, a gain value is analyzed, and a sensor SENSOR register is configured; s4, after the master control system-level chip SOC is started, obtaining first frame video data after SENSOR updates the exposure gain; s5, acquiring the current AD value and the stable value of the EV, and updating the current AD value and the stable value of the EV into the AD-EV table through the MTD node of the MTD subsystem, that is, the AD-EV table is updated after each time of starting, and the updating frequency is controllable; and S6, after the equipment completes all services, the equipment is shut down and waits for next start. The method well solves the problem that the first-frame image is abnormal due to the difference between the laboratory environment and the actual environment. And the product effect is better and better under the condition that the self-learning system is continuously quantized.

Description

Technical Field

[0001] The present invention belongs to the technical field of smart home, and in particular relates to a method for optimizing the quick-start effect of low-power electronic products. Background Art

[0002] Existing low-power electronic products, take the product form doorbell as an example. The first frame image effect is very demanding. In real commercial scenarios, the first frame of the picture may be too dark or overexposed. Taking the product doorbell as an example, in actual commercial application scenarios, if there are other events such as guests visiting or passers-by passing by the door, the visitor or passer-by may only appear for a few seconds in the normal viewing angle of the doorbell. When the doorbell detects a person passing by, it will normally wake up the SOC main control through PIR or other methods. After the main control is awakened, it will enable the SENSOR, ISP and other modules to record the scene under the lens. At this time, if no processing is done, the ISP gets a fixed EV value for exposure convergence, and it may take several frames or even more time to converge the picture to a stable EV value for the current scene. At this time, the visitor may have staggered the doorbell's field of view or left. The photos captured in the product may be some pictures with dark or overexposed scenes. In this scenario, the visitor's information cannot be normally recognized, and the product experience is poor.

[0003] In the prior art, the AD-EV table is pre-configured. There are corresponding pre-processing operations based on the environmental background, which can solve the problem of overexposure or over-darkness in the first few frames at startup to a certain extent. In the dark room environment of the laboratory, the ambient brightness is adjusted from 0 to the maximum value of the AD acquisition value through a light box, and the EV values ​​after the environment stabilizes under different AD values ​​are counted and counted in the table. The smaller the interval between the AD values, the better the effect, the more detailed the EV value information, the closer it is to the actual environment during use, and the better the effect of the first frame of the image at startup.

[0004] After the SOC is started, the embedded system reads the photosensitivity AD value during the boot phase, and traverses the AD-EV table to obtain the EV value counted in the laboratory environment based on the read current environment AD value. The SOC with co-processing can enable the ISP module through the co-processor to download the EV value to the ISP module to parse the gain value and send the corresponding register configuration to the SENSOR. At this time, the image output by the SENSOR is close to the normal scene.

[0005] The existing technologies mentioned above can alleviate the overexposure or overdarkness of the first frame in actual scenes to a certain extent. However, the obvious disadvantage is that the laboratory environment is different from the actual use environment. It can be understood that the EV values ​​of two different scenes with the same photosensitivity AD value obtained by the same device are different.

[0006] The fundamental reason for the difference is that the acquisition method and range of the photosensitive AD value are different from the acquisition method and range of the SESNOR photosensitivity. The statistical values ​​of the two may be coupled to a certain extent, but under normal circumstances, they cannot be completely consistent.

[0007] This shortcoming can be manifested even in a laboratory environment. Simulate an extreme scenario, such as modifying the brightness of the light box to a stable value in a dark room environment, assuming that the AD acquisition in this environment is 1500. At this time, within the photosensitivity range of the sensor lens, a light source is lit outside the photosensitivity range. At this time, the AD value remains unchanged, while the EV value of the sensor will change.

[0008] This is a relatively extreme example. In an actual environment, the situation may be more complicated. Therefore, the above method cannot solve the problems existing in the prior art without increasing the hardware cost.

[0009] In addition, the commonly used terms in the prior art include:

[0010] AE: Automatic exposure. The module in ISP that adjusts the image brightness to make the image reach the appropriate brightness for human eyes.

[0011] AE mainly consists of two modules:

[0012] One is the AE statistics module, which is usually a hardware module that requires powerful computing power and outputs brightness histograms, brightness area statistics, current brightness, and ambient brightness.

[0013] The second is the AE algorithm module, which is the module we need to debug. The parameters of this part are passed in by ISP debugging, and the AE behavior is determined after calculation based on the previous AE statistical values. If AE convergence is required, set the SENSOR internal shutter and gain or ISP gain.

[0014] EV: Exposure Values. A value that reflects the exposure. In a stable environment, the larger the EV, the lower the ambient brightness, and the smaller the EV, the brighter the ambient brightness.

[0015] Overexposure: Overexposure is different from highlights. When a photo is exposed for too long, the highlights overflow and the entire photo looks very close to white and very bright, this is called overexposure. A large amount of overexposure will cause the details to be lost and the overall picture will be white.

[0016] ISP: Image Processor.

[0017] SENSOR: Digital imaging sensor. For example, digital cameras, camera phones, and digital SLR cameras all have a SENSOR that works with the lens to capture and image digital images.

[0018] AD value: The characteristic of photoresistors is that they change resistance when exposed to light. The microcontroller uses the voltage detected by AD to determine the brightness of the current ambient light.

[0019] MTD subsystem: (Memory Technology Device) is commonly referred to as Flash and other storage devices that use memory chips. The MTD subsystem corresponds to the device driver layer in the block device driver framework. It can be said that MTD is a standardized hardware driver framework designed for Flash devices. Using the MTD subsystem to write flash device drivers is very fast. Summary of the invention

[0020] In order to solve the above problems, the purpose of this application is: after the existing technology uses the AD-EV table processing method, it can meet certain practical scenarios and some products with lower requirements, but for actual abnormal scenarios and high-specification products, this method optimizes and improves product functions by adding AE self-learning technology on the basis of existing technology.

[0021] This technology can make the EV value closer to the actual value in general scenes where the actual EV value is not much different from the table EV value. The gain effect of this technology in such scenes may not be obvious, but in the environment where the actual environment EV is very different from the table EV value statistically in the laboratory, the effect is obvious, and it can be clearly seen that the previous overexposure or overdarkness will be improved. The effect will also be better in general scenes, and this application is particularly effective for such abnormal scenes.

[0022] Specifically, the present invention provides a method for optimizing the quick start effect of a low-power electronic product, the method comprising the following steps:

[0023] S1, Start: The device is awakened by any event;

[0024] S2, the device reads the current environment AD value, and looks up the AD-EV table based on the value to obtain the current AD EV value in the laboratory environment;

[0025] S3, the coprocessor RISCV starts in the boot stage and gets the EV value to start the ISP module, parses the gain value and configures the sensor SENSOR register; that is, the relevant algorithm operations of ISP AE are transplanted to the coprocessor RISCV for execution. After the RISCV firmware is loaded in the early stage of the big core CPU boot, RISCV starts and executes the ISP AE related code to perform sensorISP AE convergence statistics related operations;

[0026] S4: After the main control system-level chip SOC is started, it obtains the first frame of video data after the SENSOR updates the exposure gain. That is, after the SOC is started to the kernel stage, RISCV has already performed AE processing on multiple frames of data. At this time, RISCV directly transmits the processed data to the large core ISP, and ISP processes the first frame of video data based on the data.

[0027] S5, obtain the current AD value and the stable value of EV, and update them to the AD-EV table through the MTD node of the MTD subsystem:

[0028] The MTD subsystem is referred to as the memory device management system. Since the AD-EV information required by this function is stored in the storage medium flash, when new data is learned according to different environments, it is naturally necessary to rewrite it into the storage device to replace the previous value;

[0029] Different SOC platforms will follow the standard Linux MTD subsystem architecture and implement the underlying functional interface based on different controllers. The operation of writing Flash data can be realized by calling the write interface of the standard MTD subsystem, that is, the operation of updating the AD-EV table described above; that is, the AD-EV table is updated after each startup, and the update frequency can be set. This part is completely open to users;

[0030] S6: After completing all tasks, the device shuts down and waits for the next startup.

[0031] The step S5 further comprises:

[0032] After each system startup, the ISP module will get the EV value after stabilization in the current environment. When the EV values ​​of multiple consecutive frames of images counted by the ISP module remain unchanged, the EV value in this environment is considered to be stable. In other words, under normal circumstances, if the EV of 10 consecutive frames of image data remains unchanged or the difference between the upper and lower floating values ​​is in the single digit, the environment can be considered to be stable. At this time, the application can obtain the AD value in this environment.

[0033] The actual environment value corresponding to AD-EV at this time is overwritten by the MTD subsystem to the corresponding value in the AD-EV table obtained in the laboratory environment.

[0034] In actual usage scenarios, the method will collect the EV value obtained under the same AD value for 3 or more times to reduce errors. Theoretically, the larger the base number of collections, the smaller the error rate. The average is saved and then replaced in the AD-EV table.

[0035] The method is based on the use of AD-EV tables, combined with laboratory data to make deviation corrections, and the corrected data is used for the next start-up to optimize the start-up effect.

[0036] In the method, the moment when the same device is awakened is random. After using this technical solution, it can be ensured that the EV value information obtained when the device is started in the current scenario, that is, the next time in the AD value scenario, is closer to the actual value than the last time, and the image of the first frame of SENSOR is closer to the scene after convergence.

[0037] For low-power products, the method makes the quantized data of the function infinitely close to the actual scene as the number of startup times under different AD values ​​increases, so that the first frame obtained when the low-power product is quickly started is an image under a normal exposure environment.

[0038] The method is applicable to the first frame overexposure and overdarkness environment, and is applicable to the embedded startup low-power device fast start product industry.

[0039] The method is applied to the case of multiple starts under normal business.

[0040] Therefore, the advantage of this application is that compared with the existing solutions, this solution solves the abnormal situation of the first frame image caused by the difference between the laboratory environment and the actual environment. The existing technical solutions will bring a significantly poor experience to the product in certain specific scenarios due to the fixed EV value at the start time. This situation will not occur in this solution, and the product effect will become better and better under the continuous quantization of the self-learning system. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0042] Figure 1 It is a schematic diagram of the AE self-learning process involved in this application.

[0043] Figure 2 It is a schematic diagram of the present application method. DETAILED DESCRIPTION

[0044] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.

[0045] like Figure 1 As shown, the present application proposes a method for optimizing the quick start effect of low-power electronic products, which starts multiple times under normal business, and goes through startup, table lookup, configuration of SENSOR register, acquisition of first frame, table update, shutdown; and finally waits for restart.

[0046] Furthermore, if Figure 2 As shown, multiple starts under normal business include the following steps: S1, start: the device is awakened (any event);

[0047] S2, the device reads the current environment AD value, and looks up the AD-EV table based on the value to obtain the current AD EV value in the laboratory environment;

[0048] S3, here we use the Beijing Ingenic Integrated Circuit Co., Ltd. (Ingenic) platform T31 / T40 / T41 coprocessor RISCV to start in the boot phase and get the EV value to start the ISP module, parse the gain value and configure the SENSOR register; (other platforms or SOCs without coprocessors do not have this operation);

[0049] In the boot phase of the Linux system boot process, the big core CPU is executed serially. In this phase, the CPU is loading and decompressing the kernel and rootfs. Under normal circumstances, this process takes more than 100ms. Therefore, the CPU cannot perform AE algorithm analysis at this time. It is for this reason that the relevant algorithm operations of ISP AE are transplanted to the coprocessor RISCV for execution. After the RISCV firmware is loaded in the early stage of the big core CPU boot, RISCV starts and executes the ISP AE related code to perform sensor ISP. AE convergence statistics related operations; further, the boot process is serial, and the big core CPU cannot do other operations when loading the kernel and file system, so some ISP operations are transplanted to RISCV for processing; when the big core CPU processes the kernel node and starts the ISP module on the big core CPU, it will wait for RISCV to send a signal that the processing is completed; and after RISCV completes the processing of the set number of frames, it will send a signal to the CPU. The CPU can continue to execute after receiving the signal, otherwise it will be blocked here until it times out; due to this setting, the number of frames processed by RISCV is generally set to 3 frames. Too many frames will cause the RISCV processing time to be too long, which will make the CPU wait for RISCV processing, affecting the startup speed;

[0050] S4, after the main control SOC is started, it obtains the first frame of video data after the SENSOR updates the exposure gain; that is, the relevant algorithm operations of ISP AE are transplanted to the coprocessor RISCV for execution. After the RISCV firmware is loaded in the early stage of the big core CPU boot, RISCV starts and executes the ISP AE related code to perform sensor ISP AE convergence statistics related operations; RISCV will calculate the statistical information of the sensor during the processing process, and judge whether the current image is overexposed or too dark based on the statistical information. If it is overexposed, the exposure-related register information will be lowered, otherwise it will be raised. Therefore, RISCV obtains the relevant register information through algorithm processing;

[0051] S5, obtain the current AD value and the stable value of EV, and update them to the AD-EV table through the MTD node, that is, the AD-EV table can be updated after each startup, and the update frequency is controllable by the software developer; (update the AD-EV table through the MTD subsystem); the MTD subsystem is referred to as the memory device management system. Since the AD-EV information required for this function is stored in the storage medium flash, when new data is learned according to different environments, it is naturally necessary to rewrite it to the storage device to replace the previous value; different SOC platforms will follow the standard Linux mtd subsystem architecture based on different controllers to implement the underlying functional interface. The operation of writing Flash data can be implemented by calling the write interface of the standard MTD subsystem, that is, the operation of updating the AD-EV table described above, that is, the AD-EV table is updated after each startup, and the update frequency can be set by the user;

[0052] S6: After completing all tasks, the device shuts down and waits for the next startup.

[0053] The AE self-learning function is based on the use of AD-EV tables, combined with laboratory data to make deviation corrections. The corrected data is used for the next startup to optimize the startup effect.

[0054] The step S5 further comprises:

[0055] After each system startup, the ISP module will get the EV value after stabilization in the current environment. When the EV values ​​of more than ten consecutive frames of images counted by the ISP module remain unchanged, the EV value in this environment is considered to be stable. Normally, if the EV of 10 consecutive frames of image data remains unchanged or the difference between the upper and lower floating values ​​is in the single digit, the environment can be considered stable. At this time, the application will obtain the AD value in this environment.

[0056] The actual environment value corresponding to AD-EV at this time is overwritten by the MTD subsystem to the corresponding value in the AD-EV table obtained in the laboratory environment.

[0057] In actual usage scenarios, in order to reduce errors, the EV value obtained under the same AD value will be collected 3 or more times. In theory, the larger the base number collected, the smaller the error rate. After saving and averaging, it will be replaced in the AD-EV table.

[0058] The time when the same device is awakened is random. After using this technical solution, it can be ensured that the EV value information obtained when the device is started in the current scenario, that is, the next time in the AD value scenario, is closer to the actual value than the last time, and the image of the first frame of the SENSOR is closer to the scene after convergence.

[0059] For low-power products, as the number of startups under different AD values ​​increases, the quantified data of this function is infinitely close to the actual scene, so that the first frame obtained when the low-power product is quickly started is an image under a normal exposure environment. The method is applicable to the first frame in an over-exposure and dark environment, and is applicable to the embedded startup low-power device fast startup product industry. The method is applied to the situation of multiple startups under normal business.

[0060] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for optimizing the quick start effect of low-power electronic products, characterized in that: The method The following steps are involved: S1, Start: The device is awakened by any event; S2, the device reads the current environment AD value, and looks up the AD-EV table based on the value to obtain the current AD EV value in the laboratory environment; S3, the coprocessor RISCV starts in the boot stage and gets the EV value to start the ISP module, parses the gain value and configures the sensor SENSOR register; that is, the relevant algorithm operations of ISP AE are transplanted to the coprocessor RISCV for execution. After the RISCV firmware is loaded in the early stage of the big core CPU boot, RISCV starts and executes the ISP AE related code to perform sensorISP AE convergence statistics related operations; S4: After the main control system-level chip SOC is started, it obtains the first frame of video data after the SENSOR updates the exposure gain. That is, after the SOC is started to the kernel stage, RISCV has already performed AE processing on multiple frames of data. At this time, RISCV directly transmits the processed data to the large core ISP, and ISP processes the first frame of video data based on the data. S5, obtain the current AD value and the stable value of EV, and update them to the AD-EV table through the MTD node of the MTD subsystem: The MTD subsystem is referred to as the memory device management system. Since the AD-EV information required by this function is stored in the storage medium flash, when new data is learned according to different environments, it is naturally necessary to rewrite it into the storage device to replace the previous value; Different SOC platforms will follow the standard Linux MTD subsystem architecture and implement the underlying functional interface based on different controllers. The operation of writing Flash data can be realized by calling the write interface of the standard MTD subsystem, that is, the operation of updating the AD-EV table described above; that is, the AD-EV table is updated after each startup, and the update frequency can be set. This part is completely open to users; S6: After completing all tasks, the device shuts down and waits for the next startup.

2. A method for optimizing the quick start effect of low power consumption electronic products according to claim 1, characterized in that: The step S5 further comprises: After each system startup, the ISP module will get the EV value after stabilization in the current environment. When the EV values ​​of multiple consecutive frames of images counted by the ISP module remain unchanged, the EV value in this environment is considered to be stable. In other words, under normal circumstances, if the EV of 10 consecutive frames of image data remains unchanged or the difference between the upper and lower floating values ​​is in the single digit, the environment can be considered to be stable. At this time, the application can obtain the AD value in this environment. The actual environment value corresponding to AD-EV at this time is overwritten by the MTD subsystem to the corresponding value in the AD-EV table obtained in the laboratory environment.

3. A method for optimizing the quick start effect of low power consumption electronic products according to claim 2, characterized in that: In actual usage scenarios, the method will collect the EV value obtained under the same AD value for 3 or more times to reduce errors. Theoretically, the larger the base number of collections, the smaller the error rate. The average is saved and then replaced in the AD-EV table.

4. The method for optimizing the quick start effect of low power consumption electronic products according to claim 1, characterized in that: The method is based on the use of AD-EV tables, combined with laboratory data to make deviation corrections, and the corrected data is used for the next start-up to optimize the start-up effect.

5. The method for optimizing the quick start effect of low power consumption electronic products according to claim 1, characterized in that: In the method, the moment when the same device is awakened is random. After using this technical solution, it can be ensured that the EV value information obtained when the device is started in the current scenario, that is, the next time in the AD value scenario, is closer to the actual value than the last time, and the image of the first frame of SENSOR is closer to the scene after convergence.

6. A method for optimizing the quick start effect of low power consumption electronic products according to claim 5, characterized in that: For low-power products, the method makes the quantized data of the function infinitely close to the actual scene as the number of startup times under different AD values ​​increases, so that the first frame obtained when the low-power product is quickly started is an image under a normal exposure environment.

7. The method for optimizing the quick start effect of low power consumption electronic products according to claim 1, characterized in that: The method is applicable to the first frame overexposure and overdarkness environment, and is applicable to the embedded startup low-power device fast start product industry.

8. The method for optimizing the quick start effect of low power consumption electronic products according to claim 1, characterized in that: The method is applied to the case of multiple starts under normal business.

Images