Method and device for photographing images, electronic device and storage medium

By prioritizing the initialization of the image signal processing module and generating images in the first processing system of the intelligent monitoring device, the problem of delay in capture time after the low-power consumption device is turned on is solved, and fast capture within 200-300ms is achieved, suitable for intelligent cat eyes etc.

CN115665543BActive Publication Date: 2024-11-05FUZHOU ROCKCHIP SEMICON
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202211376077.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-11-04
Publication Date
2024-11-05
Estimated Expiration
2042-11-04

AI Technical Summary

Technical Problem

The existing intelligent monitoring equipment takes as fast as 700ms to capture the camera after booting, and cannot quickly capture images on low-power products such as smart cat eyes, hunting cameras, smart doorbells and other devices, resulting in the inability to capture effective evidence in a short period of time.

Method used

By prioritizing the image signal processing module after the first processing system is started, image processing is generated, and image processing is performed after the second processing system is started, the image shooting and processing process are separated, and the time difference between power-on and shooting is shortened.

Benefits of technology

It can quickly capture the first frame of the image within 200-300ms after booting, which is suitable for low-power devices, reduce the time difference from booting to shooting, and improve the device's fast capture capability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115665543B_ABST
    Figure CN115665543B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and device for shooting images, an electronic device and a storage medium, which start a first processing system; initialize an image signal processing module through the first processing system; generate at least one frame of image captured through the image signal processing module; and perform image processing on the at least one frame of image to generate a captured image through a second processing system; the application separates the shooting and processing of images, and only loads the image signal processing module to shoot the image when the device is turned on, without waiting until the application is loaded to load the image signal processing module, and advances the time point when the image can be shot, so that the photo can be shot more quickly after the device is turned on, further shortening the time difference from turning on the device to shooting the photo, and realizing fast snapshot after the device is turned on.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of startup optimization, and in particular to a method and device for shooting images, an electronic device and a storage medium. Background Art

[0002] The startup process of existing intelligent monitoring equipment is generally: power on → hardware initialization → kernel (system kernel) load image → kernel initialization → rootfs (root file system) initialization → APP initialization. When the APP is initialized, the camera and image signal processing module (ISP) are initialized, and then the capture begins.

[0003] In order to take photos quickly after the device is turned on, existing technologies have also optimized various processes. At present, when the optimization is optimized to the best effect, it takes about 700ms (milliseconds) to capture the photo. However, for some special products, such as low-power products, 700ms is still relatively long, such as smart cat eyes, hunting cameras, smart doorbells, smart door locks and other products. When someone flashes by, it will be impossible to capture effective evidence after 700ms. Summary of the invention

[0004] The present invention provides a method and device for shooting images, an electronic device and a storage medium, which can realize rapid capture after the device is turned on.

[0005] In one aspect of the present invention, a method for capturing an image is provided, the method comprising: starting a first processing system; initializing an image signal processing module through the first processing system; generating at least one captured image frame through the image signal processing module; and performing image processing on the at least one frame of image through a second processing system to generate a captured image.

[0006] In some embodiments, the time required for the first processing system to complete startup is shorter than the time required for the second processing system to complete startup.

[0007] In some embodiments, starting the first processing system includes: acquiring a boot program stored in the second processing system; and booting the first processing system according to the boot program.

[0008] In some embodiments, booting the first processing system according to the boot program includes: loading a first processing system image corresponding to the first processing system through the boot program to start the first processing system.

[0009] In some embodiments, the method further includes: in response to powering on, starting the second processing system.

[0010] In some embodiments, starting the second processing system includes: loading a boot program from a boot memory; initializing an external memory, loading a first processing system image corresponding to the first processing system, and loading a system kernel image through the boot program; starting an embedded system boot loader; and starting a second processing system kernel, a file system, and an application.

[0011] In some embodiments, after generating at least one frame of captured image through the image signal processing module, it also includes: storing the at least one frame of image in a random access memory; and starting an application of the second processing system.

[0012] In some embodiments, performing image processing on the at least one frame of image by the second processing system to generate a captured image includes: after successfully starting the application, reading the at least one frame of image from the random access memory; and performing image processing on the at least one frame of image by the application to generate the captured image.

[0013] In some embodiments, the first processing system is an MCU, and the second processing system is an ARM.

[0014] In some embodiments, the first processing system is a first processing core in a multi-core processor, and the second processing system is a second processing core in the multi-core processor.

[0015] In another aspect of the present invention, a device for capturing images is provided. The device includes: an image signal processing module configured to receive at least one frame of raw image signal from a camera, and perform image signal processing on the at least one frame of raw image signal to generate at least one frame of image; a first processing system configured to initialize the image signal processing module; and a second processing system configured to start the first processing system and perform image processing on the at least one frame of image to generate a captured image.

[0016] In some embodiments, the time required for the first processing system to complete startup is shorter than the time required for the second processing system to complete startup.

[0017] In some embodiments, the second processing system is configured to: obtain a boot program stored in the second processing system; and boot the first processing system according to the boot program.

[0018] In some embodiments, the second processing system is further configured to: load the boot program in response to power-on; and load a second processing system kernel corresponding to the second processing system in response to startup of the first processing system.

[0019] In some embodiments, the image signal processing module is configured to store the at least one frame of image in a random access memory, and the second processing system is further configured to start an application of the second processing system.

[0020] In some embodiments, the second processing system is further configured to: read the at least one frame of image from the random access memory after successfully starting the application; and perform the image processing on the at least one frame of image through the application to generate the captured image.

[0021] In some embodiments, the first processing system is an MCU, and the second processing system is an ARM.

[0022] In some embodiments, the first processing system is a first processing core in the device and the second processing system is a second processing core in the device.

[0023] In another aspect of the present invention, an electronic device is provided, which includes a memory configured to store a computer program and a processor configured to execute the computer program to perform each step in the above-mentioned method for capturing an image.

[0024] In another aspect of the present invention, a computer readable medium is provided, wherein a computer program is stored on the medium, and the computer program is executed by a processor to implement each step in the above-mentioned method for capturing an image.

[0025] According to the present invention, after starting the first processing system, the image signal processing module in the first processing system is initialized first, and the captured image is generated by the image signal processing module. Then, the second processing system is started and the image is subsequently processed to generate the captured image. The image signal processing module required for capturing the image is loaded first, and there is no need to wait until the application is loaded to load the image signal processing module. The time point when the image can be captured is advanced, and the image processing and image capturing are separated at the time node. Image capturing is performed first after startup, so that photos can be captured more quickly after startup, further reducing the time difference from startup to photo shooting, and realizing quick snapshot after the device is powered on. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flowchart of a method for capturing an image according to an embodiment of the present invention;

[0027] Figure 2 A schematic diagram of the startup process of the fast startup capture method according to an embodiment of the present invention;

[0028] Figure 3 is a block diagram of a device for capturing images according to an embodiment of the present invention;

[0029] Figure 4 FIG. 4 is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0031] In the existing photographing method, a user's photographing function start request is received, and an image is captured to obtain raw image data. Subsequently, the raw image data is saved in a temporary storage unit, and image processing is performed on the raw image data saved in the temporary storage unit to generate a photo. This photographing method can quickly capture a scene of about 0.5 to 1 second. In this photographing method, the camera sensor receives a photographing instruction for initialization, and then the camera sensor captures image data to obtain raw data and saves it in a temporary storage unit. After booting up, the corresponding application needs to be started to realize photographing. The only way to improve the photographing efficiency is to optimize the booting process, resulting in a large delay between booting up and taking photos.

[0032] In order to solve at least the above technical problems, the present disclosure provides a method for capturing images. According to the present disclosure, the image signal processing module is initialized immediately after the first processing system is started, and at least one frame of captured image is generated by the image signal processing module, without waiting until the entire first processing system is loaded, and without taking pictures after the application is initialized. In this way, the image signal processing module is initialized in advance, without waiting for the application to be loaded, shortening the time between booting and being able to capture images, thereby realizing fast capture after the device is turned on.

[0033] Hereinafter, the technical solution according to the present disclosure will be described with reference to specific embodiments and in conjunction with the accompanying drawings.

[0034] Figure 1 1 is a flow chart showing a method 100 for capturing an image according to an embodiment of the present disclosure. Figure 1 The method 100 includes the following steps 102 to 108.

[0035] In step 102, the first processing system is started. In some embodiments, step 102 may include: obtaining a boot program stored in the second processing system; and booting the first processing system according to the boot program. In this way, the boot program is placed in the second processing system, and the boot program wakes up the first processing system, so that the boot process of the first processing system can be configured.

[0036] In some embodiments, the first processing system image corresponding to the first processing system is loaded through the boot program to start the first processing system. In this way, the first processing system corresponding to the first processing system image is started by loading the first processing system image, thereby realizing automatic startup of the first processing system.

[0037] According to an embodiment of the present disclosure, before step 102, a step may also be included: in response to power-on, starting the second processing system. In some embodiments, starting the second processing system includes: loading a boot program from a boot memory; initializing an external memory, loading a first processing system image corresponding to the first processing system, and loading a system kernel image through the boot program; starting an embedded system boot loader; and starting the second processing system kernel, file system, and application. In this way, the nesting of the startup process of the first processing system and the second processing system is achieved, and the preparatory actions before starting the first processing system are completed at power-on, so that the first processing system can be started in time.

[0038] In step 104, the image signal processing module is initialized by the first processing system.

[0039] In step 106, the image signal processing module generates at least one frame of the captured image. In some embodiments, the image signal processing module receives at least one frame of raw image signal from a camera and performs image signal processing on the at least one frame of raw image signal to generate at least one frame of the captured image.

[0040] According to an embodiment of the present disclosure, between step 106 and step 108, the following may be included: storing the at least one frame of image in a random access memory; and starting the application of the second processing system. In this way, the captured image is stored, and the image is processed after the application is started. When starting, only the shooting action needs to be completed, so only the image signal processing module related to shooting needs to be initialized, which reduces the time between starting and shooting.

[0041] In step 108, the at least one frame of image is processed by the second processing system to generate a captured image. In some embodiments, after the application is successfully started, the at least one frame of image is read from the random access memory; and the at least one frame of image is processed by the application to generate the captured image. In this way, after the application is started, the image captured at the time of startup is processed to generate the captured image, ensuring that the captured low-latency image can be displayed smoothly, and using random access as a transfer for the capture and processing of the image, the separation of the capture and processing of the image is achieved.

[0042] According to an embodiment of the present disclosure, the startup process is divided into a first processing system and a second processing system. The first processing system invokes the second processing system for initialization by booting a startup program, and after the initialization is completed, the image signal processing module is initialized first, so that shooting can be completed without waiting for the steps from system initialization to APP loading, thereby reducing the time difference from power-on to shooting.

[0043] According to the embodiment of the present disclosure, it only takes 200 to 300 ms to capture the first frame of the image. In addition, the technical solution of the present disclosure can be applied to monitoring products such as network cameras (IPCs), without the need for users to trigger a photo request, but rather using the unique architecture of the chip.

[0044] According to an embodiment of the present disclosure, the first processing system may be an MCU (Microcontroller Unit), and the second processing system may be an ARM (Advanced RISC Machines). In some embodiments, the first processing system (i.e., MCU) in the chip can serve as a coprocessor to assist in initializing the ISP / Camera in advance, and implement fast AE (automatic exposure) and AWB (automatic white balance), thereby realizing a fast snapshot function.

[0045] According to an embodiment of the present disclosure, the first processing system may be an ARM, and the second processing system may also be an ARM.

[0046] According to an embodiment of the present disclosure, the first processing system is the first processing core in a multi-core processor, and the second processing system is the second processing core in the multi-core processor, such as running an RTOS (real-time operating system) system in the first processing core and running a Linux system in the second processing core. The second processing core is used to implement the functions of ARM, and the RTOS system is run in the first processing core to implement the functions of MCU. The remaining steps are similar, that is, the second processing core is used as a startup core, and the first processing core is guided to run the RTOS system in the boot startup program (SPL) stage, and the image signal processing module (Camera / ISP) is initialized to capture images and then stored in the DDR. After the second processing core is applied, these images are read out from the DDR for processing.

[0047] According to the embodiment of the present disclosure, the kernel in the normal startup program is trimmed, and unnecessary functional configurations are trimmed. That is, except for the configurations necessary for initializing the camera and system startup, the kernel image size is reduced, and the rootfs in the normal startup program is trimmed, and unnecessary functional configurations are trimmed.

[0048] Figure 2 FIG. 2 is a schematic diagram showing a startup process of a method for fast startup according to an embodiment of the present disclosure. Figure 2 , where Mini-kernel is obtained by cutting kernel in the normal startup program, and tinyrootfs is obtained by cutting rootfs in the normal startup program. Except for the necessary function configurations for initializing the camera and system startup, all configurations are cut to reduce the size of the root file system. In this way, the boot speed is further improved.

[0049] According to the embodiments of the present disclosure, the chip can adopt the ARM+MCU architecture. Based on the fast startup characteristics of the MCU, combined with the optimization and tailoring of the boot process, the MCU is first initialized with the ISP and the camera, and the first few frames of images are saved to the DDR and other caches as quickly as possible. Then, the overall boot time is saved by using a streamlined boot process.

[0050] According to an embodiment of the present disclosure, the chip may also adopt a dual-core ARM architecture.

[0051] According to the embodiments of the present disclosure, the chip can adopt a dual-core architecture. The A core of the chip runs the Linux system, and the B core of the chip runs the RTOS system, which can also achieve the same effect. The A core is used to implement the functions of ARM, and the B core is used to implement the functions of MCU. The A core is used as the startup core to guide the B core to run the RTOS system in the SPL stage, initialize the ISP to capture images and then store them in the DDR. After the application of the A core is running, these images are read out from the DDR for processing.

[0052] Hereinafter, an application scenario of the method 100 for capturing an image according to an embodiment of the present invention will be described by way of example.

[0053] Figure 2 The schematic diagram of the startup process of a method for fast startup according to an embodiment of the present disclosure is shown, and the first processing system is MCU and the second processing system is ARM for explanation, including the following steps 202 to 212.

[0054] In step 202, a hardware power-on operation (POR, Power-On Reset, refers to the hardware process from hardware power-on to the execution of the first instruction in maskrom (mask read-only memory)) is performed.

[0055] In step 204, the boot program SPL (Second Program Loader) is started, which is a very small set of boot (system boot file) codes, which is mainly responsible for initializing external DRAM (Dynamic Random Access Memory) and flash (flash memory), and then booting u-boot (mainly used for embedded system boot loader).

[0056] In step 206, the ARM side performs hardware initialization, MCU image loading and kernel (system kernel) image loading through SPL.

[0057] In step 208, after the MCU image is loaded, the MCU starts and starts the RTOS (real-time operating system) to initialize the image signal processing module (Camera / ISP). In some embodiments, the MCU adopts RISC-V (an open source instruction set architecture).

[0058] In step 210, the image signal processing module captures the first few frames of raw (an image file) and stores them in DDR (double rate synchronous dynamic random access memory) or DARM (dynamic random access memory). In some embodiments, RTOS startup, Camera / ISP initialization and fast image capture are implemented through RTThread (an operating system).

[0059] In step 212, the ARM end is started synchronously, and after the application is successfully started, the raw image is read from the DDR or DARM and processed to obtain the captured image. In some embodiments, the raw data is processed into YUV (a color encoding method, brightness, grayscale value and chromaticity) data through ISP (image signal processing) 3A algorithm, Demosaic (demosaicing), denoising, sharpening and other algorithms, and finally sent to the encoder for encoding.

[0060] In this way, the first frame of image can be captured 200-300ms after power-on, and the images in the period from 200-300ms to the completion of the entire system startup can be captured as much as possible.

[0061] According to another aspect of the present invention, Figure 3 is a block diagram showing an apparatus 300 for photographing an image according to an embodiment of the present invention. Figure 3 The device 300 includes an image signal processing module 302 , a first processing system 304 , and a second processing system 306 .

[0062] The image signal processing module 302 is configured to receive at least one frame of raw image signal from a camera, and perform image signal processing on the at least one frame of raw image signal to generate at least one frame of image.

[0063] The first processing system 304 is configured to initialize the image signal processing module.

[0064] The second processing system 306 is configured to start the first processing system and perform image processing on the at least one frame of image to generate a captured image.

[0065] According to an embodiment of the present disclosure, the second processing system 306 is configured to: obtain a boot program stored in the second processing system; and boot the first processing system according to the boot program. The second processing system 306 is further configured to: load the boot program upon power-on; and load the second processing system kernel corresponding to the second processing system in response to the startup of the first processing system.

[0066] According to an embodiment of the present disclosure, the image signal processing module 302 is configured to store the at least one frame of image in a random access memory, and the second processing system 306 is further configured to start an application of the second processing system.

[0067] According to an embodiment of the present disclosure, the second processing system 306 is further configured to: read the at least one frame of image from the random access memory after successfully starting the application; and perform the image processing on the at least one frame of image through the application to generate the captured image.

[0068] In some embodiments, the first processing system 304 is an MCU, and the second processing system 306 is an ARM. In other embodiments, both the first processing system 304 and the second processing system 306 are ARM. In some embodiments, the first processing system is a first processing core in the device, and the second processing system is a second processing core in the device.

[0069] The time required for the first processing system to complete startup is shorter than the time required for the second processing system to complete startup.

[0070] According to yet another aspect of the present invention, Figure 4 4 is a schematic diagram showing an electronic device 400 according to an embodiment of the present invention. Figure 4 The electronic device 400 includes a memory 402, a processor 404, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the various steps or actions of the method for compatibility with different interfaces as described above are implemented.

[0071] According to another aspect of the present invention, a computer-readable medium is provided, wherein a computer program is stored on the computer-readable medium, and the computer program is executed by a processor to implement the method for being compatible with different interface specifications as described above.

[0072] In summary, the method for capturing images provided by the present invention wakes up the first processing system through the boot program stored in the second processing system before the second processing system, which starts more slowly, completes its startup. When the image signal processing module in the first processing system completes its initialization, it is first initialized through the image signal processing module, and then the image is quickly captured by the image signal processing module after the initialization to achieve image acquisition, thereby shortening the time period from startup to image capture, achieving rapid capture after startup, and being applicable to the ARM+MCU architecture, ARM+ARM or dual-core architecture, with strong practicality and applicability.

[0073] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for capturing an image, characterized in that: include: In response to powering on, starting the second processing system; Starting a first processing system; Initialize the image signal processing module and perform automatic exposure by the first processing system; Generate at least one frame of captured image through the image signal processing module; Storing the at least one frame of image in a random access memory; Starting an application of the second processing system; as well as Performing image processing on the at least one frame of image by the second processing system to generate a captured image; Wherein starting the first processing system comprises: obtaining a boot program stored in the second processing system; and guiding the first processing system to start according to the boot program; wherein the time required for the first processing system to complete startup is shorter than the time required for the second processing system to complete startup; The first processing system is an MCU, and the second processing system is an ARM, or the first processing system is a first processing core in a multi-core processor, and the second processing system is a second processing core in the multi-core processor.

2. The method according to claim 1, characterized in that Guiding the first processing system to start up according to the boot program includes: The first processing system image corresponding to the first processing system is loaded through the boot startup program to start the first processing system.

3. The method according to claim 1, characterized in that Starting the second processing system includes: Loading a boot loader from the boot memory; Initialize the external memory, load the first processing system image corresponding to the first processing system, and load the system kernel image through the boot program; Launching the embedded system boot loader; and The second processing system kernel, file system and application program are started.

4. The method according to claim 1, characterized in that: Performing image processing on the at least one frame of image by the second processing system to generate a captured image includes: After successfully starting the application, reading the at least one frame of image from the random access memory; and The image processing is performed on the at least one frame of image by the application program to generate the captured image.

5. A device for capturing an image, characterized in that: include: An image signal processing module is configured to receive at least one frame of raw image signal from a camera, and perform image signal processing on the at least one frame of raw image signal to generate at least one frame of image; and storing the at least one frame of image in a random access memory; A first processing system is configured to initialize the image signal processing module and perform automatic exposure; as well as The second processing system is configured to: In response to powering on, loading a boot program stored in the second processing system; booting the first processing system according to the booting program; In response to starting the first processing system, loading a second processing system kernel corresponding to the second processing system; Starting an application of the second processing system; as well as Performing image processing on the at least one frame of image to generate a captured image; wherein the time required for the first processing system to complete startup is shorter than the time required for the second processing system to complete startup; The first processing system is an MCU, and the second processing system is an ARM, or the first processing system is a first processing core in a multi-core processor, and the second processing system is a second processing core in the multi-core processor.

6. The device according to claim 5, characterized in that The second processing system is configured to: After successfully starting the application, reading the at least one frame of image from the random access memory; as well as The image processing is performed on the at least one frame of image by the application program to generate the captured image.

7. An electronic device, characterized in that: include: a memory configured to store a computer program; as well as A processor configured to execute the computer program to perform the method according to any one of claims 1 to 4.

8. A computer readable medium having a computer program stored thereon, characterized in that: The computer program is executed to implement the method according to any one of claims 1 to 4.

Citation Information

Patent Citations

  • Rapid shooting method and mobile terminal

    CN105812553A

  • Rapid photographing method and electronic device

    CN105827942A

  • Image data processing method and mobile terminal

    CN106506960A

  • Shooting method and mobile terminal

    CN107277353A

  • Camera process starting method, compute device, and computer-readable storage medium

    CN108989680A