Image processing chip and its control method, image acquisition device and mobile terminal

By sequentially waking up and configuring the various modules of the image processing chip, and using independent processing pipelines to handle AI image transmission and traditional photo/video functions, the problems of slow response speed and power consumption waste in existing technologies are solved, thereby improving the performance and battery life of smart wearable devices.

CN123123742APending Publication Date: 2026-11-17GRAVITYXR ELECTRONICS & TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510641697.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-16
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

The image processing chips in existing smart wearable devices process data only after all modules are woken up, which reduces response speed and efficiency and results in unnecessary power consumption, affecting battery life and user experience.

Method used

By sequentially waking up each level of modules and reusing the data processing time of the previous level module, configuration is achieved, distinguishing between AI image transmission and traditional photo/video functions, and using independent processing pipelines for image processing to reduce power consumption.

Benefits of technology

It improves the response speed and efficiency of the image processing chip, reduces power consumption, and optimizes the user experience and device battery life.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123742A_ABST
    Figure CN123123742A_ABST
Patent Text Reader

Abstract

This invention provides an image processing chip, a control method, an acquisition device, a mobile terminal, a medium, and a product. The image processing chip includes a constantly powered-on module, a front-end processing module, and a back-end processing module. The constantly powered-on module includes a trigger unit, an instruction cache unit, and a storage unit. It powers on in response to the power-on of the image processing chip and executes the trigger unit to acquire a trigger instruction. The front-end processing module powers on in response to the trigger instruction, acquires front-end configuration data from the instruction cache unit for initialization configuration, performs front-end processing on the acquired image to be processed after initialization configuration, and powers off after writing the pre-processed image data to the storage unit. The back-end processing module powers on using the same time as the front-end processing module for front-end processing, acquires back-end configuration data from the instruction cache unit for initialization configuration, encodes the cached data in response to the cached data in the storage unit reaching the start condition, and powers off after outputting the corresponding encoded data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image processing chip, a control method for an image processing chip, an image acquisition device, a near-eye display mobile terminal, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the continuous development of Extended Reality (XR) headset technology, smart wearable devices are rapidly penetrating people's daily lives. Currently, existing smart wearable devices possess Artificial Intelligence (AI) image transmission capabilities, as well as manually controlled photo and video recording functions. After the smart wearable device recognizes the corresponding function to be performed, it activates the relevant functional modules in the image processing chip, and after configuring each module, the data is processed by each module.

[0003] However, existing image processing technology in smart wearable devices has significant drawbacks. Current technology requires all necessary modules to be woken up and configured before starting the corresponding module to process data, resulting in reduced response speed and efficiency of the image processing chip. Furthermore, since not all woken-up modules operate simultaneously, some modules continue to run even when idle, causing unnecessary power consumption by the image processing chip. This shortens the battery life of smart wearable devices and negatively impacts the user experience.

[0004] In order to overcome the above-mentioned defects in the existing technology, there is an urgent need in the field for an improved image processing chip to improve the response speed and efficiency of the image processing chip, reduce the power consumption of the image processing chip, and improve the performance of the image processing chip, thereby optimizing the user experience. Summary of the Invention

[0005] The following provides a brief overview of one or more aspects to offer a basic understanding of them. This overview is not an exhaustive summary of all conceived aspects, nor is it intended to identify key or decisive elements of all aspects, nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed descriptions that follow.

[0006] To overcome the aforementioned deficiencies in existing technologies, this invention provides an image processing chip, a control method for the image processing chip, an image acquisition device, a near-eye display mobile terminal, a computer-readable storage medium, and a computer program product. On one hand, by sequentially waking up modules at each stage according to the data transmission direction and reusing the data processing time of the preceding modules to configure the subsequent modules, the response speed and efficiency of the image processing chip are improved, while its power consumption is reduced, thereby optimizing the user experience. On the other hand, by distinguishing between the application scenarios of AI image transmission functions and traditional photo / video functions, separate processing pipelines are used for image processing of the two functions, thereby improving the performance of the image processing chip, reducing its power consumption, and further optimizing the user experience.

[0007] Specifically, the image processing chip provided according to the first aspect of the present invention includes a constantly powered-on module, a front-end processing module, and a back-end processing module. The constantly powered-on module includes a trigger unit, an instruction cache unit, and a storage unit. The constantly powered-on module powers on in response to the power-on of the image processing chip and runs the trigger unit to obtain a trigger instruction. The front-end processing module powers on in response to the trigger instruction, obtains front-end configuration data from the instruction cache unit for initialization configuration, performs front-end processing on the acquired image to be processed after completing the initialization configuration, and powers off after writing the image data processed by the front-end processing module into the storage unit. The back-end processing module powers on using the time taken by the front-end processing module to perform the front-end processing, obtains back-end configuration data from the instruction cache unit for initialization configuration, encodes the cached data in response to the cached data stored in the storage unit reaching a preset start condition, and powers off after outputting the corresponding encoded data.

[0008] Furthermore, in some embodiments of the present invention, the triggering unit includes a timing unit and / or an external interface. The always-on module, in response to the power-on of the image processing chip, powers on the timing unit and runs the timing unit to generate a first trigger instruction at regular intervals. The always-on module, in response to the power-on of the image processing chip, powers on the external interface and runs the external interface to obtain a second trigger instruction from the outside.

[0009] Furthermore, in some embodiments of the present invention, the first triggering instruction includes an instruction to trigger the AI ​​image transmission function. The second triggering instruction includes an instruction to trigger the photo taking function and / or the video recording function.

[0010] Furthermore, in some embodiments of the present invention, the always-on module further includes a first processor. The first processor powers on in response to the power-on of the image processing chip, and enters WFI state after pre-storing the configuration data of the AI ​​image transmission function, the photo-taking function, and / or the video recording function to the instruction cache unit.

[0011] Furthermore, in some embodiments of the present invention, the first processor is further configured to: in response to the first trigger instruction, obtain first front-end configuration data of the AI ​​image transmission function from the instruction cache unit, and initialize the front-end processing module according to a preset configuration mode; and in response to the power-on of the back-end processing module, obtain first back-end configuration data of the AI ​​image transmission function from the instruction cache unit, and initialize the back-end processing module according to the preset configuration mode.

[0012] Furthermore, in some embodiments of the present invention, the image processing chip further includes a second processor. The computing power and peak power consumption of the second processor are both greater than those of the first processor. The first processor is further configured to: in response to the second trigger instruction, control the second processor to power on, so that after the front-end processing module is powered on, the second processor obtains the second front-end configuration data of the photo-taking function and / or the video-taking function from the instruction cache unit, and initializes the front-end processing module accordingly; and in response to the power-on of the back-end processing module, obtain the second back-end configuration data of the photo-taking function and / or the video-taking function from the instruction cache unit, and initializes the back-end processing module accordingly.

[0013] Furthermore, in some embodiments of the present invention, the post-processing module includes a video processing unit. The second processor is also configured to: in response to the cached data stored in the storage unit being video data and reaching a preset startup condition, cooperate with the video processing unit to perform video encoding on the cached data.

[0014] Furthermore, in some embodiments of the present invention, the storage unit includes a peripheral storage management unit. The peripheral storage management unit is connected to an external memory and is configured to: power on in response to the power-on of the image processing chip and control the external memory to enter a self-refresh mode; control the external memory to exit the self-refresh mode in response to the power-on of the front-end processing module to prepare for reading and writing data; and control the external memory to enter the self-refresh mode in response to the output of corresponding encoded data by the back-end processing module.

[0015] Furthermore, in some embodiments of the present invention, the front-end processing module is also pre-set with a trigger threshold. The back-end processing module powers on in response to the amount of image data written by the front-end processing module to the storage unit reaching the trigger threshold, and begins encoding the cached data in the storage unit.

[0016] Furthermore, in some embodiments of the present invention, the storage unit includes an internal storage management unit and a RAM unit. The internal storage management unit is configured to: power on in response to power-on of the image processing chip and control the RAM unit to enter a sleep mode; control the RAM unit to exit the sleep mode in response to power-on of the front-end processing module to prepare for reading and writing data; and control the RAM unit to enter the sleep mode in response to the back-end processing module completing the output of corresponding encoded data.

[0017] Furthermore, in some embodiments of the present invention, the always-on module further includes a clock unit that operates in response to the trigger command to output a clock signal controlling the operation of the front-end processing module and / or the back-end processing module. The always-on module also stops operating the trigger unit in response to the operation of the clock unit, and / or stops operating the clock unit in response to the output of the encoded data.

[0018] Furthermore, in some embodiments of the present invention, the pre-processing module and / or the post-processing module operate or stop operating according to the clock signal provided by the clock unit. And / or, during the processing of multiple frames of images, the pre-processing module and / or the post-processing module power down after the end of each frame of the image to be processed and power up again before the beginning of each frame of the image to be processed.

[0019] Further, in some embodiments of the present invention, the step of performing pre-processing on the acquired image to be processed after completing the initialization configuration includes: responding to a first trigger command indicating an AI image transmission function, driving the pre-processing module to perform low-power first pre-processing on the image to be processed with a preset first amplitude and / or first frequency driving voltage; and responding to a second trigger command indicating a photo-taking function and / or a video-taking function, driving the pre-processing module to perform high-quality second pre-processing on the image to be processed with a preset second amplitude and / or second frequency driving voltage. The first amplitude is less than the second amplitude. The first frequency is less than the second frequency.

[0020] Further, in some embodiments of the present invention, the pre-processing module includes a camera interface and at least two ISP (Image Signal Processing) units. The first ISP unit for the AI ​​image transmission function is connected to a first driving power supply. In response to the first trigger command, the camera interface inputs the acquired image to be processed into the first ISP unit, so that the image to be processed is subjected to a first pre-processing at a first frequency via the first ISP unit. The second ISP unit for the photo-taking function and / or the video-taking function is connected to a second driving power supply. In response to the second trigger command, the camera interface inputs the acquired image to be processed into the second ISP unit, so that the image to be processed is subjected to a second pre-processing at a second frequency via the second ISP unit. The output voltage of the first driving power supply is lower than the output voltage of the second driving power supply. The first frequency is lower than the second frequency.

[0021] Furthermore, in some embodiments of the present invention, the first ISP unit is connected to a first camera lens via the camera interface to acquire a first image to be processed for AI image transmission function captured by the first camera lens. The second ISP unit is connected to a second camera lens via the camera interface to acquire a second image to be processed for photo and / or video recording function captured by the second camera lens. The second camera lens has a wider equivalent focal length range, higher resolution, and / or a larger angle of view than the first camera lens.

[0022] Furthermore, in some embodiments of the present invention, the pre-processing module further includes a network processing unit, which is used to cooperate with the second ISP unit to perform second pre-processing on the image to be processed based on neural network inference.

[0023] Furthermore, in some embodiments of the present invention, the post-processing module includes an image encoder and a video encoder. The image encoder powers on during the pre-processing of the AI ​​image transmission function or the photo-taking function in the pre-processing module, obtains post-processing configuration data from the instruction cache unit for initialization configuration, performs image encoding on the cached data in response to the cached data stored in the storage unit reaching a preset start condition, and powers off after outputting the corresponding image-encoded data. The video encoder powers on during the pre-processing of the camera function in the pre-processing module, obtains post-processing configuration data from the instruction cache unit for initialization configuration, performs video encoding on the cached data in response to the cached data stored in the storage unit reaching a preset start condition, and powers off after outputting the corresponding video-encoded data.

[0024] Furthermore, in some embodiments of the present invention, the post-processing module further includes an encryption unit for encrypting the image encoding data of the AI ​​image transmission function output by the image encoder and outputting it to an external image AI processor.

[0025] Furthermore, the control method for the image processing chip provided according to the second aspect of the present invention includes the following steps: powering on a constantly powered-on module in response to power-on of the image processing chip. The constantly powered-on module includes a trigger unit, an instruction cache unit, and a storage unit; running the trigger unit to obtain a trigger instruction; powering on the front-end processing module in response to the trigger instruction and obtaining front-end configuration data from the instruction cache unit to initialize the front-end processing module; after completing the initialization configuration, performing front-end processing on the acquired image to be processed via the front-end processing module, and powering off the front-end processing module after writing the image data processed by the front-end processing module into the storage unit; reusing the time of the front-end processing module performing the front-end processing, powering on the back-end processing module, and obtaining back-end configuration data from the instruction cache unit to initialize the back-end processing module; and powering off the back-end processing module in response to the cached data stored in the storage unit reaching a preset start condition, encoding the cached data via the back-end processing module, and powering off the back-end processing module after outputting the corresponding encoded data.

[0026] Furthermore, the image processing chip provided according to a third aspect of the present invention includes a camera interface, a timing unit, an external interface, and at least two image transmission paths. The camera interface is used to acquire an image to be processed. The timing unit is used to generate a first trigger command at regular intervals. The external interface is used to acquire a second trigger command input from an external source. The first image transmission path is activated in response to the first trigger command and processes the image to be processed acquired by the camera interface based on a first voltage and a first frequency. The second image transmission path is activated in response to the second trigger command and processes the image to be processed acquired by the camera interface based on a second voltage and a second frequency. The first voltage is less than the second voltage. The first frequency is less than the second frequency.

[0027] Furthermore, in some embodiments of the present invention, the first triggering instruction includes an instruction to trigger the AI ​​image transmission function, and the second triggering instruction includes an instruction to trigger the photo taking function and / or the video recording function.

[0028] Furthermore, in some embodiments of the present invention, the first image transmission path includes a first ISP unit, which performs a first pre-processing on the image to be processed based on the first voltage and the first frequency. The second image transmission path includes a second ISP unit and / or a network processing unit, which performs a second pre-processing on the image to be processed based on the second voltage and the second frequency.

[0029] Furthermore, in some embodiments of the present invention, the first image transmission path and the second image transmission path multiplex an image encoder for image encoding of the image processed by the first pre-processing stage or the second pre-processing stage. And / or the second image transmission path includes a video encoder for video encoding of multiple frames of images processed by the second pre-processing stage.

[0030] Furthermore, the control method for the image processing chip provided according to a fourth aspect of the present invention includes the following steps: in response to receiving a first trigger command generated by a timing unit, activating a first image transmission path and processing an image to be processed acquired from a camera interface based on a first voltage and a first frequency; and in response to receiving a second trigger command from an external interface, activating a second image transmission path and processing an image to be processed acquired from the camera interface based on a second voltage and a second frequency. The first voltage is less than the second voltage. The first frequency is less than the second frequency.

[0031] Furthermore, the image acquisition device provided according to the fifth aspect of the present invention includes a camera module and an image processing chip as provided in the first or third aspect of the present invention. The camera is configured with at least one lens. The image processing chip is connected to the camera to acquire and process the acquired images.

[0032] Furthermore, in some embodiments of the present invention, the image acquisition device further includes a human-computer interaction interface and an image AI processor. The human-computer interaction interface is used to obtain a second trigger command provided by the user to trigger the photo-taking function and / or video recording function. The image AI processor is used to perform image recognition based on the encoded data of the AI ​​image transmission function output by the image processing chip to determine the corresponding image recognition result.

[0033] Furthermore, in some embodiments of the present invention, the image acquisition device further includes a storage battery, wherein the image acquisition device is powered by the storage battery.

[0034] Furthermore, the mobile terminal with near-eye display provided according to the sixth aspect of the present invention includes an image acquisition device and a display module as provided in the fifth aspect of the present invention.

[0035] Furthermore, the computer-readable storage medium provided according to the seventh aspect of the present invention stores computer instructions thereon. When the computer instructions are executed by a processor, they implement the control method of the image processing chip as provided in the second or fourth aspect of the present invention.

[0036] Furthermore, the computer program product provided according to the eighth aspect of the present invention includes computer instructions. When the computer instructions are executed by a processor, they implement the control method for the image processing chip as provided in the second or fourth aspect of the present invention. Attached Figure Description

[0037] The above-described features and advantages of the present invention will be better understood after reading the following detailed description of embodiments of the present disclosure in conjunction with the accompanying drawings. In the drawings, components are not necessarily drawn to scale, and components having similar related characteristics or features may have the same or similar reference numerals.

[0038] Figure 1 A schematic diagram of the frame of an image processing chip according to some embodiments of the present invention is shown.

[0039] Figure 2 A flowchart illustrating a control method for an image processing chip according to some embodiments of the present invention is shown.

[0040] Figure 3 A schematic diagram of the standby state of an image processing chip provided according to some embodiments of the present invention is shown.

[0041] Figure 4 A schematic diagram illustrating the initialization configuration of a front-end processing module according to some embodiments of the present invention is shown.

[0042] Figure 5 A schematic diagram of a clock unit provided according to some embodiments of the present invention is shown.

[0043] Figure 6 A schematic diagram of pre-processing via a pre-processing module is shown, according to some embodiments of the present invention.

[0044] Figure 7 A schematic diagram illustrating the initialization configuration of a post-processing module according to some embodiments of the present invention is shown.

[0045] Figure 8 A schematic diagram illustrating encoding via a post-processing module is shown, according to some embodiments of the present invention.

[0046] Figure 9 A schematic diagram of the exit state of an image processing chip provided in some embodiments of the present invention is shown.

[0047] Figure 10 A schematic diagram illustrating the initialization configuration of a front-end processing module according to some embodiments of the present invention is shown.

[0048] Figure 11 A schematic diagram of pre-processing via a pre-processing module is shown, according to some embodiments of the present invention.

[0049] Figure 12 A schematic diagram illustrating the initialization configuration of a post-processing module according to some embodiments of the present invention is shown.

[0050] Figure 13 A schematic diagram illustrating encoding via a post-processing module is shown, according to some embodiments of the present invention.

[0051] Figure 14 A schematic diagram of the exit state of an image processing chip provided in some embodiments of the present invention is shown.

[0052] Figure 15 A schematic diagram of the frame of an image processing chip provided according to some embodiments of the present invention is shown.

[0053] Figure 16 A schematic diagram of the standby state of an image processing chip provided according to some embodiments of the present invention is shown.

[0054] Figure 17 A flowchart illustrating a control method for an image processing chip according to some embodiments of the present invention is shown.

[0055] Figure 18 A schematic diagram illustrating the principle of a control method for an image processing chip according to some embodiments of the present invention is shown.

[0056] Figure 19 A schematic diagram illustrating the principle of AI image transmission function provided according to some embodiments of the present invention is shown.

[0057] Figure 20 A schematic diagram illustrating the principles of the photo-taking and video-taking functions provided according to some embodiments of the present invention is shown.

[0058] Figure label:

[0059] 10. Image processing chip; 11. Always-on power module; 111. Instruction cache unit; 112. Storage unit; 113. Timing unit; 114. External interface; 115. First processor; 116. Clock unit; 12. Front-end processing module; 121. Camera interface; 122. First ISP unit; 123. Second ISP unit; 124. Network processing unit; 13. Back-end processing module; 131. Video processing unit; 132. Image encoder; 133. Encryption unit; 14. Second processor;

[0060] 20. External storage;

[0061] 30. Image processing chip; 31. Always-on power module; 311. Instruction cache unit; 312. Storage unit; 313. Timing unit; 314. External interface; 315. First processor; 316. Clock unit; 32. Front-end processing module; 321. Camera interface; 322. First ISP unit; 323. Second ISP unit; 324. Network processing unit; 33. Back-end processing module; 331. Video processing unit; 332. Image encoder; 333. Encryption unit; 34. Second processor. Detailed Implementation

[0062] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Although the description of the present invention is presented in conjunction with preferred embodiments, this does not mean that the features of the invention are limited to these embodiments. On the contrary, the purpose of describing the invention in conjunction with embodiments is to cover other options or modifications that may be derived based on the claims of the present invention. To provide a thorough understanding of the invention, many specific details will be included in the following description. The invention may also be implemented without using these details. Furthermore, to avoid confusion or obscuring the focus of the invention, some specific details will be omitted in the description.

[0063] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0064] Furthermore, the terms "upper," "lower," "left," "right," "top," "bottom," "horizontal," and "vertical" used in the following description should be understood as the orientations shown in the relevant paragraphs and accompanying drawings. These relative terms are for illustrative purposes only and do not imply that the described apparatus must be manufactured or operated in a specific orientation, and therefore should not be construed as limiting the invention.

[0065] It is understood that although terms such as "first," "second," and "third" may be used herein to describe various components, regions, layers, and / or parts, these components, regions, layers, and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers, and / or parts. Therefore, the first components, regions, layers, and / or parts discussed below may be referred to as second components, regions, layers, and / or parts without departing from some embodiments of the present invention.

[0066] As mentioned above, existing image processing technologies for smart wearable devices have significant drawbacks. On one hand, existing technologies require all necessary modules to be activated and configured before starting the corresponding module to process data, leading to reduced response speed and efficiency of the image processing chip. Furthermore, since not all activated modules operate simultaneously, some modules remain running even when idle, causing unnecessary power consumption waste in the image processing chip, thus shortening the battery life of the smart wearable device and impacting the user experience. On the other hand, the inventors of this application have discovered that because traditional smart wearable devices reuse the processing pipeline for images generated by the camera function to process image data transmitted by AI images, this significantly exceeds the image quality requirements of large AI models, resulting in wasted performance and power consumption of the image processing chip.

[0067] To overcome the aforementioned deficiencies in existing technologies, this invention provides an image processing chip, a control method for the image processing chip, an image acquisition device, a near-eye display mobile terminal, a computer-readable storage medium, and a computer program product. On one hand, by sequentially waking up modules at each stage according to the data transmission direction and reusing the data processing time of the preceding modules to configure the subsequent modules, the response speed and efficiency of the image processing chip are improved, while its power consumption is reduced, thereby optimizing the user experience. On the other hand, by distinguishing between the application scenarios of AI image transmission functions and traditional photo / video functions, separate processing pipelines are used for image processing of the two functions, thereby improving the performance of the image processing chip, reducing its power consumption, and further optimizing the user experience.

[0068] In some non-limiting embodiments, the image acquisition device provided in the fifth aspect of the present invention can be configured and implemented in the near-eye display mobile terminal provided in the sixth aspect of the present invention. Specifically, the near-eye display mobile terminal includes, but is not limited to, head-mounted near-eye display terminals such as smart glasses, which include a battery, an image acquisition device, and / or a display screen. Since the mobile terminal is powered by its battery, there is a need to reduce power consumption to improve battery life, reduce battery size and weight, and reduce battery costs.

[0069] Furthermore, the image acquisition device provided in the fifth aspect of the present invention may be configured with the image processing chip provided in the first aspect of the present invention to implement the control method of the image processing chip provided in the second aspect of the present invention. Alternatively, the image acquisition device provided in the fifth aspect of the present invention may be configured with the image processing chip provided in the third aspect of the present invention to implement the control method of the image processing chip provided in the fourth aspect of the present invention.

[0070] Furthermore, the aforementioned image acquisition device includes a camera module and an image processing chip. The camera module is equipped with at least one lens. The image processing chip is connected to the camera to acquire and process the acquired images.

[0071] Furthermore, in some optional embodiments, the image acquisition device provided in the fifth aspect of the present invention further includes a human-computer interaction interface and an image AI processor. Here, the human-computer interaction interface is used to obtain a second trigger command provided by the user to trigger the photo-taking function and / or video recording function. The image AI processor is used to perform image recognition based on the encoded data of the AI ​​image transmission function output by the image processing chip to determine the corresponding image recognition result.

[0072] Furthermore, in some alternative embodiments, the aforementioned battery can also be configured in the image acquisition device provided in the fifth aspect of the present invention. By configuring the image processing chip provided in the first aspect of the present invention, and / or the image processing chip provided in the third aspect of the present invention, the present invention can effectively reduce the power consumption of the image acquisition device to improve the battery's battery life, reduce the battery's size and weight, and lower the battery cost.

[0073] Furthermore, in some non-limiting embodiments, the image acquisition apparatus provided in the fifth aspect of the present invention includes a memory and a processor. Here, the memory includes, but is not limited to, the computer-readable storage medium provided in the seventh aspect, on which the computer program product provided in the eighth aspect of the present invention is stored. The computer program product includes computer instructions. The processor is connected to the memory and configured to execute the computer instructions in the computer program product to implement the control method of the image processing chip as provided in the second or fourth aspect of the present invention.

[0074] Please refer to the following first. Figure 1 , Figure 1 A schematic diagram of the frame of an image processing chip according to some embodiments of the present invention is shown.

[0075] exist Figure 1 In the embodiment shown, the image processing chip 10 provided by the first aspect of the present invention includes a constantly powered module 11, a front-end processing module 12, and a back-end processing module 13.

[0076] The always-on module 11 includes a trigger unit, an instruction cache unit 111 (e.g., a CMD buffer), and a storage unit 112. Here, the always-on module 11 powers on in response to the power-on of the image processing chip 10, runs the trigger unit, and acquires a trigger instruction. The front-end processing module 12 powers on in response to the trigger instruction provided by the trigger unit, acquires front-end configuration data from the instruction cache unit 111 for initialization configuration, performs front-end processing on the acquired image to be processed after completing the initialization configuration, and powers off after writing the pre-processed image data into the storage unit 112. The back-end processing module 13 can reuse the power-on time of the front-end processing module 12 for front-end processing, acquires back-end configuration data from the instruction cache unit 111 for initialization configuration, encodes the cached data in response to the cached data stored in the storage unit 112 reaching a preset start condition, and powers off after outputting the corresponding encoded data.

[0077] Furthermore, in some embodiments, the triggering unit may include a timing unit 113. The constantly powered-on module 11 powers on the timing unit 113 in response to the power-on of the image processing chip 10, and runs the timing unit 113 to generate a first trigger instruction at regular intervals when it counts to a specified value. Here, the first trigger instruction includes an instruction to trigger the AI ​​image transmission function at regular intervals. The AI ​​image transmission function includes, but is not limited to, the function of transmitting input images to various models that process images using artificial intelligence (AI) technology, such as image recognition models and object tracking models.

[0078] Furthermore, in some embodiments, the triggering unit may also include an external interface 114 (e.g., GPIO). The constantly powered-on module powers on the external interface 114 in response to the power-on of the image processing chip and operates the external interface to obtain a second trigger command indicating the photo-taking and / or video-taking function from the outside. Here, the second trigger command includes an instruction to externally trigger the photo-taking and / or video-taking function of the image acquisition device, and the triggering method includes, but is not limited to, manual triggering, voice triggering, and motion-sensing triggering.

[0079] In addition, Figure 1 In the illustrated embodiment, the always-on module 11 may further include a low-computing-power, low-power first processor 115 (e.g., a CPU-S with a clock frequency of 170MHz, a computing power of 213DMIPS, and a peak power consumption of 26.69mW) and a clock unit 116 (e.g., a Phase-Locked Loop, PLL). The first processor 115 powers on in response to the power-on of the image processing chip 10, pre-stores configuration data for AI image transmission, photo capture, and / or video recording functions into the instruction cache unit 111, and enters a standby (Wait for Interrupt, WFI) state after pre-storage is complete to reduce power consumption. The clock unit 116 operates in response to a trigger instruction and outputs a clock signal, allowing the front-end processing module 12 and / or the back-end processing module 13 to operate or stop based on the clock signal provided by the clock unit. It is understood that the above description of the parameters of the first processor 115 is merely a non-limiting example. In actual use, those skilled in the art can make appropriate adjustments to the clock frequency, computing power and peak power consumption of the first processor 115 according to actual needs.

[0080] In addition, Figure 1 In the illustrated embodiment, the aforementioned pre-processing module can specifically be an ISP pipeline, which is mainly used to perform pre-processing on the acquired image to be processed, including Lens Shading Correction (LSC), Demosaic processing, 3A processing, Color Correction Matrix (CCM) processing, noise reduction processing, and purple fringing removal. Here, the 3A processing includes Auto Focus (AF) processing, Auto Exposure (AE) processing, and Auto White Balance (AWB) processing.

[0081] In addition, the aforementioned post-processing module can be specifically an encoder, which is mainly used to perform format-adaptive encoding processing on the image after pre-processing, so as to output the image to the image AI processor and / or display module at the back end via the image processing chip 10.

[0082] The working principle of the image processing chip described above will be described below with reference to embodiments of control methods for some image processing chips. Those skilled in the art will understand that these embodiments of control methods are merely non-limiting implementations provided by the present invention, intended to clearly demonstrate the main concepts of the invention and provide specific solutions convenient for public implementation, rather than limiting all functions or operating modes of the image processing chip. Similarly, the image processing chip is also only one non-limiting implementation provided by the present invention, and does not limit the executing entity or execution order of each step in the control methods of these image processing chips.

[0083] Please refer to the reference. Figure 2 and Figure 9 . Figure 2 A flowchart illustrating a control method for an image processing chip according to some embodiments of the present invention is shown. Figure 3 A schematic diagram of the standby state of an image processing chip provided according to some embodiments of the present invention is shown. Figure 4 A schematic diagram illustrating the initialization configuration of a front-end processing module according to some embodiments of the present invention is shown. Figure 5 A schematic diagram of a clock unit provided according to some embodiments of the present invention is shown. Figure 6 A schematic diagram of pre-processing via a pre-processing module is shown, according to some embodiments of the present invention. Figure 7 A schematic diagram illustrating the initialization configuration of a post-processing module according to some embodiments of the present invention is shown. Figure 8 A schematic diagram illustrating encoding via a post-processing module is shown, according to some embodiments of the present invention. Figure 9 A schematic diagram of the exit state of an image processing chip provided in some embodiments of the present invention is shown.

[0084] like Figure 2 and Figure 3 As shown, in the multi-level wake-up process of the image processing chip 10, in response to the power-on of the image processing chip 10, the image processing chip 10 can first power on the first processor 115, the trigger unit and the instruction cache unit 111 of its always-on module 11. After the configuration data of the AI ​​image transmission function, the photo taking function and / or the video recording function are pre-stored to the instruction cache unit 111 via the first processor 115, the first processor 115 enters the standby (WFI) state of being powered on but not running.

[0085] Subsequently, in the application of the AI ​​image transmission function, the image processing chip 10 can run the aforementioned timing unit 13 to time the first trigger instruction generated indicating the AI ​​image transmission function. Then, as... Figure 2 and Figure 4As shown, in response to the first trigger command indicating the AI ​​image transmission function, the image processing chip 10 can first power on the camera interface 121 (e.g., CSI) and the first ISP unit 122 of the front-end processing module 12, and control the first processor 115 to obtain the first front-end configuration data of the AI ​​image transmission function from the instruction cache unit 111 via the Direct Memory Access (DMA) interface, and then initialize the front-end processing module 12 according to the fast configuration mode pre-installed in the ISP pipeline. Here, the first front-end configuration data includes, but is not limited to, image size, allocated address space, data specifications, and algorithm-related parameters such as image compensation gain / image compensation threshold.

[0086] In addition, such as Figure 4 and Figure 5 As shown, in response to the first trigger command generated by the timing unit 113, the image processing chip 10 can start the AI ​​image transmission function via the first processor 115 and start the running clock unit 116 to output a clock signal that controls the front-end processing module 12 and / or the back-end processing module 13 to run synchronously. Preferably, in response to the operation of the clock unit 116, the always-on module 11 can also stop the operation of the timing unit 113, keeping it in a power-on but sleep state to prevent multiple commands from being triggered simultaneously, reduce power consumption, and ensure that the next valid trigger command can be obtained in real time.

[0087] After that, such as Figure 2 and Figure 6 As shown, after completing the initialization configuration of the aforementioned pre-processing module 12, the image processing chip 10 can perform pre-processing on the acquired image to be processed via the first ISP unit 122, including lens shading correction (LSC), demosaic processing, 3A (AF, AE, AWB) processing, and color correction matrix (CCM) processing. After writing the pre-processed image data (i.e., pre-processing result data) into the storage unit 112, the pre-processing module 12 is powered off. Thus, by writing the pre-processing result data into the storage unit 112 and directly powering off the pre-processing module 12, this invention effectively decouples the power-on and power-off operations between the pre-processing module 12 and the post-processing module 13, thereby effectively reducing the power consumption of the image processing chip 10.

[0088] In addition, such as Figure 2 and Figure 7 As shown, the image processing chip 10 can reuse the time of the front-end processing module 12 to power on the image encoder 132 of the back-end processing module 13, and obtain the back-end configuration data from the instruction cache unit 111 to initialize the back-end processing module 13.

[0089] Specifically, in the application of the AI ​​image transmission function, in response to the power-on of the back-end processing module 13, the first processor 115 can, as described above, obtain the first back-end configuration data of the AI ​​image transmission function from the instruction cache unit 111 via the direct memory access interface (DMA), and initialize the back-end processing module 13 according to a preset configuration mode. Here, the aforementioned first back-end configuration data includes, but is not limited to, parameters such as image size, allocated address space, and data specifications, and it is consistent with the configuration data of the ISP pipeline. Thus, by using the low-computing-power, low-power first processor 115 and performing the initial configuration of the ISP pipeline and encoder in a fixed, fast configuration mode, this invention can adapt to the low image quality required by the AI ​​image transmission function, effectively reducing the power consumption of the AI ​​image transmission function of the image processing chip 10 and improving its real-time performance.

[0090] Furthermore, such as Figure 2 and Figure 8 As shown, the aforementioned storage unit 112 may include a peripheral memory-controller IP (IPM). This IPM is connected to external memory (EXMEM) 20 for off-chip storage of the pre-processing result data. Specifically, the external memory 20 may be double-data-rate synchronous dynamic random access memory (DDR SDRAM). During image processing in the AI ​​image transmission function, the IPM can power on in response to the power-on of the image processing chip 10 and control the external memory 20 to enter self-refresh mode. Subsequently, in response to the power-on of the pre-processing module 12, the IPM can control the external memory 20 to exit self-refresh mode to prepare for reading and writing data. Then, in response to the output of the corresponding encoded data by the post-processing module 13, the IPM can control the external memory 20 to re-enter self-refresh mode to reduce the power consumption of the image processing chip 10.

[0091] Furthermore, the aforementioned front-end processing module 12 can also have a pre-set trigger threshold. For example, for the maximum amount of data to be read using H.264 (Advanced Video Coding, AVC) encoding, the trigger threshold set in the front-end processing module 12 can be 16 lines. Here, H.264 encoding can be used to support the transition from standard definition (SD) video to high definition (HD, i.e., 1080P resolution).

[0092] For example, the trigger threshold set in the front-end processing module 12 can be 64 lines for the maximum amount of data read in H.265 (High Efficiency Video Coding, HEVC). Here, H.265 encoding reduces the bitrate by 50% to 70% compared to H.264 encoding at the same image quality, and can be used to support ultra-high-definition video transmission and storage at resolutions of 4K (2160P) and 8K.

[0093] For example, for the requirement of stitching multiple images into a single complete image, the trigger threshold can be the amount of data from all the images to be stitched. The post-processing module 13 can power on in response to the amount of image data written to the storage unit 112 by the pre-processing module 12 reaching the trigger threshold, and begin encoding the cached data in the storage unit 112. After outputting the corresponding encoded data, it powers off to reduce the power consumption of the image processing chip 10.

[0094] Thus, compared to the on-chip storage solution, although the off-chip storage solution implemented by connecting the external memory 20 to the peripheral storage management unit requires waiting for a certain interval after completing a read request before a read request can be made, it can still reduce the access frequency by increasing the amount of data in a single request, thereby improving the efficiency of reading the data of the previous processing results.

[0095] In other embodiments, the aforementioned storage unit 112 may include an internal storage management unit and a random access memory (RAM) unit. The internal storage management unit powers on in response to the power-on of the image processing chip 10 and controls the RAM unit to enter a sleep mode. Subsequently, in response to the power-on of the front-end processing module 12, the internal storage management unit can control the RAM unit to exit the sleep mode to prepare for reading and writing data. Then, in response to the back-end processing module 13 completing the output of the corresponding encoded data, the internal storage management unit can re-control the RAM unit to enter the sleep mode.

[0096] Furthermore, the aforementioned RAM unit can have both deep sleep and light sleep modes. The internal storage management unit can control the RAM unit to enter either deep sleep or light sleep mode according to the wake-up latency requirements of the scenario. Specifically, for low-resolution and low-frame-rate scenarios (e.g., AI image transmission scenarios), the processor's computing power requirements are low and the idle time is long. The internal storage management unit can place the inactive RAM unit into deep sleep mode to further reduce the power consumption of the image processing chip 10. Conversely, for high-resolution and high-frame-rate scenarios (e.g., photography or video recording scenarios), the processor's computing power requirements are high and the idle time is short. The internal storage management unit can place the inactive RAM unit into light sleep mode to quickly wake up the RAM unit for accessing the pre-processing result data. Thus, compared to the aforementioned off-chip storage solution, although this on-chip storage solution incurs higher costs, it can further reduce the power consumption of the image processing chip 10 and further reduce the access latency of the pre-processing result data.

[0097] In addition, such as Figure 9 As shown, in response to the output of encoded data, the always-on module 11 can preferably stop running the clock unit 116 and keep it in a sleep state to further reduce power consumption.

[0098] Those skilled in the art will understand that the above-described multi-level wake-up process based on AI image transmission function is merely a non-limiting implementation provided by the present invention, intended to clearly demonstrate the main concept of the present invention and provide a specific solution that is easy for the public to implement, rather than intended to limit the scope of protection of the present invention.

[0099] Optionally, in other embodiments, in response to the power-on of the image processing chip 10, the normally powered module 11 can power on its external interface 114 and operate the external interface 114 to obtain a second trigger command from the outside indicating the photo-taking function and / or video-taking function. Here, the second trigger command includes, but is not limited to, a manual trigger command for clicking the shutter button, a voice trigger command containing the meaning of shooting, and a motion trigger command with the meaning of shooting (e.g., making a specific gesture in front of the lens, or moving the sensor along a special path).

[0100] Please refer to the reference. Figures 1-3 , Figure 5 and Figures 10-14 . Figure 10 A schematic diagram illustrating the initialization configuration of a front-end processing module according to some embodiments of the present invention is shown. Figure 11 A schematic diagram of pre-processing via a pre-processing module is shown, according to some embodiments of the present invention. Figure 12 A schematic diagram illustrating the initialization configuration of a post-processing module according to some embodiments of the present invention is shown. Figure 13 A schematic diagram illustrating encoding via a post-processing module is shown, according to some embodiments of the present invention. Figure 14 A schematic diagram of the exit state of an image processing chip provided in some embodiments of the present invention is shown.

[0101] like Figure 2 , Figure 5 and Figure 10 As shown, in the application of the photo-taking and / or video-taking functions, in response to the second trigger command provided by the external interface 114, the image processing chip 10 can power on the camera interface 121 (e.g., CSI) and the second ISP unit 123 of the front-end processing module 12, and power on the high-computing-power, high-power second processor 14 (e.g., a CPU-L with a main frequency of 600MHz, a computing power of 1386DMIPS, and a peak power consumption of 500mW) via the first processing 115 to wake up the second processor 14. Then, the second processor 14, in conjunction with the direct memory access interface (DMA), obtains the second front-end configuration data for the photo-taking and / or video-taking functions from the instruction cache unit 111, and initializes the front-end processing module 12 accordingly to start the photo-taking and / or video-taking functions. Here, the second front-end configuration data includes, but is not limited to, image size, allocated address space, data specifications, encoding category, and algorithm-related parameters such as image compensation gain / image compensation threshold. By keeping the high-power second processor 14 powered down and only responding to the second trigger instruction, and then waking it up with the low-power first processor 115, the present invention can effectively reduce the standby power consumption of the image processing chip 10. It is understood that the above description of the parameters of the second processor 14 is only a non-limiting example. In actual use, those skilled in the art can make appropriate adjustments to the clock frequency, computing power, and peak power consumption of the second processor 14 according to actual needs.

[0102] In addition, such as Figure 10 As shown, in response to the aforementioned second trigger command, the image processing chip 10 can also run the clock unit 116 to output a clock signal controlling the operation of the front-end processing module 12 and / or the back-end processing module 13. Furthermore, the always-on module 11 can preferably stop operating the external interface 114 in response to the operation of the clock unit 116. Thus, the external interface 114 remains powered on and in a sleep state to prevent simultaneous triggering of multiple commands, reduce power consumption, and ensure that the next valid trigger command can be obtained in real time.

[0103] After that, as Figure 2 and Figure 11As shown, after completing the initialization configuration of the aforementioned pre-processing module 12, the image processing chip 10 can perform pre-processing on the acquired image to be processed via the pre-processing module 12, including lens shading correction (LSC), demosaic processing, noise reduction, and purple fringing removal. After writing the pre-processed image data (i.e., the pre-processing result data) into the storage unit 112, the pre-processing module 12 is powered off. Thus, by writing the pre-processing result data into the storage unit 112 and directly powering off the pre-processing module 12, this invention effectively decouples the power-on and power-off operations between the pre-processing module 12 and the subsequent processing module 13, thereby effectively reducing the power consumption of the image processing chip 10.

[0104] In addition, such as Figure 2 and Figure 12 As shown, the image processing chip 10 can reuse the time of the front-end processing module 12 to power on the image encoder 132 of the back-end processing module 13, and obtain the back-end configuration data from the instruction cache unit 111 to initialize the back-end processing module 13.

[0105] Specifically, in the application of the photo-taking and video-taking functions, in response to the power-on of the post-processing module 13, the second processor 14 can obtain the second post-processing configuration data for the photo-taking and / or video-taking functions from the instruction cache unit 111, and initialize the post-processing module accordingly. Here, the aforementioned second post-processing configuration data includes, but is not limited to, image size, allocated address space, data specifications, and encoding category, and it is consistent with the configuration data of the ISP pipeline. Thus, by using the high-computing-power, high-power second processor 14 to initialize the ISP pipeline and encoder, this invention can adapt to the high image quality requirements of the photo-taking and video-taking functions, utilizing a large-core processor to accurately configure the ISP pipeline and encoder, thereby improving the quality of the acquired images.

[0106] Furthermore, such as Figure 2 and Figure 13As shown, the aforementioned storage unit 112 may include a peripheral memory management unit (IPM). This IPM is connected to an external memory (EX MEM) 20 for off-chip storage of the pre-processing result data. Specifically, the external memory 20 may be a double-data-rate synchronous dynamic random access memory (DDR SDRAM). During image processing in the photo-taking and / or video-taking functions, the IPM can power on in response to the power-on of the image processing chip 10 and control the external memory 20 to enter self-refresh mode. Subsequently, in response to the power-on of the pre-processing module 12, the IPM can control the external memory 20 to exit self-refresh mode to prepare for reading and writing data. Then, in response to the output of the corresponding encoded data by the post-processing module 13, the IPM can control the external memory 20 to re-enter self-refresh mode to reduce the power consumption of the image processing chip 10.

[0107] Furthermore, the aforementioned front-end processing module 12 may also have a pre-set trigger threshold. The back-end processing module 13 can power on in response to the amount of image data written by the front-end processing module 12 to the storage unit 112 reaching the trigger threshold, and begin encoding the cached data in the storage unit 112. After outputting the corresponding encoded data, it powers off to reduce the power consumption of the image processing chip 10.

[0108] Furthermore, in some embodiments of the present invention, the aforementioned post-processing module 13 may also include a video processing unit (VPU) 131. In response to the cached data stored in the storage unit 112 being video data and meeting preset startup conditions, the second processor 14 may preferably cooperate with the video processing unit 131 to perform video encoding on the cached data to meet the application requirements for shooting and outputting video stream data.

[0109] In addition, such as Figure 14 As shown, in response to the output of encoded data, the always-on module 11 can preferably stop running the clock unit 116 and keep it in a sleep state to further reduce power consumption.

[0110] In addition, in some embodiments, during the processing of multiple frames of images, the aforementioned front-end processing module 12 and / or back-end processing module 13 may preferably be powered down after the end of each frame of the image to be processed and powered up again before the beginning of each frame of the image to be processed, so as to further reduce the overall power consumption of the image processing chip 10.

[0111] Thus, the image processing chip, image processing chip control method, image acquisition device, near-eye display mobile terminal, computer-readable storage medium, and computer program product provided by the present invention can sequentially wake up each level of modules according to the data transmission direction, reuse the data processing time of the previous level module, and configure the subsequent level module to improve the response speed and efficiency of the image processing chip, reduce the power consumption of the image processing chip, and thereby optimize the user experience.

[0112] In addition to the energy-saving scheme of cascading wake-up of each module of the image processing chip 10, as described above, this invention also provides an image processing chip, an image processing chip control method, an image acquisition device, a near-eye display mobile terminal, a computer-readable storage medium, and a computer program product. By distinguishing between the application scenarios of AI image transmission functions and traditional photo / video functions, different driving voltage amplitudes and / or driving voltage frequencies are used to drive the processing pipeline for pre-processing image processing. Thus, compared to existing technologies that reuse high-performance processing pipelines for images generated by photo-taking functions to perform high-quality processing of image data from AI image transmission, this invention can effectively reduce the waste of computing power and power consumption in AI image transmission functions, thereby improving battery life, reducing battery size and weight, and lowering battery costs.

[0113] Please refer to Figure 15 , Figure 15 A schematic diagram of the frame of an image processing chip provided according to some embodiments of the present invention is shown.

[0114] exist Figure 15 In the illustrated embodiment, the image processing chip 30 provided in the third aspect of the present invention includes a camera interface (CSI) 321, a timing unit 311, an external interface 312, and at least two image transmission paths. Here, the camera interface 321 is used to acquire an image to be processed via an external camera module. The timing unit 311 is used to generate a first trigger command at regular intervals. The external interface 312 is used to acquire a second trigger command input from an external source.

[0115] The aforementioned timing unit 311 and external interface 312 are configured in the always-on module 31. Here, the always-on module 31 powers on in response to the power-on of the image processing chip 30, and runs the timing unit 31 and / or external interface 312 to obtain a first trigger command or a second trigger command. The first trigger command includes, but is not limited to, a trigger command that automatically transmits the input image to the AI ​​model. The AI ​​model includes, but is not limited to, various models that process images using artificial intelligence technology, such as image recognition models and object tracking models. The second trigger command includes, but is not limited to, a manual trigger command by clicking the shooting button, a voice trigger command containing shooting meaning, and a motion-sensing trigger command with shooting meaning (e.g., making a specific gesture in front of the lens, or moving the sensor along a specific path).

[0116] In addition, the aforementioned always-on module 31 also includes an instruction cache unit 313 (e.g., CMD Buffer), a storage unit 314, and a first processor 315. The first processor 315 powers on in response to the power-on of the image processing chip 30, and enters WFI state after pre-storing the configuration data of the AI ​​image transmission function, photo taking function, and / or video recording function to the instruction cache unit 313 to reduce power consumption.

[0117] In addition, the aforementioned always-on module 31 also includes a clock unit 316 (e.g., Phase-Locked Loop, PLL), which operates in response to a trigger command and outputs a clock signal to control the operation of the front-end processing module 32 and / or the back-end processing module 33 in the image processing chip 30.

[0118] Specifically, the aforementioned camera interface 321 is configured in the pre-processing module 32 of the image processing chip 30. The pre-processing module 32 is also configured with a first ISP unit 322 for AI image transmission function and a second ISP unit 323 for photo taking and / or video recording function. It is powered on in response to a trigger command, obtains pre-processing configuration data from the command cache unit 313 for initialization configuration, performs pre-processing on the acquired image to be processed after completing the initialization configuration, and powers off after writing the pre-processed image data into the storage unit 314.

[0119] Furthermore, the image processing chip 30 provided in the third aspect of the present invention also includes a post-processing module 33. This post-processing module 33 includes an image encoder 331 and a video encoder 332. It powers on during the pre-processing time of the pre-processing module 32, obtains post-processing configuration data from the instruction cache unit 313 for initialization configuration, encodes the cached data in a corresponding format in response to the cached data stored in the storage unit 314 reaching a preset startup condition, and powers off after outputting the corresponding encoded data.

[0120] Furthermore, in some embodiments, the first ISP unit 322 of the AI ​​image transmission function can be connected to a first driving power supply with an output voltage of a first voltage (e.g., 0.63V), while the second ISP unit 323 can be connected to a second driving power supply with an output voltage of a second voltage (e.g., 0.8V).

[0121] The first ISP unit 322, the storage unit 314, and the image encoder 331 in the subsequent processing module 33 can constitute the first image transmission path for the AI ​​image transmission function. In response to a first trigger command indicating the AI ​​image transmission function, the image processing chip 30 can activate the first image transmission path, and the image to be processed acquired by the camera interface 321 is input into the first ISP unit 322, so that the image to be processed is subjected to first pre-processing at a first frequency (e.g., 200MHz) via the first ISP unit 322.

[0122] Furthermore, the second ISP unit 323, the storage unit 314, and the image encoder 331 and video encoder 332 in the post-processing module 33 can respectively constitute the second image transmission path for the photo-taking and video-taking functions. In response to a second trigger command indicating the photo-taking and / or video-taking functions, the image processing chip 30 can activate the second image transmission path, and the acquired image to be processed is input into the second ISP unit 323 via the camera interface 321, so that the image to be processed is subjected to second pre-processing at a second frequency (e.g., 500MHz) via the second ISP unit 323. Thus, by using a smaller first voltage and first frequency for pre-processing of the AI ​​image transmission function in the first image transmission path, and reusing the image encoder 331 of the post-processing module 33 for image encoding processing of both the AI ​​image transmission and photo-taking functions, this invention can meet the lower-quality image processing requirements of the AI ​​image transmission function with lower computing power and power consumption, thereby improving battery life, reducing battery size and weight, and lowering battery costs.

[0123] Furthermore, in some embodiments, a network processing unit (NPU) 324 may preferably be configured in the second image transmission path described above. This network processing unit 324 can be configured in the front-end processing module 32 to cooperate with the second ISP unit 323 in performing second front-end processing on the image to be processed based on neural network inference.

[0124] In addition, in some embodiments, the post-processing module 33 of the image processing chip 30 may preferably be equipped with an encryption unit 333, which is used to encrypt the image encoding data of the AI ​​image transmission function output by the image encoder 331 and output it to an external image AI processor.

[0125] Furthermore, for the aforementioned differentiated first and second image transmission paths, the camera module of the image acquisition device can preferably be configured with a low-specification first camera lens and a high-specification second camera lens. The low-specification first camera lens can have an equivalent focal length of 4-6mm, a resolution of 2-5 megapixels, and a viewing angle of 60-70°. The high-specification second camera lens can have an equivalent focal length of 24-70mm, a resolution of at least 10 million pixels (e.g., 50 million), and a viewing angle of 75-85°. The aforementioned first ISP unit 322 can be connected to the first camera lens via the camera interface 321 to acquire a low-quality first image to be processed for AI image transmission, while the aforementioned second ISP unit can be connected to the second camera lens via the camera interface 321 to acquire a high-quality second image to be processed for photo and / or video functions, thereby reducing the power consumption of the image acquisition device during AI image acquisition and improving the efficiency of AI image acquisition.

[0126] The working principle of the image processing chip described above will be described below with reference to embodiments of control methods for some image processing chips. Those skilled in the art will understand that these embodiments of control methods are merely non-limiting implementations provided by the present invention, intended to clearly demonstrate the main concepts of the invention and provide specific solutions convenient for public implementation, rather than limiting all functions or operating modes of the image processing chip. Similarly, the image processing chip is also only one non-limiting implementation provided by the present invention, and does not limit the executing entity or execution order of each step in the control methods of these image processing chips.

[0127] Please refer to the reference. Figures 16-18 . Figure 16 A schematic diagram of the standby state of an image processing chip provided according to some embodiments of the present invention is shown. Figure 17 A flowchart illustrating a control method for an image processing chip according to some embodiments of the present invention is shown. Figure 18 A schematic diagram illustrating the principle of a control method for an image processing chip according to some embodiments of the present invention is shown.

[0128] like Figure 16As shown, in response to the power-on of the image processing chip 30, the always-on module 31 can also be powered on, enabling the image processing chip 30 to enter the startup state. Afterwards, the image processing chip 30 can use the first processor 315 to preload configuration data into the instruction cache unit 311 for storage, then put the first processor 315 into a sleep state, and put the timing unit 311 and external interface 312 into a running state, enabling the image processing chip 30 to enter a standby state. At this time, in the always-on module 31, except for the timing unit 311 and external interface 312 which are in a running state, all other modules or units are in an idle state.

[0129] After that, as Figure 17 As shown, the image processing chip 30 can obtain a first trigger instruction generated by the timing unit 311 or a second trigger instruction obtained from the outside via the external interface 312 through the running timing unit 311 and the external interface 312.

[0130] After that, such as Figure 18 and Figure 19 As shown, in response to receiving a first trigger command generated by the timing unit 311, the image processing chip 30 can activate the clock unit 316, activate the first image transmission path using the first processor 315, and then use DMA to move the pre-loaded commands in the instruction cache unit 311 to each module or unit to complete the initialization configuration. Afterwards, the image processing chip 30 can control the first ISP unit 322 of the first image transmission path to process the image to be processed acquired from the camera interface 321 based on a first voltage (e.g., 0.65V) and a first frequency (e.g., 20MHz), and store the corresponding pre-processing result data in the storage unit 314. Then, in response to the amount of pre-processing result data written by the first ISP unit 322 to the storage unit 314 reaching a trigger threshold, the image encoder 331 can power on and begin image encoding of the cached data in the storage unit 314 to output encoded data of a first image with a first resolution (e.g., 480P, i.e., 720 horizontal pixels × 480 vertical pixels), and transmit it to the AI ​​model. Here, the AI ​​model includes, but is not limited to, image recognition models, object tracking models, and other models that process images using artificial intelligence (AI) technology.

[0131] Furthermore, after completing the front-end data processing, the image processing chip 30 can also power down the first ISP unit 322 of the front-end processing module 32, power down the image encoder 331 of the back-end processing module 33 after outputting the image encoded data, turn off the clock unit 316, and put the first processor 315 into WFI state, so that the image processing chip 30 returns to the standby state, thereby reducing its power consumption.

[0132] Furthermore, in some embodiments, in response to a second trigger command received from the external interface 312, the image processing chip 30 can activate the clock unit 316 and wake up the second processor 34 using the first processor 315. The second image transmission path is then activated via the second processor 34, and its initial configuration is completed. Here, the first processor 315 can be a small-core processor (CPU-S) with a clock speed of 170MHz, a computing power of 213DMIPS, and a peak power consumption of 26.69mW to reduce power consumption in AI image transmission and standby mode. The second processor 34 can be a large-core processor (CPU-L) with a clock speed of 600MHz, a computing power of 1386DMIPS, and a peak power consumption of 500mW to meet the high-quality image processing requirements of the photo-taking and / or video-taking functions. Subsequently, the image processing chip 30 can control the second ISP unit 323 of the second image transmission path to process the image to be processed acquired from the camera interface 321 based on the second voltage (e.g., 0.8V) and the second frequency (e.g., 500MHz), and store the corresponding pre-processing result data in the storage unit 314. Then, in response to the amount of pre-processing result data written by the second ISP unit 323 to the storage unit 314 reaching a trigger threshold, the back-end processing module 33 can select either the image encoder 331 or the video encoder 332 according to the target output format to perform image encoding or video encoding on the cached data in the storage unit 314, so as to output a second image with a larger second resolution (e.g., 1440P, i.e., 2560 horizontal pixels × 1440 vertical pixels) compared to the first resolution, and transmit it to the back-end storage terminal.

[0133] Furthermore, after completing the front-end data processing, the image processing chip 30 can also power down the second ISP unit 323 of the front-end processing module 32, and after outputting image encoding data or video encoding data, power down the image encoder 331 and video encoder 332 of the back-end processing module 33, turn off the clock unit 316, and power down the second processor 34, so that the image processing chip 30 returns to the standby state, thereby reducing its power consumption.

[0134] Thus, the image processing chip, image processing chip control method, image acquisition device, near-eye display mobile terminal, computer-readable storage medium, and computer program product provided by the present invention can distinguish between the application scenarios of AI image transmission function and traditional photo and video functions, and use independent processing pipelines to perform image processing on the two functions respectively, thereby improving the performance of the image processing chip, reducing the power consumption of the image processing chip, and thus optimizing the user experience.

[0135] In summary, the image processing chip, image processing chip control method, image acquisition device, near-eye display mobile terminal, computer-readable storage medium, and computer program product provided by this invention can, on the one hand, improve the response speed and efficiency of the image processing chip and reduce its power consumption by sequentially waking up modules at each stage according to the data transmission direction and reusing the data processing time of the preceding module to configure the subsequent module, thereby optimizing the user experience. On the other hand, by distinguishing between the application scenarios of AI image transmission function and traditional photo / video functions, and using independent processing pipelines for image processing of the two functions respectively, the performance of the image processing chip can be improved, and its power consumption reduced, thereby optimizing the user experience.

[0136] Although the methods described above are illustrated and depicted as a series of actions for the sake of simplicity, it should be understood and appreciated that these methods are not limited by the order of the actions, as some actions may occur in a different order and / or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but which may be understood by those skilled in the art, according to one or more embodiments.

[0137] Those skilled in the art will understand that information, signals, and data can be represented using any of a variety of different techniques and arts. For example, the data, instructions, commands, information, signals, bits, symbols, and chips described throughout the above description can be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0138] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in a generalized manner in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the invention.

[0139] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but in alternatives, it may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0140] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor such that the processor can read and write information to / from the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside as discrete components in the user terminal.

[0141] In one or more exemplary embodiments, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functionality may be stored or transmitted as one or more instructions or code on or through a computer-readable medium. A computer-readable medium includes both computer storage media and communication media, encompassing any medium that facilitates the transfer of a computer program from one location to another. A storage medium may be any available medium accessible to a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and is accessible to a computer. Any connection is also legitimately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of a medium. As used in this article, disk and disc include compact discs (CDs), laser discs, optical discs, digital multi-purpose discs (DVDs), floppy disks, and Blu-ray discs. Disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of these should also be included within the scope of computer-readable media.

[0142] The prior description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing chip, characterized in that, include: The always-on module includes a trigger unit, an instruction cache unit, and a storage unit. The always-on module powers on in response to the power-on of the image processing chip and runs the trigger unit to obtain a trigger instruction. The pre-processing module powers on in response to the trigger command, retrieves pre-processing configuration data from the command cache unit for initialization configuration, performs pre-processing on the acquired image to be processed after completing the initialization configuration, and powers off after writing the pre-processed image data into the storage unit; and The post-processing module reuses the power-on time of the pre-processing module to perform the pre-processing, obtains post-processing configuration data from the instruction cache unit for initialization configuration, encodes the cache data in response to the cache data stored in the storage unit reaching the preset start conditions, and powers off after outputting the corresponding encoded data.

2. The image processing chip as described in claim 1, characterized in that, The triggering unit includes a timing unit and / or an external interface, wherein, The always-on module responds to the power-on of the image processing chip by powering on the timing unit and running the timing unit to generate a first trigger command at regular intervals. The always-on module responds to the power-on of the image processing chip by powering on the external interface and running the external interface to obtain a second trigger command from the outside.

3. The image processing chip as described in claim 2, characterized in that, The first triggering instruction includes an instruction to trigger the AI ​​image transmission function, and the second triggering instruction includes an instruction to trigger the photo taking function and / or video recording function.

4. The image processing chip as described in claim 3, characterized in that, The always-on module further includes a first processor, wherein the first processor powers on in response to the power-on of the image processing chip, and enters WFI state after pre-storing the configuration data of the AI ​​image transmission function, the photo taking function and / or the video recording function to the instruction cache unit.

5. The image processing chip as described in claim 4, characterized in that, The first processor is also configured to: In response to the first trigger command, the first front-end configuration data of the AI ​​image transmission function is obtained from the command cache unit, and the front-end processing module is initialized and configured according to a preset configuration mode; and In response to the power-on of the post-processing module, the first post-processing configuration data of the AI ​​image transmission function is obtained from the instruction cache unit, and the post-processing module is initialized and configured according to the preset configuration mode.

6. The image processing chip as described in claim 5, characterized in that, The image processing chip further includes a second processor, wherein the computing power and peak power consumption of the second processor are both greater than those of the first processor, and the first processor is further configured to: In response to the second trigger command, the second processor is powered on so that, after the front-end processing module is powered on, the second processor obtains the second front-end configuration data of the photo-taking function and / or the video-taking function from the instruction cache unit, and initializes the front-end processing module accordingly; and In response to the power-on of the post-processing module, the second processor obtains the second post-processing configuration data of the photo-taking function and / or the video recording function from the instruction cache unit, and performs initialization configuration of the post-processing module accordingly.

7. The image processing chip as described in claim 6, characterized in that, The post-processing module includes a video processing unit, and the second processor is further configured to: In response to the cached data stored in the storage unit being video data and meeting preset startup conditions, the video processing unit performs video encoding on the cached data.

8. The image processing chip as described in claim 1, characterized in that, The storage unit includes a peripheral storage management unit, which is connected to an external memory and configured to: Power on in response to the power-on of the image processing chip, and control the external memory to enter self-refresh mode; In response to the power-on of the front-end processing module, the external memory is controlled to exit the self-refresh mode in preparation for reading and writing data; as well as In response to the output of the corresponding encoded data by the subsequent processing module, the external memory is controlled to enter the self-refresh mode.

9. The image processing chip as described in claim 8, characterized in that, The front-end processing module is also pre-set with a trigger threshold. The back-end processing module powers on in response to the amount of image data written by the front-end processing module to the storage unit reaching the trigger threshold, and begins to encode the cached data in the storage unit.

10. The image processing chip as described in claim 1, characterized in that, The storage unit includes an internal storage management unit and a RAM unit, wherein the internal storage management unit is configured as follows: Power on in response to the power-on of the image processing chip, and control the RAM unit to enter sleep mode; In response to the power-on of the front-end processing module, the RAM unit is controlled to exit the sleep mode to prepare for reading and writing data; and In response to the output of the corresponding encoded data by the subsequent processing module, the RAM unit is controlled to enter the sleep mode.

11. The image processing chip as described in claim 1, characterized in that, The constantly powered module further includes a clock unit, which operates in response to the trigger command to output a clock signal controlling the operation of the front-end processing module and / or the back-end processing module, wherein... The constantly powered module also stops operating the trigger unit in response to the operation of the clock unit, and / or stops operating the clock unit in response to the output of the encoded data.

12. The image processing chip as described in claim 1, characterized in that, The front-end processing module and / or the back-end processing module operate or stop operating according to the clock signal provided by the clock unit, and / or During the processing of multiple frames of images, the front-end processing module and / or the back-end processing module are powered down after the end of each frame of the image to be processed, and powered up again before the beginning of each frame of the image to be processed.

13. The image processing chip as described in claim 1, characterized in that, The step of performing pre-processing on the acquired image after completing the initialization configuration includes: In response to a first trigger command instructing the AI ​​image transmission function, the pre-processing module is driven to perform low-power first pre-processing on the image to be processed using a preset first amplitude and / or first frequency driving voltage; and In response to a second trigger command indicating a photo-taking function and / or a video-taking function, the pre-processing module is driven to perform high-quality second pre-processing on the image to be processed with a preset second amplitude and / or second frequency driving voltage, wherein the first amplitude is less than the second amplitude and the first frequency is less than the second frequency.

14. The image processing chip as described in claim 13, characterized in that, The front-end processing module includes a camera interface and at least two ISP units, wherein... The first ISP unit of the AI ​​image transmission function is connected to a first driving power supply. The camera interface responds to the first trigger command and inputs the acquired image to be processed into the first ISP unit, so that the image to be processed is subjected to a first pre-processing at a first frequency via the first ISP unit. The second ISP unit of the photo-taking function and / or the video-taking function is connected to the second driving power supply. The camera interface responds to the second trigger command and inputs the acquired image to be processed into the second ISP unit so that the image to be processed is subjected to a second pre-processing at a second frequency via the second ISP unit. The output voltage of the first driving power supply is lower than the output voltage of the second driving power supply, and the first frequency is lower than the second frequency.

15. The image processing chip as described in claim 14, characterized in that, The first ISP unit connects to the first camera lens via the camera interface to acquire the first image to be processed via the AI ​​image transmission function captured by the first camera lens. The second ISP unit is connected to the second camera lens via the camera interface to acquire a second image to be processed by the second camera lens for taking pictures and / or recording videos. The second camera lens has a wider equivalent focal length range, higher resolution, and / or a larger angle of view than the first camera lens.

16. The image processing chip as described in claim 14, characterized in that, The pre-processing module further includes a network processing unit, which works in conjunction with the second ISP unit to perform second pre-processing on the image to be processed based on neural network inference.

17. The image processing chip as described in claim 13, characterized in that, The post-processing module includes an image encoder and a video encoder, wherein... The image encoder powers on during the pre-processing of the AI ​​image transmission function or the photo-taking function in the pre-processing module, obtains back-end configuration data from the instruction cache unit for initialization configuration, encodes the cached data in response to the cached data in the storage unit reaching the preset start condition, and powers off after outputting the corresponding image encoded data. The video encoder is powered on during the pre-processing of the camera function in the pre-processing module, obtains the back-end configuration data from the instruction cache unit for initialization configuration, performs video encoding on the cache data in response to the cache data stored in the storage unit reaching the preset start condition, and powers off after outputting the corresponding video encoded data.

18. The image processing chip as described in claim 15, characterized in that, The post-processing module also includes an encryption unit for encrypting the image encoding data of the AI ​​image transmission function output by the image encoder and outputting it to an external image AI processor.

19. A control method for an image processing chip, characterized in that, Includes the following steps: In response to the power-on of the image processing chip, its constantly powered-on module is powered on, wherein the constantly powered-on module includes a trigger unit, an instruction cache unit, and a storage unit; The triggering unit is activated to obtain a trigger command; In response to the trigger command, the front-end processing module is powered on, and the front-end configuration data is obtained from the command cache unit to initialize the front-end processing module. After the initialization configuration is completed, the acquired image to be processed is processed by the pre-processing module, and the pre-processed image data is written into the storage unit before the pre-processing module is powered off. The time allotted for the pre-processing module to perform the pre-processing is reused to power on the post-processing module, and post-processing configuration data is obtained from the instruction cache unit to initialize and configure the post-processing module; and In response to the cached data stored in the storage unit reaching the preset start-up conditions, the cached data is encoded by the post-processing module, and the post-processing module is powered down after outputting the corresponding encoded data.

20. An image processing chip, characterized in that, include: Camera interface, used to acquire images to be processed; The timing unit is used to generate the first trigger command at regular intervals. An external interface is used to obtain a second trigger command from external input; as well as At least two image transmission paths are provided, wherein a first image transmission path is activated in response to the first trigger command and processes the image to be processed acquired by the camera interface based on a first voltage and a first frequency, and a second image transmission path is activated in response to the second trigger command and processes the image to be processed acquired by the camera interface based on a second voltage and a second frequency, wherein the first voltage is less than the second voltage and the first frequency is lower than the second frequency.

21. The image processing chip as described in claim 20, characterized in that, The first triggering instruction includes an instruction to trigger the AI ​​image transmission function, and the second triggering instruction includes an instruction to trigger the photo taking function and / or video recording function.

22. The image processing chip as described in claim 20, characterized in that, The first image transmission path includes a first ISP unit, which performs a first pre-processing on the image to be processed based on the first voltage and the first frequency. The second image transmission path includes a second ISP unit and / or a network processing unit, which performs a second pre-processing on the image to be processed based on the second voltage and the second frequency.

23. The image processing chip as described in claim 22, characterized in that, The first image transmission path and the second image transmission path multiplex a single image encoder for encoding the image after the first pre-processing or the second pre-processing, and / or The second image transmission path includes a video encoder for video encoding of multiple frames of images processed by the second pre-processing stage.

24. A control method for an image processing chip, characterized in that, Includes the following steps: In response to receiving a first trigger command generated by the timing unit, the first image transmission path is activated and processed based on the first voltage and the first frequency to obtain the image to be processed from the camera interface. as well as In response to a second trigger command received from an external interface, the second image transmission path is activated and processes the image to be processed from the camera interface based on a second voltage and a second frequency, wherein the first voltage is less than the second voltage and the first frequency is less than the second frequency.

25. An image acquisition device, characterized in that, include: Camera module, equipped with at least one lens; as well as The image processing chip as described in any one of claims 1 to 18 and 20 to 23 is connected to the camera to acquire the images it captures and to perform image processing on them.

26. The image acquisition device as described in claim 25, characterized in that, Also includes: The human-computer interaction interface is used to obtain a second trigger command provided by the user to trigger the photo-taking function and / or video recording function; as well as An image AI processor is used to perform image recognition based on the encoded data of the AI ​​image transmission function output by the image processing chip, so as to determine the corresponding image recognition result.

27. The image acquisition device as described in claim 25, characterized in that, The image acquisition device also includes a storage battery, wherein the image acquisition device is powered by the storage battery.

28. A mobile terminal with a near-eye display, characterized in that, include: The image acquisition device as described in any one of claims 25 to 27; as well as Display module.

29. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the computer instructions are executed by the processor, the control method of the image processing chip as described in claim 19 or 24 is implemented.

30. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, the control method of the image processing chip as described in claim 19 or 24 is implemented.