Image processing apparatus, method, electronic device, and storage medium
By decomposing and fusing image signals, the problem of high AI processor resource consumption in existing technologies is solved, achieving more efficient image processing results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU HIKVISION DIGITAL TECHNOLOGY CO LTD
- Filing Date
- 2022-12-08
- Publication Date
- 2026-07-07
Smart Images
Figure CN115830434B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of electronic technology, and in particular to an image processing apparatus, method, electronic device, and storage medium. Background Technology
[0002] Image signal processors (ISPs) provide a series of image processing algorithms to process image signals output from image sensors, and the processed image signals are then provided to backend processors for further processing. However, the image processing algorithms provided by existing ISPs are largely rigid and have limitations in image processing. To improve image processing performance, it is proposed to combine traditional image processing algorithms with artificial intelligence (AI) technology. For example, an AI processor can be used to replace one or more functions in the ISP. In this approach, the AI processor has high data processing complexity and consumes significant resources. Summary of the Invention
[0003] In view of this, this application provides an image processing apparatus, method, electronic device, and storage medium, which aims to reduce the resource consumption of artificial intelligence (AI) processors during image processing.
[0004] This application provides an image processing apparatus, comprising:
[0005] An image signal decomposition device is used to decompose a first image signal to obtain a first sub-image signal and a second sub-image signal, wherein the first image signal is an image signal output by a camera device or is obtained based on an image signal output by a camera device;
[0006] An artificial intelligence (AI) processor is used to perform first image signal processing on the first sub-image signal to obtain a third sub-image signal;
[0007] An image signal processor (ISP) is used to perform second image signal processing on the second sub-image signal to obtain a fourth sub-image signal.
[0008] An image signal fusion device is used to fuse the third sub-image signal and the fourth sub-image signal to obtain a second image signal.
[0009] In one possible embodiment of this application, the image signal processor (ISP) includes multiple cascaded processor modules, and the image signal processor (ISP) is further configured to:
[0010] Based on one or more of its processor modules, the second image signal is further processed by the third image signal, and the image processing result is output.
[0011] Alternatively, the second image signal can be directly used as the output image processing result.
[0012] In one possible embodiment of this application, the image signal processor (ISP) includes multiple cascaded processor modules, and the image signal processor (ISP) is further configured to:
[0013] The first image signal is obtained by performing fourth image signal processing on the image signal output by the camera device based on one or more of its processor modules.
[0014] In one possible embodiment of this application, the image processing apparatus further includes:
[0015] A control device is used to analyze and process the first image signal to obtain the attribute parameters of the first image signal;
[0016] It is also used to determine the decomposition mode corresponding to the image signal decomposition device based on the attribute parameters, and to control the image signal decomposition device to decompose the first image signal based on the decomposition mode.
[0017] It is also used to determine the fusion mode corresponding to the image signal fusion device based on the attribute parameters, and to control the image signal fusion device to perform fusion processing on the third sub-image signal and the fourth sub-image signal based on the fusion mode.
[0018] In one possible implementation of this application, the attribute parameters include at least one of the following: gain intensity, noise pattern, moving / stationary region, exposure information, or image brightness, and the attribute parameters correspond to the decomposition mode and / or the fusion mode;
[0019] The control device is used to determine, based on the attribute parameters, the decomposition mode corresponding to the attribute parameters, as the decomposition mode corresponding to the image signal decomposition device; and / or
[0020] The control device is used to determine the fusion mode corresponding to the attribute parameters based on the attribute parameters, and use it as the fusion mode corresponding to the image signal decomposition device.
[0021] In one possible embodiment of this application, the image processing apparatus further includes:
[0022] A memory for transferring the first sub-image signal between the image signal decomposition device and the image signal processor (ISP);
[0023] It is also used to transfer the third sub-image signal between the image signal fusion device and the image signal processor (ISP);
[0024] The memory includes on-chip memory or off-chip memory.
[0025] This application provides an image processing method, including the following steps:
[0026] The first image signal is decomposed to obtain a first sub-image signal and a second sub-image signal. The first image signal is an image signal output by the camera device or is obtained based on the image signal output by the camera device.
[0027] The first sub-image signal is subjected to a first image signal processing to obtain a third sub-image signal; wherein, the first image signal processing is executed by an artificial intelligence (AI) processor;
[0028] The second sub-image signal is subjected to a second image signal processing to obtain a fourth sub-image signal; wherein, the second image signal processing is performed by an image signal processor (ISP);
[0029] The third sub-image signal and the fourth sub-image signal are fused to obtain the second image signal.
[0030] In one possible embodiment of this application, the image signal processor (ISP) includes multiple cascaded processor modules, and the image processing method further includes:
[0031] Based on one or more processor modules of the image signal processing ISP, the second image signal is further subjected to third image signal processing, and the image processing result is output.
[0032] Alternatively, the second image signal can be directly used as the output image processing result.
[0033] In one possible embodiment of this application, the image signal processor (ISP) includes multiple cascaded processor modules, and before the step of decomposing the first image signal to obtain a first sub-image signal and a second sub-image signal, the method further includes:
[0034] The image signal output by the camera device is processed by one or more processor modules of the image signal processing ISP to obtain the first image signal.
[0035] In one possible embodiment of this application, before the step of decomposing the first image signal to obtain the first sub-image signal and the second sub-image signal, the method further includes:
[0036] The first image signal is analyzed and processed to obtain the attribute parameters of the first image signal;
[0037] Based on the attribute parameters, determine the decomposition mode and fusion mode corresponding to the attribute parameters;
[0038] The step of decomposing the first image signal to obtain a first sub-image signal and a second sub-image signal includes:
[0039] Based on the decomposition mode, the first image signal is decomposed to obtain a first sub-image signal and a second sub-image signal.
[0040] The step of fusing the third sub-image signal and the fourth sub-image signal to obtain the second image signal includes:
[0041] Based on the fusion mode, the third sub-image signal and the fourth sub-image signal are fused to obtain the second image signal.
[0042] This application provides an electronic device, the device comprising: a memory, a processor, and an image processing program stored in the memory and executable on the processor, the image processing program being configured to implement the steps of the image processing method described above.
[0043] This application provides a storage medium storing an image processing program, which, when executed by a processor, implements the steps of the image processing method described above.
[0044] This application discloses an image processing apparatus, method, electronic device, and storage medium. Compared with the prior art, which uses an AI processor to replace one or more functions of an image signal processor (ISP), resulting in high data processing complexity and resource consumption of the AI processor, this application includes: an image signal decomposition apparatus for decomposing a first image signal to obtain a first sub-image signal and a second sub-image signal, wherein the first image signal is an image signal output by a camera device or is acquired based on an image signal output by a camera device; an artificial intelligence (AI) processor for performing a first image signal processing on the first sub-image signal to obtain a third sub-image signal; an image signal processor (ISP) for performing a second image signal processing on the second sub-image signal to obtain a fourth sub-image signal; and an image signal fusion apparatus for fusing the third sub-image signal and the fourth sub-image signal to obtain a second image signal. Therefore, in the image processing process, this application decomposes the image signal processed by the image signal processor (ISP), enabling flexible adaptation to the AI processor. The decomposed sub-image signals are then processed by the ISP and the AI processor (also known as the AI processing network). Since the image data input to the AI processor has undergone pre-adaptive processing, the amount of data processing by the AI processor is reduced, lowering the computational power requirements and hardware power consumption. This helps reduce the cost of choosing an AI processor for image data processing. Simultaneously, the sub-image signals obtained from the ISP and AI processor are fused, and the image processing result is output based on the fused image signal. Depending on the adapted environment, the AI processing network can perform de-mosaic, wide dynamic range, noise reduction, data domain conversion, etc. The fusion of the images obtained by the AI processor and the ISP processor can jointly complete the implementation or optimization of one or more functions in image processing, achieving the combination of the AI processor and ISP with fewer resources, thereby improving the image processing effect. Attached Figure Description
[0045] Figure 1 This is a schematic diagram of the structure of the first embodiment of the image processing apparatus of this application;
[0046] Figure 2 This is a schematic diagram of the structure of the second embodiment of the image processing apparatus of this application;
[0047] Figure 3 This is a schematic diagram of the structure of the third embodiment of the image processing apparatus of this application;
[0048] Figure 4 This is a schematic diagram of the structure of the fourth embodiment of the image processing apparatus of this application;
[0049] Figure 5This is a flowchart illustrating the first embodiment of the image processing method of this application;
[0050] Figure 6 This is a schematic diagram of the structure of an electronic device in the hardware operating environment involved in the embodiments of this application.
[0051] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0052] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.
[0053] This application provides an image processing apparatus, with reference to... Figure 1 , Figure 1 This is a schematic diagram of the structure of the first embodiment of the image processing apparatus of this application.
[0054] In this embodiment, the image processing device includes:
[0055] The image signal decomposition device 400 is used to decompose the first image signal to obtain a first sub-image signal and a second sub-image signal. The first image signal is an image signal output by the camera device 500 or is obtained based on the image signal output by the camera device 500.
[0056] The artificial intelligence (AI) processor 300 is used to perform first image signal processing on the first sub-image signal to obtain the third sub-image signal;
[0057] The image signal processor ISP100 is used to perform second image signal processing on the second sub-image signal to obtain the fourth sub-image signal;
[0058] The image signal fusion device 200 is used to fuse the third sub-image signal and the fourth sub-image signal to obtain the second image signal.
[0059] The image processing device provided in this application can be applied to various scenarios requiring image or video input, such as access control systems, monitoring systems, video conferencing, or video calls. For example, in an access control system, the image used for identification during operation can be an image processed by the image processing device provided in this application. Similarly, in a monitoring system, the image displayed on the monitoring platform's screen during operation can be an image processed by the image processing device provided in this application. And in a video conferencing or video call, the images displayed on the screens of different users' smart terminals (mobile phones, computers, smartwatches capable of video calls) can be images processed by the image processing device provided in this application.
[0060] In this embodiment, the camera device 500 can be any device with image acquisition function. For example, the camera device 500 can include, but is not limited to, an analog camera, a digital camera, a monocular camera, or a binocular camera.
[0061] In this embodiment, the image signal output from the camera device 500 undergoes multiple image processing procedures to obtain the image processing result. These image processing procedures include, but are not limited to: black level compensation, dead pixel removal, depixelation, noise reduction, wide dynamic range synthesis, gamma correction, or image domain conversion. The image signal processor ISP100 can execute one or more of these image processing procedures, and the artificial intelligence (AI) processor 300 can also execute one or more of these image processing procedures. Specifically, the AI processor 300 executes the processing of the first sub-image signal (… Figure 1 The process of performing the first image signal processing on the image signal A1 in the image signal processor (ISP100) and the process of performing the second sub-image signal processing on the image signal A1 in the image signal processor (ISP100) are similar. Figure 1 The image signal processing process (A2) and the second image signal processing process can be the same or different. Specifically, the first image signal processing executed by the AI processor 300 may include one or more of the aforementioned image processing processes, and the second image signal processing executed by the image signal processor ISP100 may include one or more of the aforementioned image processing processes.
[0062] In this embodiment, the first image signal can be an image signal output by the camera device 500, or it can be acquired based on the image signal output by the camera device 500. When the first image signal is an image signal output by the camera device 500, the camera device 500 directly transmits the image signal to the image signal decomposition device 400. When the first image signal is acquired based on the image signal output by the camera device 500, the camera device 500 transmits the image signal to the image signal processor ISP100, and the image signal processor ISP100 performs fourth image processing on the image signal output by the camera device 500 to obtain the first image signal. Figure 1 The first image signal (A) is then transmitted by the image signal processor ISP100 to the image signal decomposition device 400. It should be noted that when the image signal processor ISP100 performs a fourth image processing on the image signal output by the camera device 500, the fourth image processing process may include one or more of the above-mentioned image processing processes.
[0063] In this embodiment, the image signal fusion device 200 pairs the third sub-image signal ( Figure 1 Image signal B1) and the fourth sub-image signal ( Figure 1 The image signal B2 in the image signal is fused to obtain the second image signal. Figure 1 After the second image signal (B) is processed, it is transmitted to the image signal processor ISP100 for further processing to obtain an image processing result. The image signal processor ISP100 can directly output the second image signal as the image processing result; alternatively, it can perform third image processing on the second image signal and output the resulting image processing result. It should be noted that when the image signal processor ISP100 performs third image processing on the second image signal, this third image processing process can include one or more of the aforementioned image processing steps.
[0064] In this embodiment, the data interaction between the camera device 500, the image signal decomposition device 400, the artificial intelligence AI processor 300, the image signal processor ISP100, and the image signal fusion device 200 is achieved through communication lines.
[0065] Compared to directly replacing one or more functions of the image signal processor (ISP100) with an AI processor, which results in high data processing complexity and resource consumption for the AI processor, the image processing apparatus of this application decomposes the image signal processed by the ISP100 during image processing. The decomposed sub-image signals are then processed separately by the ISP100 and the AI processor 300, reducing the data processing load and resource consumption of the AI processor 300. Simultaneously, the sub-image signals obtained from the ISP100 and AI processor 300 are fused, and the image processing result is output based on the fused image signal, thus improving the image processing effect. In other words, the image processing apparatus provided in this application improves the image processing effect while reducing the resource consumption of the AI processor 300.
[0066] Figure 2 This is a schematic diagram of the structure of the second embodiment of the image processing apparatus of this application. Figure 1 Based on the image processing apparatus shown, the image signal processor ISP100 in the image processing apparatus includes multiple cascaded processor modules 101.
[0067] Based on the above, the image signal processing (ISP) is further used for:
[0068] Based on one or more of its processor modules 101, the second image signal is further processed into a third image signal, and the image processing result is output.
[0069] Alternatively, the second image signal can be directly used as the output image processing result.
[0070] Furthermore, the image signal processor ISP100 is also used for:
[0071] Based on one or more processor modules 101, the image signal output by the camera device 500 is processed into a fourth image signal to obtain a first image signal.
[0072] In this embodiment, each processor module 101 is configured to perform an image processing procedure. The image processing procedures performed by different processor modules 101 may be the same or different. It should be noted that the image processing procedures performed by the processor module 101 include, but are not limited to: black level compensation, dead pixel removal, depixelation, noise reduction, wide dynamic range synthesis, gamma correction, or image domain conversion.
[0073] In this embodiment, during the entire image processing process from the image signal output by the camera device 500 to obtain the image processing result, multiple cascaded processor modules 101 can execute their respective set image processing processes sequentially; or the multiple cascaded processor modules 101 can execute their respective set image processing processes according to a preset execution order. The preset execution order can be set according to actual application requirements, and this embodiment does not impose specific limitations on it.
[0074] like Figure 2 As shown, the image signal processor ISP100 includes processor modules 01, ..., processor module i, processor module i+1, ..., processor module i+n, processor module i+(n+1), ..., processor module i+(n+m) connected in sequence.
[0075] In this embodiment, when the first image signal is acquired based on the image signal output by the camera device 500, the camera device 500 transmits the image signal to the processor module 101 connected to the image signal processor ISP 100. Figure 2 The processor module 01 in the image signal processor ISP100 performs fourth image processing on the image signal output by the camera device 500 to obtain the first image signal. Figure 2 The image signal A output by the processor module i is then transmitted by the image signal processor ISP100 to the image signal decomposition device 400. It should be noted that when the image signal processor ISP100 performs a fourth image processing step on the image signal output by the camera device 500, this fourth image processing step may include one or more of the steps described above. Figure 2 In the process, the fourth image processing step is as follows: processor modules 01 to i execute their respective set image processing steps in sequence.
[0076] In this embodiment, the second image signal processing performed by the image signal processor ISP100 may include one or more processes from the aforementioned image processing process. Figure 2 In the second image signal processing, the processor modules i+1 to i+n execute their respective set image processing processes in sequence.
[0077] In this embodiment, the image signal fusion device 200 pairs the third sub-image signal ( Figure 2 Image signal B1) and the fourth sub-image signal ( Figure 2 The image signal B2 in the image signal is fused to obtain the second image signal. Figure 2After the second image signal (B) is processed, it is transmitted to the image signal processor ISP100 for further processing to obtain the image processing result. Specifically, if the fourth sub-image signal is output by the last processor module 101 sequentially connected in the image signal processor ISP100... Figure 2 If the processor module i+(n+m) in the image signal processor ISP100 outputs the second image signal as the image processing result, then the processor module 101 that outputs the fourth sub-image signal can directly output the second image signal as the image processing result. Figure 2 If the processor module i+(n+1) is the last processor module 101 sequentially connected in the non-image signal processor ISP100, then the image signal processor ISP100 performs third image processing on the second image signal and outputs the image processing result. It should be noted that when the image signal processor ISP100 performs third image processing on the second image signal, this third image processing process can include one or more of the aforementioned image processing processes. Figure 2 In the process, the third image processing step is as follows: processor modules i+(n+1) to i+(n+m) execute their respective set image processing steps in sequence.
[0078] It should be noted that, during the entire image processing process, the image signal used by the image signal decomposition device 400 for decomposition can be the image signal output by the camera device 500, or it can be obtained by the image signal processor ISP100 performing a fourth image signal processing on the image signal output by the camera device 500 based on one or more processor modules 101. The second image output by the image signal fusion device 200 can be directly used as the output image processing result and can be transmitted to the image signal processor ISP100 so that it can continue to perform a third image signal processing on the second image signal based on one or more processor modules 101 and output the image processing result. Therefore, the embodiments of this application can flexibly configure the position of the artificial intelligence processor 300 in the entire image processing process, meet the requirements of different application scenarios for image processing effects, expand the application scope of the image processing device, and make the image processing device universal.
[0079] Figure 3 This is a schematic diagram of the structure of the third embodiment of the image processing apparatus of this application. Figure 2 Based on the image processing apparatus shown, the image processing apparatus further includes:
[0080] The control device 600 is used to analyze and process the first image signal to obtain the attribute parameters of the first image signal;
[0081] It is also used to determine the decomposition mode corresponding to the image signal decomposition device 400 based on the attribute parameters, and to control the image signal decomposition device 400 to decompose the first image signal based on the decomposition mode.
[0082] It is also used to determine the fusion mode corresponding to the image signal fusion device 200 based on attribute parameters, and to control the image signal fusion device 200 to perform fusion processing on the third sub-image signal and the fourth sub-image signal based on the fusion mode.
[0083] In this embodiment, the control device 600 controls the first image signal ( Figure 3 After analyzing and processing the image signal A), attribute parameters of the first image signal are obtained. These attribute parameters include, but are not limited to, gain intensity, noise pattern, moving / stationary areas, exposure information, or image brightness. Furthermore, these attribute parameters correspond to the decomposition mode and / or fusion mode. The control device 600 can control the image signal decomposition device 400, the memory, and the image signal fusion device 200 by recognizing the environment, thereby controlling the decomposition, synthesis, and storage modes of the image signal.
[0084] In different decomposition modes, the image signal decomposition device 400 decomposes the processed signal and extracts different feature values. Based on the control sent by the control device 600, the image signal decomposition device 400 can perform operations such as image domain single-component data decomposition of video images, image signal decomposition at different frequency levels, background and foreground decomposition, image segmentation, and upsampling / downsampling.
[0085] As an example, the gain intensity corresponds to the first decomposition mode. When the image signal decomposition device 400 corresponds to the first decomposition mode, it decomposes the first image signal into high, medium, and low frequencies according to the gain intensity, obtaining a high-frequency image signal, a medium-frequency image signal, and a low-frequency image signal. It then uses one of the high-frequency, medium-frequency, and low-frequency image signals as the first sub-image signal and the remaining two frequency band signals as the second sub-image signal; or it uses two of the high-frequency, medium-frequency, and low-frequency image signals as the first sub-image signal and the remaining frequency band signal as the second sub-image signal.
[0086] As an example, the moving / stationary region corresponds to the second decomposition mode. When the image signal decomposition device 400 corresponds to the second decomposition mode, it divides the first image signal into moving and stationary regions based on the moving / stationary regions, obtaining moving region image signals and stationary region image signals, and uses one of the moving region image signals and the stationary region image signals as the first sub-image signal and the other as the second sub-image signal.
[0087] As an example, the noise pattern corresponds to the third decomposition mode. When the image signal decomposition device 400 corresponds to the third decomposition mode, it decomposes the first image signal into high, medium, and low frequencies according to the noise pattern, obtaining a high-frequency image signal, a medium-frequency image signal, and a low-frequency image signal. It then uses one of the high-frequency, medium-frequency, and low-frequency image signals as the first sub-image signal and the remaining two frequency band signals as the second sub-image signal; or it uses two of the high-frequency, medium-frequency, and low-frequency image signals as the first sub-image signal and the remaining frequency band signal as the second sub-image signal.
[0088] As an example, the gain intensity corresponds to the first fusion mode. When the fusion mode corresponding to the image signal fusion device 200 is the first fusion mode, the third sub-image signal and the fourth sub-image signal are a high-frequency image signal, a mid-frequency image signal, and a low-frequency image signal, respectively. Based on the gain intensity, the high-frequency image signal, the mid-frequency image signal, and the low-frequency image signal are fused to obtain the second image signal.
[0089] As an example, the moving / stationary region corresponds to the second fusion mode. When the fusion mode corresponding to the image signal fusion device 200 is the second fusion mode, the third sub-image signal and the fourth sub-image signal are the moving region image signal and the stationary region image signal, respectively. Based on the moving / stationary region, the moving region image signal and the stationary region image signal are fused to obtain the second image signal.
[0090] As an example, the noise pattern corresponds to the third fusion mode. When the fusion mode corresponding to the image signal fusion device 200 is the third fusion mode, the third sub-image signal and the fourth sub-image signal are high-frequency image signal, mid-frequency image signal and low-frequency image signal, respectively. Based on the noise pattern, the high-frequency image signal, mid-frequency image signal and low-frequency image signal are fused to obtain the second image signal.
[0091] It should be noted that in this embodiment, the decomposition mode and the fusion mode can be coupled or independent. For example, when the attribute parameter is gain intensity, the decomposition mode is the first decomposition mode corresponding to the gain intensity, and the fusion mode is the first fusion mode corresponding to the gain intensity; that is, there is a coupling relationship between the decomposition mode and the fusion mode. Conversely, when the attribute parameters are gain intensity and noise shape, the decomposition mode can be the first decomposition mode corresponding to the gain intensity, and the fusion mode can be the third fusion mode corresponding to the noise shape; that is, the decomposition mode and the fusion mode are independent.
[0092] It should be noted that the selection of decomposition and fusion modes is related to the image scene dimension. For example, when the image scene is a sports scene, the decomposition mode is the second decomposition mode corresponding to the moving / stationary region, and the fusion mode is the second fusion mode corresponding to the moving / stationary region. That is, the attribute parameters obtained by analyzing and processing the first image signal are the moving / stationary regions. When the image scene is a face recognition scene, the decomposition mode is the third decomposition mode corresponding to the noise pattern, and the fusion mode is the third fusion mode corresponding to the noise pattern. That is, the attribute parameters obtained by analyzing and processing the first image signal are the noise pattern.
[0093] As an example, the decomposition and fusion modes are flexibly customizable. Several decomposition modes can be selected to convert the first image signal into a first sub-image signal suitable for processing by an AI processor and a second sub-image signal suitable for processing by an image signal processor (ISP). Similarly, several fusion modes can be customized to ensure that the resulting second image signal meets further image usage requirements.
[0094] In the image processing process, this application enables flexible self-adaptation of the AI processor through custom configuration of decomposition and fusion modes. Since the image data input to the AI processor undergoes pre-adaptive decomposition processing, the data processing load of the AI processor is reduced, lowering its implementation complexity and hardware power consumption, thus reducing the cost of selecting an AI processor for image data processing. Simultaneously, based on the adapted environment, the sub-image signals obtained from the ISP and AI processor are fused. Because the AI processing network can effectively solve some image processing problems, such as de-mosaicing, wide dynamic range, noise reduction, and data domain conversion, the fusion of the images obtained from the AI processor and ISP processor can jointly achieve or optimize one or more functions in image processing. This allows for the integration of the AI processor and ISP with fewer resources, optimizing the image signal processing results.
[0095] Furthermore, the control device 600 controls the image signal decomposition device 400 to decompose the first image signal based on the decomposition mode, and controls the image signal fusion device 200 to decompose the third sub-image signal based on the fusion mode. Figure 3 Image signal B1) and the fourth sub-image signal ( Figure 3 In addition to fusing the image signal B2 in the image sensor, the control device 600 can also control the acquisition position of the first image signal. This can be understood as the control device 600 controlling whether the first image signal is acquired from the camera device 500 or from a processor module in the image signal processor ISP100.
[0096] Figure 4 This is a schematic diagram of the structure of the fourth embodiment of the image processing apparatus of this application. Figure 3 Based on the image processing apparatus shown, the image processing apparatus further includes:
[0097] The memory 700 is used to transfer the first sub-image signal between the image signal decomposition device 400 and the image signal processor ISP 100;
[0098] It is also used to transfer a third sub-image signal between the image signal fusion device 200 and the image signal processor ISP100.
[0099] In this embodiment, the image signal decomposition device 400 decomposes the first image signal ( Figure 4 After decomposing the image signal A in the image, the first sub-image signal is obtained. Figure 4 Image signal A1) and second sub-image signal ( Figure 4 The first sub-image signal (A2) is written to the memory 700. The AI processor 300 reads the first sub-image signal from the memory 700, thereby completing the transfer of the first sub-image signal between the image signal decomposition device 400 and the image signal processor ISP100.
[0100] In this embodiment, the artificial intelligence (AI) processor 300 performs first image signal processing on the first sub-image signal to obtain the third sub-image signal. Figure 4 The image signal B1 in the image signal is written to the memory 700, and the image signal fusion device 200 reads the third sub-image signal from the memory 700, thereby completing the transfer of the third sub-image signal between the image signal fusion device 200 and the image signal processor ISP100.
[0101] Furthermore, the memory 700 includes on-chip memory or off-chip memory.
[0102] In this embodiment, the control device 600 flexibly selects either on-chip memory or off-chip memory as the memory 700 for transferring data between the image signal fusion device 200 and the image signal processor ISP 100, or between the image signal decomposition device 400 and the image signal processor ISP 100, based on the amount of data supported by the image processing device. Specifically, when the amount of data supported by the image processing device is greater than a preset data threshold, selecting off-chip memory as the memory 700 in this embodiment ensures sufficient storage space for storing image signals; when the amount of data supported by the image processing device is less than or equal to the preset data threshold, selecting on-chip memory as the memory 700 in this embodiment ensures data interaction while reducing resource waste.
[0103] This application also provides an image processing method applied to a control device. (See attached image processing method.) Figure 5 , Figure 5 This is a flowchart illustrating the first embodiment of the image processing method of this application.
[0104] In this embodiment, the image processing method includes the following steps:
[0105] Step S10: Control the image signal decomposition device to decompose the first image signal to obtain a first sub-image signal and a second sub-image signal. The first image signal is the image signal output by the camera device or is obtained based on the image signal output by the camera device.
[0106] Step S20: Control the artificial intelligence (AI) processor to perform first image signal processing on the first sub-image signal to obtain the third sub-image signal; wherein, the first image signal processing is performed by the artificial intelligence (AI) processor; the AI processor performs hardware network processing on the signal A1 decomposed by the image signal decomposition device, which can perform functions such as noise reduction, de-mosaic, and wide dynamic range.
[0107] Step S30: Control the image signal processor (ISP) to perform second image signal processing on the second sub-image signal to obtain the fourth sub-image signal; wherein, the second image signal processing is performed by the image signal processor (ISP);
[0108] Step S40: Control the image signal fusion device to perform fusion processing on the third sub-image signal and the fourth sub-image signal to obtain the second image signal;
[0109] Step S50: Control the image signal processing (ISP) to output the image processing result based on the second image signal.
[0110] In one possible embodiment of this application, the image signal processor (ISP) includes multiple cascaded processor modules, and the image processing method further includes the step of controlling the image signal processor (ISP) to output an image processing result based on the second image signal.
[0111] The image signal processing ISP is controlled to continue performing third image signal processing on the second image signal based on one or more processor modules of the image signal processing ISP, and output the image processing result;
[0112] Alternatively, the image signal processing (ISP) can be controlled to directly output the second image signal as the image processing result.
[0113] In one possible embodiment of this application, the image signal processor (ISP) includes multiple cascaded processor modules. Before the step of controlling the image signal decomposition device to decompose the first image signal to obtain the first sub-image signal and the second sub-image signal, the method further includes:
[0114] The image signal processing ISP is controlled to perform fourth image signal processing on the image signal output by the camera device based on one or more processor modules of the image signal processing ISP to obtain the first image signal.
[0115] In one possible embodiment of this application, before the step of the control image signal decomposition device decomposing the first image signal to obtain the first sub-image signal and the second sub-image signal, the method further includes:
[0116] The first image signal is analyzed and processed to obtain the attribute parameters of the first image signal;
[0117] Based on the attribute parameters, determine the decomposition mode and fusion mode corresponding to the attribute parameters;
[0118] The step of the control image signal decomposition device decomposing the first image signal to obtain a first sub-image signal and a second sub-image signal includes:
[0119] The control image signal decomposition device decomposes the first image signal based on the decomposition mode to obtain a first sub-image signal and a second sub-image signal.
[0120] The step of the control image signal fusion device fusing the third sub-image signal and the fourth sub-image signal to obtain the second image signal includes:
[0121] The control image signal fusion device performs fusion processing on the third sub-image signal and the fourth sub-image signal based on the fusion mode to obtain the second image signal.
[0122] The implementation principle and beneficial effects of the image processing method shown in the embodiments of this application can be found in the implementation principle and beneficial effects of the image processing device shown in any of the above embodiments, and will not be repeated here.
[0123] Reference Figure 6 , Figure 6 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of this application.
[0124] like Figure 6As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.
[0125] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0126] like Figure 6 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and an image processing program.
[0127] exist Figure 6 In the electronic device shown, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the electronic device of this application can be set in the electronic device, and the electronic device calls the image processing program stored in the memory 1005 through the processor 1001 and executes the image processing method provided in the embodiment of the above image processing method.
[0128] This application also provides a storage medium that stores one or more programs, which can be executed by one or more processors to implement the steps of the above-described image processing method.
[0129] The specific implementation of the storage medium in this application is basically the same as the various embodiments of the image processing method described above, and will not be repeated here.
[0130] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.
[0131] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0132] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0133] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. An image processing apparatus, characterized in that, include: An image signal decomposition device is used to decompose a first image signal to obtain a first sub-image signal and a second sub-image signal, wherein the first image signal is an image signal output by a camera device or is obtained based on an image signal output by a camera device; An artificial intelligence (AI) processor is used to perform first image signal processing on the first sub-image signal to obtain a third sub-image signal; An image signal processor (ISP) is used to perform second image signal processing on the second sub-image signal to obtain a fourth sub-image signal. An image signal fusion device is used to fuse the third sub-image signal and the fourth sub-image signal to obtain a second image signal; The image processing device further includes: The control device is configured to analyze and process the first image signal to obtain attribute parameters of the first image signal; and to determine the decomposition mode corresponding to the image signal decomposition device based on the attribute parameters, and to control the image signal decomposition device to decompose the first image signal based on the decomposition mode.
2. The image processing apparatus as claimed in claim 1, characterized in that, The image signal processor (ISP) includes multiple cascaded processor modules, and the image signal processing ISP is also used for: Based on one or more of its processor modules, the second image signal is further processed by the third image signal, and the image processing result is output. Alternatively, the second image signal can be directly used as the output image processing result.
3. The image processing apparatus as claimed in claim 1, characterized in that, The image signal processor (ISP) includes multiple cascaded processor modules, and the ISP is further used for: The first image signal is obtained by performing fourth image signal processing on the image signal output by the camera device based on one or more of its processor modules.
4. The image processing apparatus according to any one of claims 1 to 3, characterized in that, The control device is further configured to determine the fusion mode corresponding to the image signal fusion device based on the attribute parameters, and control the image signal fusion device to perform fusion processing on the third sub-image signal and the fourth sub-image signal based on the fusion mode.
5. The image processing apparatus as described in claim 4, characterized in that, The attribute parameters include at least one of the following: gain intensity, noise pattern, moving / stationary region, exposure information, or image brightness, and the attribute parameters correspond to the decomposition mode and / or the fusion mode; The control device is used to determine the decomposition mode corresponding to the attribute parameters based on the attribute parameters, and use it as the decomposition mode corresponding to the image signal decomposition device. and / or The control device is used to determine the fusion mode corresponding to the attribute parameters based on the attribute parameters, and use it as the fusion mode corresponding to the image signal decomposition device.
6. The image processing apparatus according to any one of claims 1 to 3, characterized in that, The image processing device further includes: A memory for transferring the first sub-image signal between the image signal decomposition device and the image signal processor (ISP); It is also used to transfer the third sub-image signal between the image signal fusion device and the image signal processor (ISP); The memory includes on-chip memory or off-chip memory.
7. An image processing method, characterized in that, Includes the following steps: The first image signal is decomposed to obtain a first sub-image signal and a second sub-image signal. The first image signal is an image signal output by the camera device or is obtained based on the image signal output by the camera device. The first sub-image signal is subjected to a first image signal processing to obtain a third sub-image signal; wherein, the first image signal processing is executed by an artificial intelligence (AI) processor; The second sub-image signal is subjected to a second image signal processing to obtain a fourth sub-image signal; wherein, the second image signal processing is performed by an image signal processor (ISP); The third sub-image signal and the fourth sub-image signal are fused to obtain the second image signal; Before the step of decomposing the first image signal to obtain the first sub-image signal and the second sub-image signal, the method further includes: The first image signal is analyzed and processed to obtain the attribute parameters of the first image signal; Based on the attribute parameters, determine the decomposition mode corresponding to the attribute parameters; The step of decomposing the first image signal to obtain a first sub-image signal and a second sub-image signal includes: Based on the decomposition mode, the first image signal is decomposed to obtain a first sub-image signal and a second sub-image signal.
8. The image processing method as described in claim 7, characterized in that, The image signal processor (ISP) includes multiple cascaded processor modules, and the image processing method further includes: Based on one or more processor modules of the image signal processing ISP, the second image signal is further processed by a third image signal, and the image processing result is output. Alternatively, the second image signal can be directly used as the output image processing result.
9. The image processing method as described in claim 7, characterized in that, The image signal processor (ISP) includes multiple cascaded processor modules. Before the step of decomposing the first image signal to obtain the first sub-image signal and the second sub-image signal, the method further includes: Based on one or more processor modules of the image signal processing ISP, the image signal output by the camera device is subjected to fourth image signal processing to obtain the first image signal.
10. The image processing method according to any one of claims 7 to 9, characterized in that, Before the step of decomposing the first image signal to obtain the first sub-image signal and the second sub-image signal, the method further includes: The first image signal is analyzed and processed to obtain the attribute parameters of the first image signal; Based on the attribute parameters, determine the fusion mode corresponding to the attribute parameters; The step of fusing the third sub-image signal and the fourth sub-image signal to obtain the second image signal includes: Based on the fusion mode, the third sub-image signal and the fourth sub-image signal are fused to obtain the second image signal.
11. An electronic device, characterized in that, The device includes: a memory, a processor, and an image processing program stored in the memory and executable on the processor, the image processing program being configured to implement the steps of the image processing method as claimed in any one of claims 7 to 10.
12. A storage medium, characterized in that, The storage medium stores an image processing program, which, when executed by a processor, implements the steps of the image processing method as described in any one of claims 7 to 10.