Image processing apparatus, image processing method, electronic apparatus, and medium

A dual-image acquisition and processing system with separate modules connected via a connection interface addresses processor overload, improving image processing quality and efficiency by distributing the workload across multiple processors.

CN119094902AInactive Publication Date: 2024-12-06VIVO MOBILE COMM CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411089418.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-12-06
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing image processing systems face challenges with low quality and efficiency when processing multiple camera data due to processor overload, limited bandwidth, and insufficient computing power.

Method used

Implementing a dual-image acquisition and processing system with separate modules connected via a connection interface, where each module processes its own data and shares the load with the main processor, reducing the burden on the primary processor by distributing the workload across multiple processors.

Benefits of technology

This approach effectively alleviates processor overload, enhancing image processing quality and efficiency by distributing the processing tasks across multiple processors, thereby improving overall system performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119094902A_ABST
    Figure CN119094902A_ABST
Patent Text Reader

Abstract

The invention discloses an image processing device, an image processing method, electronic equipment and a medium, and belongs to the technical field of image processing. The image processing method comprises the following steps: acquiring first image data through a first image acquisition module, and processing the first image data to obtain second image data; acquiring second image data through the second image acquisition module, and processing the second image data to obtain fourth image data; and processing the second image data and the fourth image data to obtain a target image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to an image processing device, an image processing method, an electronic device and a medium. Background Art

[0002] At present, electronic devices generally use the camera connection interface provided by the platform manufacturer. Most platforms have data interfaces for multiple cameras and can drive multiple cameras at the same time. However, if multiple cameras are turned on at the same time, the image data collected by multiple cameras needs to be processed at the same time. This processing requires high bandwidth and computing power. However, the number and performance of platform processors are limited. Processing the image data collected by multiple cameras at the same time will cause the processor to be overloaded, resulting in problems with image processing quality and efficiency. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide an image processing device, an image processing method, an electronic device and a medium, which can solve the problems of low image processing quality and efficiency caused by excessive processor load when simultaneously processing multiple image data in the related art.

[0004] In a first aspect, an embodiment of the present application provides an image processing device, comprising: a first image acquisition module, a second image acquisition module, a connection module and an image processing module; the first image acquisition module and the second image acquisition module are connected via the connection module;

[0005] The first image acquisition module includes at least one first image acquisition device and a first processor; the first image acquisition device is used to acquire first image data and transmit the first image data to the first processor; the first processor is used to process the first image data to obtain second image data and transmit the second image data to the image processing module;

[0006] The second image acquisition module includes at least one second image acquisition device and a second processor; the second image acquisition device is used to acquire third image data and transmit the third image data to the second processor; the second processor is used to process the third image data to obtain fourth image data and transmit the fourth image data to the image processing module;

[0007] The image processing module is used to process the second image data and the fourth image data to obtain a target image.

[0008] In a second aspect, an embodiment of the present application provides an image processing method, which is applied to the image processing device described in the first aspect, and the method includes:

[0009] Acquire first image data through the first image acquisition module, and process the first image data to obtain second image data;

[0010] Acquire third image data through the second image acquisition module, and process the third image data to obtain fourth image data;

[0011] The second image data and the fourth image data are processed to obtain a target image.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the second aspect are implemented.

[0013] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the second aspect are implemented.

[0014] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the second aspect.

[0015] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the second aspect.

[0016] In the embodiment of the present application, the first image data is collected by the first image acquisition module, and the first image data is processed to obtain the second image data; the second image acquisition module is collected by the second image acquisition module, and the second image data is processed to obtain the fourth image data; and the target image is obtained by processing the second image data and the fourth image data. Since the second image acquisition module can collect the second image data through the second image acquisition device and process the second image data, it can share the task of processing all image data of the first image acquisition module, and reduce the processor load pressure when the first image acquisition module processes a large amount of image data. Therefore, it can effectively avoid the situation where the processor load is too high when the first image acquisition module processes multiple image data at the same time, and can improve the quality and efficiency of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of an existing image processing architecture in the related art;

[0018] Figure 2is a schematic diagram of the module composition of an image processing device provided in an embodiment of the present application;

[0019] Figure 3 is a schematic diagram of the architecture of an image processing device provided in an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of the working principle of an image processing device provided in an embodiment of the present application;

[0021] Figure 5 is a schematic diagram of the architecture of another image processing device provided in an embodiment of the present application;

[0022] Figure 6 is a schematic diagram of the working principle of another image processing device provided in an embodiment of the present application;

[0023] Figure 7 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0024] Figure 8 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0025] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0027] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0028] In order to facilitate the description of this application, the image processing architecture in the related art is first described. Figure 1 A schematic diagram of an existing image processing architecture provided by an embodiment of the present application is shown. Figure 1As shown, in the existing image processing architecture, the system-on-chip (System on Chip, SOC) provides multiple mobile industry processor interfaces (Mobile Industry Processor Interface, MIPI), which can connect multiple cameras including color (Red, Green, Blue, RGB) cameras and black and white cameras, and receive image data collected from multiple cameras. For example, the SOC chip connects two RGB cameras and four black and white cameras. When the above multiple cameras are turned on at the same time, the generated data stream will increase, but the bus in the SOC chip transmits data to each camera separately. When the data stream increases, it will inevitably lead to insufficient bus bandwidth or increased bandwidth busyness. Moreover, the processing capacity of the image signal processor (Image Signal Processor, ISP) is limited and cannot process a large amount of image data. In addition, after the data is transmitted to the hardware abstraction layer (Hardware Abstraction Layer, HAL), the HAL layer also performs independent algorithm processing on the image data collected by each camera, which will consume more central processing unit (Central Processing Unit, CPU) resources, resulting in reduced performance. The HAL layer is a software layer that provides an abstract interface between the operating system or application software and the hardware. The HAL layer can optimize the image data processing process according to the characteristics of the underlying hardware, such as the ISP processor, CPU, camera, etc., thereby improving the image processing performance.

[0029] The image processing device, image processing method, electronic device and medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0030] Figure 2 A schematic diagram of the module composition of an image processing device provided in an embodiment of the present application is shown. Figure 2 As shown, the image processing device includes: a first image acquisition module 21, a second image acquisition module 22, a connection module 23 and an image processing module 24, and the first image acquisition module 21 and the second image acquisition module 22 are connected via the connection module 23;

[0031] The first image acquisition module 21 includes at least one first image acquisition device 211 and a first processor 212; the first image acquisition device 211 is used to acquire first image data and send the first image data to the first processor 212; the first processor 212 is used to process the first image data to obtain second image data and send the second image data to the image processing module 24;

[0032] The second image acquisition module 22 includes at least one second image acquisition device 221 and a second processor 222; the second image acquisition device 221 is used to acquire third image data and send the third image data to the second processor 222; the second processor 222 is used to process the third image data to obtain fourth image data and send the fourth image data to the image processing module 24;

[0033] The image processing module 24 is used to process the second image data and the fourth image data to obtain a target image.

[0034] The image processing device in the embodiment of the present application can be applied to an extended reality (ER) device or other mobile device with multiple cameras, and the multiple cameras may include at least one RGB camera and at least one black and white camera.

[0035] In the first image acquisition module 21 in the embodiment of the present application, the first image acquirer 211 is used to acquire first image data and send the first image data to the first processor 212. The first processor 212 is used to perform denoising, bad pixel correction, lens shading correction, gamma correction, edge enhancement and automatic exposure control on the first image data to obtain second image data, and send the second image data to the image processing module 24.

[0036] In the second image acquisition module 22, the second image acquirer 221 is used to acquire third image data and send the third image data to the second processor 222. The second processor 222 is used to perform denoising, bad pixel correction, and automatic exposure control on the third image data to obtain fourth image data, and send the fourth image data to the image processing module 24.

[0037] The image processing module 24 receives the second image data and the fourth image data, and can perform processing such as color space conversion, image compression, image enhancement, image analysis and recognition on the second image data and the fourth image data. For example, in a scenario applied to an XR device, the image processing module 24 can perform image preprocessing on the second image data and the fourth image data, such as denoising, contrast enhancement, etc., perform feature detection and feature matching on the second image data and the fourth image data, and perform three-dimensional reconstruction and posture estimation, and finally output the user's posture (pose) data. In addition, the image processing module 24 can also perform the same image processing process as the ISP processor on the first image data collected by the first image collector 211 and the third image data collected by the second image collector 221.

[0038] The first image acquisition module 21 in the embodiment of the present application may include a SOC chip, the first processor 212 may be deployed on the SOC chip, and an image acquisition interface is provided on the SOC chip for connecting the first image acquisition device 211. The image acquisition interface may be, for example, a MIPI interface, and the first image data acquired by the first image acquisition device 211 may be transmitted to the first processor 212 through the image acquisition interface. The second image acquisition module may include an integrated circuit chip (IC) chip, the second processor 222 may be deployed on the IC chip, and an image acquisition interface is provided on the IC chip for connecting the second image acquisition device 221. The image acquisition interface may be, for example, a MIPI interface, and the third image data acquired by the second image acquisition device 221 may be transmitted to the second processor 222 through the image acquisition interface. The first image acquisition device 211 may be an RGB camera, and the second image acquisition device 221 may be a black and white camera. In another embodiment, the first image acquisition device 211 may also be a black and white camera, and the second image acquisition device 221 may also be an RGB camera. The image processing module 24 may be a module capable of performing image processing based on the Android operating system (hereinafter referred to as the system), and is used to process the second image data and the fourth image data, and finally output a target image. The image processing module 24 receives the second image data and the fourth image data, and can perform processing such as color space conversion, image compression, image enhancement, image analysis and recognition on the second image data and the fourth image data.

[0039] In one example, the first image collector 211 is an RGB camera, the first processor 212 is an ISP processor deployed on a SOC chip, the second image collector 221 is a black and white camera, and the second processor 222 is an ISP processor deployed on an IC chip. The image data collected by the first image collector 211, i.e., the RGB camera, is the first image data, which is the original RGB value sensed by each photosensitizing point of the RGB camera sensor. Then, the first image data is sent to the first processor 212, and the first processor 212 performs denoising, bad pixel correction, lens shading correction, gamma correction, edge enhancement, automatic exposure control and other processing on the first image data to obtain the second image data, and sends the second image data to the image processing module 24. The image data collected by the second image collector 221, that is, the black and white camera is the second image data, which is the image data collected by the black and white camera and only contains brightness information. Then, the second image data is sent to the second processor 222, and the second processor 222 performs denoising, bad pixel correction, contrast enhancement, automatic exposure control and other processing on the second image data to obtain fourth image data, and sends the fourth image data to the image processing module 24. The image processing module 24 can perform image enhancement processing, image conversion and encoding processing on the second image data and the fourth image data, such as sharpening, contrast adjustment, format conversion and image compression processing, feature detection and feature matching, and three-dimensional reconstruction and posture estimation. The image processing module 24 can also send the target image obtained after processing to the corresponding application in the mobile device.

[0040] In the embodiment of the present application, the first image acquisition module 21 and the second image acquisition module 22 are connected via a connection module. In one implementation, the connection module 23 includes a connection interface;

[0041] The connection interface is used to connect the first processor and the second processor.

[0042] In an embodiment of the present application, the first processor 212 and the second processor 222 can be connected via a connection interface, which can be a data interface or a data transmission interface, and the connection interface can be, for example, a MIPI interface. The first image acquisition module 21 includes a SOC chip, and the first processor 212 is deployed on the SOC chip. The second image acquisition module 22 includes an IC chip, and the second processor 222 is deployed on the IC chip. In practical applications, the IC chip can also be connected to the SOC chip via a connection interface. After the IC chip and the SOC chip are connected, the second processor 222 in the IC chip can receive the third image data collected by the second image collector 221 and process the third image data. In this way, the IC chip can share the image processing task of the SOC chip, and can reduce the performance burden of the existing SOC chip when it needs to process a large amount of image data collected by multiple cameras.

[0043] Figure 3 A schematic diagram of the architecture of an image processing device provided in an embodiment of the present application is shown. Figure 3 As shown, an ISP processor is deployed in the SOC chip, and the ISP processor is connected to two RGB cameras through a MIPI interface. The ISP processor can receive the first image data collected by the two RGB cameras, process the first image data, and send the processed first image data to the image processing module. As an example, the image processing module can be a module capable of image processing based on the Android operating system. After the ISP processor on the SOC chip processes the first image data, the second image data obtained after processing can be sent to the HAL layer of the Android operating system, and the HAL layer performs data format conversion and post-processing on the second image data. The HAL layer is a software layer that provides an abstract interface between the operating system or application software and the hardware. The HAL layer can optimize the image data processing process according to the characteristics of the underlying hardware, such as the ISP processor, CPU, first image collector, etc., thereby improving the image processing performance. The SOC chip is connected to the IC chip via a data interface, and an ISP processor is deployed in the IC chip. The ISP processor is connected to the four black-and-white cameras via a MIPI interface. The ISP processor receives the third image data collected by the four black-and-white cameras and processes the third image data, and sends the fourth image data obtained after processing to the HAL layer of the Android operating system, and the HAL layer performs data format conversion and post-processing on the second image data. In addition, an algorithm unit can also be provided in the IC chip, and the algorithm unit is used to further process the fourth image data through an image processing algorithm, such as image enhancement processing, feature extraction, etc. In practical applications, the fourth image data can also be further processed by the algorithm unit to generate a target image without passing through the image processing module 24.

[0044] Figure 4 A schematic diagram showing the working principle of an image processing device provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the ISP processor in the SOC chip receives the first image data collected by the two RGB cameras and processes the first image data. After the SOC chip completes the processing of the first image data, the second image data obtained after the processing can be transmitted to the HAL layer of the system through the Driver inside the SOC chip. The HAL layer of the system performs data format conversion and data post-processing, etc., passes through the service layer of the system, and transmits the final data to the corresponding application. The ISP processor in the IC chip receives the second image data collected by the four black and white cameras and processes the second image data. After the ISP processor of the IC chip processes the second image data, the fourth image data obtained after the processing can be transmitted to the HAL layer of the system through the Driver inside the IC chip. The HAL layer of the system performs data format conversion and data post-processing, etc., passes through the service layer of the system, and transmits the final data to the corresponding application.

[0045] The above two RGB cameras and four black-and-white cameras are all cameras on the mobile device. When multiple cameras in the mobile device are turned on at the same time, the SOC chip is responsible for processing the first image data collected by the two RGB cameras, and the IC chip is responsible for processing the third image data collected by the four black-and-white cameras. After the data processing is completed, the SOC chip sends the second image data obtained after processing to the system's HAL layer, and the IC chip sends the fourth image data obtained after processing to the system's HAL layer, and the system's HAL layer further processes the second image data and the fourth image data. After that, the system HAL layer sends the final data to the corresponding application. The image acquisition and processing processes of the SOC chip and the IC chip do not interfere with each other. In this way, by connecting the SOC chip with the IC chip, the bus bandwidth required for the SOC chip to transmit and process a large amount of image data can be reduced, which greatly reduces the load pressure on the processor (including the ISP processor and the CPU) when the SOC chip processes a large amount of image data collected by multiple cameras.

[0046] In one implementation, the first image acquisition module further includes: a data bus;

[0047] The connection module is also used to connect the first processor and the second processor to the data bus of the first image acquisition module.

[0048] In the embodiment of the present application, the first image acquisition module 21 includes a SOC chip, and a data bus is deployed on the SOC chip, and the second image acquisition module 22 includes an IC chip. The first processor may be an ISP processor deployed on the SOC chip, and the second processor may be an ISP processor deployed on the IC chip, and the first processor and the second processor may be connected via the data bus on the SOC chip. In practical applications, the IC chip and the SOC chip may be packaged in the same single wafer area, and the IC chip and the SOC chip may be connected via the data bus of the SOC chip.

[0049] Figure 5 FIG. 4 shows a schematic diagram of the architecture of another image processing device provided in an embodiment of the present application. Figure 5 As shown, the SOC chip is connected to multiple RGB cameras and multiple black and white cameras through multiple MIPI interfaces, and the image data collected by multiple cameras is processed by the ISP processor. The IC chip is connected to the SOC chip through the data bus on the SOC chip. The IC chip is equipped with an ISP processor, which can receive and process the image data collected by the SOC chip when the SOC chip cannot process a large amount of image data, so as to reduce the load pressure of the SOC chip processor. In practical applications, the IC chip and the SOC chip can be packaged in the same wafer area. Figure 5 As shown in FIG. 1 , on the basis of connecting the SOC chip with the IC chip, a double data rate (DDR) module can also be set on the SOC chip to improve the data transmission efficiency of the SOC chip. The DDR module is a high-speed cache memory and a technical standard of synchronous dynamic random access memory (SDRAM) used to increase the bandwidth of the memory. DDR technology transmits data at both the rising and falling edges of the clock signal, thereby achieving a higher data transmission rate than the traditional single data rate (SDR) memory at the same clock frequency.

[0050] It is understandable that when multiple cameras are turned on at the same time, the SOC needs to collect and process multiple image data. However, there is only one ISP processor in the SOC chip. When multiple cameras are turned on at the same time and need to be processed by the ISP processor, the processor needs to allocate resources to these cameras at the same time. However, the processing power of the ISP processor is limited. If the number of cameras processed at the same time is too large, or the processing complexity of a single camera is too high, it may lead to insufficient resource allocation, thereby affecting the processing speed and efficiency. In addition, the clock frequency of the ISP processor is limited. When there are too many processing tasks, the clock frequency may not be able to meet all processing requirements, resulting in a decrease in processing speed, which in turn causes frame loss. In this case, the interrupt processing situation can be used to determine whether the SOC chip is unable to process multiple image data at the same time, thereby causing frame loss.

[0051] In one implementation, the first image acquisition module further includes an interrupt processing unit;

[0052] An interrupt processing unit, configured to detect whether the first image acquisition module fails to respond to the interrupt request within a preset time threshold; and, if the first image acquisition module fails to respond to the interrupt request within the preset time threshold, trigger the second image acquisition module to process the interrupt request;

[0053] The interrupt request is used to trigger the first image acquisition module to process the first image data.

[0054] In the embodiment of the present application, the interrupt processing unit of the first image acquisition module 21 can detect whether the first image acquisition module 21 does not respond to the interrupt request within the preset time threshold, and the interrupt request is used to trigger the first image acquisition module 21 to process the first image data. The interrupt processing unit is also used to trigger the second image acquisition module 22 to process the interrupt request when the first image acquisition module 21 does not respond to the interrupt request within the preset time threshold. That is, the interrupt detection unit can trigger the second image acquisition module 22 to process the interrupt request when it is detected that the first image acquisition module 21 does not process the interrupt request within the preset time threshold.

[0055] An interrupt request is a signal sent by an interrupt source to the CPU, requesting the CPU to suspend the currently executing program and instead process the event or task raised by the interrupt source. The purpose of an interrupt request is to improve the CPU utilization and the system response speed, so that the CPU can handle various urgent or important events in a timely manner, rather than waiting for the completion of a program. During the interrupt processing, in order to improve the efficiency of interrupt processing and the responsiveness of the system, the interrupt processing program can be decomposed into the upper half and the lower half. When there is a large amount of image data to be processed, the upper half of the interrupt in the interrupt processing unit will receive the interrupt request, and the lower half of the interrupt will allocate the specific processing process. When the ISP processor of the SOC chip cannot process a large amount of image data, the lower half of the interrupt will allocate the image processing task to the IC chip for processing.

[0056] In actual applications, the IC chip can be added to the driver of the SOC chip. When multiple cameras are started, the IC chip will be in a low-power state. If the SOC chip does not lose interrupts, that is, the interrupt request is not processed within the preset time threshold, the IC chip does not need to process the image.

[0057] In one example, the interrupt request may be issued by an RGB camera, requesting the SOC chip to process the collected RGB image data. Based on this, the interrupt processing unit can detect whether the SOC chip has not processed the interrupt request issued by the RGB camera within a preset time threshold. If it is detected that the SOC chip has not processed the interrupt request within the preset time threshold, it means that the SOC chip cannot process the RGB image data normally, and its processor may be under load. Then, the interrupt processing unit can trigger the IC chip to process the interrupt request, that is, to process the RGB image data collected by the above-mentioned RGB camera. If it is detected that the SOC chip has processed the interrupt request within the preset time threshold, then the IC chip can always maintain a low power consumption state without participating in image processing.

[0058] In one implementation, the interrupt processing unit is further configured to generate an interrupt processing signal when the first image acquisition module does not respond to the interrupt request within a preset time threshold; transmit the interrupt processing signal to the second image acquisition module; and the interrupt processing signal is used to trigger the second image acquisition module to process the interrupt request;

[0059] The second image acquisition module is further used to acquire the first image data acquired by the first image acquisition module and process the first image data under the triggering of the interrupt processing signal.

[0060] The interrupt processing unit in the embodiment of the present application can also generate an interrupt processing signal when the SOC chip does not respond to the interrupt request within a preset time threshold, and send the interrupt processing signal to the IC chip, and the interrupt processing signal is used to trigger the IC chip to process the interrupt request. The IC chip obtains the first image data collected by the SOC chip under the triggering of the interrupt processing signal, and processes the first image data through the ISP processor.

[0061] In the embodiment of the present application, the IC chip may acquire the first image data collected by the SOC chip by actively acquiring the first image data, or the interrupt processing unit in the SOC chip may send the first image data to the IC chip while sending an interrupt processing signal.

[0062] After the IC chip completes processing the interrupt request, it can directly transmit the processed target image data to the driver of the SOC chip, and the driver of the SOC chip finally transmits the target image data to the corresponding application.

[0063] In one example, the SOC chip receives first image data collected by four RGB cameras and processes the first image data. Figure 6 A schematic diagram of the working principle of another image processing device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, after the SOC chip completes processing the first image data, the second image data obtained after processing can be transmitted to the HAL layer of the system through the Driver driver, and the HAL layer of the system performs data format conversion and data post-processing, etc., passes through the service layer of the system, and transmits the final data to the corresponding application. It can be judged whether the SOC chip is in a load state by the processing of the interrupt request. If it is detected that the SOC chip does not process the interrupt request within the preset time threshold, it means that the SOC chip cannot process the first image data normally, and it may be in a load state. Then, the IC chip can process the interrupt request and process the first image data collected by the above-mentioned RGB camera. The IC chip can receive the first image data collected by the SOC chip and process the image data. After the ISP processor of the IC chip processes the first image data, the processed first image data can be transmitted to the HAL layer of the system through the Driver driver of the SOC chip, and the HAL layer of the system performs data format conversion and data post-processing, etc., passes through the service layer of the system, and transmits the final data to the corresponding application.

[0064] The above-mentioned image processing device provided in the embodiment of the present application acquires the first image data through the first image acquisition module, and processes the first image data to obtain the second image data; acquires the second image data through the second image acquisition module, and processes the second image data to obtain the fourth image data; and obtains the target image by processing the second image data and the fourth image data. Since the second image acquisition module can acquire the second image data through the second image acquisition device and process the second image data, it can share the task of processing all image data for the first image acquisition module, and reduce the processor load pressure when the first image acquisition module processes a large amount of image data. Therefore, it can effectively avoid the situation where the processor load is too high when the first image acquisition module processes multiple image data at the same time, and can improve the quality and efficiency of image processing.

[0065] See also Figure 7 , is a flow chart of an image processing method provided in an embodiment of the present application. Figure 7 As shown, the image processing method provided in the embodiment of the present application can be applied to Figure 2 The image processing device shown in the figure specifically includes the following steps.

[0066] S702: collecting first image data through a first image collection module, and processing the first image data to obtain second image data;

[0067] S704: collecting second image data through a second image collection module, and processing the second image data to obtain fourth image data;

[0068] S706: Process the second image data and the fourth image data to obtain a target image.

[0069] In an embodiment of the present application, the image processing device collects first image data through a first image acquisition module, and processes the first image data to obtain second image data; collects second image data through a second image acquisition module, and processes the second image data to obtain fourth image data; and processes the second image data and the fourth image data through an image processing module to obtain a target image.

[0070] In one embodiment, the method further comprises:

[0071] Generate an interrupt request; the interrupt request is used to trigger the first image acquisition module to process the first image data;

[0072] When the first image acquisition module fails to respond to the interrupt request within a preset time threshold, the second image acquisition module is triggered to process the interrupt request.

[0073] In an embodiment of the present application, the image processing device may generate an interrupt request during the image data processing process, and the interrupt request is used to trigger the first image acquisition module to process the first image data. The interrupt request may be issued by the first image acquisition device, requesting the first image acquisition module to process the first image data.

[0074] The interrupt processing unit can detect, based on the interrupt request, whether the first image acquisition module does not respond to the interrupt request within a preset time threshold. If the first image acquisition module does not respond to the interrupt request within the preset time threshold, the interrupt processing unit triggers the second image acquisition module to process the interrupt request.

[0075] In one implementation, when the first image acquisition module does not respond to the interrupt request within a preset time threshold, triggering the second image acquisition module to process the interrupt request includes:

[0076] When the first image acquisition module fails to respond to the interrupt request within a preset time threshold, an interrupt processing signal is generated; the interrupt processing signal is used to trigger the second image acquisition module to process the interrupt request;

[0077] Based on the interrupt processing signal, the second image acquisition module is triggered to acquire the first image data and process the first image data.

[0078] In an embodiment of the present application, when the first image acquisition module does not respond to the interrupt request within a preset time threshold, the interrupt processing unit generates an interrupt processing signal, which is used to trigger the second image acquisition module to process the interrupt request. Based on the interrupt processing signal, the second image acquisition module is triggered to obtain the first image data and process the first image data.

[0079] The above method provided by the embodiment of the present application acquires the first image data through the first image acquisition module, and processes the first image data to obtain the second image data; acquires the second image data through the second image acquisition module, and processes the second image data to obtain the fourth image data; and obtains the target image by processing the second image data and the fourth image data. Since the second image acquisition module can acquire the second image data through the second image acquisition module and process the second image data, it can share the task of processing all image data for the first image acquisition module, and reduce the processor load pressure when the first image acquisition module processes a large amount of image data. Therefore, it can effectively avoid the situation where the processor load is too high when the first image acquisition module processes multiple image data at the same time, and can improve the quality and efficiency of image processing.

[0080] See also Figure 8 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 8As shown, an embodiment of the present application further provides an electronic device 800, including a processor 801 and a memory 802, wherein the memory 802 stores programs or instructions that can be executed on the processor 801, and when the program or instructions are executed by the processor 801, the various steps of the above-mentioned image acquisition method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0081] It should be noted that the electronic devices in the embodiments of the present application include mobile electronic devices and non-mobile electronic devices.

[0082] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device according to an embodiment of the present application.

[0083] The electronic device 900 includes but is not limited to: a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, and a processor 910 and other components.

[0084] Those skilled in the art will appreciate that the electronic device 900 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 910 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0085] The processor 910 is configured to acquire first image data through the first image acquisition module, and process the first image data to obtain second image data;

[0086] Acquire third image data through the second image acquisition module, and process the third image data to obtain fourth image data;

[0087] The second image data and the fourth image data are processed to obtain a target image.

[0088] In the embodiment of the present application, the first image data is collected by the first image acquisition module, and the first image data is processed to obtain the second image data; the second image acquisition module is collected by the second image acquisition module, and the second image data is processed to obtain the fourth image data; and the target image is obtained by processing the second image data and the fourth image data. Since the second image acquisition module can collect the second image data through the second image acquisition device and process the second image data, it can share the task of processing all image data of the first image acquisition module, and reduce the processor load pressure when the first image acquisition module processes a large amount of image data. Therefore, it can effectively avoid the situation where the processor load is too high when the first image acquisition module processes multiple image data at the same time, and can improve the quality and efficiency of image processing.

[0089] Optionally, the processor 910 is further configured to generate an interrupt request; the interrupt request is used to trigger the first image acquisition module to process the first image data;

[0090] When the first image acquisition module fails to respond to the interrupt request within the preset time threshold, the second image acquisition module is triggered to process the interrupt request.

[0091] Optionally, the processor 910 is further configured to generate an interrupt processing signal when the first image acquisition module fails to respond to the interrupt request within the preset time threshold; the interrupt processing signal is used to trigger the second image acquisition module to process the interrupt request;

[0092] Based on the interrupt processing signal, the second image acquisition module is triggered to acquire the first image data and process the first image data.

[0093] It should be understood that in the embodiment of the present application, the input unit 904 may include a graphics processor (Graphics Processing Unit, GPU) 9041 and a microphone 9042, and the graphics processor 9041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 906 may include a display panel 9061, and the display panel 9061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 907 includes a touch panel 9071 and at least one of other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include two parts: a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0094] The memory 909 can be used to store software programs and various data. The memory 909 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 909 may include a volatile memory or a non-volatile memory, or the memory 909 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 909 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0095] The processor 910 may include one or more processing units; optionally, the processor 910 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 910.

[0096] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0097] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0098] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0099] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0100] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0101] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0102] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0103] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing device, characterized in that: include: A first image acquisition module, a second image acquisition module, a connection module and an image processing module; The first image acquisition module and the second image acquisition module are connected via the connection module; The first image acquisition module includes at least one first image acquisition device and a first processor; the first image acquisition device is used to acquire first image data and transmit the first image data to the first processor; The first processor is used to process the first image data to obtain second image data, and transmit the second image data to the image processing module; The second image acquisition module includes at least one second image acquisition device and a second processor; the second image acquisition device is used to acquire third image data and transmit the third image data to the second processor; the second processor is used to process the third image data to obtain fourth image data and transmit the fourth image data to the image processing module; The image processing module is used to process the second image data and the fourth image data to obtain a target image.

2. The image processing device according to claim 1, characterized in that The connection module includes a connection interface; The connection interface is used to connect the first processor and the second processor.

3. The image processing device according to claim 1, characterized in that The first image acquisition module further includes: a data bus; The connection module is also used to connect the first processor and the second processor to the data bus of the first image acquisition module.

4. The image processing device according to claim 1, characterized in that: The first image acquisition module also includes an interrupt processing unit; The interrupt processing unit is used to detect whether the first image acquisition module fails to respond to the interrupt request within a preset time threshold; and, if the first image acquisition module fails to respond to the interrupt request within the preset time threshold, trigger the second image acquisition module to process the interrupt request; The interrupt request is used to trigger the first image acquisition module to process the first image data.

5. The image processing device according to claim 4, characterized in that: The interrupt processing unit is further configured to generate an interrupt processing signal when the first image acquisition module fails to respond to the interrupt request within the preset time threshold; Transmitting the interrupt processing signal to the second image acquisition module; the interrupt processing signal is used to trigger the second image acquisition module to process the interrupt request; The second image acquisition module is further used to acquire the first image data acquired by the first image acquisition module and process the first image data under the triggering of the interrupt processing signal.

6. The image processing device according to claim 4, characterized in that: The first image acquisition module includes a SOC chip; the second image acquisition module includes an IC chip.

7. An image processing method, characterized in that: Applied to the image processing device according to any one of claims 1 to 6, the method comprises: Acquire first image data through the first image acquisition module, and process the first image data to obtain second image data; Acquire third image data through the second image acquisition module, and process the third image data to obtain fourth image data; The second image data and the fourth image data are processed to obtain a target image.

8. The method according to claim 7, characterized in that The method further comprises: Generate an interrupt request; the interrupt request is used to trigger the first image acquisition module to process the first image data; When the first image acquisition module fails to respond to the interrupt request within the preset time threshold, the second image acquisition module is triggered to process the interrupt request.

9. The method according to claim 8, characterized in that The triggering of the second image acquisition module to process the interrupt request when the first image acquisition module fails to respond to the interrupt request within the preset time threshold comprises: If the first image acquisition module does not respond to the interrupt request within the preset time threshold, an interrupt processing signal is generated; the interrupt processing signal is used to trigger the second image acquisition module to process the interrupt request; Based on the interrupt processing signal, the second image acquisition module is triggered to acquire the first image data and process the first image data.

10. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method as described in any one of claims 7 to 9 are implemented.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image processing method according to any one of claims 7 to 9 are implemented.

Citation Information

Patent Citations

  • Terminal device and method of taking panoramic image

    CN107623804A

  • Shooting control method and device, electronic equipment and computer readable storage medium

    CN110971836A

  • Control method and system

    CN111923747A

  • Identification equipment

    CN207571756U

  • Image processing method and apparatus, electronic device and computer-readable storage medium

    WO2020088133A1