Image signal processing drive method, control module, processing module, and medium

Abstract

The present application provides an image signal processing driving method, a control module, a processing module, and a medium. The method is used in an image signal processing module including multiple ISP cores or an ISP core supporting time division multiplexing, and a driving architecture is mounted on the image signal processing module. The method includes: an interface layer receives an image signal processing ISP command including an initial control parameter and an image size parameter from an upper application; a connection layer determines a processing mode of an image signal based on the image size parameter, and determines target control parameters for different processing modes based on the initial control parameters; and a driving layer transmits the target control parameters to a target ISP core, thereby driving the target ISP core to perform image signal processing based on the target control parameters, and the processing mode includes a single-core mode or a multi-core mode. By providing a connection layer in the driving architecture, low resolution by a single core and high resolution by a multi-core can be realized under the same driving architecture, and multiple processing modes can be supported.

Description

[Technical field]

[0001] The present application relates to the technical field of image processing, and specifically to an image signal processing driving method, a control module, a processing module and a medium. [Background technology]

[0002] ISP (Image Signal Processor) technology is a technology that improves the imaging quality of image signals. With the rapid development of electronic devices, image resolution has increased from 2K to 4K, 8K, 16K, etc., and people's demands for image resolution are also increasing. In conventional technology, multi-core is required to achieve a resolution of 8K or higher. Electronic devices are required to achieve a resolution of 8K or higher as well as a lower resolution, which is a technical problem that this field is trying to solve. Summary of the Invention

[0003] The embodiments of the present application aim to provide an image signal processing driving method, a control module, a processing module and a medium for achieving technical effects that satisfy both the realization of high resolution and low resolution.

[0004] In a first aspect of an embodiment of the present application, there is provided an image signal processing driving method, which is used in an image signal processing module including a plurality of ISP cores or a time division multiplexed ISP core, the image signal processing module is equipped with a driving architecture including an interface layer, a connection layer and a driving layer, the driving layer is connected to the ISP cores, the method includes a step of the interface layer receiving an image signal processing ISP command from an upper application, the image signal processing command including an initial control parameter and an image size parameter, the connection layer determining a processing mode of an image signal based on the image size parameter, and determining target control parameters for different processing modes based on the initial control parameters, and the driving layer transmitting the target control parameters to a target ISP core, thereby driving the target ISP core to perform image signal processing based on the target control parameters, the processing mode including a single-core mode or a multi-core mode.

[0005] In the above realization process, the image signal processing module includes multiple ISP cores or an ISP core supporting time division multiplexing. The image signal processing module is also equipped with a driving architecture. An ISP command from a higher-level application includes an initial control parameter and an image size parameter. By providing a connection layer in the driving architecture, the connection layer can determine which processing mode to use based on the image size parameter. In addition, in any processing mode, the connection layer can construct different target control parameters based on the initial control parameters. Meanwhile, the driving layer transmits the target control parameters to the target ISP core, thereby driving the target ISP core to perform image signal processing. In this way, both low resolution by a single core and high resolution by a multi-core can be realized under the same driving architecture, and multiple processing modes can be supported.

[0006] Furthermore, the processing mode is a multi-core mode, and the step of determining target control parameters based on the initial control parameters includes the step of constructing a plurality of control parameters based on the initial control parameters to define the target control parameters.

[0007] In the above realization process, the connection layer establishes multiple control parameters under the multi-core mode as target control parameters, thereby realizing the technical effect of supporting multiple processing modes.

[0008] Further, the ISP instruction includes a target function to be called, and the step of constructing a plurality of control parameters based on the initial control parameters to become the target control parameters includes a step of calling a membership function pointed to by a function pointer provided in the multi-core mode so as to construct the initial control parameters as a plurality of control parameters to become target control parameters, and the membership function corresponds to the target function.

[0009] In the above implementation process, the function pointer can point to different membership functions. Under multi-core mode, multiple control parameters can be constructed by calling the membership functions pointed to by the corresponding function pointers and corresponding to the target functions. In this way, desired target control parameters can be constructed according to the processing mode, and the target ISP core can perform image signal processing according to the corresponding target control parameters under different processing modes, thereby achieving the technical effect of supporting multiple processing modes.

[0010] Furthermore, the objective function includes one or more of a function module open function, a function module close function, and a driving interface function.

[0011] In the above implementation process, different target functions included in the ISP command can construct corresponding target control parameters by calling the membership functions pointed to by the corresponding function pointers under different processing modes, so that the target ISP core can perform image signal processing according to the corresponding target control parameters under different processing modes, thereby achieving the technical effect of supporting multiple processing modes.

[0012] Furthermore, the ISP command indicates a target image processing function to be realized, the target control parameters include size-related control parameters, and the size-related control parameters include one or more of an image width, a parameter related to the image width, an image height, and a parameter related to the image height, the target ISP core performs image signal processing to realize the target image processing function, and the step of constructing a plurality of control parameters based on the initial control parameters to become the target control parameters includes, if the target image processing function is a default function, a step of constructing a plurality of different control parameters to become the target control parameters, and if the target image processing function is not the default function, a step of constructing a plurality of the same control parameters to become the target control parameters, and the default function is realized by the target ISP core performing image signal processing based on the size-related control parameters.

[0013] In the above realization process, the target control parameters constructed under the multi-core mode are different depending on the image processing function. For the default function, the target control parameters used for its realization are size-related control parameters, so the constructed multiple control parameters are different. In this way, image signal processing can be performed using different control parameters. For image processing functions other than the default function, the constructed multiple control parameters are the same. In this way, image signal processing can be performed using the same control parameters. As a result, all of the different image processing functions can be realized under the multi-core mode.

[0014] Furthermore, the step of constructing the plurality of different control parameters includes a step of setting the plurality of different control parameters calculated based on a predetermined image position as the target control parameters.

[0015] In the above realization process, if multiple different control parameters need to be obtained, the target ISP performs image signal processing according to the different control parameters by calculating based on a predetermined image position, and realizes the target image processing function.

[0016] Further, under a multi-core mode, the method further includes a step in which the connection layer responds to an interrupt signal of an image signal processing corresponding to a target ISP core, and after receiving all the interrupt signals, synchronously merges and sends all the interrupt signals to the interface layer.

[0017] In the above implementation process, when an interruption occurs in image signal processing, the interruption signals are synchronously merged and reported after receiving all interruption signals. In any mode, the upper application receives the interruption signal only once, so it does not recognize the existence of two processing modes.

[0018] Furthermore, the method further includes a step of storing an interrupt signal corresponding to a target ISP core in a corresponding queue after receiving the interrupt signal, and a step of synchronously merging and sending all interrupt signals to the interface layer after receiving all interrupt signals includes a step of reading the interrupt signals in each queue, synchronously merging, and sending them to the interface layer when interrupt signals are stored in all queues.

[0019] In the above implementation process, when an interrupt signal of the target ISP core is received, the interrupt signal is stored in a corresponding queue, and waits until all interrupt signals are received. By providing a queue for storing interrupt signals, the technical effect of synchronously merging and reporting after all interrupt signals are received can be achieved.

[0020] Furthermore, when the target ISP core is multiple, the all interrupt signals include interrupt signals corresponding to all the target ISP cores, and the corresponding queues are queues corresponding to the target ISP cores that transmit the interrupt signals; when the target ISP core supports time division multiplexing, the all interrupt signals include interrupt signals transmitted by the target ISP core in all multiple time segments, and the corresponding queues are queues corresponding to the multiple time segments that transmit the interrupt signals.

[0021] In the above implementation process, a corresponding queue is set for each target ISP core or each multiple time segment, and the interrupt signal corresponding to the target ISP core is stored in the corresponding queue, so that the technical effect of synchronously merging and reporting after receiving all the interrupt signals can be achieved.

[0022] Furthermore, the interrupt signal includes a plurality of types, and the step of storing the interrupt signal in a corresponding queue includes a step of storing the interrupt signal in a target position in the queue corresponding to the target type based on the target type to which the interrupt signal belongs, and storing the interrupt signal in all queues includes storing the interrupt signal in target positions in all of the queues.

[0023] In the above implementation process, the interrupt signal includes multiple types, and the interrupt signals of the same type are synchronously merged and reported. Therefore, different positions in the queue are set to correspond to different types of interrupt signals, so that different types of interrupt signals can be stored in the same queue. And when all the interrupt signals are stored in the same position in all the queues, it is considered that all the interrupt signals are received. This achieves the technical effect of synchronously merging and reporting after all the interrupt signals are received.

[0024] Furthermore, the processing mode is a single-core mode, and determining a target control parameter based on the initial control parameter includes determining the initial control parameter as the target control parameter.

[0025] In the above implementation process, the connection layer directly determines the initial control parameters as the target control parameters under the single-core mode, thereby achieving the technical effect of supporting multiple processing modes.

[0026] In a second aspect of an embodiment of the present application, there is provided an image processing control module, the image processing control module being connected to a plurality of ISP cores or an ISP core supporting time division multiplexing, and configured to implement any one of the steps of the method of the first aspect.

[0027] In a third aspect of an embodiment of the present application, an image signal processing module is provided, the image signal processing module including an image processing control module of the second aspect and a plurality of ISP cores or an ISP core supporting time division multiplexing, and the image processing control module and the plurality of ISP cores or the ISP core supporting time division multiplexing are connected.

[0028] Furthermore, the image signal processing module is an IP core, chip or electronic device for image signal processing.

[0029] In a fourth aspect of an embodiment of the present application, there is provided a computer readable storage medium storing computer instructions which, when executed by a processor, effect the steps of any one of the methods of the first aspect. [Brief description of the drawings]

[0030] In order to more clearly describe the technical solutions of the embodiments of the present application, the drawings necessary for describing the embodiments of the present application are briefly described below. The drawings described are only for illustrating some embodiments of the present application, and are not intended to limit the scope. Those skilled in the art can obtain other related drawings based on these drawings without using inventive abilities. [Figure 1] FIG. 1 is a schematic block diagram of a drive architecture based on a single core according to the prior art; [Diagram 2]FIG. 2 is a block diagram of a drive architecture configuration according to an embodiment of the present application. [Diagram 3] FIG. 2 is a schematic block diagram of another drive architecture according to an embodiment of the present application. [Figure 4] 2 is a flowchart of an image signal processing driving method according to an embodiment of the present application. [Diagram 5] 10 is a flowchart of another image signal processing driving method according to an embodiment of the present application. [Figure 6] 10 is a flowchart of another image signal processing driving method according to an embodiment of the present application. [Figure 7] 1 is a flowchart of a process for constructing a target control parameter according to an embodiment of the present application. [Figure 8] 11 is a flowchart of another target control parameter construction process according to an embodiment of the present application. [Figure 9] FIG. 2 is a schematic configuration diagram of an image signal processing module according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] Hereinafter, the technical solutions according to the embodiments of the present application will be described with reference to the drawings of the embodiments of the present application.

[0032] Since like reference numerals refer to like things in the drawings, when defined in one drawing, further definition and interpretation in the other drawings is unnecessary. Also, in the description of this application, the terms "first", "second", etc., are for descriptive purposes only and should not be understood to indicate or imply any relative importance.

[0033] ISP (Image Signal Processor) technology is a technology that improves the imaging quality of image signals. With the rapid development of electronic devices, image resolution has increased from 2K to 4K, 8K, 16K, etc., and people's demands for image resolution are also increasing. In conventional technology, low-resolution images such as 2K, 4K, etc. are realized by a single core. That is, processing of low-resolution images can be realized with only one ISP core. In contrast, multi-cores are required to realize resolutions of 8K or higher. That is, processing of high-resolution images can be realized by using multiple ISP cores, such as two or more ISP cores.

[0034] Electronic devices sometimes process low-resolution images and sometimes process high-resolution images. Therefore, electronic devices need to meet the requirements for processing both high-resolution and low-resolution images. However, in the prior art, the driving architecture of the ISP is written with its driving program based on a single core. The ISP driving architecture based on a single core shown in FIG. 1 includes a test layer (Test), an image engine layer (cam_engine), an interface layer (Cameric), a memory input management control (Memory Input Manager, mim), a buffer pool (buffer_pool), a memory output management control (Memory Output Manager, mom_ctrl), a snapshot output module control (Snapshot Output Module, som_ctrl), a hardware abstraction layer (HAL), an ISP driving (ISPdrv), and a memory interface driving (MIdrv). This driving architecture can only drive a single core to perform image signal processing and realize low-resolution image processing.

[0035] To achieve high-resolution image processing, it is necessary to upgrade the driving architecture so that the driving architecture can drive multiple ISP cores to perform image signal processing. Considering that electronic devices need to simultaneously meet high-resolution image processing and low-resolution image processing, the upgraded driving architecture must not only realize multi-core driving, but also have the original single-core driving function.

[0036] Therefore, the present application proposes an image signal processing driving method for use in an image signal processing module including multiple ISP cores or an ISP core supporting time division multiplexing.

[0037] An ISP core that supports time division multiplexing means that one ISP core can execute different tasks at different times, realizing multitasking execution, and fully utilizing the computing resources of the ISP core. Multiple time segments can appear alternately according to a predetermined period.

[0038] Exemplarily, the image signal processing module is a hardware module, including, but not limited to, IP cores (Intellectual Property cores), chips, electronic devices, and other hardware devices or hardware modules for image signal processing, including ISP cores. The IP core refers to an integrated circuit IP core, which is a multiplexing module having a specific circuit function that has already been designed and verified. Depending on the degree of coupling with a specific integrated circuit manufacturing process (Foundry), it is classified into soft cores, firm cores, and hard cores, among which, the soft cores are independent of the final realization process. The electronic devices include, but are not limited to, other devices that require image processing, such as smartphones / mobile phones, personal digital assistants (PDAs), media content players, video game stations / systems, virtual reality systems, augmented reality systems, and wearable devices (e.g., watches, smart bands, gloves, hats, helmets, virtual reality headsets, augmented reality headsets, head-mounted displays (HMDs), headbands, pendants, shoulder patches, leg bands, shoes, or vests, etc.).

[0039] The image signal processing module includes a driving architecture 200 shown in Fig. 2, which includes an interface layer 210, a connection layer 220, and a driving layer 230. The driving layer 230 is connected to an ISP core. If the image signal processing module includes multiple ISP cores, the driving layer 230 is connected to the multiple ISP cores.

[0040] Exemplarily, the driving layer 230 includes the HAL layer, ISPdrv, and MIdrv shown in FIG. 1. The driving architecture 200 may further include other layers for driving the ISP core, such as a test layer and an image engine layer. As an example, FIG. 3 shows a driving architecture obtained by improving the ISP driving architecture based on a single core. As shown in FIG. 3, this driving architecture provides a connection layer (binder) between the interface layer (Camerice) and the hardware abstraction layer (HAL). The improved driving architecture supports both high-resolution image processing and low-resolution image processing. In this way, by providing the connection layer, the driving architecture that drives a single core can also drive multiple cores to perform image signal processing. And the improved driving architecture can support both high-resolution image processing and low-resolution image processing, and the changes to the conventional driving architecture are relatively small, which can save the human costs of development and upgrade.

[0041] Based on the above embodiments, the image signal processing driving method according to the present application is realized by the driving architecture shown in FIG. 2 or FIG. 3, and includes the steps shown in FIG.

[0042] Step S1: The interface layer receives an image signal processing ISP command from an upper application.

[0043] The ISP command includes initial control parameters and image size parameters.

[0044] Step S2: The connection layer determines a processing mode of an image signal according to the image size parameter, and determines target control parameters for different processing modes according to the initial control parameters.

[0045] The processing mode includes a single-core mode or a multi-core mode.

[0046] Step S3: The driving layer sends the target control parameters to a target ISP core, thereby driving the target ISP core to perform image signal processing based on the target control parameters.

[0047] Exemplarily, the steps of the above method can be realized by a dedicated device or module based on hardware that realizes a predetermined function or executes a predetermined operation. For example, the steps of the above method are realized by an image processing control module. The image processing control module is a hardware module including a circuit for realizing a specific function. The image processing control module is connected to a plurality of ISP cores or an ISP core supporting time division multiplexing, and drives a target ISP core connected to perform image signal processing by executing the above steps S1 to S3. For example, the image processing control module may be an IP core. Optionally, the image signal processing module may include the above image processing control module, a plurality of ISP cores, or an ISP core supporting time division multiplexing.

[0048] The upper application can issue an ISP command to drive the ISP core to process the image signal. The ISP command includes an initial control parameter and an image size parameter. The connection layer determines the processing mode of the image signal based on the image size parameter so that the processing mode can be determined at startup.

[0049] The processing mode includes a single-core mode or a multi-core mode. The single-core mode is image signal processing by one ISP core of the image signal processing module. When the image signal processing module includes multiple ISP cores, image signal processing is performed using one of the multiple ISP cores. The multi-core mode is image signal processing using at least two of the multiple ISP cores of the image signal processing module, or the ISP performs different tasks related to image signal processing in different multiple time segments to perform image signal processing. For example, when the image signal processing module is a dual-core module, the multi-core mode performs image signal processing using two ISP cores. In this case, the multi-core mode is also called a dual-core mode. For example, when the image signal processing module has more than two ISP cores, the multi-core mode may perform image signal processing using all ISP cores, or may perform image signal processing using at least two ISP cores.

[0050] The connection layer determines a processing mode of the image signal based on the image size parameter. Exemplarily, if the image size parameter is equal to or greater than a predetermined size threshold, the processing mode is determined to be a multi-core mode, and if the image size parameter is less than the predetermined size threshold, the processing mode is determined to be a single-core mode. The image size parameter may be a resolution of the image. In this case, it may be configured to determine that if the image resolution is 8K or greater, the processing mode is determined to be a multi-core mode, and if the image resolution is less than 8K, the processing mode is determined to be a single-core mode.

[0051] And the connection layer determines target control parameters based on the initial control parameters for different processing modes. Under different processing modes, the determined target control parameters are different. The control parameters are parameters set in the registers to enable the chip. The control parameters can be called by the user and set according to the actual needs of the user. The control parameters include multiple types, for example, but not limited to, size-related control parameters and function module-related control parameters. Exemplarily, the size-related control parameters include image width, parameters related to image width, image height, parameters related to image height. The function module-related control parameters include CAC (Chromatic Aberration Correction) module parameters, DPF (De-noising Pre-Filter) module parameters, etc. The initial control parameters and the target control parameters are the same type of control parameters.

[0052] After determining the target control parameters, the driving layer sends the target control parameters to the target ISP core, which is an ISP core that performs image signal processing.

[0053] In the case of an image signal processing module including multiple ISP cores, all ISP cores in the image signal processing module may be target ISP cores, i.e., all ISP cores are involved in image signal processing, and some ISP cores may be target ISP cores, i.e., only some ISP cores are involved in image signal processing. On the other hand, in the case of an image signal processing module including an ISP core supporting time division multiplexing, the target ISP is an ISP core that supports time division multiplexing in the image signal processing module and performs image signal processing. A method for determining the target ISP core will be described later.

[0054] In the image signal processing driving method according to the present application, a connection layer is provided in the driving architecture, so that the connection layer can construct target control parameters based on the initial control parameters in any processing mode. Compared with the prior art, the connection layer separates the interface layer and the driving layer, and constructs corresponding target control parameters under different processing modes. The driving layer also transmits the target control parameters to the target ISP core, thereby driving the target ISP core to perform image signal processing. In this way, both low resolution by a single core and high resolution by multiple cores can be realized with the same driving architecture, and multiple processing modes can be supported.

[0055] In some embodiments, step S2 may include the steps shown in FIG.

[0056] Step S20: The connection layer determines a processing mode of an image signal according to the image size parameter.

[0057] Step S21: Under the single-core mode, the initial control parameters are determined as the target control parameters.

[0058] Step S22: Under the multi-core mode, a number of control parameters are constructed based on the initial control parameters, and are set as the target control parameters.

[0059] Under the single-core mode, the initial control parameters are directly determined as target control parameters, there is one target control parameter, and there is one target ISP core. In this way, the driving layer sends the one initial control parameter as the target control parameter to the one target ISP core, so that the one target ISP core performs image signal processing according to the target control parameter.

[0060] Under the multi-core mode, the target control parameters include multiple control parameters, and the target ISP core or the multiple time segments of the target ISP core are multiple. The number of control parameters is the same as the number of the target ISP core or the multiple time segments. In this way, the driving layer sends the multiple control parameters to the multiple target ISP cores respectively, so that the multiple target ISP cores perform image signal processing based on the respective control parameters. Or, the target ISP core performs image signal processing based on different control parameters in different multiple time segments.

[0061] In this way, the connection layer realizes the technical effect of supporting multiple processing modes by respectively constructing different numbers of control parameters as target control parameters under different processing modes.

[0062] In some embodiments, the ISP command indicates a target image processing function to be realized. There are a variety of image processing functions, including but not limited to Demosaic, Gamma Correction, Auto White Balance (AWB), etc. The upper application issues a corresponding ISP command to operate the target ISP core to perform image signal processing based on the target control parameters, thereby realizing the image processing function. As described above, the control parameters include size-related control parameters and function module-related control parameters. If the target control parameters used to realize the image processing function are size-related control parameters including one or more of image width, parameters related to image width, image height, and parameters related to image height, the image processing function is a default function. In other words, when realizing the default function, the target ISP core performs image signal processing based on the size-related control parameters.

[0063] Also, under the multi-core mode, the corresponding image processing function is realized by processing each image by multiple target ISP cores simultaneously or multiple multiple time segments by a target ISP core respectively. For example, taking AWB as an example, one image can be divided into multiple sub-images, and the number of sub-images may be the same as the number of target ISP cores or the number of multiple time segments of the target ISP core. In this way, one target ISP core performs AWB processing on one sub-image, or performs AWB processing on one sub-image in one multiple time segment of the target ISP core. When performing AWB processing, the target ISP core performs image signal processing corresponding to the AWB algorithm based on the size-related control parameters. Therefore, AWB is the above-mentioned default function. After the image signal processing is completed, the image processing results of each target ISP core or each multiple time segment of the target ISP core can be stored in consecutive addresses, and an image after image processing can be obtained without overlapping areas caused by uneven division of the image.

[0064] By way of example, default features include, but are not limited to, LSC (Lens Shading Correction), CAC (Chromatic Aberration Correction), AE (Auto Exposure), AWB, DPCC (Defect Pixel Cluster Correction), IS (Image Stabilization), Crop, Scale, VSM (Video Stabilization Measurement), PDAF (Phase Detection Auto Focus), and the like. On the other hand, image functions that are not image signal processing performed based on size-related control parameters, i.e., image processing functions other than default functions, include, but are not limited to, 2DNR (2D Digital Noise Reduction), 3DNR (3D Digital Noise Reduction), EE (Edge Enhance), DPF (De-noising Pre-Filter), CCM (Color Correction Matrix), CSM (Color Space Matrix), WDR (Wide Dynamic Range), BLS (Black Level Subtraction), DGAIN (Digital Gain), and the like.

[0065] The target image processing function indicated by the ISP command may be the above-mentioned default function or may not be the default function. The step of constructing a plurality of control parameters under the multi-core mode in the above-mentioned step S22 may include the steps shown in FIG.

[0066] Step S221: If the target image processing function is a default function, a plurality of different control parameters are constructed as the target control parameters.

[0067] Step S222: If the target image processing function is not the default function, construct a number of identical control parameters as the target control parameters.

[0068] For example, the step of constructing a plurality of different control parameters in the above step S221 may include a step of setting a plurality of different control parameters obtained by calculation based on a predetermined image position as the target control parameters.

[0069] As described above, for the preset function, an image is usually divided into multiple sub-images, and image signal processing is performed on the different sub-images according to different control parameters. When constructing the different control parameters, calculations are performed based on the image positions of the sub-images in the image, and corresponding control parameters can be obtained. Since the image positions of the sub-images in the image are different, the constructed control parameters are also different. Since the sub-images are divided by a predetermined division method, the image positions are preset. For example, a calculation is performed by an image algorithm based on the image positions to obtain multiple different control parameters. The image algorithm includes a mapping algorithm. After the initial control parameters are processed by the mapping algorithm, multiple different control parameters can be obtained as the target control parameters.

[0070] Exemplarily, in the above step S222, constructing the plurality of identical control parameters may include performing an operation according to an image algorithm to obtain the plurality of identical control parameters.

[0071] When constructing a plurality of identical control parameters, it is not necessary to consider the image positions of the sub-images, and the same control parameters can be obtained by directly performing calculations using image algorithms.

[0072] In this embodiment, under the multi-core mode, the construction process of the multiple control parameters is different for different target image processing functions. For the default function, the target control parameters used for its realization are size-related control parameters, so two different size-related control parameters need to be constructed by the mapping algorithm to become the target control parameters. Correspondingly, the target ISP core performs image signal processing based on the different size-related control parameters. Other image processing functions other than the default function do not need to use size-related control parameters for their realization. Therefore, the multiple control parameters are the same, and there is no need to process by the mapping algorithm. The target ISP core performs image signal processing based on the same control parameters.

[0073] In some embodiments, the ISP command includes a target function to be called. The target function includes, but is not limited to, one or more of a function module open function (Open), a function module close function (Close), and a driving interface (ioctl) function. The Open function is used to open a function module, and the Close function is used to close a function module. A function module related to an image processing function is called by the ioctl function. The above target function may be an API (Application Programming Interface) function for calling an API in a driving layer. Exemplarily, the driving layer includes a hardware abstraction layer, ISPdrv and MIdrv, and the target function can be used to call an API in the hardware abstraction layer.

[0074] A function pointer is provided in the multi-core mode. The function pointer is a pointer variable of a function, and can point to and call its membership function. In this case, the step of constructing a plurality of control parameters under the multi-core mode in the above step S22 includes a step of calling the membership function pointed to by the function pointer provided under the multi-core mode, thereby constructing the initial control parameters as a plurality of control parameters to be target control parameters. The membership function corresponds to the target function.

[0075] Taking a dual-core device as an example, that is, the multi-core mode is the dual-core mode. As shown in FIG. 7, when the target function is an Ioctl function, that is, when transferring control parameters, the driving layer provides an API for realizing the image processing module, that is, the API BinderHalIoctl. An upper layer such as an interface layer can call the API BinderHalIoctl. Then, when it is determined that the processing mode is the single-core mode, the membership function GeneralIoctl(0) pointed to by the function pointer bindGeneralOps is called. The membership function GeneralIoctl(0) corresponds to the target function Ioctl function. Then, the initial control parameters are transferred to the halIoctl(0) function of the driving layer as target control parameters. When it is determined that the processing mode is the dual-core mode (TileMode), the membership function TileModeIoctl(0) pointed to by the function pointer bindTileModeOps provided in the multi-core mode is called, and the initial control parameters are constructed as two control parameters to be the target control parameters. The membership function TileModeIoctl(0) corresponds to the target function Ioctl function, and transmits two control parameters to the halIoctl(0) and halIoctl(1) functions of the actuation layer, respectively.

[0076] As shown in FIG. 8, when the target function is a Close function, that is, when the functional module needs to be closed, the driving layer provides an API for realizing the closing of the functional module, that is, the API BinderHalClose. An upper layer such as an interface layer can call the API BinderHalClose. Then, when it is determined that the processing mode is the single-core mode, the membership function GeneralClose(0) pointed to by the function pointer bindGeneralOps is called. The membership function GeneralClose(0) corresponds to the target function Close function. Then, the initial control parameters are transferred to the halclose(0) function of the driving layer as target control parameters. When it is determined that the processing mode is the dual-core mode (TileMode), the membership function TileModeClose(0) pointed to by the function pointer bindTileModeOps provided in the multi-core mode is called, and the initial control parameters are constructed as two control parameters to be the target control parameters. The membership function TileMode(0) corresponds to the target function Close function. Then, the two control parameters are transferred to the halclose(0) function and the halclose(1) function of the driving layer, respectively.

[0077] In this embodiment, the multi-core mode is provided with a function pointer that can point to different membership functions. In the multi-core mode, the membership function pointed to by the function pointer and corresponding to the target function can be called to construct multiple control parameters. In this way, desired target control parameters can be constructed according to the processing mode, and the target ISP core can perform image signal processing based on the corresponding target control parameters under different processing modes, thereby achieving the technical effect of supporting multiple processing modes.

[0078] In some embodiments, the ISP command includes an ISP core identification number, also called an ISP core ID, for uniquely identifying the ISP core. The image signal processing mode can also be determined by the calling format of the image processing control module to the ISP core identification number. When the image processing control module calls one ISP core identification number only once, it is determined that the processing mode is a single-core mode, and the target ISP core corresponds to the ISP core identification number, i.e., the target ISP core is the ISP core indicated by the ISP core identification number. When the image processing control module calls multiple ISP core identification numbers, it is determined that the processing mode is a multi-core mode, and the target ISP core is multiple, and the multiple target ISP cores correspond one-to-one to the multiple ISP core identification numbers, respectively, i.e., the ISP core indicated by each of the multiple ISP core identification numbers is the target ISP core. When the image processing control module calls one ISP core identification number multiple times, it is determined that the processing mode is a multi-core mode, and the image signal processing is performed by time division multiplexing of the target ISP core under the multi-core mode, and the target ISP core is the ISP core indicated by the ISP core identification number.

[0079] In this embodiment, the ISP command includes an ISP core identification number, so that the connection layer can determine whether the processing mode is a single-core mode or a multi-core mode according to the manner in which the ISP core identification number is called, and can determine that the ISP core indicated by the ISP core identification number is a target ISP core for image signal processing, thereby achieving the technical effect of supporting multiple processing modes.

[0080] The target ISP core generates an interrupt signal after completing the image signal processing, and the target ISP core sends the interrupt signal to the upper layer. In some embodiments, under the multi-core mode, the connection layer responds to the interrupt signal of the image signal processing corresponding to the target ISP core, and after receiving all the interrupt signals, synchronously merges all the interrupt signals and sends them to the interface layer.

[0081] If there are multiple target ISP cores, the connection layer responds to an interruption signal of image signal processing corresponding to at least one of the target ISP cores, and after receiving interruption signals corresponding to all the target ISP cores, synchronously merges all the interruption signals and transmits them to the interface layer.

[0082] If the target ISP core supports time division multiplexing, the connection layer responds to an interrupt signal of image signal processing corresponding to at least one multiple time segment of the target ISP core, and after receiving interrupt signals corresponding to all the multiple time segments, synchronously merges all the interrupt signals and sends them to the interface layer.

[0083] Illustratively, all the interrupt signals are merged into one interrupt signal and reported to the interface layer, informing the interface layer to take appropriate action.

[0084] Exemplarily, synchronization of all interrupt signals can be realized by an interrupt handling function provided by the connection layer: The interface layer can be provided with an interrupt handling function by the connection layer.

[0085] As mentioned above, according to different target image functions, the constructed control parameters may be the same or different. For example, the control parameter is the image width. If the control parameters are the same, image signal processing is performed on sub-images of the same size (resolution) by multiple target ISP cores or multiple multiple time segments of the target ISP core. In this way, the image signal processing of each sub-image is completed simultaneously or the difference in the completion time is within a predetermined time difference range. In this way, the connection layer can receive all interrupt signals simultaneously or within a predetermined time difference range.

[0086] When the multiple control parameters are different, the multiple target ISP cores perform image signal processing on sub-images with different sizes (resolutions). In this way, the image signal processing of different sub-images is completed at different times. In this way, the connection layer receives interrupt signals corresponding to each target ISP core or interrupt signals corresponding to each multiple time segment back and forth.

[0087] In this way, in order to realize synchronization of all interrupt signals, in some embodiments, the interrupt signal that came first may be stored in a queue. When there are multiple target ISP cores, a queue may be set for each target ISP core. In this case, when an interrupt signal corresponding to a certain target ISP core is received, the interrupt signal is stored in the queue corresponding to the target ISP core. When the target ISP core supports time division multiplexing, a queue may be set for each multiple time segment. In this case, when an interrupt signal corresponding to a certain multiple time segment is received, the interrupt signal is stored in the queue corresponding to the multiple time segment in which the interrupt signal is transmitted.

[0088] When interrupt signals are stored in all the queues, it is assumed that all interrupt signals have been received, and the interrupt signals in each queue are read, synchronously merged, and transmitted to the interface layer.

[0089] In some embodiments, the interruption of the image signal processing includes multiple types, including, but not limited to, a YUV interruption output from a sp1 (self path 1) path in a memory interface (MI), a YUV interruption output from a sp2 (self path 2) path, a Raw interruption, and a DMA interruption of a Direct Memory Access (DMA). Different types of interruptions correspond to different interrupt signals, that is, the interrupt signal also includes multiple types. Thus, each type of interrupt signal has a corresponding storage position in the queue. In other words, different types of interrupt signals are stored at different positions in the queue.

[0090] In this way, after receiving an interrupt signal, the target position of the interrupt signal in the queue can be determined according to the type of the interrupt signal, and the interrupt signal can be stored in the target position in the queue. If the interrupt signals are stored in the target positions in all queues, it is considered that the same type of interrupt signals corresponding to all target ISP cores or all multiple time segments of the target ISP cores are received, and the same type of interrupt signals in the target positions in each queue are read, synchronously merged, and sent to the interface layer.

[0091] Illustratively, a register in the interface layer is set.

[0092] In the image signal processing driving method according to the present application, a connection layer is provided in the driving architecture, so that in any processing mode, the connection layer can construct different target control parameters based on the initial control parameters. The driving layer transmits the target control parameters to the target ISP core, and drives the target ISP core to perform image signal processing accordingly. In this way, low resolution using a single core and high resolution using multiple cores can be realized under the same driving architecture, and multiple processing modes can be supported.

[0093] Based on the image signal processing driving method according to any of the above embodiments, the present application further provides a driving architecture that is mounted on an image signal processing module including multiple ISP cores or an ISP core supporting time division multiplexing, and is used to drive image signal processing. As shown in Figure 2, the driving architecture 200 includes an interface layer 210, a connection layer 220, and a driving layer 230, and the driving layer 230 is connected to the ISP cores.

[0094] The interface layer 210 receives image signal processing ISP commands from an upper application, and the ISP commands include initial control parameters and image size parameters.

[0095] The connection layer 220 determines a processing mode of an image signal based on the image size parameter, and determines target control parameters for different processing modes based on the initial control parameters, where the processing modes include a single-core mode or a multi-core mode.

[0096] The drive layer 230 sends the target control parameters to a target ISP core, thereby driving the target ISP core to perform image signal processing based on the target control parameters.

[0097] In some embodiments, specifically, the connection layer 220 determines the initial control parameters as the target control parameters under the single-core mode, and constructs multiple control parameters based on the initial control parameters as the target control parameters under the multi-core mode.

[0098] In some embodiments, the ISP command indicates a target image processing function to be realized, the target control parameters include size-related control parameters, the size-related control parameters include one or more of an image width, a parameter related to the image width, an image height, and a parameter related to the image height, and the target ISP core performs image signal processing to realize the target image processing function. Specifically, the connection layer 220 constructs a plurality of different control parameters as the target control parameters when the target image processing function is a default function, and constructs a plurality of the same control parameters as the target control parameters when the target image processing function is not the default function. The default function is realized by the target ISP core performing image signal processing based on the size-related control parameters.

[0099] Specifically, in some embodiments, the connection layer 220 computes an image algorithm based on a given image location to obtain a number of different control parameters.

[0100] In some embodiments, the ISP command includes a target function to be called, and specifically, the connectivity layer 220 calls a membership function pointed to by a function pointer provided in the multi-core mode, thereby constructing the initial control parameters into a plurality of control parameters as target control parameters, and the membership function corresponds to the target function.

[0101] In some embodiments, the objective function includes one or more of a function module open function, a function module close function, and a drive interface function.

[0102] In some embodiments, under multi-core mode, the connectivity layer 220 further responds to an interrupt signal of the image signal processing corresponding to the target ISP core, and after receiving all the interrupt signals, synchronously merges and sends all the interrupt signals to the interface layer.

[0103] In some embodiments, the connection layer 220 further stores the interrupt signal in a corresponding queue after receiving an interrupt signal corresponding to a target ISP core, and when the interrupt signals are stored in all queues, reads out the interrupt signals in each queue, synchronously merges them, and sends them to the interface layer.

[0104] In some embodiments, when the target ISP core is multiple, the all interrupt signals include interrupt signals corresponding to all the target ISP cores, and the corresponding queues are queues corresponding to the target ISP cores that send the interrupt signals.

[0105] If the target ISP core supports time division multiplexing, the all interrupt signals include interrupt signals sent by the target ISP core in all multiple time segments, and the corresponding queues are queues corresponding to the multiple time segments in which the interrupt signals are sent.

[0106] In some embodiments, the interrupt signal includes multiple types, and specifically, the connection layer 220 stores the interrupt signal in the queue at a target location corresponding to the target type based on the target type to which the interrupt signal belongs.

[0107] In some embodiments, the image size parameters include an image resolution.

[0108] The process of realizing the functions and effects of each module in the above device can be referred to the process of realizing the corresponding steps in the above method, so the description will be omitted here.

[0109] Based on the image signal processing driving method according to any of the above embodiments, the present application further provides an image processing control module. The image processing control module is a hardware module including circuits for realizing specific functions. The image processing control module is connected to a plurality of ISP cores or an ISP core supporting time division multiplexing, and drives the target ISP cores connected to perform image signal processing by executing the image signal processing driving method according to any of the above embodiments.

[0110] In some embodiments, the image processing control module may be an IP core.

[0111] Based on the image signal processing driving method according to any of the above embodiments, the present application further provides a schematic configuration diagram of an image signal processing module shown in FIG. 9. Optionally, as shown in the left diagram of FIG. 9, the image signal processing module includes an image processing control module and a plurality of ISP cores. The image processing control module is connected to the plurality of ISP cores. The image processing control module may call the plurality of ISP cores to realize image signal processing under multi-core mode, may call an ISP core multiple times and realize image signal processing under multi-core mode by time division multiplexing of the called ISP cores, or may call an ISP core only once to realize image signal processing under single-core mode.

[0112] Optionally, as shown in the right diagram of FIG. 9, the image signal processing module includes an image processing control module and one ISP core. The image processing control module is connected to an ISP core supporting time division multiplexing. The image processing control module may call the ISP core multiple times to realize image signal processing under a multi-core mode by time division multiplexing of the ISP core, or may call the ISP core only once to realize image signal processing under a single-core mode. The image processing control module drives a target ISP core connected to the ISP core to perform image signal processing by executing the image signal processing driving method according to any of the above embodiments.

[0113] In some embodiments, the image processing control module and the ISP core may be configured together as an IP core, i.e., the image signal processing module may be an IP core. Optionally, the IP core may be implemented in another hardware device, such as a chip.

[0114] In some embodiments, the image signal processing module may be another hardware device or module that includes an ISP core, such as a chip or electronic device for image signal processing.

[0115] The present application further provides a computer storage medium storing a computer program, which, when executed by a processor, performs the image signal processing driving method according to any of the above embodiments.

[0116] In some embodiments provided by the present application, the described apparatus and method may be realized in other manners. The above-described apparatus examples are merely illustrative, and for example, the flowcharts and block diagrams in the drawings show possible architectures, functions, and operations based on the apparatus, method, and computer program products according to the embodiments of the present application. Here, each block in the flowchart or block diagram may represent a module, program segment, or part of code including one or more executable instructions capable of implementing a certain logical function. In alternative implementations, the implementation of the functions described in the blocks may be different from the order shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or may be executed in reverse order depending on the required functions. This is determined by the required functions. In addition, each block in the block diagram and / or flowchart, and a combination of blocks in the block diagram and / or flowchart may be realized by a dedicated hardware-based system that executes a certain function or operation, or may be realized by a combination of dedicated hardware and computer instructions.

[0117] In addition, each functional module according to each embodiment of the present application may be integrated to form a single independent part, each module may exist independently, or two or more modules may be integrated to form a single independent part.

[0118] The functions can be realized in the form of software functional modules and stored in a computer-readable storage medium when sold or used as an independent product. From this understanding, the technical solution of the present application itself, or a part that can contribute to the prior art, or a part of the technical solution can be realized in the form of a software product. The computer software product is stored in a storage medium and includes a number of commands for a computer device (such as a personal computer, a server, or a network device) to execute all or part of the steps of the above method in each embodiment of the present application. The above storage medium includes various media that can store program code, such as a USB disk, a portable hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0119] The above description is merely an embodiment of the present application and does not limit the scope of protection of the present application. Those skilled in the art may have various modifications and changes to the present application. As long as it does not deviate from the spirit and principle of the present application, any modifications, equivalent replacements, improvements, etc., are within the scope of protection of the present application. In addition, since the same symbols indicate the same things in the drawings, when a definition is made in one drawing, it is not necessary to further define and interpret in other drawings.

[0120] The above description is merely a specific embodiment of the present application, and does not limit the scope of protection of the present application. Those skilled in the art may modify or replace the technical solutions within the technical scope disclosed in the present application, and these modifications or replacements are also included in the scope of protection of the present application. Therefore, the scope of protection of the present application is subject to the contents set forth in the claims.

[0121] In this specification, relational terms such as first and second are merely used to distinguish one entity or action from another entity or action, and do not necessarily require or imply any actual relationship or order between these entities or actions. Furthermore, terms such as "have" and "include" and any variations thereof are intended to cover non-exclusive inclusions. Thus, a process, method, article, or apparatus that includes a set of elements is not necessarily limited to those elements, and may include other elements that are not specified or inherent in those processes, methods, articles, or apparatus. Unless otherwise specified, an element limited by the expression "includes" does not exclude a situation in which the process, method, article, or apparatus that includes the element also includes other similar elements.

Claims

1. An image signal processing driving method, comprising: The image signal processing driving method is used in an image signal processing module including a plurality of ISP cores or an ISP core supporting time division multiplexing, the image signal processing module is equipped with a driving architecture including an interface layer, a connection layer and a driving layer, the driving layer is connected to the ISP cores, and the image signal processing driving method includes: the interface layer receiving an image signal processing ISP command including an initial control parameter and an image size parameter from an upper application; the connection layer determines a processing mode of an image signal based on the image size parameter, and determines target control parameters for different processing modes based on the initial control parameters; the driving layer transmits the target control parameters to a target ISP core, thereby driving the target ISP core to perform image signal processing based on the target control parameters; The processing mode includes a single-core mode or a multi-core mode.

2. An image signal processing driving method comprising:

2. the processing mode is a multi-core mode, The step of determining a target control parameter based on the initial control parameter includes: constructing a plurality of control parameters based on the initial control parameters to define the target control parameters.

2. The image signal processing and driving method according to claim 1.

3. The ISP instruction includes a target function to be called, and the step of constructing a plurality of control parameters based on the initial control parameters as the target control parameters includes: calling a membership function pointed to by a function pointer provided in the multi-core mode so as to construct the initial control parameters into a plurality of control parameters to become target control parameters; The membership function corresponds to the objective function.

3. The image signal processing and driving method according to claim 2.

4. The objective function includes one or more of a function module open function, a function module close function, and a driving interface function.

4. The image signal processing and driving method according to claim 3.

5. the ISP command indicates a target image processing function to be realized, the target control parameters include size-related control parameters, the size-related control parameters include one or more of an image width, a parameter related to the image width, an image height, and a parameter related to the image height, the target ISP core performs image signal processing to realize the target image processing function, and a step of constructing a plurality of control parameters based on the initial control parameters to become the target control parameters includes: if the target image processing function is a default function, constructing a plurality of different control parameters as the target control parameters; if the target image processing function is not the default function, constructing a plurality of identical control parameters as the target control parameters; The predetermined function is realized by the target ISP core performing image signal processing based on the size-related control parameters.

5. The image signal processing driving method according to claim 2, wherein the image signal processing driving method is a driving method for driving a pixel.

6. The step of constructing a plurality of different control parameters as the target control parameter includes: A step of setting a plurality of different control parameters calculated based on a predetermined image position as the target control parameters.

6. The image signal processing and driving method according to claim 5.

7. In a multi-core mode, the image signal processing driving method includes: The connection layer further includes a step of synchronously merging all the interrupt signals and sending them to the interface layer in response to an interrupt signal of the image signal processing corresponding to the target ISP core after receiving all the interrupt signals.

2. The image signal processing and driving method according to claim 1.

8. The image signal processing driving method includes: after receiving a corresponding interrupt signal from the target ISP core, storing the interrupt signal in a corresponding queue; After receiving all the interrupt signals, synchronously merging all the interrupt signals and sending them to the interface layer, When the interruption signals are stored in all the queues, the interruption signals in the queues are read out, synchronously merged, and transmitted to the interface layer.

8. The image signal processing and driving method according to claim 7.

9. When the target ISP core is a plurality of target ISP cores, the all interrupt signals include interrupt signals corresponding to all the target ISP cores, and the corresponding queues are queues corresponding to the target ISP cores that transmit the interrupt signals; If the target ISP core supports time division multiplexing, the all interrupt signals include interrupt signals sent by the target ISP core in all multiple time segments, and the corresponding queues are queues corresponding to the multiple time segments in which the interrupt signals are sent.

9. The image signal processing driving method according to claim 8.

10. The interrupt signal includes a plurality of types, and the step of storing the interrupt signal in a corresponding queue includes: storing the interrupt signal in the queue at a target location corresponding to the target type based on a target type to which the interrupt signal belongs; The fact that the interrupt signal is stored in all cues storing an interrupt signal at a target position in all of said queues; 9. The image signal processing driving method according to claim 8.

11. the processing mode is a single-core mode, The step of determining a target control parameter based on the initial control parameter includes: determining said initial control parameters as said target control parameters.

2. The image signal processing and driving method according to claim 1.

12. The image signal processing method according to claim 1, further comprising: a first ISP core for processing an image signal from the first ISP core to the second ISP core; 2. An image processing control module comprising:

13. 13. An image processing control module according to claim 12, comprising a plurality of ISP cores or an ISP core supporting time division multiplexing, the image processing control module and the plurality of ISP cores or the ISP core supporting time division multiplexing are connected to each other.

13. An image signal processing module comprising:

14. The image signal processing module is an IP core, chip or electronic device for image signal processing.

14. An image signal processing module according to claim 13.

15. A computer readable storage medium storing computer instructions, which, when executed by a processor, effect the steps of the image signal processing driving method according to any one of claims 1 to 11. A computer-readable storage medium comprising:

Images