Image processing apparatus and image processing circuit

Abstract

To provide an image processing apparatus which implements various kinds of image processing by a comparatively smaller-scale circuit while securing a certain degree of freedom and minimizes access to a memory.SOLUTION: An image processing unit of the image processing apparatus includes: a control unit which incorporates a program memory allowing for writing thereto from an external CPU and executes a program stored in the program memory; a plurality of execution circuits each of which performs operation of which the type is compliant with setting information, on input data and outputs an operation result; a plurality of internal memories for temporarily storing operation results of the execution circuits; a register unit including a plurality of registers in which information designating respective processing object input data sources for the execution circuits, information designating output destinations of operation result data of the execution circuits, and respective setting information for the execution circuits are held; and a selection circuit which performs processing of, in accordance with the setting information in the register unit, selecting image data in a memory or an internal memory as input data to an execution circuit and selecting a memory or an internal memory as the output destination of the operation result of the execution circuit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device and an image processing circuit. [Background technology]

[0002] Recent digital camera products, such as digital still cameras, digital video cameras, security cameras, and vehicle-mounted cameras, are equipped with image processing devices that realize complex image processing. Image processing devices are generally equipped with various image processing circuits, and the desired processing is realized by combining these circuits.

[0003] Conventionally, after an image processing device is completed, new requirements arise that cannot be met by the image processing device alone. For this reason, the processing results by the image processing device are temporarily written out to an external large-capacity memory such as a DRAM. Then, new processing to be added to the intermediate processing results is performed by a separate circuit or software. Then, for the image data that has been subjected to the additional processing, the image processing device is made to process again the parts that the image processing device can perform. However, writing out the intermediate processing results to the large-capacity memory causes problems such as consuming the capacity and access bandwidth of the large-capacity memory.

[0004] Several solutions to this problem have been proposed. For example, Patent Documents 1 and 2 propose a method of increasing the flexibility of image processing by using a programmable circuit called a DSP (Digital Signal Processor) as the image processing circuit, rather than a specific arithmetic processing circuit.

[0005] As another example, there is a method that uses programmable and highly flexible circuits such as FPGAs (Field Programmable Gate Arrays) and GPGPUs (General Purpose Graphic Processing Units). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2007-310721 A [Patent Document 2] Japanese Patent Application Publication No. 11-53526 Summary of the Invention [Problem to be solved by the invention]

[0007] However, in the conventional technologies disclosed in the above-mentioned Patent Documents 1 and 2, a DSP is used, and the processing speed of the DSP directly affects the processing speed of the entire image processing. For this reason, it is necessary to have a high-performance processor that can process several tens of pixels per clock. Alternatively, it is necessary to install many DSPs to compensate for the low processing power of each processor. A high-performance DSP has the problem that it becomes large-scale in itself. Also, when DSPs are installed in parallel, there is a problem that the scale increases in proportion to the number of DSPs installed.

[0008] FPGAs have a high degree of freedom in calculations and high processing speed. On the other hand, FPGAs have the problem that they require a circuit scale about 10 to 20 times larger than a dedicated image processing circuit. In addition, when using GPGPUs, there are problems similar to those when installing DSPs in parallel.

[0009] The present invention has been made in consideration of such problems, and aims to provide a technology that realizes various types of image processing using relatively small-scale circuits while ensuring a certain degree of freedom, and also reduces access to memory. [Means for solving the problem]

[0010] In order to solve this problem, for example, an image processing device according to the present invention has the following arrangement. An image processing device, The image processing device includes a CPU that controls the image processing device, an image processing unit that is connected to the CPU via a system bus and performs image processing under the control of the CPU, a plurality of bus masters, and a memory that is accessed by the plurality of bus masters and the image processing unit, The image processing unit includes: a control unit that includes a program memory that is writable by the CPU and controls the image processing unit by executing a program stored in the program memory; a plurality of execution circuits, each of which performs a type of operation on input data according to setting information and outputs an operation result; a plurality of internal memories for temporarily storing the results of the calculations of the execution circuit; a register unit including a plurality of registers for storing information specifying an input source of data to be processed by each of the execution circuits, information specifying an output destination of data resulting from the operation of each of the execution circuits, and the setting information for each of the execution circuits; a selection circuit that performs a process of selecting image data from one of the memory and the internal memory as input data to the execution circuit according to information set in the register unit, and a process of selecting one of the memory and the internal memory as an output destination of an operation result of the execution circuit, The control unit is By executing the program stored in the program memory, the setting information for the execution circuit, as well as the input source and output destination of the execution circuit, are set each time the execution circuit processes image data of a predetermined processing unit. Effect of the Invention

[0011] According to the present invention, it is possible to realize various types of image processing using a relatively small-scale circuit while ensuring a certain degree of freedom, and to minimize memory access. [Brief description of the drawings]

[0012] [Figure 1] FIG. 2 is a block diagram of a main part according to the embodiment. [Diagram 2] 4 is a flowchart of a control process executed by a CPU. [Diagram 3] 4 is a flowchart of a control process executed by a microcomputer of an image processing unit. [Figure 4] FIG. 2 is a circuit configuration diagram of a first type of image processing circuit according to the embodiment. [Diagram 5] FIG. 4 is a circuit configuration diagram of a second type of image processing circuit according to the embodiment. [Figure 6] FIG. 1 is a block diagram showing the configuration of an imaging apparatus to which an embodiment is applied. [Figure 7] FIG. 13 is an equivalent circuit diagram when the image processing unit functions as a smoothing filter. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0014] FIG. 6 is a block diagram showing the configuration of an image capturing apparatus 1000 to which the image processing apparatus of the embodiment is applied.

[0015] The photographing lens 601 takes in light from a subject to be photographed, and forms a subject image on the imaging surface of the image sensor 603 via the aperture 602. The aperture 602 adjusts the amount of light transmitted through the photographing lens. A large number of photoelectric conversion elements that generate an electric signal according to the intensity of the incident light are arranged in a matrix on the imaging surface of the image sensor 602. Therefore, the signal obtained by the photoelectric conversion elements arranged in a matrix represents an analog signal that represents the formed image. The image sensor 602 outputs the analog signal (analog image signal) obtained by each photoelectric conversion element to an AFE (Analog Front End) 604. The image sensor 602 can output, for example, an image with one frame of 4K (3840 pixels horizontal x 2160 pixels horizontal) or 8K (7680 pixels horizontal x 4320 pixels horizontal) at 60 frames per second (fps) or 120 fps. The AFE 604 converts the input analog image signal into digital image data. The AFE 604 can convert the image data into, for example, 8-bit digital data per pixel, but may also be configured to convert the image data into 10-bit or 12-bit digital data. The AFE 604 then outputs the image data obtained by the conversion to the image processing core 600. The AFE 604 is configured as a one-chip semiconductor integrated circuit (IC).

[0016] Next, a description will be given of the internal configuration of the image processing core 600 that processes image data of a subject. The image processing core 600 is configured as a one-chip semiconductor integrated circuit (IC) independent of the AFE 604.

[0017] The correction unit 605 performs correction processing on the image data received from the AFE 604 based on the inherent characteristics of the image sensor 603. The camera signal processing unit 608 performs development processing such as filtering and color conversion on the image data. The noise reduction unit 609 removes noise from the image data. The resizing unit 610 converts the image data into the number of pixels of a recorded image. The resizing unit 610 also performs conversion processing into the number of pixels suitable for the subject detection processing of the subject analysis unit 607.

[0018] The still image encoding unit 611 performs encoding processing to convert image data into an image format for recording still images, such as JPEG (Joint Photographic Experts Group). The moving image encoding unit 612 performs encoding processing to convert image data into an image format for recording moving images, such as H.264. The display I / F 613 is an interface unit for controlling image output to the display unit 618. The recording I / F unit 114 performs processing for recording image data, such as moving images and still images, onto a recording medium 619.

[0019] The subject analysis unit 607 detects a subject in the image data resized by the resizing unit 610 and performs processing to track the movement of the subject. Furthermore, the subject analysis unit 607 calculates parameters for AF (autofocus) control that adjusts the focus of the photographing lens 603 and AE (auto exposure) control that adjusts the amount of light of the aperture 602, and outputs them to the photographing control unit 615. The photographing control unit 615 controls the driving of the photographing lens 601, aperture 602, and image sensor 603 according to the parameters input from the subject analysis unit 607 and a trigger signal based on a user's operation instruction.

[0020] A CPU (Central Processing Unit) 100 controls the entire system. The CPU 100 controls the entire image processing core 600 based on a program stored in a built-in memory, and controls the setting and execution of operation parameters of each functional module, including the object analysis unit 607 and the camera signal processing unit 608 described above. The programs executed by the CPU 100 may be stored in an external memory instead of the built-in memory.

[0021] The system bus 121 is a bus that connects functional modules that act as bus masters, such as the correction unit 605, CPU 100, object analysis unit 607, camera signal processing unit 608, noise reduction unit 609, resizing unit 610, still image encoding unit 611, video encoding unit 612, display I / F 613, and recording I / F 614, to functional modules that act as bus slaves of the DRAM control unit 616.

[0022] The DRAM control unit 616 controls (arbitrates) the writing and reading of data to the DRAM 101 based on a write transaction or a read transaction issued by the above-mentioned bus master.

[0023] The image processing unit 102 performs processing for adding various effects to image data displayed on the display unit 618 or image data recorded on the recording medium 619. For example, the image processing unit 102 performs peaking processing for adding a predetermined color to a predetermined portion of the displayed image, such as a high-luminance portion or a portion likely to cause subject blur. The image processing unit 102 also performs processing for highlighting an in-focus area in the image by adding a predetermined color based on the analysis result from the subject analysis unit 607. The detailed configuration of the image processing unit 102 will be described later.

[0024] The DRAM (Dynamic Random Access Memory) 101 is a large-capacity volatile storage device connected to the image processing core 100. The DRAM 101 is a memory device conforming to a standard such as LPDDR4 SDRAM (Low Power Double Data Rate 4 Synchronous DRAM). The DRAM 101 is controlled by a predetermined command from the DRAM control unit 616, and exchanges data with the DRAM control unit 616. The DRAM 101 is also used to store temporary image data to be processed by the functional module serving as the bus master described above, or programs to be executed, etc.

[0025] The operation unit 617 includes operation members such as various switches, buttons, and a touch panel that can be operated by the user. The user issues instructions to the imaging device 1000 by operating the operation unit 617. The display unit 618 includes a display member such as a liquid crystal panel, and displays captured images, various menu screens, and the like. The recording medium 619 is a random-access recording medium such as a memory card.

[0026] Next, the image processing unit 102 will be described with reference to Fig. 1. Fig. 1 is a diagram showing the image processing unit 102 in the image processing core 600 and the configuration related to it.

[0027] The image processing unit 102 includes a plurality of image processing circuits 103-1, 103-2, . . . , 103-m, a plurality of memories 104-1, 104-2, .

[0028] Each of the image processing circuits 103-1, 103-2, ..., 103-m performs relatively simple arithmetic processing on an image. Each of the memories 104-1, 104-2, ..., 104-n is configured with a memory element having an access speed much faster than that of the DRAM 101. Hereinafter, unless otherwise specified, any one of the image processing circuits 103-1, 103-2, ..., 103-n will be simply referred to as the "image processing circuit 103." The same is true for the memories 104-1, 104-2, ..., 104-n, and any one of them will be simply referred to as the "memory 104."

[0029] The selection circuit 105 selects or specifies the input source of data to be processed by the image processing circuit 103 and selects or specifies the output destination of the image processing circuit 103 , thereby controlling the flow of image data within the image processing unit 102 .

[0030] Specifically, the selection circuit 105 selects whether to input data from the DRAM 101 or the memory 104 as an input source to the image processing circuit 103, and in the case of the memory 104, selects which of the memories 104-1 to 104-n to use as the input source. One image processing circuit has a maximum of two input sources. Furthermore, the selection circuit 105 selects whether to output processed data resulting from execution by the image processing circuit 103 to the DRAM 101 or the memory 104, and in the case of the memory 104, selects which of the memories 104-1 to 104-n to output to.

[0031] The internal register unit 106 is made up of a plurality of registers that hold setting values ​​related to the selection of the selection circuit 105, as well as information indicating the operation modes of the image processing circuits 103-1 to 103-m and various parameters.

[0032] The microcomputer 107 is a dedicated controller for the image processing unit 102, provided separately from the CPU 100. The microcomputer 107 has an internal program memory 107a. The CPU 100 can write a program into the program memory 107a in the microcomputer 107. Under the control of the CPU 100, the microcomputer 107 executes the program stored in the program memory 107a upon receiving an instruction from the CPU 100 to start execution. The microcomputer 107 then determines and writes a value to be set in the internal register unit 106. As a result, the image processing unit 102 can execute various types of image processing.

[0033] As described above, the CPU 100 controls each block in the image processing core 600. As described above, the DRAM 101 is a storage area for storing image data to be processed and supplied to the image processing unit 102, and image data after processing by the image processing unit 102. In addition, information other than image data, such as a control program for the CPU 100, may also be stored.

[0034] As described above, image processing unit 102 has a plurality of image processing circuits 103-1 to 103-m each performing a simple arithmetic process, a plurality of memories 104-1 to 104-n, selection circuit 105, internal register unit 106, and microcomputer 107. All of these components in image processing unit 102 are configured as hardware circuits. Image processing unit 102 uses these components to perform processing for adding various effects to images.

[0035] The image processing unit 102 in the embodiment divides the image data to be processed stored in the DRAM 101 into predetermined processing units smaller than one frame, and executes processing in these division units. For example, the division unit is one line of image data. This division unit may be another unit as long as it is equal to or smaller than the capacity of the memory 104. For example, one processing unit may be a predetermined number of pixels. For the sake of simplicity, the division unit in the embodiment is one horizontal line of image data. Furthermore, the input source of the image data to be processed does not have to be the DRAM 101, and the output of another image processing circuit may be received as the input image.

[0036] Image processing circuit 103 starts processing upon receiving a request to start execution from microcomputer 107. Image processing circuit 103 also has built-in calculators that perform the types of calculations set by internal register section 106, and counters that count the number of input pixels. When image processing circuit 103 counts the number of pixels set by internal register section 106 using this counter, it notifies microcomputer 107 that it has finished processing the target number of pixels. When image processing circuit 103 has made this notification, it also resets the count value of the counter, stops processing, and waits for an instruction from microcomputer 107 to start processing.

[0037] The memory 104 is a memory that can be accessed at a sufficiently high speed compared to the DRAM 101. The memory 104 is, for example, an SRAM (Static Random Access Memory). The memory 104 has a capacity for storing at least one image data unit for processing by the image processing circuit 103. The memory 104 is used as a memory for temporarily storing intermediate calculation results in image processing by the image processing unit 102. That is, the image processing unit 102 in the embodiment does not output intermediate calculation results during image processing to the external DRAM 101. As a result, at least the access to the DRAM 101 by the image processing unit 102 is limited to the initial read and the writing of the final processing result. Therefore, since the image processing unit 102 does not need to access the DRAM 101 to write intermediate calculation results during image processing, an increase in the bus bandwidth of the system bus 121 can be suppressed. In addition, the influence of the system bus 121 on the access of other bus masters can be suppressed.

[0038] Also, the memory 104 in the embodiment operates in either a write mode or a read mode according to an instruction from the microcomputer 107. The memory 104 has a counter for counting the number of pixels, similar to the image processing circuit 103. For example, in the write mode, the memory 104 starts inputting image data upon receiving a storage start instruction from the microcomputer 107, performs a storage process of the input data, and counts the number of input data by the counter. Then, when the counter value reaches a value set by the internal register unit 106, the memory 104 notifies the microcomputer 107 of the completion of storage, resets the counter value, and waits for a storage start instruction from the microcomputer 107. On the other hand, in the read mode, the memory 104 starts outputting the data stored in the memory upon receiving a read start instruction from the microcomputer 107, and counts the number of output data. When the count value of the counter reaches the value set by the internal register unit 106, the memory 104 notifies the microcomputer 107 of the completion of output, resets the count value of the counter, and waits for an instruction from the microcomputer 107 to start reading.

[0039] The internal register unit 106 is composed of a plurality of registers (storage elements) for storing values, parameters, etc., to be set in the image processing circuit 103, memory 104, and selection circuit 105. The value of the internal register 106 is controlled not to be changed while the image processing unit 102 is processing one line of data, which is a division unit. The value of the internal register unit 106 is changed to a value for processing the next line of data at the timing when the processing of one line of image data is completed. For this reason, the internal register unit 106 adopts a double bank configuration. That is, while the image processing unit 102 is processing one line according to the value set in the first bank of the internal register unit 106, the microcomputer 107 sets various parameters in the second bank for processing the next line. Then, when the image processing unit 102 finishes processing one line according to the value set in the first bank, the microcomputer 107 changes the value of the internal register 106 to the value already set in the second bank, and instantly switches to processing the next line and starts the processing. Naturally, while the image processor 102 is processing one line according to the values ​​set in the second bank, the microcomputer 107 sets various parameters in the first bank for processing the next line.

[0040] The microcomputer 107 has a dedicated program memory 107a, and based on a program written in the program memory 107a, performs settings in the internal register unit 106 and controls the image processing circuit 103 and memory 104. The CPU 100 writes the program in the program memory 107a of the microcomputer 107 and instructs the microcomputer 107 to start execution.

[0041] It takes a certain amount of time for the image processing circuit 103 to process image data for one division unit. For example, if the throughput of the image processing circuit 103 is one pixel per clock, it takes 2000 clocks to process image data of 2000 pixels. The microcomputer 107 only needs to have the performance to complete calculation and setting of the setting value for the next division unit in the internal register unit 106 within this time. If the microcomputer 107 has such performance, the processing speed of the microcomputer 107 will not cause a decrease in processing of the image processing unit 102. In other words, the microcomputer 107 only needs to calculate the setting value for the next division processing unit within the processing time of one division unit, and high performance is not required, which allows costs to be reduced accordingly.

[0042] Next, a specific example of the multiple image processing circuits 103-1 to 103-m included in the image processing unit 102 of the embodiment will be described. The image processing circuits 103-1 to 103-m in the embodiment have one of two types of circuit configurations. One is the circuit configuration of FIG. 4, and the other is the circuit configuration of FIG. 5. Hereinafter, the image processing circuit 103 having the configuration of FIG. 4 will be referred to as a first type image processing circuit, and the image processing circuit 103 having the configuration of FIG. 5 will be referred to as a second type image processing circuit. Note that for simplicity, counters are not shown in FIGS. 4 and 5.

[0043] 4 has two input terminals 501 and 502, one output terminal 503, a selector 504, and a calculator 505. The calculator 505 performs a calculation according to an opcode set in the internal register unit 106 on inputs A and B, and outputs the calculation result from output C. The types of calculations are the four arithmetic operations (+, -, ×, / ), logical operations (AND, OR, XOR), and bit shift.

[0044] The selector 504 selects either the data from the input terminal 502 or a value set by the internal register unit 106 in accordance with a selection signal from the internal register unit 106 , and supplies the selected value as input B to the arithmetic unit 505 .

[0045] In the above configuration, for example, assume that "addition" is set as the opcode for arithmetic unit 505, and selector 504 is set by a selection signal to select input terminal 502. Now, if input terminal 501 receives data of the i-th line of image data, and input terminal 502 receives data of the (i+1)-th line of image data, the first type image processing circuit in Fig. 4 functions as a circuit that outputs the result of adding two pixels aligned vertically.

[0046] Also, assume that "bit shift" is set as the opcode for the arithmetic unit 505, and the selector 504 is set to select a set value from the internal register unit 106. When performing a bit shift operation, the arithmetic unit 505 shifts the value of the input A to the left by the value of the input B if the value of the input B is positive, and shifts the value of the input A to the right by the value of the input B if the value of the input B is negative. Now, assume that data of a certain line of image data is input to the input terminal 501, and the selector 504 selects the set value "-1" set by the internal register unit 106. In this case, the arithmetic unit 505 functions as a circuit that outputs the result of shifting the value of the input A to the right by 1 bit in accordance with "-1" indicated by the input B (equivalent to calculating 1 / 2 the value of the input A). Other types of opcodes will be clear from the above, so their explanation will be omitted.

[0047] The second type image processing circuit shown in FIG. 5 is composed of one input terminal 501, one output terminal 503, delays 511 to 514, and a calculation unit 550. Each of the delays 511 to 514 latches input data in response to a clock signal. That is, the delays 511 to 514 hold data of four pixels arranged horizontally (in time series). Therefore, when combined with the current input data, data of five pixels consecutive in the horizontal direction can be processed. The calculation unit 550 is composed of calculators 551 to 554. Each of the calculators 551 to 554 performs calculation processing according to an opcode, similar to the calculator 505 of the first type image processing circuit. However, the calculators 551 to 554 of the second type image processing circuit only need to be of two types, addition and subtraction, and the circuit configuration of the calculator can be simplified accordingly. Furthermore, each of the calculators 551 to 554 can also perform calculations with input B invalid (or with the value of input B forcibly set to 0) according to an opcode.

[0048] For example, if an opcode for "addition" is set to the calculator 551 and opcodes for other calculators 552 to 554 are set to invalidate input B, the addition result of calculator 551 will be output as is from calculator 554. As a result, the second type image processing circuit functions as a circuit that outputs the addition result of two horizontally adjacent pixels.

[0049] It will be understood that, depending on the settings, it can also function as a circuit that outputs the sum of 3 to 5 pixels arranged in the horizontal direction. Note that there is no particular limit to the number of latches and arithmetic units in Figure 5, and it is sufficient to have a configuration according to the required number of horizontal pixels.

[0050] By combining several first and second type image processing circuits and the memory 104, it is possible to configure a smoothing filter having a size of, for example, 4×4 pixels.

[0051] 7 is an equivalent circuit diagram of the 4×4 pixel smoothing filter. In the figure, it should be understood that each of reference numerals 701 to 705 is the image processing circuit 103, and each of reference numerals 751 to 758 is an individual memory in the memory 104.

[0052] Adder 701 inputs data of the i-th line and data of the (i+1)-th line of an image stored in DRAM 101, adds pixels at the same horizontal position on each line, and stores the result in either memory 751 or 752 (memory 751 in the figure). Note that adder 701 alternately switches the output memory in cycle units.

[0053] The adder 702 inputs data of the (i+2)th line and data of the (i+3)th line of the image stored in the DRAM 101, adds pixels at the same horizontal position on each line, and stores the result in either memory 753 or 754 (memory 753 in the figure). The adder 702 alternates between output memories in cycle units.

[0054] The adder 703 inputs data from two memories (memories 752 and 754 in the illustrated example) that store the results of the arithmetic processing performed by the adders 701 and 702 in the previous cycle, and adds them together. The adder 703 then stores the result of the addition in either memory 755 or 756 (memory 755 in the illustrated example). The adder 703 also alternates between output memories on a cycle-by-cycle basis.

[0055] Horizontal summer 704 calculates the sum of four horizontally consecutive data from a memory (memory 756 in the illustrated case) that stores the calculation result of the previous cycle by adder 703. Then, horizontal summer 704 stores the sum result in either memory 757 or 758 (memory 757 in the illustrated case). Horizontal summer 704 also alternately switches the output memory in cycle units.

[0056] The divider 705 inputs data from a memory (memory 758 in the illustrated example) that stores the calculation result of the previous cycle by the horizontal adder 704, divides the data by 16, and outputs the result to the DRAM 101.

[0057] In the above, each of the adders 701, 702, and 703 is a first type image processing circuit. By making the selector 504 in the first type image processing circuit select the input terminal 502 and setting the opcode of "addition" to the arithmetic unit 505, the first type image processing circuit can function as a two-input, one-output adder.

[0058] The horizontal adder 704 is a second type image processing circuit. By setting an opcode of "addition" to the calculators 551 to 553 in the calculation unit 550 in the second type image processing circuit and setting the input B to the calculator 554 as invalid, the second type image processing circuit can function as an adder of four pieces of data that are continuous in the horizontal direction.

[0059] The divider 705 is a first type image processing circuit. By making the selector 504 in the first type image processing circuit select the set value ("-4") from the internal register unit 106 and setting the operation code of "bit shift" to the calculator 505, the first corresponding image processing circuit can function as a divider that shifts input data right by 4 bits (equivalent to dividing by "16"). Note that, by making the selector 504 in the first type image processing circuit select the set value ("16") from the internal register unit 106 and setting the operation code of "division" to the calculator 505, the first type image processing can also function as a divider. The divider 705 outputs the division result to the DRAM 101 as a smoothing processing result.

[0060] It is assumed that the transfer of image data from the DRAM 101 to the adders 701 and 702, and the transfer of data from the divider 705 to the DRAM 101 are performed using DMA (Direct Memory Access) transfer. Therefore, the image processing unit 102 may include a built-in DMAC (Direct Memory Access Controller).

[0061] 7 is switched between writing and reading when data processing for one line is completed. This switching also occurs at the timing of bank switching in the internal register unit 106 described above. The microcomputer 107 determines that data processing for one line has ended when, as described above, it receives a notification from the image processing circuit 103 (counter) in use in the current cycle that arithmetic processing for the number of pixels in the division unit has ended.

[0062] 7, it takes four cycles from when the first four lines of data of an image stored in DRAM 101 are supplied to adders 701 and 702 until divider 705 outputs the smoothing processing result, as shown in the figure. However, after the smoothing processing result for the first four lines of data of an image is output to DRAM 101, pipeline processing can be realized by parallel execution of adders 701 and 702, adder 703, horizontal adder 704, and divider 705. In other words, it can function as a four-input, one-output high-speed smoothing filter circuit.

[0063] 7, the settings for the internal register unit 106 are the same. Also, the settings for the internal register unit 106 are the same in the second and fourth cycles. In other words, after the microcomputer 107 sets the internal register unit 106 for the first two cycles, it is sufficient for the microcomputer 107 to perform bank switching every time processing for one line is completed. Also, during the period from the first cycle to the third cycle in which data for the first four lines of the image in the DRAM 101 is input to the adders 701 and 702, meaningless data is output from the divider 705, so the microcomputer 107 controls so as not to write data to the DRAM 101 during that period.

[0064] Furthermore, each time microcomputer 107 finishes processing one division unit according to the program stored in program memory 107a, it notifies CPU 100 of the end of processing of that division unit. Microcomputer 107 receives an execution instruction for the next processing unit from CPU 100 and starts the corresponding processing, and repeats this process.

[0065] The above is an example in which the image processing unit 102 functions as a smoothing filter of 4×4 pixel size, but it is clear that by combining the required number of first type image processing circuits and second type image processing circuits in the image processing circuit 103, and further the required number of memories 104, it is possible to make it function as other types of image processing units.

[0066] The features of the image processing unit 102 according to the embodiment have been described above. Next, a control method for the image processing device according to the embodiment will be described with reference to the flowchart of FIG.

[0067] 2 is a flowchart of a program executed by the CPU 100, and is an excerpt of a processing flow when using the image processing unit 102 of the embodiment. Processing other than that related to the use of the image processing unit 102 by the CPU 100 is not the main focus of this embodiment, so it will be omitted.

[0068] In S200, CPU 100 reads out a program (microcode) for performing the type of image processing targeted by image processing unit from an internal memory of CPU 100, and transfers it to program memory 107a of microcomputer 107 in image processing unit .

[0069] In S201, the CPU 100 instructs the microcomputer 107 of the image processing unit 102 to start execution. As a result, the microcomputer 107 starts executing the program stored in the program memory 107a, sets various information in the internal register unit 106, and starts processing one division unit (one cycle).

[0070] In S202, the CPU 100 waits for a notification from the microcomputer 107 that processing for one division unit has been completed. If the CPU 100 receives a notification of the completion of processing from the microcomputer 107, the CPU 100 advances the process to S203.

[0071] In S203, CPU 100 determines whether there is an unprocessed processing unit. If CPU 100 determines that there is an unprocessed processing unit, CPU 100 returns the process to S201. On the other hand, if CPU 100 determines that all processing units have been processed, CPU 100 ends this process.

[0072] 2, CPU 100 determines whether or not it has received a notification from microcomputer 107 indicating that processing of a division unit has been completed. However, CPU 100 may receive the notification of completion of processing of a division unit from microcomputer 107 as an interrupt signal. In this case, CPU 100 only needs to determine whether or not there is an unprocessed processing unit within the interrupt process, and the processing capacity of CPU 100 can be used more effectively for other tasks.

[0073] Next, the processing of the microcomputer 107 will be described with reference to the flowchart of Fig. 3. This processing is started when the CPU 100 stores a program to be executed by the microcomputer 107 in the program memory 107a and issues an execution start instruction to the microcomputer 107 (S201).

[0074] In S300, the microcomputer 107 sets various settings required for one cycle, that is, one division unit, in one bank of the internal register unit 106. The settings include an opcode for the image processing circuit 103 to be used, various parameters, the number of pieces of processing data (target value of the counter), information indicating where the image processing circuit 103 inputs the data to be processed, and where the processing result is to be output.

[0075] In S301, the microcomputer 107 requests all image processing circuits 103 to be used in the current cycle of processing to start processing.

[0076] In S305, the microcomputer 107 determines whether or not there are any unprocessed division units remaining in the image to be processed. If it is determined that there are any unprocessed division units remaining in the image to be processed, the microcomputer 107 advances the process to S302.

[0077] In S302, the microcomputer 107 sets a value for the next cycle of processing in another bank of the internal register unit 106.

[0078] In S303, the microcomputer 107 waits for the completion of the set one cycle of processing. Specifically, the microcomputer 107 waits for notifications of the completion of processing from all image processing circuits 103 that started processing in S301.

[0079] When the microcomputer 107 completes one cycle of processing, it notifies the CPU 100 that one cycle of processing has been completed, and changes the value of the internal register unit 106 to the value set in S302 (switches the bank). Then, the microcomputer 107 returns the process to S301 and waits for a start instruction from the CPU 100.

[0080] Thereafter, as long as there are unprocessed division units, the processes from S301 to S303 are repeated.

[0081] When the processing of the last division unit is started, the microcomputer 107 advances the process from S305 to S304. In S304, the microcomputer 107 waits for the processing of the last division unit to be completed. When the processing of the last division unit is completed, the processing of the entire image data is completed, and the process ends.

[0082] As described above, the image processing unit of this embodiment has a plurality of image processing circuits that perform relatively simple arithmetic processing, a plurality of internal memories that can be accessed faster than an external memory, a selection circuit for selecting a path through which image data passes, a register unit for setting parameters and operation codes related to the operations of the selection circuit and each image processing circuit, and a programmable microcomputer that writes data to the register unit. As a result, it is possible to function as various types of image processing circuits by combining the image processing circuits to be used, the parameters and operation codes to be set thereto, and the internal memory. Moreover, intermediate data generated in the process of image processing is stored in the internal memory, not in a memory external to the image processing unit. Therefore, it is possible to minimize the impact on the bandwidth of the external bus and to minimize the impact on other bus masters connected to the external bus.

[0083] Usually, an image processing device that employs programmable image processing has multiple image processes to be realized, and these are processed by one image processing device. In this embodiment, the image processes to be realized are realized by combining multiple image processing circuits, each of which performs a simple operation. Therefore, the image processes to be realized are broken down into small operations, and one image processing circuit 103 is responsible for each unit. Naturally, there are few cases in which the image processes to be realized can be neatly broken down into a set of identical operations, so there may be three or more variations of the image processing circuit 03 so that all the necessary operations can be implemented.

[0084] Furthermore, in the above embodiment, the image processing core 600 is configured as a one-chip IC, but the image processing unit 102 in the image processing core 600 may also be configured as a one-chip IC.

[0085] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0086] The disclosure of this specification includes the following image processing device and image processing circuit. (Item 1) An image processing device, The image processing device includes a CPU that controls the image processing device, an image processing unit that is connected to the CPU via a system bus and performs image processing under the control of the CPU, a plurality of bus masters, and a memory that is accessed by the plurality of bus masters and the image processing unit, The image processing unit includes: a control unit that includes a program memory that is writable by the CPU and controls the image processing unit by executing a program stored in the program memory; a plurality of execution circuits, each of which performs a type of operation on input data according to setting information and outputs an operation result; a plurality of internal memories for temporarily storing the results of the calculations of the execution circuit; a register unit including a plurality of registers for storing information specifying an input source of data to be processed by each of the execution circuits, information specifying an output destination of data resulting from the operation of each of the execution circuits, and the setting information for each of the execution circuits; a selection circuit that performs a process of selecting image data from one of the memory and the internal memory as input data to the execution circuit according to information set in the register unit, and a process of selecting one of the memory and the internal memory as an output destination of an operation result of the execution circuit, The control unit is By executing the program stored in the program memory, the setting information for the execution circuit and the input source and output destination of the execution circuit are set every time the execution circuit processes image data of a predetermined processing unit. 13. An image processing device comprising: (Item 2) The image processing device described in item 1, characterized in that the control unit sets the memory as an input source of the arithmetic circuit that initially processes image data of one of the processing units, and sets the internal memory as an output destination of intermediate arithmetic results until a desired arithmetic result is obtained for the data of one of the processing units. (Item 3) The image processing device according to item 2, characterized in that, when a desired calculation result is obtained for data of one of the processing units, the control unit sets the output destination of the calculation result data of the execution circuit to the memory. (Item 4) The setting information includes the type of calculation and parameters for the calculation. 4. The image processing device according to any one of items 1 to 3. (Item 5) 5. The image processing device according to any one of items 1 to 4, wherein the operations executed by the execution circuit include arithmetic operations, logical operations, and bit shifts. (Item 6) The execution circuit includes a first type execution circuit with two inputs and one output that outputs one calculation result for two pieces of data, and a second type execution circuit that calculates and outputs a set number of pieces of continuous time-series data. 6. The image processing device according to any one of items 1 to 5, (Item 7) the register unit is composed of two banks, each of which holds the information indicating the input source, the information indicating the output destination, and the setting information; While the execution circuit processes image data of one of the processing units in accordance with the information indicating the input source, the information indicating the output destination, and the setting information held in one of the two banks, the control unit sets the information indicating the input source, the information indicating the output destination, and the setting information for processing image data of the next processing unit in the other bank. 2. The image processing device according to item 1, (Item 8) The predetermined processing unit is one line of image data. 8. The image processing device according to any one of items 1 to 7, (Item 9) The access speed of the internal memory is faster than that of the memory; 9. The image processing device according to any one of items 1 to 8, wherein the internal memory has a capacity capable of storing image data of at least one of the processing units. (Item 10) 10. The image processing device according to any one of items 1 to 9, wherein the image processing unit is a one-chip IC. (Item 11) A one-chip image processing circuit that functions as an image processing unit and is connected to a system bus to which a CPU, a plurality of bus masters, and a memory that control an image processing device are connected, a control unit that includes a program memory that is writable by the CPU and controls the image processing unit by executing a program stored in the program memory; a plurality of execution circuits, each of which performs a type of operation on input data according to setting information and outputs an operation result; a plurality of internal memories for temporarily storing the results of the calculations of the execution circuit; a register unit including a plurality of registers for storing information specifying an input source of data to be processed by each of the execution circuits, information specifying an output destination of data resulting from the operation of each of the execution circuits, and the setting information for each of the execution circuits; a selection circuit that performs a process of selecting image data from one of the memory and the internal memory as input data to the execution circuit according to information set in the register unit, and a process of selecting one of the memory and the internal memory as an output destination of an operation result of the execution circuit, The control unit is By executing the program stored in the program memory, the setting information for the execution circuit and the input source and output destination of the execution circuit are set every time the execution circuit processes image data of a predetermined processing unit. 1. An image processing circuit comprising:

[0087] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0088] 100: CPU, 101: DRAM, 121: system bus, 102: image processing unit, 103-i to 103-m: image processing circuits, 104-1 to 104-n: memories, 105: selection circuit, 106: internal register unit, 107: microcomputer, 107a: program memory

Claims

1. An image processing apparatus, comprising: a CPU connected to a system bus and controlling the image processing apparatus; a plurality of bus masters connected to the system bus; an image processing unit connected to the system bus; a memory connected to the system bus and accessed by the CPU, the plurality of bus masters, and the image processing unit via the system bus; wherein the image processing unit includes: a control unit that incorporates a program memory into which the CPU can write data and controls the image processing unit by executing a program stored in the program memory; a plurality of arithmetic circuits each performing an arithmetic operation of a type according to setting information on input data and outputting an arithmetic result; a plurality of internal memories for temporarily storing the arithmetic results of the plurality of arithmetic circuits; a register unit including a plurality of registers that hold first information for designating an input source of data to be processed by each of the plurality of arithmetic circuits, second information for designating an output destination of data of the arithmetic result of each of the plurality of arithmetic circuits, and the setting information for each of the plurality of arithmetic circuits when set by the control unit; a selection circuit that performs a process of selecting, as input data for each of the plurality of arithmetic circuits, image data from one of the memory and the internal memory connected to the system bus according to the first information held in the register unit, and a process of selecting, as an output destination of the arithmetic result of each of the plurality of arithmetic circuits, one of the memory and the internal memory connected to the system bus according to the second information held in the register unit; wherein the control unit: sets the first information, the second information, and the setting information in the plurality of registers each time each of the plurality of arithmetic circuits processes image data of one predetermined processing unit by executing a program stored in the program memory. The image processing apparatus according to claim 1.

2. The plurality of registers are each composed of two banks that hold the first information, the second information, and the setting information. While the control unit is processing the image data of the one predetermined processing unit according to the first information used for processing the image data of the one predetermined processing unit, the second information used for processing the image data of the one predetermined processing unit, and the setting information used for processing the image data of the one predetermined processing unit, which are held in one of the two banks by the plurality of arithmetic circuits, the control unit sets the first information used for processing the image data of the next predetermined processing unit, the second information used for processing the image data of the next predetermined processing unit, and the setting information used for processing the image data of the next predetermined processing unit in the other bank of the two banks. The image processing apparatus according to claim 1, characterized in that.

3. The control unit sets the first information used for processing the image data of the one predetermined processing unit in the register so that the memory connected to the system bus is selected as one input source of the plurality of arithmetic circuits that first processes the image data of the one predetermined processing unit among the plurality of arithmetic circuits, and for the plurality of arithmetic circuits, sets the second information used for processing the image data of the one predetermined processing unit in the register so that one of the plurality of internal memories is selected as the output destination of the intermediate arithmetic result until the target arithmetic result is obtained for the image data of the one predetermined processing unit. The image processing apparatus according to claim 1, characterized in that.

4. The control unit sets the second information used for processing the image data of the one predetermined processing unit in the register so that the memory connected to the system bus is selected as the output destination of the arithmetic result data of the arithmetic circuit that outputs the target arithmetic result for the image data of the one predetermined processing unit among the plurality of arithmetic circuits. The image processing apparatus according to claim 3, characterized in that.

5. The setting information includes the type of operation and the parameters for the operation. The image processing apparatus according to any one of claims 1 to 4, characterized in that.

6. The operations executed by the arithmetic circuit include arithmetic operations, logical operations, and bit shifts. The image processing apparatus according to any one of claims 1 to 4, characterized in that.

7. The plurality of arithmetic circuits include a first type of arithmetic circuit with two inputs and one output that outputs one arithmetic result for two pieces of data, and a second type of arithmetic circuit that performs arithmetic operations on a predetermined number of consecutive time-series data and outputs the result. The image processing apparatus according to any one of claims 1 to 4, characterized in that.

8. The one predetermined processing unit is image data for one line. The image processing apparatus according to any one of claims 1 to 4, characterized in that.

9. The control unit is a dedicated microcomputer for the image processing unit, and is provided separately from the CPU. The image processing apparatus according to any one of claims 1 to 4, characterized in that.

10. The access speed of the internal memory is faster than that of the memory connected to the system bus. Each of the plurality of internal memories has a capacity capable of storing at least the image data of the one predetermined processing unit. The image processing apparatus according to any one of claims 1 to 4, characterized in that.

11. Each of the plurality of internal memories is a SRAM, and the memory connected to the system bus is a DRAM. The image processing apparatus according to claim 10, characterized in that.

12. The image processing unit is an IC on one chip. The image processing apparatus according to any one of claims 1 to 4, characterized in that.

13. The CPU writes a program according to the type of target image processing into the program memory. The image processing apparatus according to any one of claims 1 to 4, characterized in that.

14. A one-chip image processing circuit that functions as an image processing unit for connecting a CPU that controls an image processing apparatus, a plurality of bus masters, and a system bus to which a DRAM is connected. It incorporates a program memory that can be written by the CPU, and controls the image processing unit by executing the program stored in the program memory. A microcomputer. Each of them performs an arithmetic operation of a type according to setting information on the input data and outputs an arithmetic result. A plurality of arithmetic circuits. A plurality of internal memories for temporarily storing the arithmetic results of the plurality of arithmetic circuits. A register unit including a plurality of registers for holding, when set by the microcomputer, first information specifying an input source of data to be processed by each of the plurality of arithmetic circuits, second information specifying an output destination of data of arithmetic results of the plurality of arithmetic circuits, and the setting information for each of the plurality of arithmetic circuits. A selection circuit that performs a process of selecting, as input data for each of the plurality of arithmetic circuits, image data from one of the DRAM and the internal memory connected to the system bus according to the first information held in the register unit, and a process of selecting, as an output destination of arithmetic results of the plurality of arithmetic circuits, one of the DRAM and the internal memory connected to the system bus according to the second information held in the register unit. The microcomputer By executing the program stored in the program memory, each time each of the plurality of arithmetic circuits processes image data of one predetermined processing unit, sets the first information, the second information, and the setting information in the plurality of registers. An image processing circuit characterized by the above.

Images