Image processing device, imaging device, image processing method, and program

Abstract

To provide an image processing device, an imaging device, an image processing method, and a program that selectively use, as a memory for temporarily storing a target image to be subjected to geometric transformation processing, a memory having a storage capacity smaller than a data amount of the target image and a memory having a storage capacity equal to or greater than the data amount of the target image.SOLUTION: An image processing device includes: a signal processing unit that performs signal processing on a first image to generate a second image; a first temporary storage unit having a storage capacity smaller than a data amount of the second image; a second temporary storage unit having a storage capacity equal to or greater than the data amount of the second image; a second selection unit that selects the first temporary storage unit or the second temporary storage unit; and a geometric transformation unit that performs geometric transformation processing based on the second image stored in the temporary storage unit to successively generate a third image line by line. The second selection unit selects the first temporary storage unit when a rotation angle of the second image by the geometric transformation processing does not exceed a predetermined angle, and selects the second temporary storage unit when the rotation angle exceeds the predetermined angle.SELECTED DRAWING: Figure 1B

Description

[Technical field]

[0001] The present invention relates to an image processing device, an imaging device, an image processing method, and a program. [Background technology]

[0002] Digital cameras are known that perform geometric transformation of images for distortion correction, electronic image stabilization, rolling shutter distortion correction, etc. Known geometric transformation techniques include techniques that randomly access a large-capacity memory (e.g., DRAM). Patent Document 1 discloses a technique for improving the efficiency of access to a large-capacity memory in geometric transformation.

[0003] Incidentally, imaging devices have recently become more and more pixelated, and high-resolution video recording is becoming common. In addition, processes other than geometric transformation processing may also generate a large amount of access to the DRAM. This may lead to a shortage of the DRAM access bandwidth. Patent Document 2 discloses a technology for reducing access to the DRAM by equipping an imaging device with a high-speed memory (e.g., SRAM) dedicated to geometric transformation processing. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 4781229 [Patent Document 2] JP 2013-125401 A Summary of the Invention [Problem to be solved by the invention]

[0005] Even if an imaging device has multiple memories (e.g., SRAM and DRAM), not all of the memories necessarily have a capacity capable of storing the entire image to be processed. For example, since SRAM is more expensive than DRAM, it is conceivable to reduce the capacity of the SRAM mounted in the imaging device in order to reduce costs. If the memory cannot hold the entire image to be processed, the geometric transformation process may not be executed properly depending on the content of the geometric transformation process.

[0006] The present invention has been made in view of the above circumstances, and has an object to provide a technique that makes it possible to selectively use, as necessary, a memory having a storage capacity smaller than the amount of data of a target image and a memory having a storage capacity equal to or larger than the amount of data of the target image, as a memory for temporarily storing an image to be subjected to geometric transformation processing. [Means for solving the problem]

[0007] In order to solve the above problems, the present invention provides a signal processing device that generates a second image by performing signal processing on a first image; a first temporary storage means having a first storage capacity smaller than the data amount of the second image; a second temporary storage means having a second storage capacity equal to or greater than the data amount of the second image; a selection means for selecting the first temporary storage means or the second temporary storage means as a temporary storage means for temporarily storing the second image, and the signal processing means outputs the second image to the temporary storage means sequentially line by line; a geometric transformation unit that performs a geometric transformation process based on the second image stored in the temporary storage unit to sequentially generate a third image line by line; Equipped with The selection means is selects the first temporary storage means as the temporary storage means when a rotation angle of the second image by the geometric transformation process does not exceed a predetermined angle based on the first storage capacity and the data amount of the second image; When the rotation angle exceeds the predetermined angle, the second temporary storage means is selected as the temporary storage means; When the first temporary storage means is selected as the temporary storage means, the first temporary storage means sequentially stores the second image output from the signal processing means line by line so that the first storage capacity does not overflow, and sequentially discards portions of the second image that have been used as the geometric transformation process progresses. The present invention provides an image processing device comprising: Effect of the Invention

[0008] According to the present invention, it is possible to selectively use, as needed, a memory having a storage capacity smaller than the data volume of the target image and a memory having a storage capacity equal to or greater than the data volume of the target image as a memory for temporarily storing an image to be subjected to geometric transformation processing.

[0009] Other features and advantages of the present invention will become apparent from the accompanying drawings and the following detailed description of the preferred embodiments of the present invention. [Brief description of the drawings]

[0010] [Figure 1A] FIG. 1 is a block diagram showing a hardware configuration of an image processing device 100. [Figure 1B] 1 is a block diagram showing the functional configuration of an image processing apparatus 100 according to a first embodiment. [Figure 1C] 11A and 11B are diagrams for explaining examples of storage capacity and predetermined angles required for performing geometric transformation processing; [Figure 1D] 11A and 11B are diagrams for explaining examples of storage capacity and predetermined angles required for performing geometric transformation processing; [Diagram 2] 4A to 4C are diagrams for explaining the processing contents of a geometric transformation unit 156. [Diagram 3] FIG. 13 is a conceptual diagram of an input image to a first temporary storage unit 153. [Figure 4] FIG. 4 shows an example of the configuration of a first temporary storage unit 153. [Diagram 5]FIG. 11 is a block diagram showing the functional configuration of an image processing device 100 according to a second embodiment. [Figure 6] FIG. 4 shows an example of the configuration of a first temporary storage unit 506. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0012] [First embodiment] Hardware configuration of the image processing device 100 1A is a block diagram showing a hardware configuration of an image processing device 100. The image processing device 100 is configured to perform geometric transformation processing on an image. The image to be subjected to the geometric transformation processing is, for example, an image captured by an imaging device such as a digital camera.

[0013] In the image processing device 100, the CPU 101, the DRAM 102, the storage device 103, and the image processing circuit 104 are interconnected via a bus 105. The CPU 101 executes a program stored in the storage device 103 to perform overall control of the image processing device 100. The DRAM 102 is used as a work memory when the CPU 101 executes the program. The DRAM 102 is also used to temporarily store various data including an image to be processed by the image processing circuit 104. The storage device 103 stores the program executed by the CPU 101, the image processed by the image processing circuit 104, etc.

[0014] The image processing circuit 104 includes a signal processing circuit 141, a first selection circuit 142, an SRAM control circuit 143, an SRAM 144, a second selection circuit 145, and a geometric transformation circuit 146. The operation of the image processing circuit 104 will be described later with reference to FIG.

[0015] The image processing device 100 may include an image sensor (not shown). In this case, the image processing device 100 can also function as an image capturing device that captures the image to be processed.

[0016] Functional configuration of the image processing device 100 FIG. 1B is a block diagram showing a functional configuration of the image processing device 100. The function of the signal processing unit 151 is realized by the signal processing circuit 141 that operates under the control of the CPU 101. The signal processing unit 151 performs signal processing on an input image. The signal processing to be performed is not particularly limited, and includes, for example, peripheral light amount falloff correction, magnification chromatic aberration correction, de-Bayer processing, noise reduction processing, gamma correction, and sharpness correction. The input image is input to the signal processing unit 151 line by line from top to bottom. The input lines are temporarily held in an SRAM in the signal processing circuit 141. The signal processing unit 151 performs signal processing on the lines input in sequence, and outputs the processing results (processed lines) to the first selection unit 152 in sequence.

[0017] The function of the first selection unit 152 is realized by the first selection circuit 142 which operates under the control of the CPU 101. The first selection unit 152 selectively outputs the processing result of the signal processing unit 151 to the first temporary storage unit 153 or the second temporary storage unit 154. The selection criterion by the first selection unit 152 will be described later.

[0018] The function of the first temporary storage unit 153 is realized by the SRAM control circuit 143 and the SRAM 144, which operate under the control of the CPU 101. The first temporary storage unit 153 controls the temporary storage of the processing results of the signal processing unit 151 in the SRAM 144. The SRAM 144 has a capacity capable of holding only a part of the image to be processed. The capacity of the SRAM 144 is larger than the capacity of the SRAM in the geometric transformation circuit 146 and smaller than the capacity of the DRAM 102. In accordance with a read request from the geometric transformation unit 156, the first temporary storage unit 153 holds a part of the processing results of the signal processing unit 151 in the SRAM 144 for a predetermined period of time and outputs it to the geometric transformation unit 156 via the second selection unit 155.

[0019] The function of the second temporary storage unit 154 is realized by the CPU 101 and the DRAM 102. The second temporary storage unit 154 performs control to temporarily store the processing result of the signal processing unit 151 in the DRAM 102. The DRAM 102 has a capacity capable of holding the entire image to be processed.

[0020] The memory of the first temporary storage unit 153 is not limited to the SRAM 144. Any memory can be used as the memory of the first temporary storage unit 153 as long as it is independent of the memory of the second temporary storage unit 154 (for example, the DRAM 102) and has a storage capacity smaller than the data amount of the target image (second image) of the geometric transformation processing. The memory of the second temporary storage unit 154 is not limited to the DRAM 102. Any memory can be used as the memory of the second temporary storage unit 154 as long as it is independent of the memory of the first temporary storage unit 153 (for example, the SRAM 144) and has a storage capacity equal to or larger than the data amount of the target image (second image) of the geometric transformation processing. For example, the image processing device 100 may include a DRAM for the second temporary storage unit 154 in addition to the DRAM 102. In this case, the function of the second temporary storage unit 154 is realized by the CPU 101 and a DRAM separate from the DRAM 102.

[0021] For example, the memory of the first temporary storage unit 153 may be a dedicated memory (a memory dedicated to the geometric transformation process by the geometric transformation unit 156). The memory of the second temporary storage unit 154 may be a shared memory (a memory used not only for the geometric transformation process by the geometric transformation unit 156 but also for other processes). Furthermore, the memory of the first temporary storage unit 153 (e.g., SRAM 144) may be a memory faster than the memory of the second temporary storage unit 154 (e.g., DRAM 102).

[0022] The function of the second selection unit 155 is realized by the second selection circuit 145 which operates under the control of the CPU 101. The second selection unit 155 selects the first temporary storage unit 153 or the second temporary storage unit 154 based on the same selection criteria as the first selection unit 152. The second selection unit 155 connects the selected one of the first temporary storage unit 153 and the second temporary storage unit 154 to the geometric transformation unit 156.

[0023] The function of the geometric transformation unit 156 is realized by the geometric transformation circuit 146 that operates under the control of the CPU 101. The geometric transformation unit 156 performs a geometric transformation process on the processing result of the signal processing unit 151 that is input via the second selection unit 155.

[0024] Selection criteria used by the first selection unit 152 and the second selection unit 155 The first selection unit 152 and the second selection unit 155 select the first temporary storage unit 153 or the second temporary storage unit 154 depending on whether the capacity of the memory (SRAM 144) of the first temporary storage unit 153 is sufficient to perform the geometric transformation process. The storage capacity required to perform the geometric transformation process on a specific image is determined according to the rotation angle of the geometric transformation (the angle at which the image is rotated by the geometric transformation), and the larger the rotation angle, the larger the required storage capacity. Therefore, the first selection unit 152 and the second selection unit 155 select the first temporary storage unit 153 when the rotation angle of the geometric transformation does not exceed a predetermined angle, and select the second temporary storage unit 154 when the rotation angle of the geometric transformation exceeds the predetermined angle. The case where the geometric transformation does not involve rotation (i.e., the case where the rotation angle of the geometric transformation is 0) is included in the case where the rotation angle of the geometric transformation does not exceed the predetermined angle.

[0025] The predetermined angle used here is determined so as to satisfy the condition that the storage capacity required when the rotation angle of the geometric transformation is the predetermined angle does not exceed the memory capacity of the first temporary storage unit 153. As an example, the predetermined angle is determined so that the storage capacity required when the rotation angle of the geometric transformation is the predetermined angle matches the memory capacity of the first temporary storage unit 153. In this case, the first temporary storage unit 153 is selected as long as the geometric transformation process can be performed. Unlike the memory of the second temporary storage unit 154 (e.g., DRAM 102), the memory of the first temporary storage unit 153 (e.g., SRAM 144) has a dedicated access band, so that when the first temporary storage unit 153 is selected, the geometric transformation unit 156 can perform the geometric transformation process at high speed. Therefore, adopting a configuration in which the first temporary storage unit 153 is selected as long as the geometric transformation process can be performed has the advantage of improving the possibility of performing the geometric transformation process at high speed.

[0026] An example of the storage capacity required to perform the geometric transformation process and a predetermined angle will be described with reference to Figures 1C and 1D. In Figure 1C, input image 181 is an image that is the target of the geometric transformation process, and output image 182 is an output image of geometric transformation section 156 (an image generated as a result of the geometric transformation process). Figure 1C shows how input image 181 is tilted, and a horizontal output image 182 is generated by performing geometric transformation to rotate input image 181.

[0027] The geometric transformation unit 156 generates and outputs the output image 182, which is the result of the geometric transformation process, line by line, starting from the top line of the output image 182 downward. In order for the geometric transformation unit 156 to output line 184, the geometric transformation unit 156 needs to use (refer to) line 183 of the input image 181.

[0028] As described above, the signal processing unit 151 sequentially performs signal processing on the sequentially input lines, and sequentially outputs the processing results (processed lines) to the first selection unit 152. Since each line is output only once, when the first selection unit 152 selects the first temporary storage unit 153, the first temporary storage unit 153 needs to continue to hold the data of each input line as long as there is a possibility that the data will be referred to for geometric transformation.

[0029] For example, in order for the geometric transformation unit 156 to output the line 184 of the output image 182 as shown in FIG. 1C, the geometric transformation unit 156 needs to refer to the line 183 of the input image 181. In this case, as shown by a plurality of rectangles in FIG. 1D, the geometric transformation unit 156 needs to hold data corresponding to a plurality of partial regions 185. Here, among the plurality of partial regions 185, those that do not overlap with the line 183 are not referred to when outputting the line 184. However, the partial region 185 that does not overlap with the line 183 is also referred to when the geometric transformation unit 156 later outputs a line below the line 184 of the output image 182. Therefore, during the processing of the line 184, the first temporary storage unit 153 needs to continue to hold all of the data corresponding to the plurality of partial regions 185 shown in FIG. 1D, including those that do not overlap with the line 183.

[0030] The range of the input image 181 corresponding to the plurality of partial regions 185 to be held by the first temporary storage unit 153 is determined according to the rotation angle of the geometric transformation. The data amount of the input image is determined according to the data format (number of pixels, bit depth, etc.) of the input image, and the data amount of the input image corresponding to the plurality of partial regions 185 determined according to the rotation angle of the geometric transformation is determined according to the data amount of the input image. Therefore, the storage capacity required to perform the geometric transformation process is determined from the rotation angle and the data amount of the input image. Based on the storage capacity determined in this way and the memory capacity of the first temporary storage unit 153, the predetermined angle can be determined so as to satisfy the condition that the storage capacity required when the rotation angle of the geometric transformation is a predetermined angle does not exceed the memory capacity of the first temporary storage unit 153. Therefore, the predetermined angle is based on the memory capacity of the first temporary storage unit 153 and the data amount of the input image.

[0031] In the example of FIG. 1D, the first temporary storage unit 153 manages data in units of partial regions 185. By adopting such a configuration, the process of holding and discarding data is simplified, and the structure of the SRAM control circuit 143 can be simplified. However, this embodiment is not limited to the configuration of managing data in units of partial regions 185. For example, the first temporary storage unit 153 may manage data in units of pixels of the input image 181.

[0032] In the above, with reference to Figs. 1C and 1D, it has been described that when the first selection unit 152 selects the first temporary storage unit 153, the data of each input line must be held as long as there is a possibility that the data will be referred to for the geometric transformation. This is also true when the first selection unit 152 selects the second temporary storage unit 154. However, since the memory of the second temporary storage unit 154 has a capacity capable of holding the entire image to be processed, the second temporary storage unit 154 can hold all necessary data even if the rotation angle of the geometric transformation is large. Therefore, when the rotation angle exceeds a predetermined angle, the second temporary storage unit 154 can be selected to perform the geometric transformation process.

[0033] ●Control to limit rotation angle As described above, when the rotation angle of the geometric transformation exceeds a predetermined angle, the memory capacity of the first temporary storage unit 153 may be insufficient, so the first selection unit 152 and the second selection unit 155 select the second temporary storage unit 154. Conversely, if the rotation angle is limited to be equal to or less than a predetermined angle, it is possible to prevent the memory capacity of the first temporary storage unit 153 from being insufficient, and it becomes possible to select the first temporary storage unit 153 having a memory with a dedicated access bandwidth. As a result, it becomes possible to perform the geometric transformation process at high speed, and to prevent the geometric transformation process from consuming the shared access bandwidth of the DRAM 102.

[0034] Therefore, when a predetermined condition is satisfied, CPU 101 performs control to limit the rotation angle of the geometric transformation so as not to exceed a predetermined angle, and causes first selection unit 152 and second selection unit 155 to select first temporary storage unit 153. Examples of cases in which a predetermined condition is satisfied include cases in which the geometric transformation process needs to be performed at high speed, and cases in which consumption of the access bandwidth of DRAM 102 needs to be suppressed.

[0035] For example, consider a case where the geometric transformation unit 156 performs geometric transformation for electronic image stabilization of a captured image. When a moving image is captured at a relatively high data rate (first data rate) such as 8K60P, the geometric transformation process needs to be performed at high speed. In this case, since the amount of access bandwidth of the DRAM 102 consumed by the moving image capture is large, the access bandwidth of the DRAM 102 may become congested if the access bandwidth of the DRAM 102 is used for the geometric transformation process. Therefore, when a moving image is captured in the 8K60P moving image recording mode (when the target image for the geometric transformation process is a frame of a moving image input to the image processing device 100 at a relatively high data rate), the CPU 101 performs control to limit the rotation angle of the geometric transformation so as not to exceed a predetermined angle. As a result, the CPU 101 causes the first selection unit 152 and the second selection unit 155 to select the first temporary storage unit 153. Conversely, when video shooting is performed in a 4K60P video recording mode with a relatively low data rate (when the target image for geometric transformation processing is a frame of a video input to the image processing device 100 at a relatively low data rate), the CPU 101 does not perform control to limit the rotation angle of the geometric transformation so as not to exceed a predetermined angle. As a result, the CPU 101 allows the first selection unit 152 and the second selection unit 155 to select the second temporary storage unit 154.

[0036] In addition, when multiple input images need to be processed continuously, such as in the case of video shooting, the first selection unit 152 and the second selection unit 155 may perform selection based on the above-mentioned selection criteria for each input image. For example, when video shooting is performed in a 4K60P video recording mode (when the rotation angle is not limited), the first selection unit 152 and the second selection unit 155 may select the first temporary storage unit 153 or the second temporary storage unit 154 for each input image based on whether the rotation angle exceeds a predetermined angle. Alternatively, the first selection unit 152 and the second selection unit 155 may perform selection with one entire video shooting as one selection unit. In this case, since there is a possibility that the rotation angle exceeds the predetermined angle, the first temporary storage unit 153 or the second temporary storage unit 154 selects the second temporary storage unit 154 for all input images.

[0037] Details of processing by the geometric transformation unit 156 The geometric transformation unit 156 requests the first temporary storage unit 153 or the second temporary storage unit 154 to read out an area necessary for generating division units obtained by dividing the output image of the geometric transformation unit 156 into predetermined small areas. The geometric transformation unit 156 collectively issues a predetermined number of requests for a predetermined number of division units without waiting for reception of partial images corresponding to requests that have already been issued. The first temporary storage unit 153 or the second temporary storage unit 154 determines an image range to be temporarily held based on the multiple read requests issued collectively, and receives and stores the image range to be held from the processing result of the signal processing unit 151.

[0038] In this embodiment, necessary area information is transmitted from the geometric transformation unit 156 provided at the rear stage of the data flow to the first temporary storage unit 153 or the second temporary storage unit 154 at the front stage. Then, a partial image cut out based on the necessary area information is transmitted from the first temporary storage unit 153 or the second temporary storage unit 154 to the geometric transformation unit 156, thereby realizing the necessary geometric transformation processing.

[0039] The processing contents of the geometric transformation unit 156 will be described with reference to Fig. 2. An output image 200 is an image output from the geometric transformation unit 156. The geometric transformation unit 156 divides this output image 200 into a grid pattern, and sequentially processes each divided area line by line.

[0040] At the timing when the input position of the divided region 201 of the output image is obtained, the geometric transformation unit 156 performs a transformation from output coordinates to input coordinates using a coordinate transformation circuit or coordinate transformation software. As a result of the transformation, a necessary input region 203 is obtained. That is, in order to generate an image of the divided region 201 of the output image, it is necessary to read the necessary input region 203. Since the shape of the necessary input region 203 may become complicated depending on the contents of the coordinate transformation, the geometric transformation unit 156 selects an input rectangular region 204 that includes the input region 203. The geometric transformation unit 156 requests this input rectangular region 204 from the first temporary storage unit 153 or the second temporary storage unit 154.

[0041] Regardless of whether a partial image corresponding to the input rectangular area 204 specified by the request is returned from the first temporary storage unit 153 or the second temporary storage unit 154, the geometric transformation unit 156 sequentially proceeds with the calculation of the input rectangular area of ​​the next divided area, and issues a predetermined number of requests to the first temporary storage unit 153 or the second temporary storage unit 154. The coordinate transformation circuit or coordinate transformation software supports various geometric transformation processes. The type of geometric transformation to be supported can be determined depending on the image processing to be performed. In general, image processing includes distortion aberration correction, rolling shutter distortion removal, electronic image stabilization processing, equirectangular transformation, perspective projection transformation, rotation correction according to the orientation of the display panel, pixel aspect ratio transformation, and electronic zoom.

[0042] Details of Processing by the Second Temporary Storage Unit 154 As described above, the second temporary storage unit 154 includes a memory having a capacity capable of holding the entire image to be processed. This memory is, for example, the DRAM 102, but is not limited to this.

[0043] The second temporary storage unit 154 reads out an image corresponding to the input rectangular area 204 requested by the geometric transformation unit 156 from the DRAM 102 via a memory controller (not shown) or the like. The second temporary storage unit 154 returns the read result to the geometric transformation unit 156. The read result is temporarily stored in the SRAM of the geometric transformation circuit 146 that realizes the function of the geometric transformation unit 156. When the second temporary storage unit 154 is selected by the first selection unit 152 and the second selection unit 155, the geometric transformation process can be realized by moving data from the DRAM 102 to the SRAM of the geometric transformation circuit 146 in a manner similar to the method described in Patent Document 1.

[0044] Details of Processing by First Temporary Storage Unit 153 The details of the processing by the first temporary storage unit 153 will be described with reference to Fig. 3 and Fig. 4. Fig. 3 is a conceptual diagram of an input image, and Fig. 4 is a diagram showing an example of the configuration of the first temporary storage unit 153. As described above, the function of the first temporary storage unit 153 is realized by the SRAM control circuit 143 and the SRAM 144 which operate under the control of the CPU 101, but in Fig. 4, the description corresponding to the SRAM 144 is omitted.

[0045] The first temporary storage unit 153 divides the input image, which is the processing result of the signal processing unit 151, into a plurality of rectangular regions 301 (rectangular regions 301A to 301D in the example of FIG. 3), and stores the divided images in the SRAM 144. In this specification, when there is no need to strictly distinguish between the rectangular regions 301A to 301D, they are collectively referred to as rectangular regions 301. It is not necessary for all rectangular regions 301 to have the same width. The SRAM 144 includes a plurality of storage elements each having a capacity sufficient to store one line of the rectangular region 301 with the widest width. Hereinafter, the storage element for one line is referred to as a line memory.

[0046] As shown in Fig. 4, each strip area 301 has a corresponding strip memory management section 400. In the example of Fig. 4, strip memory management section 400A corresponds to strip area 301A, strip memory management section 400B corresponds to strip area 301B, strip memory management section 400C corresponds to strip area 301C, and strip memory management section 400D corresponds to strip area 301D. In this specification, when there is no need to strictly distinguish between each of the strip memory management sections 400A to 400D, the term strip memory management section 400 is used as a general term.

[0047] First, each strip memory management unit 400 initializes the vertical range to be held next to "none". A request for an input rectangular area 204 from the geometric transformation unit 156 is input from an input area request 407. The request for the input rectangular area 204 is distributed to all strip memory management units 400. The request for the input rectangular area 204 is also stored in a FIFO unit 405 separately from the strip memory management units 400.

[0048] When a rectangular area indicated by the request for the input rectangular area 204 overlaps with the corresponding rectangular area 301, each strip memory management unit 400 recognizes it as the input rectangular area 204 to be managed. Each strip memory management unit 400 expands the vertical range to be held next to include the vertical range from the top to the bottom of the rectangular area indicated by the request for the input rectangular area 204. For example, when a request for an upper end of 100 and a lower end of 110 is input, the range to be held is changed from 100 to 110. Next, when a request for an upper end of 105 and a lower end of 115 is input, the range to be held is expanded from 100 to 115. Each strip memory management unit 400 repeats such an expansion process of the range to be held a predetermined number of times to determine the vertical range to be held next. This predetermined number is the number of rectangular areas arranged in a line so as to cover at least one rectangular area vertically and from the left end to the right end of the output image horizontally.

[0049] After the vertical range of each strip memory management section 400 is determined, each strip memory management section 400 aligns the bottom end of the vertical range to be held next to the lowest position among strip memory management sections 400A to 400D. For example, consider a case where strip memory management section 400A's vertical range to be held next is from 100 to 110, strip memory management section 400B's vertical range to be held next is from 90 to 100, strip memory management section 400C's vertical range to be held next is from 95 to 105, and strip memory management section 400D's vertical range to be held next is from 102 to 112. In this case, the lowest lower end is 112 for strip memory management section 400D, so all strip memory management sections 400 change their bottom ends to 112.

[0050] During output processing by geometric transformation unit 156, if the output of the processing result from signal processing unit 151 reaches the lower end of the vertical range to be held, first temporary storage unit 153 issues a temporary suspension request to signal processing unit 151. Conversely, if the pixel value requested by geometric transformation unit 156 is not yet held in first temporary storage unit 153, first temporary storage unit 153 will not output the pixel value to geometric transformation unit 156 until that pixel value is input from signal processing unit 151.

[0051] When the bottom ends of the vertical ranges to be held for all the strip memory management units 400 are aligned, the output unit 406 starts to operate. The output unit 406 receives a request for the input rectangular area 204 previously input from the FIFO unit 405. The output unit 406 starts to output the output image 408 based on the information indicated by the request. The output image 408 is a pixel value obtained by the output unit 406 inquiring of the strip memory management unit 400 corresponding to the request for the input rectangular area 204 and receiving the necessary pixels. The pixel value output from the first temporary storage unit 153 is temporarily stored in the SRAM of the geometric transformation circuit 146 that realizes the function of the geometric transformation unit 156.

[0052] For example, since the input rectangular region 302A is in the strip region 301A of the input image, the output unit 406 requests pixel values ​​from the strip memory management unit 400A corresponding to the strip region 301A of the input image, and outputs the obtained pixel values. The same applies to the input rectangular region 302B and the input rectangular region 302C. Since the input rectangular region 302D straddles the strip region 301A of the input image and the strip region 301B of the input image, the output unit 406 requests pixel values ​​from both the strip memory management unit 400A and the strip memory management unit 400B. In this specification, when it is not necessary to strictly distinguish between the input rectangular regions 302A to 302D, the input rectangular regions 302A to 302D are collectively referred to as the input rectangular region 302. Note that in FIG. 3, the input rectangular region 302 is illustrated as being close to a parallelogram, but it may be a shape with four corners at angles of 90 degrees, similar to the input rectangular region 204.

[0053] If the pixel requested by the output unit 406 has not yet been stored in the line memory, the strip memory management unit 400 suspends response to the output unit 406 until the pixel is stored. The strip memory management unit 400 continues to receive the image input from the input image 409 until the image reaches at least the lower end of the vertical range to be held. The strip memory management unit 400 may receive the input image up to a line that is a predetermined number of lines below the lower end. If the image input from the input image 409 is within the strip area corresponding to the strip memory management unit 400 and within the vertical range to be held, the strip memory management unit 400 stores the input image in the line memory. If the line memory corresponding to the line number of the image input from the input image 409 has already been secured, the strip memory management unit 400 writes the image into the line memory. If the line memory has not been secured, the strip memory management unit 400 issues a request to the line memory management unit 404, and defines the secured line memory as a line memory associated with the line number and uses it.

[0054] During this output process, the strip memory management unit 400 obtains the next "vertical range to be held next" in the same manner as described above. When the output unit 406 completes both the process of reading a predetermined number of requests from the FIFO unit 405 and outputting pixel values ​​and the input of the input area request 407 reaches a predetermined number, the first temporary storage unit 153 moves to the next operation. When the first temporary storage unit 153 moves to the next operation, all the strip memory management units 400 update the "vertical range to be held next" to the newly obtained range. During this update, each strip memory management unit 400 returns the line memory corresponding to the unnecessary upper line to the line memory management unit 404. In this way, the returned line memory is reused as the next lower line memory. The first temporary storage unit 153 continues to update the range to be held while retaining a part of the image (sequentially updating), and can output the image of the required range to the geometric transformation unit 156.

[0055] Summary of the first embodiment As described above, according to the first embodiment, the signal processing unit 151 performs signal processing on the target image (first image) of the geometric transformation processing. The first temporary storage unit 153 has a storage capacity smaller than the data amount of the processing result (second image) of the signal processing unit 151. The storage capacity of the first temporary storage unit 153 is configured, for example, by the SRAM 144. The second temporary storage unit 154 has a storage capacity equal to or larger than the data amount of the processing result (second image) of the signal processing unit 151. The storage capacity of the second temporary storage unit 154 is configured, for example, by the DRAM 102. The first selection unit 152 and the second selection unit 155 select the first temporary storage unit 153 or the second temporary storage unit 154 as the temporary storage unit that temporarily stores the processing result (second image) of the signal processing unit 151. The signal processing unit 151 outputs the processed beriberi (second image) line by line in sequence to the selected temporary storage unit. The geometric transformation unit 156 performs geometric transformation processing based on the processing result (second image) stored in the selected temporary storage unit, thereby sequentially generating an output image (third image) line by line.

[0056] If the rotation angle of the geometric transformation process does not exceed a predetermined angle based on the storage capacity of first temporary storage unit 153 and the data amount of the processing result (second image) of signal processing unit 151, first selection unit 152 and second selection unit 155 select first temporary storage unit 153. If the rotation angle of the geometric transformation process exceeds the predetermined angle, first selection unit 152 and second selection unit 155 select second temporary storage unit 154 as the temporary storage unit.

[0057] When the first temporary memory unit 153 is selected by the first selection unit 152 and the second selection unit 155, the first temporary memory unit 153 sequentially stores the processing result (second image) output from the signal processing unit 151 line by line so as to prevent the storage capacity from overflowing, and sequentially discards parts of the processing result (second image) that have been used as the geometric transformation processing progresses.

[0058] Therefore, according to the first embodiment, it is possible to selectively use, as necessary, a memory having a storage capacity smaller than the data amount of the target image (e.g., SRAM 144) and a memory having a storage capacity equal to or larger than the data amount of the target image as a memory for temporarily storing the target image for geometric transformation processing.

[0059] [Second embodiment] In the second embodiment, a configuration for realizing faster processing than in the first embodiment will be described. In this embodiment, the basic configuration of the image processing device 100 is similar to that of the first embodiment. Below, differences from the first embodiment will be mainly described.

[0060] Functional configuration of the image processing device 100 FIG. 5 is a block diagram showing a functional configuration of an image processing device 100 according to a second embodiment. In the second embodiment, there are a plurality of signal processing units 501 (in the example of FIG. 5, signal processing units 501A to 501D) and a plurality of geometric transformation units 509 (in the example of FIG. 5, geometric transformation units 509A to 509D). In this specification, when it is not necessary to strictly distinguish between the signal processing units 501A to 501D, the term "signal processing unit 501" is used as a general term. Also, when it is not necessary to strictly distinguish between the geometric transformation units 509A to 509D, the term "geometric transformation unit 509" is used as a general term. The number of signal processing units 501 and the number of geometric transformation units 509 do not necessarily need to be the same. The number of signal processing units 501 and the number of geometric transformation units 509 may be determined according to the processing performance requirements of the system and the processing performance of the signal processing units 501 and the geometric transformation units 509.

[0061] The configuration of each signal processing unit 501 is similar to that of the signal processing unit 151. However, in the second embodiment, an input image is divided into the number of signal processing units 501, and each divided input image is input to the corresponding signal processing unit 501.

[0062] The configurations of the first selection unit 505 and the second selection unit 508 are similar to those of the first selection unit 152 and the second selection unit 155 .

[0063] Based on a read request from the geometric transformation units 509A to 509D, the first temporary storage unit 506 holds a part of the processing results of the signal processing units 501A to 501D for a predetermined period and outputs them to the geometric transformation units 509A to 509D. A detailed configuration of the first temporary storage unit 506 will be described later with reference to FIG.

[0064] The second temporary storage unit 507 is similar to the second temporary storage unit 154 .

[0065] The configuration of each geometric transformation unit 509 is similar to that of the geometric transformation unit 156. However, in the second embodiment, the output image is divided into the number of geometric transformation units 509, and each geometric transformation unit 509 performs geometric transformation processing on the corresponding divided output image.

[0066] Details of Processing by the First Temporary Storage Unit 506 6 is a diagram showing an example of the configuration of first temporary storage unit 506. An input area request of geometric transformation unit 509A is input from input area request 609A. An input area request of geometric transformation unit 509B is input from input area request 609B. An input area request of geometric transformation unit 509C is input from input area request 609C. An input area request of geometric transformation unit 509D is input from input area request 609D. The input area requests 609A to 609D are connected to all of the rectangular memory management units 600A to 600D. Also, the input area requests 609A to 609D are connected to all of the FIFO units 613A to 613D.

[0067] The output of signal processing unit 501A is input from input image 605A. The output of signal processing unit 501B is input from input image 605B. The output of signal processing unit 501C is input from input image 605C. The output of signal processing unit 501D is input from input image 605D. Each input image 605 is connected to a corresponding strip memory management unit 600. In this specification, when it is not necessary to strictly distinguish between the input images 605A to 605D, the term input image 605 is used as a general term. Also, when it is not necessary to strictly distinguish between the strip memory management units 600A to 600D, the term strip memory management unit 600 is used as a general term.

[0068] Although four strip memory management units 600 are shown in FIG. 6, the number of strip memory management units 600 may be any number equal to or greater than the number of input images 605. When the number of strip memory management units 600 is greater than the number of input images 605, a plurality of strip memory management units 600 correspond to one input image 605. In this case, a plurality of strip memory management units 600 connected to one input image 605 are adjacent to each other. Then, each strip memory management unit 600 manages a strip area obtained by further dividing a strip area corresponding to the left end to the right end of the image data input from the input image 605. Each strip memory management unit 600 determines whether each pixel value of the input image is within the strip area it is responsible for, and if it is within the strip area it is responsible for, stores the pixel value in the line memory. The method of allocating the line memory is the same as that of the first embodiment, and each strip memory management unit 600 allocates the line memory by inquiring of the line memory management unit 604 and stores the pixel value.

[0069] When multiple strip memory management units 600 are connected to one input image 605, the bottom ends of the vertical ranges to be held by these strip memory management units 600 must be aligned. However, the bottom ends of the vertical ranges to be held by strip memory management units 600 connected to different input images 605 do not necessarily have to be aligned. On the other hand, the top ends of the vertical ranges to be held may be different for all strip memory management units 600 (it is possible that the top ends of multiple strip memory management units 600 may coincidentally coincide).

[0070] An input image input from signal processing unit 501A is input to strip memory management unit 600A. Similarly, an input image input from signal processing unit 501B is input to strip memory management unit 600B, an input image input from signal processing unit 501C is input to strip memory management unit 600C, and an input image input from signal processing unit 501D is input to strip memory management unit 600D.

[0071] On the other hand, input area requests 609A-609D from the geometric transformation unit 509A are notified to all of the strip memory management units 600A-600D. This is because, particularly in geometric transformations such as distortion correction and rotation, the input area request of the geometric transformation unit 509A may intrude into the area managed by the strip memory management unit 600B.

[0072] The rectangular memory management units 600A-600D determine the images to be held in the SRAM 144 based on the input area requests from multiple input sources. This operation results in mutually correlated operations rather than simple parallel operations, and satisfies the request from the geometric transformation units 509A-509D requesting curved readout. Since processing is performed on rectangular areas that are correlated with each other, if the rectangular memory management units 600A-600D are holding images that meet the necessary input area requests, then the operation will naturally be synchronous.

[0073] Other operations are the same as those in the first embodiment. In the geometric transformation process, since the input image is read out obliquely or using curvilinear coordinates, adjacent output units among the output units 617A to 617D may read out overlapping areas. For example, the output units 617A and 617B may read out overlapping areas. The first temporary storage unit 506 is configured to perform reading taking such overlaps into consideration. The output units 617A to 617D output output images 621A to 621D.

[0074] Summary of the second embodiment As described above, according to the second embodiment, the image processing device 100 includes a plurality of signal processing units 501 and a plurality of geometric transformation units 509. Therefore, in the second embodiment, it is possible to complete the geometric transformation process faster than in the first embodiment.

[0075] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0076] [summary] The above-described embodiment discloses at least the inventions shown in the following items, but is not limited to these inventions. [Item 1] a signal processing means for performing signal processing on the first image to generate a second image; a first temporary storage means having a first storage capacity smaller than the data amount of the second image; a second temporary storage means having a second storage capacity equal to or greater than the data amount of the second image; a selection means for selecting the first temporary storage means or the second temporary storage means as a temporary storage means for temporarily storing the second image, and the signal processing means outputs the second image to the temporary storage means sequentially line by line; a geometric transformation unit that performs a geometric transformation process based on the second image stored in the temporary storage unit to sequentially generate a third image line by line; Equipped with The selection means is selects the first temporary storage means as the temporary storage means when a rotation angle of the second image by the geometric transformation process does not exceed a predetermined angle based on the first storage capacity and the data amount of the second image; When the rotation angle exceeds the predetermined angle, the second temporary storage means is selected as the temporary storage means; When the first temporary storage means is selected as the temporary storage means, the first temporary storage means sequentially stores the second image output from the signal processing means line by line so that the first storage capacity does not overflow, and sequentially discards portions of the second image that have been used as the geometric transformation process progresses. 13. An image processing device comprising: [Item 2] When the first temporary storage means is selected as the temporary storage means, said geometric transformation means outputs a request for designating an area of ​​said second image corresponding to each line of said third image to said temporary storage means, for each line in sequence; The temporary storage means sequentially updates the stored portion of the second image based on the requests sequentially output from the geometric transformation means. 2. The image processing device according to item 1, [Item 3] The present invention further includes a control unit that performs control to limit the rotation angle so as not to exceed the predetermined angle when a predetermined condition is satisfied. 3. The image processing device according to item 1 or 2. [Item 4] When the first image is a frame of a moving image input to the image processing device at a first data rate, the control means performs the control to limit the rotation angle so as not to exceed the predetermined angle; When the first image is a frame of a moving image input to the image processing device at a second data rate lower than the first data rate, the control means does not perform the control of limiting the rotation angle so as not to exceed the predetermined angle. 4. The image processing device according to item 3, [Item 5] the first storage capacity is constituted by a dedicated memory; The second storage capacity is configured from a shared memory. 5. The image processing device according to any one of items 1 to 4, [Item 6] The first storage capacity is configured with a faster memory than the second storage capacity. 6. The image processing device according to any one of items 1 to 5, [Item 7] A plurality of the signal processing means; A plurality of the geometric transformation means; Equipped with each of the plurality of signal processing means performs the signal processing on a different region of the first image to generate a different region of the second image, and outputs the different region of the second image line by line to the temporary storage means; each of the plurality of geometric transformation means sequentially generates different regions of the third image line by line; When the first temporary storage means is selected as the temporary storage means, the first temporary storage means sequentially stores each of the multiple areas of the second image output from the multiple signal processing means line by line so that the first storage capacity does not overflow, and sequentially discards the portion of the second image that has been used as the geometric transformation process progresses. 7. The image processing device according to any one of items 1 to 6, [Item 8] An image processing device according to any one of items 1 to 7, An imaging means for generating the first image; An imaging device comprising: [Item 9] An image processing method executed by an image processing device, comprising: a signal processing step in which a signal processing means of the image processing device performs signal processing on a first image to generate a second image; a selection step in which a selection means of the image processing device selects a first temporary storage means of the image processing device or a second temporary storage means of the image processing device as a temporary storage means for temporarily storing the second image, the first temporary storage means having a first storage capacity smaller than the data amount of the second image, the second temporary storage means having a second storage capacity equal to or larger than the data amount of the second image, and the signal processing means sequentially outputs the second image line by line to the temporary storage means; a geometric transformation step in which a geometric transformation means of the image processing device performs a geometric transformation process based on the second image stored in the temporary storage means to sequentially generate a third image line by line; Equipped with In the selection step, the selection means selects the first temporary storage means as the temporary storage means when a rotation angle of the second image by the geometric transformation process does not exceed a predetermined angle based on the first storage capacity and the data amount of the second image; When the rotation angle exceeds the predetermined angle, the second temporary storage means is selected as the temporary storage means; The image processing method includes: When the first temporary storage means is selected as the temporary storage means, the first temporary storage means sequentially stores the second image output from the signal processing means line by line so that the first storage capacity does not overflow, and sequentially discards portions of the second image that have been used as the geometric transformation process progresses. 13. An image processing method comprising: [Item 10] A program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 7.

[0077] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0078] 100: image processing device, 151: signal processing unit, 152: first selection unit, 153: first temporary storage unit, 154: second temporary storage unit, 155: second selection unit, 156: geometric transformation unit

Claims

1. a signal processing means for performing signal processing on the first image to generate a second image; a first temporary storage means having a first storage capacity smaller than the data amount of the second image; a second temporary storage means having a second storage capacity equal to or greater than the data amount of the second image; a selection means for selecting the first temporary storage means or the second temporary storage means as a temporary storage means for temporarily storing the second image, and the signal processing means outputs the second image to the temporary storage means sequentially line by line; a geometric transformation unit that performs a geometric transformation process based on the second image stored in the temporary storage unit to sequentially generate a third image line by line; Equipped with The selection means is selects the first temporary storage means as the temporary storage means when a rotation angle of the second image by the geometric transformation process does not exceed a predetermined angle based on the first storage capacity and the data amount of the second image; When the rotation angle exceeds the predetermined angle, the second temporary storage means is selected as the temporary storage means; When the first temporary storage means is selected as the temporary storage means, the first temporary storage means sequentially stores the second image output from the signal processing means line by line so that the first storage capacity does not overflow, and sequentially discards portions of the second image that have been used as the geometric transformation process progresses.

13. An image processing device comprising:

2. When the first temporary storage means is selected as the temporary storage means, the geometric transformation means outputs a request for designating an area of ​​the second image corresponding to each line of the third image to the temporary storage means, line by line, in sequence; The temporary storage means sequentially updates the stored portion of the second image based on the requests sequentially output from the geometric transformation means.

2. The image processing device according to claim 1,

3. The present invention further includes a control unit that performs control to limit the rotation angle so as not to exceed the predetermined angle when a predetermined condition is satisfied.

2. The image processing device according to claim 1,

4. when the first image is a frame of a moving image input to the image processing device at a first data rate, the control means performs the control to limit the rotation angle so as not to exceed the predetermined angle; When the first image is a frame of a moving image input to the image processing device at a second data rate lower than the first data rate, the control means does not perform the control of limiting the rotation angle so as not to exceed the predetermined angle.

4. The image processing device according to claim 3.

5. the first storage capacity is constituted by a dedicated memory; The second storage capacity is configured from a shared memory.

2. The image processing device according to claim 1,

6. The first storage capacity is configured with a memory faster than the second storage capacity.

2. The image processing device according to claim 1,

7. A plurality of the signal processing means; A plurality of the geometric transformation means; Equipped with each of the plurality of signal processing means performs the signal processing on a different region of the first image to generate a different region of the second image, and outputs the different region of the second image line by line to the temporary storage means; each of the plurality of geometric transformation means sequentially generates different regions of the third image line by line; When the first temporary storage means is selected as the temporary storage means, the first temporary storage means sequentially stores each of the multiple areas of the second image output from the multiple signal processing means line by line so that the first storage capacity does not overflow, and sequentially discards the portion of the second image that has been used as the geometric transformation process progresses.

2. The image processing device according to claim 1,

8. An image processing device according to any one of claims 1 to 7, An imaging means for generating the first image; An imaging device comprising:

9. An image processing method executed by an image processing device, comprising: a signal processing step in which a signal processing means of the image processing device performs signal processing on a first image to generate a second image; a selection step in which a selection means of the image processing device selects a first temporary storage means of the image processing device or a second temporary storage means of the image processing device as a temporary storage means for temporarily storing the second image, the first temporary storage means having a first storage capacity smaller than the data amount of the second image, the second temporary storage means having a second storage capacity equal to or larger than the data amount of the second image, and the signal processing means sequentially outputs the second image line by line to the temporary storage means; a geometric transformation step in which a geometric transformation means of the image processing device performs a geometric transformation process based on the second image stored in the temporary storage means to sequentially generate a third image line by line; Equipped with In the selection step, the selection means selects the first temporary storage means as the temporary storage means when a rotation angle of the second image by the geometric transformation process does not exceed a predetermined angle based on the first storage capacity and the data amount of the second image; When the rotation angle exceeds the predetermined angle, the second temporary storage means is selected as the temporary storage means; The image processing method includes: When the first temporary storage means is selected as the temporary storage means, the first temporary storage means sequentially stores the second image output from the signal processing means line by line so that the first storage capacity does not overflow, and sequentially discards portions of the second image that have been used as the geometric transformation process progresses.

13. An image processing method comprising:

10. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 7.

Images