Image processing device, image processing method, and imaging device

Abstract

This image processing device comprises a filter processing unit that performs: first processing for generating a plurality of pieces of processed image data having different rotation angles from original image data; second processing for detecting bright spot pixels for each of the plurality of pieces of processed image data and propagating pixel values based on the pixel values of the bright spot pixels to other propagation pixels; third processing for rotating all or a portion of the plurality of pieces of processed image data subjected to the second processing; and fourth processing for synthesizing propagation pixels in the plurality of pieces of processed image data subjected to the third processing.

Description

Image processing device, image processing method, and imaging device

[0001] The present technology relates to an image processing device, an image processing method, and an imaging device, and in particular to filtering using image processing.

[0002] One method for expanding the range of image expression is the use of optical lens filters, such as cross filters that add and emphasize oblique light rays and soft filters that soften contrast. Optical filters are installed in front of the image sensor and are dedicated lens filters, so if you want to change the number of light rays or the degree of diffusion, you need to physically replace the optical filter each time. Furthermore, because they are dedicated lens filters, it is necessary to prepare a separate optical lens filter for each parameter you want to use. This is cost-inefficient. In contrast, there are also known techniques for achieving image effects similar to those of optical lens filters using digital signal processing. For example, Patent Document 1 listed below discloses a technique for achieving a cross filter effect using digital signal processing.

[0003] JP 2012-104932 A

[0004] However, the method of Patent Document 1 and other general hardware image processing methods suffer from the following problems: General two-dimensional filter processing requires a buffer equal to the length of the ray, which directly affects the circuit size. Drawing the longest diagonal ray requires storing pixel information for one image frame and performing filter calculations, making it an unrealistic method. Furthermore, while the method of Patent Document 1 is considered to be software processing by a CPU, the area that can be processed within one frame is limited, so processing the entire area requires at least 10 frames, creating a processing delay problem. The delay time also varies depending on the number of bright spots. Furthermore, reducing frame delay using general hardware image processing requires an enormous circuit size.

[0005] Therefore, the present disclosure proposes a technology for realizing digital signal processing that achieves the same effect as an optical filter with a relatively simple configuration and without processing delay.

[0006] The image processing device according to the present technology includes a filter processing unit that performs a first process of generating a plurality of processed image data with different rotation angles from original image data, a second process of detecting bright point pixels in each of the plurality of processed image data and propagating pixel values ​​based on the pixel values ​​of the bright point pixels to other propagation pixels, a third process of rotating all or part of the plurality of processed image data that have undergone the second process, and a fourth process of combining the propagation pixels in the plurality of processed image data that have undergone the third process. The propagation is a process of setting a value based on the pixel value of the bright point pixel as the value of other pixels. For example, the pixel value of the bright point pixel that has been given a predetermined attenuation rate is set to a surrounding pixel.

[0007] 1 is a block diagram of an imaging device according to an embodiment of the present technology; 2 is a block diagram of an image processing unit according to an embodiment; 3 is an explanatory diagram of an image effect due to a cross filter; 4 is an explanatory diagram of an image effect due to a soft filter; 5 is an explanatory diagram of processing by the image processing unit according to an embodiment; 6 is an explanatory diagram of processed image data by the image processing unit according to an embodiment; 7 is an explanatory diagram of image data that is raster-input according to an embodiment; 8 is an explanatory diagram of propagation to the fourth quadrant according to an embodiment; 9 is an explanatory diagram of propagation in a lower right direction according to an embodiment; 10 is an explanatory diagram of propagation in a lower right direction according to an embodiment; 11 is an explanatory diagram of propagation to multiple pixels according to an embodiment;

[0008] The embodiments will be described below in the following order: <1. Configuration of imaging device> <2. Processing of digital lens filter> <3. Summary and modified examples>

[0009] In the embodiment, an example of an imaging device 1 including an image processing unit 10 (image processing device) that realizes an optical filter by digital signal processing will be described.

[0010] 1 shows the configuration of an imaging device 1 according to an embodiment. The imaging device 1 is a so-called digital still camera, and can capture both still images and moving images by switching the imaging mode, for example. Note that in this embodiment, the imaging device 1 is not limited to a digital still camera, and may be a video camera that is primarily used for capturing moving images but can also capture still images.

[0011] The imaging device 1 is composed of a main body 2 that constitutes the camera body, and a lens barrel 3. Although an example will be described in which the imaging device 1 is composed of the main body 2 and the lens barrel 3 separately, the portion corresponding to the lens barrel 3 may be integrated with the main body 2.

[0012] The imaging device 1 has a main body 2 which includes an image processing unit 10, a DRAM (Dynamic Random Access Memory) 11, an imaging element (image sensor) 12, a recording control unit 14, a display unit 15, an output unit 16, an operation unit 17, a camera control unit 30, and a memory unit 31. The lens barrel 3 has a lens system 21, a driver unit 22, and a lens control unit 23.

[0013] The lens system 21 in the lens barrel 3 includes lenses such as a zoom lens and a focus lens, and an iris mechanism. Light (incident light) from a subject is guided by this lens system 21 and collected on the image sensor 12 in the imaging device 1.

[0014] The image sensor 12 is configured as, for example, a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal Oxide Semiconductor) type. The image sensor 12 performs, for example, CDS (Correlated Double Sampling) processing, AGC (Automatic Gain Control) processing, etc. on the electrical signals obtained by photoelectrically converting the received light, and further performs A / D (Analog / Digital) conversion processing. The captured image data is then output as digital data to the image processing unit 10 and the camera control unit 30 in the subsequent stages.

[0015] The image processing unit 10 is configured as an image processor, for example, using a DSP (Digital Signal Processor). The image processing unit 10 performs various signal processing on the captured image data from the image sensor 12. For example, it performs development processing, resolution conversion processing depending on the output destination, metadata addition processing, formatting processing, etc. In the case of this embodiment, the image processing unit 10 also performs digital lens filter processing equivalent to an optical filter.

[0016] The DRAM 11 is shown as a storage area used in the processing steps by the image processing unit 10. The DRAM 11 may be built into the processor chip that constitutes the image processing unit 10, or may be a separate memory chip.

[0017] The recording control unit 14 performs processing to record image content such as still image data and video data, and metadata associated with the image content, onto a recording medium such as a non-volatile memory. The actual configuration of the recording control unit 14 can be various. For example, the recording control unit 14 may be a circuit that writes to and reads from a flash memory built into the imaging device 1, or may be in the form of a memory card (e.g., a portable flash memory) that can be attached to and detached from the imaging device 1 and a card recording / playback unit that accesses the memory card for recording / playback.

[0018] The display unit 15 is a display unit that displays various information to the photographer, and specifically refers to a display panel or viewfinder such as a liquid crystal display (LCD) or an organic electroluminescence (EL) display provided on the back of the main body unit 2.

[0019] The display unit 15 executes various displays on the display screen based on instructions from the camera control unit 30. For example, the display unit 15 displays a playback image of image data read from the recording medium by the recording control unit 14. The display unit 15 is also supplied with image data of captured images whose resolution has been converted for display by the camera signal processing unit 13. The display unit 15 displays a so-called through image (a monitoring image of the subject), which is an image captured during release standby, based on the image data of the captured image in response to instructions from the camera control unit 30. The display unit 15 also executes displays on the screen of various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the camera control unit 30.

[0020] The output unit 16 performs wired or wireless data communication and network communication with external devices. For example, it transmits and outputs image content to an external display device, recording device, playback device, etc. The output unit 16 may also be a network communication unit that performs communication via various networks such as the Internet, a home network, a LAN (Local Area Network), etc., and transmits and receives various data to and from servers, terminals, etc. on the network.

[0021] The operation unit 17 collectively represents input devices that allow the user to input various operations. Specifically, the operation unit 17 represents various operators (such as a shutter button) provided on the main body 2. The operation unit 17 detects operations by the user, and sends a signal corresponding to the input operation to the camera control unit 30.

[0022] The operation unit 17 may be implemented not only by physical keys or other operators but also by a touch panel. For example, a touch panel may be formed on a display panel, and various operations may be performed by touch panel operations using icons, menus, etc. displayed on the display panel. Alternatively, the operation unit 17 may be configured to detect user tap operations, etc. using a touch pad or the like. Furthermore, the operation unit 17 may also be configured as a receiving unit for an external operation device such as a separate remote controller.

[0023] The camera control unit 30 is configured by a microcomputer (arithmetic processing device) equipped with a CPU (Central Processing Unit). The memory unit 31 stores information and the like used for processing by the camera control unit 30. The illustrated memory unit 31 collectively represents, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, and the like. The memory unit 31 may be a memory area built into the microcomputer chip that constitutes the camera control unit 30, or may be configured by a separate memory chip.

[0024] The camera control unit 30 executes programs stored in the ROM, flash memory, etc. of the memory unit 31, thereby controlling the entire imaging device 1 and lens barrel 3. For example, the camera control unit 30 controls the shutter speed of the image sensor 12, instructs the camera signal processing unit 13 to perform various signal processing, controls the imaging and recording operations in response to user operations, plays back recorded image files, and controls the operations of the lens system 21, such as zoom, focus, and aperture adjustment in the lens barrel 3, and controls the user interface operations. With regard to aperture adjustment, the camera control unit 30 controls the F-number variable in response to user operations and instructs the F-number for automatic control (auto iris).

[0025] The RAM in the memory unit 31 is used to temporarily store data, programs, etc. as a working area when the CPU of the camera control unit 30 processes various types of data. The ROM and flash memory (non-volatile memory) in the memory unit 31 are used to store the OS (Operating System) used by the CPU to control each unit, content files such as image files, application programs for various operations, firmware, etc.

[0026] When the lens barrel 3 is attached to the main body 2, the camera control unit 30 communicates with the lens control unit 23 and issues various instructions. The lens barrel 3 is equipped with the lens control unit 23, which is implemented by, for example, a microcomputer, and various data communications are possible between the camera control unit 30. For example, the camera control unit 30 issues drive instructions to the lens control unit 23 to drive the zoom lens, focus lens, aperture mechanism, etc. The lens control unit 23 controls the driver unit 22 in response to these drive instructions, causing the lens system 21 to operate. Note that when the lens barrel 3 is attached to the main body 2, wired communication is performed between the camera control unit 30 and the lens control unit 23. However, the camera control unit 30 and the lens control unit 23 may also be configured to communicate wirelessly.

[0027] The driver unit 22 is provided with, for example, a motor driver for a zoom lens drive motor, a motor driver for a focus lens drive motor, a motor driver for a diaphragm mechanism motor, etc. These motor drivers apply drive currents to the corresponding drivers in response to instructions from the lens control unit 23, and execute operations such as moving the focus lens or zoom lens and opening and closing the diaphragm blades of the diaphragm mechanism.

[0028] Note that the above configuration is an example, and does not show all the components typically included in an imaging device. Furthermore, the technology of this embodiment can be applied even to imaging devices that do not include some of the components shown in the figure. The imaging device 1 of this embodiment has a configuration such as that shown in FIG. 1 , in which the image processing unit 10 has a function of performing digital filter processing.

[0029] 2 shows an example of the configuration of the image processing unit 10. The image processing unit 10 includes a brightness and color processing unit 41, a digital lens filter 42, an OETF (Opto-Electronic Transfer Function) processing unit 43, and a formatter 44.

[0030] The brightness and color processing section 41 performs RGB gain processing, white balance processing, master gain processing, etc. on the captured image data supplied from the image sensor 12 .

[0031] The digital lens filter 42 performs signal processing to achieve image effects equivalent to those of optical filters such as cross filters and soft filters. For example, Figure 3 shows an image with the cross filter effect. The image appears to have light extending in four directions from a bright spot, i.e., a pixel with high brightness, at the center. This image effect emphasizes brightness. In the example shown, the light extends diagonally up, down, left, and right, but it can also be achieved in a variety of directions, such as horizontally and vertically, or diagonally at different angles. Figure 4 shows the original image (left) and an image with the soft filter effect (right). For example, when the soft filter effect is applied to the original lamp image, the light appears to be softly diffused around the periphery.

[0032] The digital lens filter 42 uses signal processing to achieve effects such as spreading light in multiple directions or spreading it around. For example, the digital lens filter 42 uses the DRAM 11 to generate multiple pieces of processed image data by rotating one frame (original image data) of captured image data, and then uses this data to perform processing to add effects such as a cross filter. Details will be described later.

[0033] The captured image signal that has passed through the digital lens filter 42 is converted by an OETF processing unit 43 using a transfer function used to convert the light of the real scene into an image signal, and then a formatter 44 performs resolution conversion and formatting processing according to the output destination before outputting.

[0034] 2. Processing by the Digital Lens Filter An example of processing by the digital lens filter 42 will now be described. Fig. 5 is a schematic representation of the flow of signal processing by the digital lens filter 42, and also shows the state of the image at each stage. The processing flow will be explained by dividing it into a first process ST1 to a fifth process ST5.

[0035] The captured image data input to the digital lens filter 42 is referred to as the "original image data" in Figure 6. For example, a white circle represents a bright spot 50. In this case, it is assumed that there is one bright spot 50 in one frame of image. The bright spot 50 is a pixel that has a luminance value equal to or greater than a predetermined threshold value.

[0036] As shown in Fig. 5, in the first process ST1, the digital lens filter 42 rotates the original image data to generate multiple pieces of processed image data, each with a different rotation angle. For example, as shown in Fig. 6, the original image data may be rotated by 0 degrees, 90 degrees, 180 degrees, and 270 degrees to generate four pieces of processed image data. The four pieces of processed image data are stored in, for example, the DRAM 11 while subsequent processes are carried out.

[0037] As the second process ST2, the digital lens filter 42 detects pixels that are bright spots 50 for each of the four processed image data, and propagates pixel values ​​based on the pixel values ​​of the pixels of the bright spots 50 to other pixels. In this case, raster input in image processing is applied to the bright spots detected in each processed image data, causing light to propagate in the fourth quadrant direction, creating a light ray 51. Figure 6 shows the state in which the light ray 51 propagates from the bright spots 50 in the fourth quadrant direction for the four processed image data.

[0038] Propagation toward the fourth quadrant is explained in Figures 7, 8, 9, and 10. Since the processing target for the processed image data is the pixels input as a raster, as shown in Figure 7, processing proceeds horizontally from the upper left pixel of the frame, proceeding to pixels on the lines below each horizontal line, and proceeding to the rightmost pixel on the final line. In this case, as shown in Figure 8, if the pixel of the bright spot 50 is considered to be the origin, the fourth quadrant of the frame coordinates (X-Y coordinates on the frame plane) is the range indicated by the shaded area. Luminance values ​​are propagated from the bright spot 50 to the range of this fourth quadrant. Note that the fourth quadrant (shaded area) in this disclosure also includes the direction directly to the right on the X-axis (horizontal direction) and the direction directly below on the Y-axis (vertical direction) when the bright spot 50 is considered to be the origin.

[0039] The following description will be given assuming propagation in the direction of 45 degrees downward to the right within the fourth quadrant. In each of the figures from Figure 9 onwards, it is assumed that one square represents a pixel. Now, assume that pixel P1 in Figure 9 is the pixel of bright spot 50. In reality, it is assumed that the bright spot, i.e., the area that becomes the light source in the subject, is not a single pixel but a cluster of multiple pixels, but since the concept of propagation is the same, the description will be given assuming that one pixel is a bright spot.

[0040] The digital lens filter 42 uses a raster input method, sequentially selecting pixels as target pixels starting from the upper left pixel, and performs bright spot detection and propagation processing for each target pixel. First, the digital lens filter 42 sequentially determines whether each target pixel is a pixel that will become a bright spot 50 (hereinafter also referred to as a "bright spot pixel"). For example, if pixel P1 in FIG. 9 has a brightness value equal to or greater than a predetermined threshold, the digital lens filter 42 stores the fact that pixel P1 is a bright spot pixel at least until pixel P2, located 45 degrees to the lower right, becomes the target pixel. In this case, for example, it is sufficient to store bright spot pixels using a memory area equivalent to one line. For example, while a pixel on a certain horizontal line is selected as the target pixel, the brightness values ​​of each pixel on the previous horizontal line may be stored.

[0041] Furthermore, the digital lens filter 42 sequentially determines whether each pixel selected as a pixel of interest is a bright point pixel and whether the pixel of interest is a propagation pixel. When propagation is performed 45 degrees to the lower right, the digital lens filter 42 determines whether the pixel 45 degrees to the upper left of the pixel of interest is a bright point pixel or a propagation pixel. If the pixel 45 degrees to the upper left is a bright point pixel or a propagation pixel, this means that the current pixel of interest is also a pixel located in the direction in which light propagates and is a pixel that should be a propagation pixel. Therefore, when pixel P2 in Figure 9 becomes the pixel of interest, the digital lens filter 42 determines that pixel P2 is a propagation pixel because the upper left pixel P1 is a bright point pixel, and performs processing to propagate light.

[0042] In this case, the pixel value of pixel P2 is set to a pixel value based on the pixel value of bright pixel P1, i.e., a light propagation process is performed. For example, a predetermined attenuation rate K (K<1) is set, and the value of pixel P2 is rewritten to a value obtained by multiplying the pixel value of bright pixel P1 by the attenuation rate K. The updated luminance value of pixel P2, which has become a propagation pixel, is also stored, for example, until the processing of pixels on the next line.

[0043] 10 becomes the pixel of interest, pixel P3 is also determined to be a propagation pixel because pixel P2, which is 45 degrees to the upper left of the pixel of interest, is a propagation pixel. The digital lens filter 42 then rewrites the value of pixel P3 to a value obtained by multiplying the pixel value of propagation pixel P2 by the attenuation rate K.

[0044] For example, in the process of sequentially processing each pixel in this way, a process of propagating a ray in a direction within the fourth quadrant can be performed. As a result, a ray 51 can be formed in each of the four processed image data, as shown in the second process ST2 in Figure 6.

[0045] 9 and 10 show an example in which light is propagated in a 45-degree direction to the lower right, but the direction in which light is propagated can be changed by changing the criteria for determining the propagation pixel. For example, by propagating light in various directions (0 degrees, 60 degrees, etc.), it is possible to create light rays with different angles.

[0046] In the second process ST2, as in the above example, a process is performed to propagate the light from the bright spot 50 for each of the four processed image data. The length of the propagated light beam is determined by the set attenuation rate K.

[0047] As the third process ST3 shown in FIG. 5 , the digital lens filter 42 rotates the four processed image data through which light propagated in the second process ST2. Images of each processed image data in the rotated state are shown in FIG. 6 . In the example shown, in the processed image data rotated 0 degrees, the light ray 51 is rotated 0 degrees (i.e., not rotated), resulting in the light ray 51 being formed in the fourth quadrant. In the processed image data rotated 90 degrees from the original image data, the light ray 51 is rotated 90 degrees, resulting in the light ray 51 being formed from the bright spot 50 in the first quadrant. In the processed image data rotated 180 degrees from the original image data, the light ray 51 is rotated 180 degrees, resulting in the light ray 51 being formed from the bright spot 50 in the second quadrant. In the processed image data rotated 270 degrees from the original image data, the light ray 51 is rotated 270 degrees, resulting in the light ray 51 being formed from the bright spot 50 in the third quadrant.

[0048] As shown in Fig. 5, in the fourth process ST4, the digital lens filter 42 combines the propagation pixels of the four processed image data that were rotated in the third process ST3. Fig. 6 shows an image in which the propagation pixels are combined to create a composite image with light rays 51 formed by each piece of processed image data. In this case, a cross-filter effect image is formed, with light rays 51 extending in four directions. Through the processes up to this point, an image to be added for an image effect that did not exist in the original image data has been generated.

[0049] As shown in Fig. 5, in the fifth process ST5, the digital lens filter 42 combines the image data combined in the fourth process ST4 with the original image data. Fig. 6 shows an image in which four cross-filter-like rays 51 are combined with the original image. Through the processes up to this point, an image with a cross-filter effect added to the original image data has been generated.

[0050] As a result, it is possible to obtain a captured image with a cross filter effect without physically attaching an optical filter.

[0051] Although the above describes an example of a cross filter, processing as a soft filter, for example, is also possible using a similar concept. In the case of soft filter processing, if any of the upper left pixel, the pixel immediately above, or the pixel to the left of the pixel of interest is a bright point pixel or a propagation pixel, the pixel of interest is designated as a propagation pixel. Then, as shown in FIG. 11, for example, when pixel P1 is a bright point pixel, its pixel value is propagated to the right pixel, the lower right pixel, and the pixel immediately below. In other words, a luminance value based on the luminance value of the bright point pixel is propagated to the surroundings. Therefore, an image effect in which light spreads softly can be obtained by using the attenuation rate K during propagation.

[0052] In addition to cross filters and soft filters, various image effects can be obtained by setting the propagation pixels and attenuation rate K. Furthermore, by performing propagation processing using hardware rather than software calculations, there is also the advantage that the length of the light beam can be extended infinitely without imposing a processing load. Furthermore, with software calculations, the amount of calculation increases depending on the number of bright spots, so the number of bright spots is limited by resources. On the other hand, the processing of this embodiment simply propagates the pixel values ​​of the bright spots, so there is no limit to the number of bright spots 50. Therefore, filter processing is possible without increasing the load even for images containing a large number of bright spots 50.

[0053] Furthermore, the frame delay is extremely small compared to software calculations. For example, if the original image data has a 0-frame delay, the filtered image data can be output with a delay of 2 or 1 frame.

[0054] Furthermore, this technology offers the following advantages. First, the brightness for bright spot detection can be set arbitrarily. The determination of pixels as bright spots 50 is based on the setting of a threshold brightness value. The threshold setting determines the brightness level that is determined as a bright spot. Therefore, for example, by allowing the user to variably set the threshold, it is possible to perform the bright spot determination desired by the user for the original image data.

[0055] The length of the light beam 51 can be set arbitrarily. The length of the light beam 51 is determined by the attenuation rate K. Therefore, by setting the attenuation rate K, the user can obtain an image with the length of the light beam 51 desired by the user.

[0056] The brightness of the light ray 51 can also be set by the propagation strength of the light ray 51. For example, the brightness of the light ray can be adjusted by adjusting the brightness value of the bright pixel. Rather than using the brightness value of the bright pixel as is, it is possible to make the light ray 51 brighter by, for example, increasing the brightness value by a predetermined percentage before propagating the light ray 51. Of course, it is also possible to make the light ray 51 darker by decreasing the brightness value of the bright pixel by a predetermined percentage before propagating the light ray 51.

[0057] The color of the light ray 51 can also be modulated to a different color. For example, it is possible to obtain an effect in which a light ray 51 of any color, such as blue, green, or red, extends from a bright spot. The color of the light ray 51 can be set by individually adjusting the propagation method (attenuation rate) for each of the R, G, and B values ​​of the bright spot pixel.

[0058] In addition, the line width of the light ray 51 can be set arbitrarily. That is, the range of pixels that are to be treated as the light ray 51 can be set according to the conditions for determining the propagation pixel.

[0059] Furthermore, any number of light rays 51 can be set. For example, if one light ray 51 is generated in the direction of the fourth quadrant for each of the four processed image data, four light rays 51 will be represented in the composite image that has undergone the fifth processing ST5. However, if two light rays 51 are formed in each of the processed image data by propagation in the fourth quadrant, eight light rays 51 will be represented in the composite image. Furthermore, the four processed image data do not necessarily need to form the same number of light rays 51, and the light rays may be oriented in different directions. This allows for the creation of images with a variety of numbers and added light rays 51.

[0060] The angle of the light ray 51 can also be set arbitrarily. The light ray angle depends on the angle in the fourth quadrant at which the light ray 51 is formed, so the angle of the light ray 51 can be set by setting the determination conditions for the propagation pixel.

[0061] It is also possible to select the type of light beam. For example, in addition to the cross lines of a cross filter as described above, any optical filter can be set, such as a light beam that diffuses light in all directions like a soft filter.

[0062] Furthermore, the above various settings can be not only variably set by the user, but also automatically set. For example, the threshold for bright spot determination, the attenuation rate, the rate of increase / decrease in brightness of bright spot pixels, the determination conditions for propagation pixels, the light color setting, etc. may be set by AI (Artificial Intelligence) processing according to image analysis.

[0063] 3. Summary and Modifications According to the above embodiment, the following effects can be obtained.

[0064] The imaging device 1 of the embodiment includes an image processing unit 10 that functions as a filter processing unit that performs digital filter processing. The image processing unit 10 performs a first process ST1 that generates multiple pieces of processed image data with different rotation angles from the original image data, a second process ST2 that detects bright point pixels in each of the multiple pieces of processed image data and propagates pixel values ​​based on the pixel values ​​of the bright point pixels to other propagation pixels, a third process ST3 that rotates all or part of the multiple pieces of processed image data that have undergone the second process ST2, and a fourth process ST4 that combines the propagation pixels in the multiple pieces of processed image data that have undergone the third process ST3.

[0065] Propagation is a process in which a value based on the pixel value of a bright pixel is used as the value of other pixels. For example, the pixel value of a bright pixel is given a predetermined decay rate and set to the surrounding pixels. Then, by generating multiple processed image data by rotating the original image data and propagating the bright pixel, effect images that generate a variety of expressions can be realized with relatively simple hardware processing. This also allows for extremely short delays compared to processing using software calculations. Furthermore, because it is digital signal processing, it is possible to freely set the degree of diffusion and number of light rays, etc.

[0066] The image generated up to the fourth process ST4 is, for example, an image like the light ray composite image in Figure 6, but it is also possible to output only the image data at this stage, i.e., the effect image, so that the user can use it for some kind of image editing. For example, it is possible to generate a cross-filter light ray effect image from a certain image, output the light ray effect image without combining it with the original image data, and then composite it with another image.

[0067] Furthermore, in the third processing ST3, by returning the multiple processed image data, particularly the light rays 51, to the rotation angle of the original image data, for example, a filter effect image can be obtained in which the light rays 51 extend in a cross direction, but by applying various rotation angles to the multiple processed image data, a variety of effect images can also be obtained.

[0068] The image processing unit 10 of the embodiment further performs a fifth process ST5, in which the image data synthesized in the fourth process ST4 is synthesized with the original image data. This allows for the creation of a filter that imparts image effects to the original image data. Furthermore, it can achieve effects such as cross filters and soft filters without the need for optical lens filters, and can be configured as a digital filter that allows for the degree of diffusion and number of light rays to be freely set. This eliminates the need for users to prepare multiple optical lens filters to achieve a variety of effects, and also eliminates the need to physically replace optical filters each time. This results in cost advantages for users and ease of use.

[0069] In the embodiment, an example has been described in which the image processing unit 10, in the second processing ST2, propagates pixel values ​​based on the pixel value of the detected bright point pixel to other pixels within the range of the fourth quadrant of the frame coordinate system as propagation pixels for the processed image data. This means that pixel values ​​are propagated to pixels that are processed later in time than the bright point pixel for the original image data that is raster input, making it possible to realize a variety of filter processing with relatively simple hardware processing.

[0070] In the embodiment, an example has been described in which the image processing unit 10 generates processed image data rotated 0 degrees from the original image data, processed image data rotated 90 degrees from the original image data, processed image data rotated 180 degrees from the original image data, and processed image data rotated 270 degrees from the original image data in the first process ST1. By generating these three pieces of processed image data at 0 degrees, 90 degrees, 180 degrees, and 270 degrees, it becomes possible to easily realize an effect image in which light is extended in all directions from a bright point pixel by propagating brightness values ​​from the bright point image to pixels within the fourth quadrant within the frame.

[0071] For example, the effect of extending light in two directions from a bright pixel can be achieved by using two processed images, such as processed image data rotated 0 degrees from the original image data and processed image data rotated 180 degrees, propagating the bright pixel in each, and then rotating them back and combining them. For example, the effect of extending light in three directions can be achieved by using three processed images, such as processed image data rotated 0 degrees from the original image data, processed image data rotated 180 degrees, and processed image data rotated 270 degrees, propagating the bright pixel in each, and then rotating them back and combining them. In other words, the number of processed images can be changed depending on the image effect desired.

[0072] In the embodiment, an example was given in which the image processing unit 10, in the second process ST2, propagates a pixel value based on the pixel value of a detected bright pixel through a pixel that is in a predetermined positional relationship with the bright pixel, and then propagates the pixel value through other pixels that are in a predetermined positional relationship with the propagation pixel. In other words, starting from the bright pixel, a pixel in a predetermined positional relationship (for example, the lower right) is set as the propagation pixel. Then, pixel values ​​based on the pixel value of the bright pixel are propagated to the propagation pixels one after another. This allows the brightness of the bright pixel to be extended in a direction corresponding to the predetermined positional relationship.

[0073] In the embodiment, an example was given in which the pixel value propagated by the image processing unit 10 in the second processing ST2 is a pixel value obtained by applying a predetermined attenuation rate to the pixel value of the source pixel. In other words, starting from a bright pixel, the luminance value is propagated while decreasing at a predetermined attenuation rate. This allows for the effect of light attenuating as it naturally extends across the image. Note that although the attenuation rate K (where K<1) has been described, a coefficient such as K=1 or K>1 may also be used. In this case, an image in which light simply spreads, or an image in which light becomes stronger as it extends, can be obtained.

[0074] In the embodiment, the image processing unit 10 identifies pixels with pixel values ​​equal to or greater than a predetermined threshold as bright pixels in the second process ST2. For each piece of processed image data, bright pixels where the light source is the subject can be identified simply by comparing the pixel value with the predetermined threshold. Furthermore, if the threshold can be set by the user, the user can adjust the bright spots 50 that create image effects.

[0075] In the embodiment, the image processing unit 10 propagates pixel values ​​based on the pixel values ​​of detected bright pixels in one direction within the frame of processed image data in the second process ST2. This allows an image in which light is stretched in one direction from the origin within one piece of processed image data to be generated. Therefore, by combining four pieces of processed image data, a cross filter effect can be achieved.

[0076] In the embodiment, it was also mentioned that the image processing unit 10 propagates pixel values ​​based on the pixel values ​​of the detected bright pixels in multiple directions within the frame of processed image data in the second process ST2. This allows an image in which light extends in multiple directions from the origin within a single piece of processed image data to be generated. Therefore, by combining four pieces of processed image data, a soft filter-like effect can be achieved.

[0077] In the embodiment, an example was also mentioned in which the pixel values ​​propagated by the image processing unit 10 in the second process ST2 are pixel values ​​that change the color of the source pixel. For example, a different coefficient is set for each of the R, G, and B pixel values ​​of the source pixel, and these are used as the propagated pixel values. This allows for an effect image in which light extends while changing color.

[0078] The image processing unit 10 in the embodiment, i.e., the image processing device equipped with the digital lens filter 42, can be built into the imaging device 1, and can also be mounted as an image processing device in an image editing device, etc. For example, this can be realized in any equipment that is capable of acquiring images and is equipped with a chip capable of image processing, such as a system camera, a camera adapter, a single-lens reflex camera, a compact digital camera, a cinema camera, a master monitor, a mobile phone, a CCU (camera control unit), or a game console.

[0079] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0080] The present technology may also be configured as follows: (1) An image processing device including a filter processing unit that performs a first process of generating a plurality of processed image data with different rotation angles from original image data, a second process of detecting a bright point pixel in each of the plurality of processed image data and propagating a pixel value based on the pixel value of the bright point pixel to other propagation pixels, a third process of rotating all or a part of the plurality of processed image data that have undergone the second process, and a fourth process of combining the propagation pixels in the plurality of processed image data that have undergone the third process. (2) The image processing device according to (1) above, in which the filter processing unit further performs a fifth process of combining the image data combined in the fourth process with the original image data. (3) The image processing device according to (1) or (2) above, in which the filter processing unit, in the second process, propagates a pixel value based on the pixel value of the bright point pixel in the processed image data, using other pixels within a fourth quadrant in frame coordinates as propagation pixels when viewed from the detected bright point pixel. (4) The image processing device according to any of (1) to (3), wherein the filter processing unit generates, in the first processing, processed image data rotated 0 degrees from the original image data, processed image data rotated 90 degrees from the original image data, processed image data rotated 180 degrees from the original image data, and processed image data rotated 270 degrees from the original image data. (5) The image processing device according to any of (1) to (4), wherein the filter processing unit propagates, in the second processing, a pixel value based on the pixel value of a detected bright point pixel, by using pixels that are in a predetermined positional relationship with the bright point pixel as propagation pixels, and further propagating other pixels that are in the predetermined positional relationship with the propagation pixel as propagation pixels. (6) The image processing device according to any of (1) to (5), wherein the filter processing unit sets the pixel value propagated in the second processing to a pixel value obtained by applying a predetermined attenuation rate to the pixel value of the pixel from which the pixel is propagated. (7) The image processing device according to any one of (1) to (6), wherein the filter processing unit, in the second processing, classifies pixels having a pixel value equal to or greater than a predetermined threshold as bright pixels.(8) The image processing device according to any one of (1) to (7), wherein the filter processing unit propagates pixel values ​​based on pixel values ​​of detected bright point pixels in one direction within a frame of the processed image data in the second process. (9) The image processing device according to any one of (1) to (7), wherein the filter processing unit propagates pixel values ​​based on pixel values ​​of detected bright point pixels in the second process in multiple directions within a frame of the processed image data. (10) The image processing device according to any one of (1) to (9), wherein the filter processing unit sets the pixel values ​​propagated in the second process to pixel values ​​that cause a change in color of the source pixel. (11) An image processing method in which an image processing device performs the following: a first process of generating a plurality of processed image data with different rotation angles from original image data, a second process of detecting bright point pixels in each of the plurality of processed image data and propagating pixel values ​​based on the pixel values ​​of the bright point pixels to other propagation pixels, a third process of rotating all or a part of the plurality of processed image data that have undergone the second process, and a fourth process of combining the propagation pixels in the plurality of processed image data that have undergone the third process. (12) An imaging device equipped with an image processing unit that performs filter processing including the first process of generating a plurality of processed image data with different rotation angles from original image data, a second process of detecting bright point pixels in each of the plurality of processed image data and propagating pixel values ​​based on the pixel values ​​of the bright point pixels to other propagation pixels, a third process of rotating all or a part of the plurality of processed image data that have undergone the second process, and a fourth process of combining the propagation pixels in the plurality of processed image data that have undergone the third process.

[0081] REFERENCE SIGNS LIST 1 Imaging device 2 Main body 3 Lens barrel 10 Image processing section 41 Brightness and color processing section 42 Digital lens filter 43 OETF processing section 44 Formatter 50 Bright spot 51 Light beam

Claims

1. An image processing device having a filter processing unit that performs the following processes: a first process of generating multiple processed image data with different rotation angles from original image data; a second process of detecting bright pixels in each of the multiple processed image data and propagating pixel values ​​based on the pixel values ​​of the bright pixels to other propagation pixels; a third process of rotating all or part of the multiple processed image data that have undergone the second process; and a fourth process of synthesizing the propagation pixels in the multiple processed image data that have undergone the third process.

2. The image processing device according to claim 1, wherein the filter processing unit further performs a fifth process of combining the image data combined in the fourth process with the original image data.

3. The image processing device according to claim 1, wherein in the second processing, the filter processing unit propagates pixel values ​​based on the pixel value of the detected bright pixel for the processed image data, using other pixels within the fourth quadrant of the frame coordinates from the detected bright pixel as propagation pixels.

4. The image processing device according to claim 1, wherein the filter processing unit generates processed image data rotated 0 degrees from the original image data, processed image data rotated 90 degrees from the original image data, processed image data rotated 180 degrees from the original image data, and processed image data rotated 270 degrees from the original image data in the first processing.

5. The image processing device according to claim 1, wherein the filter processing unit propagates a pixel value based on the pixel value of the detected bright pixel in the second process by using pixels that are in a predetermined positional relationship with the bright pixel as propagation pixels, and further propagates other pixels that are in the predetermined positional relationship with the propagation pixel as propagation pixels.

6. The image processing device according to claim 1, wherein the filter processing unit uses the pixel value propagated in the second processing as a pixel value obtained by applying a predetermined attenuation rate to the pixel value of the pixel from which the pixel is propagated.

7. The image processing device according to claim 1, wherein the filter processing unit, in the second processing, classifies pixels having pixel values ​​equal to or greater than a predetermined threshold value as bright pixels.

8. The image processing device according to claim 1, wherein the filter processing unit propagates pixel values ​​based on pixel values ​​of bright pixels detected in the second processing in one direction within the frame of the processed image data.

9. The image processing device according to claim 1, wherein the filter processing unit propagates pixel values ​​based on pixel values ​​of detected bright pixels in the second processing in multiple directions within the frame of the processed image data.

10. The image processing device according to claim 1, wherein the pixel value propagated in the second processing is a pixel value that causes a change in color of the pixel from which the pixel is propagated.

11. An image processing method in which an image processing device performs the following steps: a first process of generating multiple processed image data with different rotation angles from original image data; a second process of detecting bright pixels in each of the multiple processed image data and propagating pixel values ​​based on the pixel values ​​of the bright pixels to other propagation pixels; a third process of rotating all or part of the multiple processed image data that have undergone the second process; and a fourth process of combining the propagation pixels in the multiple processed image data that have undergone the third process.

12. An imaging device equipped with an image processing unit that performs filter processing including: a first process of generating multiple processed image data with different rotation angles from original image data; a second process of detecting bright pixels in each of the multiple processed image data and propagating pixel values ​​based on the pixel values ​​of the bright pixels to other propagation pixels; a third process of rotating all or part of the multiple processed image data that have undergone the second process; and a fourth process of combining the propagation pixels in the multiple processed image data that have undergone the third process.

Images