Image processing device, image processing method, and program

Abstract

This image processing device comprises a moire detection unit that generates moire detection information by detecting pixel regions in which movement is different, from among pixel regions for which it is supposed that the movements thereof will be the same as a subject that moves, such subject movement being changes in the position of the subject within frames among images having different timings.

Description

[Technical Field]

[0001] The present technology relates to an image processing device, an image processing method, and a program, and in particular to a technical field regarding moire that occurs in an image. [Background technology]

[0002] In general cameras, moire is prevented by using an optical low-pass filter to cut out any part of the image sensor that exceeds the Nyquist frequency, but this cannot be completely eliminated due to the need to balance this with resolution. Therefore, Patent Document 1 below proposes a method of preparing two optical systems with different resolutions, detecting moiré from the difference between the resolutions, and reducing the moiré. Furthermore, Patent Document 2 below discloses a method for detecting and reducing moire from the difference between two frames in which the cutoff frequency is changed using a variable optical low-pass filter. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-207414 [Patent Document 2] Japanese Patent Application Laid-Open No. 2006-80845 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in both of the above cases, the difference between the two images also contains high-frequency components that are the actual image and not moiré, and the difference is only moiré in the low-frequency range. In other words, while it is possible to detect moiré that has aliased down to the low frequencies, it is not possible to distinguish between moiré and the actual high-frequency components in the high-frequency range. For this reason, there is a trade-off between maintaining the resolution of the high-frequency range and eliminating moiré.

[0005] Therefore, this technology proposes a method that can detect moiré by distinguishing it from real patterns in an image, regardless of the moiré frequency. [Means for solving the problem]

[0006] The image processing device according to the present technology includes a moiré detection unit that detects pixel areas that exhibit different movement from pixel areas that are expected to exhibit the same movement as a subject that exhibits movement as a change in position within a frame between images taken at different times, and generates moiré detection information. The images at different times may be, for example, the current image and an image from one to several frames before. If a change in the position of a certain subject within a frame is observed between frames at different times, that is, if there is movement, and if a pixel region is expected to show the same movement as that movement but the movement is different, then that pixel region is determined to be moiré. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a block diagram of an imaging device according to an embodiment of the present technology; [Figure 2] FIG. 1 is a block diagram of an information processing apparatus according to an embodiment. [Figure 3] 1 is a block diagram of a configuration example of an image processing device according to an embodiment; [Figure 4] 10 is a flowchart of a moire detection process according to an embodiment. [Figure 5] 10 is a flowchart of a moire reduction process according to an embodiment. [Figure 6] FIG. 10 is a block diagram of another example of the configuration of the image processing device according to the embodiment. [Figure 7] 10 is a flowchart of a first example of moire detection processing according to an embodiment. [Figure 8] 1A and 1B are diagrams illustrating the concept of moire detection according to an embodiment. [Figure 9] 1A and 1B are diagrams illustrating the concept of moire detection according to an embodiment. [Figure 10] 1A and 1B are diagrams illustrating the concept of moire detection according to an embodiment. [Figure 11] 10 is a flowchart of a second example of moire detection processing according to the embodiment. [Figure 12] 10 is a flowchart of a third example of moire detection processing according to the embodiment. [Figure 13] 10 is a flowchart of a first example of a moire reduction process according to an embodiment. [Figure 14] 10 is a flowchart of a second example of the moire reduction process according to the embodiment. [Figure 15] 10 is a flowchart of a third example of the moire reduction process according to the embodiment. [Figure 16] 10A and 10B are explanatory diagrams of a third example of the moire reduction processing according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] The embodiments will be described below in the following order. <1. Configuration of imaging device> <2. Configuration of information processing device> <3. Image processing configuration and processing overview> <4. Example of moire detection processing> <5. Example of moiré reduction processing> <6. Summary and Variations>

[0009] In the present disclosure, the "movement" of a subject in an image refers to a change in the position of all or part of the subject within a frame between images taken at different times. For example, a change in the position of all or part of a so-called moving subject, such as a human, animal, or machine, within a frame due to the subject moving in part or in whole is one aspect that is expressed as "movement" in this disclosure. Furthermore, the change in position within a frame of a stationary subject such as a landscape or still life due to a change in the shooting direction, such as panning or tilting of an imaging device (camera), is also one aspect that can be expressed as "movement."

[0010] Furthermore, the term "image" does not specifically refer to a still image or a video when it is recorded. An imaging device captures one frame of an image at each time point at a predetermined frame rate, and as a result, it is assumed that either a certain frame will be recorded as a still image or a video made up of consecutive frames will be recorded.

[0011] The image processing device according to the embodiment is expected to be installed as an image processing unit in an imaging device (camera) or an information processing device that performs image editing, etc. Furthermore, the imaging device or information processing device equipped with such an image processing unit can also be considered as an image processing device.

[0012] <1. Configuration of imaging device> An example of the configuration of an imaging device 1 will be described with reference to FIG. This imaging device 1 includes an image processing unit 20 that performs moire detection processing, and this image processing unit 20, or the imaging device 1 including the image processing unit 20, can be considered as an example of an image processing device of the present disclosure.

[0013] The imaging device 1 has, for example, a lens system 11, an imaging element unit 12, a recording control unit 14, a display unit 15, a communication unit 16, an operation unit 17, a camera control unit 18, a memory unit 19, an image processing unit 20, a buffer memory 21, a driver unit 22, a sensor unit 23, and a connection unit 24.

[0014] The lens system 11 includes lenses such as a zoom lens and a focus lens, an aperture mechanism, etc. The lens system 11 guides light (incident light) from a subject and focuses the light on the imaging element unit 12.

[0015] The lens system 11 can also be provided with an optical low pass filter, such as a birefringent plate, to reduce moiré. However, it is difficult to completely eliminate moiré using an optical low pass filter, and in this embodiment, moiré that cannot be completely eliminated by an optical low pass filter is detected and reduced in the image processing unit 20. Note that even if an optical low pass filter is not provided, the detection and reduction of moiré by the image processing unit 20 is effective.

[0016] The imaging element unit 12 includes an imaging element (image sensor) 12a, such as a CMOS (Complementary Metal Oxide Semiconductor) type or a CCD (Charge Coupled Device) type. The image sensor unit 12 performs processes such as CDS (Correlated Double Sampling) and AGC (Automatic Gain Control) on the electrical signals obtained by photoelectrically converting the light received by the image sensor 12a, and then performs A / D (Analog / Digital) conversion on the electrical signals.Then, the image sensor unit 12 outputs the imaging signals as digital data to the image processing unit 20 and the camera control unit 18 in the subsequent stages.

[0017] The image processing unit 20 is configured as an image processor, for example, using a DSP (Digital Signal Processor). The image processing unit 20 performs various types of signal processing on the digital signal (captured image signal) from the image sensor unit 12, that is, the RAW image data.

[0018] For example, the image processing unit 20 performs lens correction, noise reduction, synchronization processing, YC generation processing, color reproduction / sharpness processing, and the like. In the synchronization process, a color separation process is performed so that the image data for each pixel contains all the color components R, G, and B. For example, in the case of an image sensor that uses a Bayer color filter, a demosaic process is performed as the color separation process. In the YC generation process, a luminance (Y) signal and a color (C) signal are generated (separated) from R, G, and B image data. Color reproduction / sharpness processing involves adjusting gradation, saturation, tone, contrast, etc., which are used to create images.

[0019] The image processing unit 20 performs signal processing in this manner, that is, signal processing generally called development processing, to generate image data in a predetermined format. In this case, resolution conversion and file creation processing may be performed. In the file creation processing, image data is subjected to, for example, compression encoding for recording or communication, formatting, and generation and addition of metadata to create files for recording or communication. For example, still image files can be generated in formats such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), GIF (Graphics Interchange Format), HEIF (High Efficiency Image File Format), YUV422, YUV420, etc. It is also possible to generate image files in the MP4 format used for recording MPEG-4 compliant video and audio. Note that there are also cases where an image file is generated using RAW image data that has not undergone development processing.

[0020] In this embodiment, the image processing unit 20 has signal processing functions as a moire detection unit 31 and a moire reduction unit 32 . The moire detection unit 31 detects pixel areas in which different movement appears among pixel areas in which the same movement as that of a subject occurring as a change in position within a frame between images taken at different times is expected, and performs processing to generate moire detection information Sdt (see Figure 3). The moire reduction unit 32 performs moire reduction processing based on the detection information Sdt. The details of these signal processing functions will be described later. Note that the image processing unit 20 may not be provided with the moire reduction unit 32.

[0021] The buffer memory 21 is formed of, for example, a DRAM (Dynamic Random Access Memory), and is used by the image processing unit 20 to temporarily store image data during the above-mentioned development process and the like. The buffer memory 21 may be a memory chip separate from the image processing unit 20, or may be configured in an internal memory area of ​​a DSP or the like that configures the image processing unit 20.

[0022] The recording control unit 14 performs recording and reproduction on a recording medium such as a nonvolatile memory, and performs processing to record image files such as moving image data and still image data on the recording medium. The recording control unit 14 may take a variety of actual forms. For example, the recording control unit 14 may be configured as a flash memory built into the imaging device 1 and its write / read circuit. The recording control unit 14 may also take the form of a card recording / playback unit that performs recording / playback access to a recording medium that can be attached to or detached from the imaging device 1, such as a memory card (such as a portable flash memory). The recording control unit 14 may also be realized as an HDD (Hard Disk Drive) built into the imaging device 1.

[0023] The display unit 15 is a display unit that displays various information to the user, and is, for example, a display panel or viewfinder using a display device such as a liquid crystal display (LCD) or an organic electroluminescence (EL) display that is arranged on the housing of the imaging device 1. The display unit 15 executes various displays on the display screen based on instructions from the camera control unit 18. For example, the display unit 15 displays a reproduced image of image data read from a recording medium by the recording control unit 14. Furthermore, image data of the captured image that has been resolution converted for display by the image processing unit 20 is supplied to the display unit 15, and the display unit 15 may display based on the image data of the captured image in response to an instruction from the camera control unit 18. This allows the display of a so-called through image (monitoring image of the subject), which is an image captured while checking the composition or recording a video. Furthermore, based on instructions from the camera control unit 18, the display unit 15 displays various operation menus, icons, messages, etc., that is, GUI (Graphical User Interface), on the screen.

[0024] The communication unit 16 performs wired or wireless data communication and network communication with external devices, such as transmitting and outputting still image files and video files containing captured image data and metadata to external information processing devices, display devices, recording devices, playback devices, etc. The communication unit 16 also serves as a network communication unit, and can communicate over various networks such as the Internet, a home network, and a LAN (Local Area Network), and can transmit and receive various data to and from servers, terminals, and the like on the network. The imaging device 1 may also be capable of mutual information communication with, for example, a PC, a smartphone, a tablet terminal, etc., via the communication unit 16, for example, by short-range wireless communication such as Bluetooth (registered trademark), Wi-Fi (registered trademark), or NFC (Near Field Communication), infrared communication, etc. The imaging device 1 may also be capable of mutual communication with other devices via wired connection communication. Therefore, the imaging device 1 can transmit image data and metadata via the communication unit 16 to an information processing device 70 (to be described later) and the like.

[0025] The operation unit 17 collectively refers to input devices that allow the user to input various operations. Specifically, the operation unit 17 refers to various operators (keys, dials, touch panel, touch pad, etc.) provided on the housing of the imaging device 1. The operation unit 17 detects the user's operation, and a signal corresponding to the input operation is sent to the camera control unit 18 .

[0026] The camera control unit 18 is configured by a microcomputer (arithmetic processing device) equipped with a CPU (Central Processing Unit). The memory unit 19 stores information and the like used for processing by the camera control unit 18. The illustrated memory unit 19 comprehensively represents, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, and the like. The memory unit 19 may be a memory area built into the microcomputer chip that serves as the camera control unit 18, or may be configured as a separate memory chip. The camera control unit 18 controls the entire imaging device 1 by executing a program stored in the ROM or flash memory of the memory unit 19 . For example, camera control unit 18 controls the shutter speed of image sensor unit 12, instructs various signal processing in image processing unit 20, controls the operation of each necessary unit, such as image capturing operation and image recording operation in response to user operation, playback operation of recorded image files, and operation of lens system 11 such as zoom, focus, and aperture adjustment in the lens barrel. Furthermore, camera control unit 18 detects operation information from operation unit 17 and controls display on display unit 15 as a user interface operation. Camera control unit 18 also controls communication operation with external devices via communication unit 16.

[0027] The RAM in the memory unit 19 is used as a work area for the CPU of the camera control unit 18 to process various data, and is used to temporarily store data, programs, and the like. The ROM and flash memory (non-volatile memory) in the memory unit 19 are used to store an OS (Operating System) for the CPU to control each unit, and content files such as image files, etc. The ROM and flash memory in the memory unit 19 are also used to store application programs, firmware, various setting information, etc. for various operations of the camera control unit 18 and image processing unit 20.

[0028] The driver section 22 includes, for example, a motor driver for a zoom lens drive motor, a motor driver for a focus lens drive motor, a motor driver for a diaphragm mechanism motor, and the like. These motor drivers apply drive currents to the corresponding drivers in response to instructions from the camera control unit 18, and cause the focus lens and zoom lens to move, and the aperture blades of the aperture mechanism to open and close, etc.

[0029] The sensor unit 23 collectively represents various sensors mounted on the imaging device. When an IMU (inertial measurement unit), for example, is installed as the sensor unit 23, angular velocity can be detected using a three-axis angular velocity (gyro) sensor of pitch, yaw, and roll, and acceleration can be detected using an acceleration sensor. The sensor unit 23 may also include, for example, a position information sensor, an illuminance sensor, a distance measurement sensor, and the like. Various information detected by the sensor unit 23, such as location information, distance information, illuminance information, IMU data, etc., is supplied to the camera control unit 18 and can be associated with the captured image as metadata together with date and time information managed by the camera control unit 18. The camera control unit 18 can generate metadata for each frame of an image, associate it with the image frame, and record it together with the image on a recording medium using the recording control unit 14. The camera control unit 18 can also associate the metadata generated for each frame of an image with the image frame and transmit it to an external device together with the image data using the communication unit 16.

[0030] The connection unit 24 communicates with a so-called pan-tilter or tripod that is equipped with the imaging device 1 and performs panning and tilting. For example, the connection unit 24 can input operation information such as the direction and speed of panning and tilting from the pan-tilter or the like and transmit it to the camera control unit 18.

[0031] <2. Configuration of information processing device> Next, an example of the configuration of the information processing device 70 will be described with reference to FIG. The information processing device 70 is a device such as a computer that is capable of information processing, particularly image processing. Specific examples of the information processing device 70 include personal computers (PCs), mobile terminal devices such as smartphones and tablets, mobile phones, video editing devices, and video playback devices. The information processing device 70 may also be a computer configured as a server device or a computing device in cloud computing. This information processing device 70 is equipped with an image processing unit 20 that performs moire detection and moire reduction, and this image processing unit 20, or the information processing device 70 equipped with the image processing unit 20, can be considered an example of an image processing device of the present disclosure.

[0032] The CPU 71 of the information processing device 70 executes various processes in accordance with programs stored in a ROM 72 or a nonvolatile memory unit 74 such as an EEPROM (Electrically Erasable Programmable Read-Only Memory), or programs loaded from a storage unit 79 to a RAM 73. The RAM 73 also stores data necessary for the CPU 71 to execute various processes as appropriate.

[0033] The image processing unit 20 has the functions of the moiré detection unit 31 and the moiré reduction unit 32 described in the imaging device 1 above.

[0034] The moire detection unit 31 and the moire reduction unit 32 as the image processing unit 20 may be provided as functions within the CPU 71. The image processing unit 20 may also be realized by a CPU separate from the CPU 71, a graphics processing unit (GPU), a general-purpose computing on graphics processing units (GPGPU), an artificial intelligence (AI) processor, or the like.

[0035] The CPU 71, ROM 72, RAM 73, nonvolatile memory unit 74, and image processing unit 20 are interconnected via a bus 83. To this bus 83, an input / output interface 75 is also connected.

[0036] An input unit 76 consisting of operators and operation devices is connected to the input / output interface 75. For example, the input unit 76 may be various operators and operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, or a remote controller. An operation by the user is detected by the input unit 76, and a signal corresponding to the input operation is interpreted by the CPU 71. A microphone may also be used as the input unit 76. Voice uttered by the user may also be input as operation information.

[0037] Furthermore, the input / output interface 75 is connected integrally or separately to a display unit 77 made up of an LCD or organic EL panel or the like, and an audio output unit 78 made up of a speaker or the like. The display unit 77 is a display unit that displays various types of information, and is configured by, for example, a display device provided in the housing of the information processing device 70, or a separate display device connected to the information processing device 70, or the like. The display unit 77 displays images for various image processing, moving images to be processed, etc. on the display screen based on instructions from the CPU 71. Furthermore, the display unit 77 displays various operation menus, icons, messages, etc., i.e., GUI (Graphical User Interface), based on instructions from the CPU 71.

[0038] The input / output interface 75 may be connected to a storage unit 79 configured with a HDD or solid-state memory, or a communication unit 80 configured with a modem or the like.

[0039] The storage unit 79 can store data to be processed and various programs. When the information processing device 70 functions as the image processing device of the present disclosure, it is expected that the memory unit 79 will store image data to be processed, detection information Sdt obtained by moiré detection processing, or image data that has undergone moiré reduction processing. The storage unit 79 may also store programs for moire detection processing and moire reduction processing.

[0040] The communication unit 80 performs communication processing via a transmission path such as the Internet, and communication with various devices via wired / wireless communication, bus communication, and the like. Communication with the imaging device 1, for example, reception of captured image data, metadata, and the like, is performed by a communication unit 80.

[0041] A drive 81 is also connected to the input / output interface 75 as required, and a removable recording medium 82 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted thereon. Drive 81 allows data files such as image files and various computer programs to be read from removable recording medium 82. The read data files are stored in storage unit 79, and images and sounds contained in the data files are output on display unit 77 and audio output unit 78. Furthermore, computer programs and the like read from removable recording medium 82 are installed in storage unit 79 as needed.

[0042] In this information processing device 70, for example, software for the processing of this embodiment can be installed via network communication by the communication unit 80 or via a removable recording medium 82. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, etc.

[0043] <3. Image processing configuration and processing overview> The image processing unit 20 in the above-described imaging device 1 and information processing device 70 will be described. A subject that has frequency components exceeding the Nyquist frequency of the image sensor 12a of the imaging device 1 will produce moiré as aliasing distortion. Moiré can basically be prevented by using an optical low-pass filter in front of the image sensor 12a to cut out frequencies above the Nyquist frequency, but it is difficult to completely cut out moiré due to the impact on image resolution. Therefore, as a post-processing step for images in which moiré has occurred, the moiré areas (pixel areas) are detected from the movement of the subject, and the moiré is reduced by blurring those areas.

[0044] Here, when the subject is stationary relative to the imaging device 1, it is difficult to distinguish between moire and a real pattern, and the moire is not particularly distracting in the image. On the other hand, if the subject in the image is moving, it is possible to identify it as moire, since a real pattern would move in the same direction and at the same speed, whereas a moire pattern does not necessarily move in that way.

[0045] FIG. 3 shows an example of the configuration for moire detection and moire reduction in the image processing unit 20. The image data Din indicates image data to be subjected to moiré detection processing, and is image data that is input sequentially, for example, frame by frame. The image data Din for each frame is input to the memory 30, the moiré detection unit 31, and the moiré reduction unit 32, respectively.

[0046] In the case of the imaging device 1, for example, the image data Din may be RAW image data input to the image processing unit 20. Alternatively, it may be image data after undergoing some or all of the development process.

[0047] For example, in the case of the imaging device 1, the memory 30 may be a storage area of ​​the buffer memory 21 inside or outside the image processing unit 20. In the case of the information processing device 70, it is considered that the memory 30 may be a storage area of ​​the RAM 73. Either storage area may be used as the memory 30.

[0048] The moire detection unit 31 detects pixel areas in which different movement appears among pixel areas in which the same movement as that of a subject occurring as a change in position within a frame between images taken at different times is expected, and performs moire detection processing to generate moire detection information. Therefore, the image data Din is input as an image of the current frame (current image DinC), and the image data Din stored in the memory 30 is read out after one frame period and input as a past image DinP. The past image DinP does not necessarily have to be read out after one frame period. For example, it may be read out after two frame periods or several frame periods and used as the past image DinP. The moire detection unit 31 only needs to compare the current image DinC with an earlier past image DinP, and the time difference between the current image DinC and the past image DinP used in the comparison may be set to a time appropriate for moire detection.

[0049] Every time image data of a frame serving as the current image DinC is input, the moire detection unit 31 performs moire detection processing as shown in FIG. 4 using the current image DinC and the past image DinP. 4, the moire detection unit 31 detects motion within an image. For example, it detects the motion of a certain subject within the image. Alternatively, it may detect a substantially uniform motion of the entire subject within the image.

[0050] In step S102, the moiré detection unit 31 detects pixel areas exhibiting different motion from those pixel areas expected to exhibit the same motion as that detected in step S101. This "different motion" refers to motion with different direction (direction of change in position within a frame) or speed (amount of displacement between frames), for example.

[0051] In an image that includes a moving subject such as a person, animal, or machine, the area that is expected to move in the same manner as the subject is considered to be the area within the outline of the moving subject. In addition, in images that contain only still subjects such as landscapes or still lifes, movement may be detected due to panning of the imaging device 1 itself, in which case the entire pixel area within the frame becomes a pixel area that is expected to move in the same way as the subject.

[0052] The moire detection unit 31 performs processing to compare the current image DinC with the past image DinP and detect portions where a different movement appears in a pixel area where the same movement as the movement of the subject is expected.

[0053] In step S103, the moire detection unit 31 generates moire detection information Sdt based on the results of detecting different movements. The detection information Sdt may include moire presence / absence information indicating whether or not moire occurs in the current image DinC. Alternatively, the detection information Sdt may include area information indicating a pixel area where moiré occurs in the current image DinC. The detection information Sdt may include both the moire presence / absence information and the area information.

[0054] The detection information Sdt generated by the moiré detection unit 31 in FIG. Image data Din is input as a target for moiré reduction processing to the moiré reduction unit 32. The moiré reduction unit 32 performs moiré reduction processing, specifically, for example, LPF (low pass filter) processing, on the image data Din based on the detection information Sdt.

[0055] The detection information Sdt is the moire detection result for the frame that the moire detection unit 31 has determined to be the current image DinC, so the moire reduction process based on the detection information Sdt is performed on the image data Din of the same frame as the current image DinC.

[0056] The moire reduction unit 32 performs moire reduction processing on each frame of the image data Din, for example, as shown in FIG. In step S201, the moiré reduction unit 32 acquires the detection information Sdt corresponding to the frame of the image data Din that is currently being processed.

[0057] In step S202, the moiré reduction unit 32 refers to the detection information Sdt to determine whether the current frame being processed is an image in which moiré occurs. If the detection information Sdt includes moiré presence / absence information, this can be used to make the determination. Even if the detection information Sdt is only area information, the occurrence of moiré can be determined based on whether the relevant area is indicated as area information.

[0058] If it is determined that moiré has occurred, the moiré reduction unit 32 proceeds to step S203, where it reduces moiré by performing LPF processing on the image data Din of the frame to be processed. Note that although LPF processing is used, BPF (band pass filter) processing that filters a specific frequency band may also be used. If it is determined that no moiré has occurred, the moiré reduction unit 32 does not perform step S203 and ends the processing of FIG. 5 for the image data Din of the frame to be processed.

[0059] As a result of the processing in the moiré reduction unit 32 as described above, image data Dout in which moiré has been reduced (or eliminated) is obtained. For example, by performing development processing on this image data Dout, it is possible to display an image with reduced moire.

[0060] 6 is also possible as the image processing unit 20. That is, this is a configuration example in which the moiré reduction unit 32 is not provided. In this case, the detection information Sdt is generated as described above by the moire detection unit 31. For example, the camera control unit 18 of the imaging device 1 or the CPU 71 of the information processing device 70 uses this detection information Sdt as metadata associated with the frame of the image data Din. For example, the camera control unit 18 can cause the recording control unit 14 to record metadata on a recording medium in association with each frame of the image data Dout. Alternatively, the camera control unit 18 can cause the communication unit 16 to transmit the image data Dout and the metadata associated with each frame thereof to an external device.

[0061] Therefore, each frame of image data is associated with moiré detection information Sdt. In this case, the moiré reduction process shown in Fig. 5 can be performed in a device that inputs an image file containing the image data and metadata, such as the information processing device 70. In this case, the detection information obtained in step S201 is read from the metadata corresponding to the frame that is the target of the moiré reduction process. Note that the CPU 71 of the information processing device 70 can also treat the detection information Sdt as metadata associated with frames of the image data Din, similar to the camera control unit 18. In this case, the storage unit 79 or the like can record the metadata in association with each frame of the image data Dout on a recording medium, or the communication unit 80 can transmit the image data Dout and the metadata associated with each frame thereof to an external device.

[0062] <4. Example of moire detection processing> Below, we will explain specific examples (first example, second example, and third example) of moiré detection processing by the moiré detection unit 31. Each example is an example of processing performed by the moiré detection unit 31 that inputs the current image DinC and the past image DinP as shown in FIG.

[0063] A first example of the moire detection process will be described with reference to FIGS. The first example is an example in which object recognition processing is performed on subjects in an image, and pixel regions in which the movement of each object does not match the movement inside the object are determined to be moiré. FIG. 7 is a flowchart showing a first example of the moire detection unit 31.

[0064] In step S110, the moire detection unit 31 performs object recognition processing on the image and sets a target subject based on the recognition result.

[0065] In this case, the moiré detection unit 31 performs object recognition processing such as for a person, animal, or object on the subject in the current image DinC, for example, by semantic segmentation processing, pattern recognition processing, etc. For the sake of explanation, these subjects recognized as some kind of object will be referred to as object A, object B, object C, etc. Then, among these recognized objects, an object to be the target of motion detection is specified and set as the target subject.

[0066] For example, the moire detection unit 31 may consider an object that is estimated to be the same individual as an object recognized in the object recognition process for the past image DinP as a target subject for motion detection. Note that the object in the past image DinP can be determined based on the object recognition result at the time when the moire detection unit 31 previously treated the frame as the current image DinC, for example.

[0067] For example, if objects A, B, and C are recognized in the past image DinP, and objects A and B are recognized in the current image DinC in the object recognition process, then objects A and B are set as target subjects for motion detection. In this way, in step S110, one or more objects are set as target subjects based on the object recognition process for the image.

[0068] However, there are cases where a target subject cannot be set. For example, there is no object that is determined to be a common individual between the object recognized in the current image DinC and the object recognized in the past image DinP. In such cases, the moiré detection unit 31 proceeds from step S111 to step S114.

[0069] If one or more objects are set as target subjects in step S110, moire detection unit 31 proceeds from step S111 to step S112, and detects the movement of each of one or more target subjects. That is, for each target subject, the position within the frame in the previous image DinP is compared with the position within the frame in the current image DinC to detect movement. For example, the movement of each target subject is detected such that object A is moving to the left of the screen at a speed of "1," object B is moving to the top of the screen at a speed of "3," and so on. It should be noted that there may be cases where no motion is detected for a certain target subject.

[0070] In step S113, moire detection unit 31 detects pixel regions whose movement differs from that of the object, among the pixel regions of the object whose movement has been detected, from among the objects set as the target subjects. For example, within the pixel region of object A as the target subject, that is, within each pixel corresponding to the contour of the subject as object A, a portion where a movement different from the movement detected for object A occurs is detected. Specifically, when object A is detected as "moving to the left of the screen at a speed of 1," pixels that are not detected as "moving to the left of the screen at a speed of 1" are detected among the pixels in the pixel area recognized as object A. Such a pixel area or areas of one or more pixels are defined as pixel areas showing different movements.

[0071] Then, in step S114, the moire detection unit 31 generates detection information Sdt based on the detection of pixel regions that exhibit different movements. For example, if a pixel area showing different motions is detected, moiré presence / absence information indicating "moiré present" is generated as the detection information Sdt. If no pixel area showing different motions is detected, moiré presence / absence information indicating "moiré absent" is generated as the detection information Sdt. Alternatively, area information that identifies a pixel area showing different motion is generated as the detection information Sdt. If there is no pixel area showing different motion, area information indicating that no such area exists is generated. When the process proceeds from step S111 to step S114, moire presence / absence information indicating "no moire" is generated as the detection information Sdt, or area information indicating that no relevant area exists is generated.

[0072] The concept of the above processing will be explained with reference to FIGS. 8, it is assumed that an object 50 exists as a subject in the past image DinP and the current image DinC. It is assumed that a vertical striped pattern 51 (shown as stripes of shaded and non-shaded areas in the figure) is seen on the image of this object 50. By comparing the past image DinP and the current image DinC, movement in the image is detected. That is, movement of the object 50 to the left at a certain speed is detected between frames at different times. Now, if we look at the pattern 51 inside the object 50, we find that it is moving to the left at the same speed. In this case, we determine that this pattern 51 is not a moiré pattern, but an actual pattern on the object 50.

[0073] On the other hand, when comparing the past image DinP and the current image DinC in Figure 9, movement of the object 50 to the left at a certain speed is detected, but when examining the pattern 52 inside the object 50, it is found that the direction, speed, or both of the movement are not the same. In other words, the movements of the object 50 and the pattern 52 do not match. In such a case, the pattern 52 is determined to be a moire.

[0074] It should be noted that the object 50 may actually have a moving pattern. For example, FIG. 10 shows an object 55 as a subject, and it is assumed that no movement of this object 55 is detected by comparing the past image DinP with the current image DinC. However, suppose that some movement is detected in the pattern 53 inside the object 50. In this case, it is highly likely that the pattern 53 is actually moving.

[0075] Therefore, if the object selected as the target subject is determined to be "no motion" in steps S112 and S113 of FIG. 7, the pixel region inside the object may not be determined to be moiré even if there is motion. That is, in step S113, pixel regions exhibiting different movements are detected only for target subjects for which movement has been detected, thereby reducing erroneous detection of moire.

[0076] A second example of the moire detection process will be described with reference to FIG. The second example is an example in which, in a case where uniform motion information indicating that the subject in the image moves uniformly is obtained in advance, moire detection is performed using the results of overall motion detection without performing object recognition.

[0077] In step S121 of FIG. 11, the moire detection unit 31 checks whether there is prior information that all subjects move uniformly, that is, whether there is uniform movement information, and branches the process.

[0078] The prior information as uniform movement information may be, for example, a shooting mode setting made by the user, information on panning or tilting, or, in the case of processing image data Din captured in the past, information on the shooting mode or panning performed when capturing the image data Din.

[0079] For example, if a user sets a shooting mode suitable for capturing landscapes or still objects, it is expected that the subject will be stationary. In this case, the subject on the screen is expected to move uniformly, with changes corresponding to the movement of the imaging device 1. Similarly, when it is expected that the user will perform panning, such as when setting the camera to panoramic photography mode, the subject is expected to move uniformly. Furthermore, information indicating that a panning or tilting operation will be performed by a pan tilter or the like attached to the imaging device 1 can also be considered as a form of prior information indicating that a motionless subject will move within the image. However, since a motionless subject is not necessarily imaged, when there is information indicating that a panning or tilting operation will be performed, it can also be considered that the moire detection processing of the first example described above is basically applied. When a motionless subject is estimated or determined to be a motionless subject, information indicating that a panning or tilting operation will be performed serves as prior information indicating that all subjects will move uniformly.

[0080] For example, if there is no prior information that allows one to assume that all subjects move uniformly, as described above, the moiré detection unit 31 proceeds from step S121 to another process. For example, the first example of the process shown in Fig. 7 may be performed. Alternatively, it is also possible to not perform the moiré detection process.

[0081] On the other hand, if there is prior information as uniform motion information as described above, the moiré detection unit 31 proceeds to step S122, where it first sets feature points within the image. For example, in the current image DinC, one or more points where a clear edge is detected or a feature shape is selected as feature points.

[0082] In step S123, the moire detection unit 31 compares the positions of the feature points in the frames of the past image DinP and the current image DinC to detect the uniform movement (direction and speed) of the subject in the image.

[0083] In step S124, the moiré detection unit 31 compares the past image DinP with the current image DinC to detect pixels that exhibit movement that differs from the uniform movement described above. In this case, since each subject moves uniformly, the pixel region that is expected to exhibit the same movement as the subject is the entire frame. Therefore, among the pixels in the entire frame, pixels that exhibit movement that differs from the uniform movement are determined, and the region of such pixels is detected. In other words, parts that exhibit movement in a direction or at a different speed from the overall movement are detected locally.

[0084] Then, in step S125, the moiré detection unit 31 generates detection information Sdt (moiré presence / absence information, area information) based on the detection of pixel regions that exhibit motion different from uniform motion.

[0085] In this way, when the uniform movement of the subject is assumed, moire detection can be performed based on the difference in movement, as in the first example, without performing object recognition.

[0086] A third example of the moire detection process will be described with reference to FIG. The third example is an example in which, when the subject itself is not moving and information on the movement of a tripod or pan-tilter is obtained, or information on the movement of the imaging device 1 itself is obtained as IMU data from the sensor unit 23, the areas that do not match the movement are determined to be moire.

[0087] In step S131 of FIG. 12, the moire detection unit 31 checks whether uniform motion information indicating that all subjects move uniformly has been obtained as prior information, and branches the process accordingly. This is similar to step S121 in Figure 11, and the prior information as uniform movement information can be, for example, the shooting mode setting by the user, and can be thought of as information that the user has set a shooting mode suitable for shooting landscapes or still objects, information on the panoramic shooting mode, information on panning using a pan-tilt camera, etc. In this third example, it is assumed that the imaging device 1 is mounted on a pan-tilter or the like and is capable of detecting the direction and speed of its panning and tilting movements, or that the direction and speed of the movement of the imaging device 1 itself can be detected as IMU data from the sensor unit 23, for example.

[0088] If there is no prior information as uniform motion information as described above, the moiré detection unit 31 proceeds to other processing from step S131. For example, it is possible to perform the processing of the first example in Fig. 7, or to not perform the moiré detection processing.

[0089] On the other hand, if there is prior information as uniform motion information as described above, the moire detection unit 31 proceeds to step S132 and acquires motion information. For example, information on the shooting direction of the pan / tilt camera corresponding to each point in time between the past image DinP frame and the current image DinC frame, as well as IMU data corresponding to each frame, can be obtained. From this information, the uniform movement (direction and speed) of the entire subject can be detected.

[0090] In step S133, the moiré detection unit 31 compares the past image DinP with the current image DinC to detect pixels that exhibit movement different from the uniform movement described above. In this case, since each subject moves uniformly, the same as the movement of the pan / tilt camera or the image capture device 1, the pixel area in which the same movement as the subject is expected is the entire frame. Therefore, among the pixels in the entire frame, pixels that exhibit movement different from the uniform movement are determined, and the area of ​​such pixels is detected. In other words, parts where movement in a direction or speed different from the movement of the image capture device 1, such as panning, is observed locally.

[0091] Then, in step S134, the moiré detection unit 31 generates detection information Sdt (moiré presence / absence information, area information) based on the detection of pixel regions that exhibit motion different from uniform motion.

[0092] In this way, when it is assumed that the subject is not moving, the movement of the pan tilter or the image capture device 1 can be regarded as uniform movement of the subject, and portions of different movement can be detected as moire.

[0093] In the moiré detection processes such as those in the first, second, and third examples above, when pixel areas with different movements are present, they are detected as moiré, but the determination of whether the "movements" are different can be adjusted depending on various circumstances. For example, even within a recognized subject, it is possible that the movement of the subject as a whole may not strictly match the movement of that subject. For example, the overall movement of a person may differ slightly from the movement of individual parts of their clothing, or when a plant sways in the wind, the movement may differ slightly from a uniform one. It is not appropriate to include such slight differences in the determination of moiré.

[0094] Therefore, it is conceivable that the threshold value for the difference in direction or speed for determining "different movements" is set to a value that does not detect minute differences, or that it can be varied depending on the situation. For example, it is conceivable to change the threshold value depending on the type of recognized subject, or the speed of movement of the image capture device 1, etc.

[0095] <5. Example of moiré reduction processing> Next, we will explain specific examples (first example, second example, third example) of moiré reduction processing by the moiré reduction unit 32. Each example is an example of processing performed by the moiré reduction unit 32 that receives the detection information Sdt as shown in FIG.

[0096] A first example of moiré reduction processing is shown in Fig. 13. This is an example in which moiré presence / absence information is input as detection information Sdt.

[0097] In step S211, the moiré reduction unit 32 acquires moiré presence / absence information as the detection information Sdt. In step S212, the moiré presence / absence information is used to check whether moiré has occurred in the currently processed frame of image data Din. If no moiré has occurred, the moiré reduction process ends without doing anything with respect to the currently processed frame.

[0098] On the other hand, if it is confirmed that moire is occurring, the moire reduction unit 32 proceeds to step S213 and performs LPF processing on the entire image data Din currently being processed. This makes it possible to obtain image data Dout with less noticeable moire.

[0099] A second example of moire reduction processing is shown in Fig. 14. This is an example in which area information is input as detection information Sdt.

[0100] In step S221, the moiré reduction unit 32 acquires area information indicating a pixel area in which moiré is detected as the detection information Sdt. In step S222, the moiré presence / absence information is checked based on the area information to determine whether moiré has occurred in the currently processed frame of image data Din, i.e., whether one or more pixel regions are indicated in the area information. If no moiré has occurred, the moiré reduction process is terminated without doing anything for the frame currently being processed.

[0101] On the other hand, if it is confirmed that moire is occurring, the moire reduction unit 32 proceeds to step S223 and performs LPF processing on the pixel region indicated by the area information. This makes it possible to obtain image data Dout in which moire has been reduced in areas where moire has occurred.

[0102] A third example of moiré reduction processing is shown in Fig. 15. This is also an example where area information is input as detection information Sdt, and it is an example where smoothing processing is performed to smooth out changes in perceived image resolution at the boundary between areas where LPF processing is performed and areas where it is not performed.

[0103] In step S231, the moiré reduction unit 32 acquires area information indicating a pixel area in which moiré is detected as the detection information Sdt. In step S222, the moiré presence / absence information is checked based on the area information to determine whether moiré has occurred in the currently processed frame of image data Din, i.e., whether one or more pixel regions are indicated in the area information. If no moiré has occurred, the moiré reduction process is terminated without doing anything for the frame currently being processed.

[0104] On the other hand, if it is confirmed that moire has occurred, the moire reduction unit 32 proceeds to step S223, and generates an LPF-processed image by performing LPF processing on the entire frame.

[0105] In step S234, the moiré reduction unit 32 sets a blending ratio for each pixel based on the area information. The blending ratio is the mixing ratio of pixel values ​​of the LPF-processed image and its original image (image that has not been subjected to LPF processing).

[0106] The blending ratio for each pixel is set, for example, as follows: It is assumed that the area AR1 indicated by the diagonal lines in FIG. 16 is an area indicated by the area information as having moire. Areas AR2, AR3, and AR4 are set around the periphery of this area AR1, and the rest is area AR5.

[0107] Then, for each area, the blending ratio between the LPF processed image and the original image is set as follows: AR1 100:0 AR2 75:25 AR3 50:50 AR4 25:75 AR5 0:100

[0108] Note that the number of areas divided into areas AR1 to AR5 and the blend ratio are merely examples for the purpose of explanation.

[0109] In step S235 of FIG. 15, the moiré reduction unit 32 combines the LPF-processed image and the original image at the above blend ratio in each of the areas AR1 to AR5.

[0110] For example, pixels in area AR1 are assigned to pixels from the LPF-processed image. The pixel values ​​of each pixel in area AR1 are combined so that the pixel values ​​of the LPF-processed image and the original image correspond at a ratio of 75:25. Areas AR3 and AR4 are also combined using the above blend ratio. Pixels from the original image are assigned to area AR5. The image data resulting from this synthesis is taken as image data Dout for which moire has been reduced.

[0111] By doing so, it is possible to make the difference in perceived resolution at the boundary between a pixel region that has undergone LPF processing and a pixel region that has not undergone LPF processing less noticeable.

[0112] Although the moiré reduction processes in the first, second, and third examples above use LPF processing, it is possible to adjust the frequency characteristics, for example by raising the cutoff frequency, to reduce only the moiré that has aliased to high frequencies. In this way, patterns will not disappear even if they do not match the movement of the subject in the low-frequency range, and the impact of incorrect moiré detection can be reduced in cases where the pattern is actually moving within the subject.

[0113] The cutoff frequency of the LPF processing may also be adjustable by the user. For example, it is conceivable that the user could check the image after moiré reduction processing while changing the cutoff frequency, allowing the user to adjust the image resolution and moiré reduction to their desired state.

[0114] Furthermore, while this method can be used to detect and reduce moiré that has aliased to high frequencies, other methods can be used to detect moiré that has aliased to low frequencies. For example, the difference between two images with different optical properties can be detected as moiré, and that part can be reduced using LPF processing.

[0115] <6. Summary and Variations> According to the above embodiment, the following effects can be obtained. The image processing unit 20 of the embodiment includes a moire detection unit 31 that detects pixel areas that exhibit different movement from pixel areas that are expected to exhibit the same movement as a subject that exhibits movement as a change in position within a frame between images at different times, and generates moire detection information Sdt. The movement of a subject within an image, i.e., a change in the position of the subject within a frame between images taken at different times, can be a change due to the movement of the subject itself or a change due to movement of the imaging device 1, such as panning. When a specific subject is moving, the pixel area within the outline of the subject is a pixel area that is expected to move in the same way as the subject. Also, when a stationary subject is being imaged and the entire subject is moving due to panning or tilting, the entire pixel area within the frame is a pixel area that is expected to move in the same way as the subject. Therefore, when a subject is moving, if a pixel area that should show the same movement as the subject shows a different movement, it can be detected as moiré, not as the subject's intended pattern, etc. In this way, by detecting moiré from the state of movement, that is, the state of change in position within a frame between images taken at different times, it is possible to detect moiré and distinguish it from real patterns, regardless of the moiré frequency.

[0116] The image processing unit 20 according to the embodiment further includes a moiré reduction unit 32 that performs moiré reduction processing based on the detection information Sdt. Based on the information detected from the state of movement, moiré is reduced using, for example, LPF processing, etc. This makes it possible to distinguish moiré from real patterns and reduce it regardless of the moiré frequency.

[0117] In the embodiment, an example has been given in which the moire detection unit 31 detects the movement of a target subject that is the target of the detection process based on the object recognition result in the image, detects pixel areas of the target subject that have movement different from the movement of the target subject, and generates detection information Sdt (first example of moire detection process: see Figure 7). By identifying subjects in an image using object recognition, one or more target subjects can be set, such as people, objects, or animals, and their movement can be detected. The pixel area of ​​each target subject should all exhibit movement (changes in position within the frame) similar to the movement of the contour of that target subject. Therefore, if a different movement is detected, it can be determined to be moiré.

[0118] In the embodiment, an example has been given in which the moire detection unit 31 detects the movement of feature points in an image given uniform movement information indicating that the entire subject in the image moves uniformly, detects pixel areas where there is movement different from the movement of the feature points, and generates detection information Sdt (second example of moire detection processing: see Figure 11). For example, if information such as the shooting mode selected by the user or information indicating that panning or tilting will be performed using a pan-tilter or the like attached to the imaging device 1 is provided as advance information, the moiré detection unit 31 can determine that uniform movement appears across the entire subject in the image. In this case, the direction and speed of the original movement caused by panning or the like can be detected as a change in the position of a certain feature point in the image between frames. If a pixel area exhibiting movement different from this original movement is detected, it can be determined to be moiré. Depending on the situation, it may be possible to selectively use either the process of detecting pixel regions whose movement differs from the movement of the target subject based on object recognition as shown in Figure 7, or the process of detecting pixel regions whose movement differs from the movement of feature points based on prior information as shown in Figure 11.

[0119] In the embodiment, an example has been given in which the moire detection unit 31 detects pixel areas in an image that have uniform motion information indicating that the entire subject in the image moves uniformly, and that have motion that differs from the motion information indicating the motion of the imaging device at the time of imaging, and generates detection information Sdt (third example of moire detection processing: see Figure 12). For example, the movement of the imaging device 1 during image capture is indicated by inputting information on the direction and speed of movement from a pan / tilt camera or by IMU data from the sensor unit 23. If the moiré detection unit 31 can determine from prior information that uniform movement appears for the entire subject in the image, it should detect movement for the entire subject that matches the movement information indicating the movement of the imaging device 1. In this case, if a pixel area is detected that shows movement different from the movement information, it can be determined to be moiré. Depending on the situation, it may be possible to selectively use a process that detects pixel areas whose movement differs from the movement of the target subject based on object recognition as shown in Figure 7, a process that detects pixel areas whose movement differs from the movement of feature points based on prior information as shown in Figure 11, or a process that obtains information on the movement of the entire subject and detects pixel areas whose movement differs as shown in Figure 12.

[0120] In the embodiment, it has been described that the detection information Sdt may include information on the presence or absence of moire. By including at least moiré presence / absence information as the detection information Sdt, the moiré reduction unit 32 can perform moiré reduction processing only on frames in which moiré is detected. By not performing moiré reduction processing on images in which no moiré occurs, it is possible to prevent unnecessary loss of image resolution.

[0121] In the embodiment, it has been described that the detection information Sdt may include area information indicating a pixel area in which moire is detected. By including area information as the detection information Sdt, it is possible to perform moiré reduction processing only on pixel areas where moiré is detected in the moiré reduction unit 32. This prevents moiré reduction processing from being performed on image areas where no moiré occurs, making it possible to reduce only moiré without impairing the perceived resolution of the image.

[0122] In the embodiment, an example has been described in which a control unit such as the camera control unit 18 of the imaging device 1 or the CPU 71 of the information processing device 70 associates the detection information Sdt with an image as metadata corresponding to the image (see FIGS. 1, 2, and 4). The camera control unit 18 of the imaging device 1 associates the detection information Sdt detected for each frame by the moiré detection unit 31 with the image frame as metadata, records the information on a recording medium, or transmits it to an external device, allowing devices other than the imaging device 1 to perform moiré reduction processing using the detection information Sdt. This allows for effective use of the moiré detection results based on a comparison of motion. Even if such processing is performed by the CPU 71 of the information processing device 70, subsequent processing by the information processing device 70 or processing by another device can still perform moiré reduction processing using the detection information Sdt.

[0123] In the embodiment, the moire reduction unit 32 performs moire reduction processing by LPF processing on an image (see the first example (FIG. 13), the second example (FIG. 14), and the third example (FIG. 15) of the moire reduction processing). The moire reduction unit 32 is formed by an LPF, and performs LPF processing on the current image (image data Din) based on the detection information Sdt as shown in FIG. 3, thereby performing moire reduction when necessary.

[0124] In the embodiment, an example has been given in which the moiré reduction unit 32 performs moiré reduction processing by LPF processing on the pixel area indicated by the area information based on the detection information Sdt including area information indicating the pixel area where moiré is detected (second example of moiré reduction processing, see FIG. 14). For example, the LPF processing is performed only on the pixel area indicated by the area information. When area information is supplied as the detection information Sdt, the moiré reduction unit 32 can perform LPF processing only on pixel areas where moiré occurs. This allows moiré reduction to be achieved without performing LPF processing on areas other than moiré, without impairing the perceived resolution in areas where moiré does not occur.

[0125] In the embodiment, an example is given in which the moiré reduction unit 32 performs moiré reduction processing by LPF processing on the pixel area indicated by the area information based on the detection information Sdt including area information indicating the pixel area in which moiré is detected, and also performs smoothing processing on the area surrounding the pixel area indicated by the area information, gradually changing the degree of reflection of the LPF processing (third example of moiré reduction processing, see Figures 15 and 16). When area information is supplied as the detection information Sdt, if LPF processing is performed on the pixel area indicated by the area information but not on other areas, the smoothness of the image may be lost at the boundaries of those pixel areas. Therefore, smoothing processing as described in Figures 15 and 16 is performed. This prevents pixel areas where moiré is detected from looking unnatural.

[0126] In the embodiment, an example has been given in which the moiré reduction unit 32 performs moiré reduction processing by LPF processing on the entire image based on the detection information Sdt including moiré presence / absence information (first example of moiré reduction processing, see FIG. 13). When moiré presence / absence information is supplied as the detection information Sdt, the moiré reduction unit 32 can reduce moiré by performing LPF processing on the entire image in which moiré occurs. In other words, it is possible to avoid performing LPF processing on images in which moiré does not occur.

[0127] The moiré reduction unit 32 may switch the LPF processing depending on the content of the detection information Sdt. For example, if the detection information Sdt contains only information on the presence or absence of moiré, the LPF processing may be performed on the entire image, and if the detection information Sdt contains area information, the LPF processing may be performed on the pixel region indicated by the area information.

[0128] In the embodiment, an example has been described in which the moiré reduction unit 32 performs moiré reduction processing by LPF processing on an image, and variably sets the cutoff frequency of the LPF processing. For example, by changing the cutoff frequency of the LPF processing depending on the user's operation or the situation, it becomes possible to perform moiré reduction processing according to the user's thoughts and the situation. For example, when the subject is stationary or the imaging device 1 is moving, the cutoff frequency may be lowered.

[0129] The program of the embodiment is a program that causes a processing device such as a CPU, DSP, GPU, GPGPU, or AI processor, or a device including these, to execute the moire detection processing shown in Figures 4, 7, 11, and 12. In other words, the program of the embodiment is a program that causes a processing device to execute a process of detecting pixel areas that exhibit different movement from pixel areas that are expected to exhibit the same movement as a subject that exhibits movement as a change in position within a frame between images taken at different times, and generating moire detection information Sdt. Such a program allows the image processing device referred to in the present disclosure to be realized by various computer devices.

[0130] This program may further be a program that causes the moire reduction processing shown in Figures 5, 13, 14, and 15 to be executed by, for example, a CPU, DSP, GPU, GPGPU, AI processor, or a device including these.

[0131] These programs can be recorded in advance on a HDD as a recording medium built into a device such as a computer, or on a ROM in a microcomputer having a CPU. Alternatively, the program can be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, a memory card, etc. Such removable recording media can be provided as so-called packaged software. Such a program can be installed onto a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0132] Furthermore, such a program is suitable for providing the image processing device of the present disclosure to a wide range of devices. For example, by downloading the program to a mobile terminal device such as a smartphone or tablet, a mobile phone, a personal computer, a game device, a video device, a PDA (Personal Digital Assistant), or the like, these devices can function as the image processing device of the present disclosure.

[0133] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0134] The present technology can also be configured as follows. (1) A moire detection unit is provided which detects pixel areas where there is different movement from pixel areas where the same movement as that of a subject occurring as a change in position within a frame between images taken at different times is expected, and generates moire detection information. Image processing device. (2) The apparatus further includes a moiré reduction unit that performs moiré reduction processing based on the detection information. The image processing device according to (1) above. (3) The moire detection unit Detecting the movement of a target subject that is the target of detection processing based on the object recognition result in the image; Detecting a pixel area having a movement different from the movement of the target subject from among the pixel areas of the target subject, and generating the detection information. The image processing device according to (1) or (2) above. (4) The moire detection unit For an image given with uniform motion information indicating that the entire subject in the image moves uniformly, a pixel area having a motion different from the motion of the feature points is detected, and the detection information is generated. The image processing device according to any one of (1) to (3) above. (5) The moire detection unit For an image given with uniform motion information indicating that the entire subject in the image moves uniformly, a pixel area in which a motion different from the motion information indicating the motion of the imaging device at the time of image capture appears is detected, and the detection information is generated. The image processing device according to any one of (1) to (4) above. (6) The detection information includes information on whether or not moire is present. The image processing device according to any one of (1) to (5) above. (7) The detection information includes area information indicating a pixel area in which moire is detected. The image processing device according to any one of (1) to (6) above. (8) a control unit that associates the detected information with the image as metadata corresponding to the image; An image processing device according to any one of (1) to (7) above. (9) The moiré reduction unit The moire reduction process is performed by low-pass filtering the image. The image processing device according to (2) above. (10) The moiré reduction unit Based on the detection information including area information indicating a pixel area where moire is detected, The moire reduction process is performed by low-pass filtering the pixel area indicated by the area information. The image processing device according to (2) or (9) above. (11) The moiré reduction unit Based on the detection information including area information indicating a pixel area where moire is detected, The moire reduction process is performed by low-pass filtering the pixel area indicated by the area information, and A smoothing process is performed on the area around the pixel area indicated by the area information, gradually changing the degree of reflection of the low-pass filter process. The image processing device according to any one of (2), (9), and (10) above. (12) The moiré reduction unit Based on the detection information including information on whether or not moiré is present, The moire reduction process is performed by low-pass filtering the entire image. The image processing device according to any one of (2), (9), (10), and (11) above. (13) The moiré reduction unit The moire reduction process is performed by low-pass filtering the image, and Variable setting of cutoff frequency for low-pass filter processing The image processing device according to any one of (2), (9), (10), (11), and (12) above. (14) Among pixel areas where the same movement as that of a subject occurring as a change in position within a frame between images taken at different times is expected, pixel areas where different movement occurs are detected, and moiré detection information is generated. Image processing methods. (15) Among pixel areas where the same movement as that of a subject occurring as a change in position within a frame between images taken at different times is expected, pixel areas where different movement is detected are detected, and moire detection information is generated. A program executed by a processing unit. [Explanation of symbols]

[0135] 1. Imaging device 11 Lens system 12 Image sensor section 12a Image sensor 18 Camera control unit 20 Image processing section 21 Buffer memory 30 memory 31 Moire detection unit 32 Moiré reduction section 70 Information processing equipment 71 CPU

Claims

1. A moire detection unit is provided which detects pixel areas where there is different movement from pixel areas where the same movement as that of a subject occurring as a change in position within a frame between images taken at different times is expected, and generates moire detection information. Image processing device.

2. The apparatus further includes a moiré reduction unit that performs moiré reduction processing based on the detection information. The image processing device according to claim 1 .

3. The moire detection unit Detecting the movement of a target subject that is the target of detection processing based on the object recognition result in the image; Detecting a pixel area having a movement different from the movement of the target subject from among the pixel areas of the target subject, and generating the detection information. The image processing device according to claim 1 .

4. The moire detection unit For an image given with uniform motion information indicating that the entire subject in the image moves uniformly, a pixel area where a motion different from the motion of the feature points is detected, and the detection information is generated. The image processing device according to claim 1 .

5. The moire detection unit For an image given with uniform motion information indicating that the entire subject in the image moves uniformly, a pixel area in which a motion different from the motion information indicating the motion of the imaging device at the time of image capture appears is detected, and the detection information is generated. The image processing device according to claim 1 .

6. The detection information includes information on whether or not moire is present. The image processing device according to claim 1 .

7. The detection information includes area information indicating a pixel area in which moire is detected. The image processing device according to claim 1 .

8. a control unit that associates the detected information with the image as metadata corresponding to the image; The image processing device according to claim 1 .

9. The moiré reduction unit The moire reduction process is performed by low-pass filtering the image. The image processing device according to claim 2 .

10. The moiré reduction unit Based on the detection information including area information indicating a pixel area where moire is detected, The moire reduction process is performed by low-pass filtering the pixel area indicated by the area information. The image processing device according to claim 2 .

11. The moiré reduction unit Based on the detection information including area information indicating a pixel area where moire is detected, The moire reduction process is performed by low-pass filtering the pixel area indicated by the area information, and A smoothing process is performed on the area around the pixel area indicated by the area information, gradually changing the degree of reflection of the low-pass filter process. The image processing device according to claim 2 .

12. The moiré reduction unit Based on the detection information including information on whether or not moiré is present, The moire reduction process is performed by low-pass filtering the entire image. The image processing device according to claim 2 .

13. The moiré reduction unit The moire reduction process is performed by low-pass filtering the image, and Variable setting of cutoff frequency for low-pass filter processing The image processing device according to claim 2 .

14. Among pixel areas where the same movement as that of a subject occurring as a change in position within a frame between images taken at different times is expected, pixel areas with different movement are detected, and moiré detection information is generated. Image processing methods.

15. Among pixel areas where the same movement as that of a subject occurring as a change in position within a frame between images taken at different times is expected, pixel areas where different movement is detected are detected, and moire detection information is generated. A program executed by a processing unit.

Images