Processing apparatus, imaging system, processing method, storage medium, and computer program product

Abstract

The invention discloses a processing apparatus, an imaging system, a processing method, a storage medium, and a computer program product. The processing apparatus is configured to cause the display apparatus to display a first image and acquire, from the imaging apparatus, a second image generated by photographing at a viewing angle including a display area of the display apparatus. The processing apparatus includes: an acquisition unit configured to acquire a distinguishing result for distinguishing between a region in which a subject existing between the display region and the imaging apparatus has been photographed and a display region in the second image; a detector configured to detect a change in the second image by comparing the first image with an image of a display area in the second image; and a control unit configured to perform first control for notifying information related to the detection result.

Description

Technical Field

[0001] The present disclosure relates to a processing device, a camera system, a processing method, and a storage medium. Background Art

[0002] Recently, virtual production in which a background image is displayed on an LED display (LED wall) and simultaneously photographed together with a real subject has rapidly become popular. Any minute moiré patterns (interference fringes) on the LED display may not be noticed during photography, but may be noticed during post-production and result in re-photography, so moiré patterns may be detected during photography. Japanese Patent Application Laid-Open No. 2008-011334 discloses a method for converting a spatial-axis signal representing an image including moiré patterns into a frequency-axis signal, and then converting the frequency-axis signal from which frequency components corresponding to the moiré patterns have been removed into a spatial-axis signal.

[0003] The method disclosed in Japanese Patent Application Laid-Open No. 2008-011334 uses a plurality of images taken from different photographing angles to identify moiré patterns, and thus prepares a camera for moiré detection separate from the camera used for photography. Summary of the Invention

[0004] A processing device according to an aspect of the present disclosure is configured to cause a display device to display a first image, and acquire a second image generated by photographing from a perspective including a display area of the display device from a photographing device. The processing device includes: an acquisition unit configured to acquire a discrimination result for discriminating between an area where a subject existing between the display area and the photographing device has been photographed and the display area in the second image; a detector configured to detect a change in the second image by comparing the first image with an image of the display area in the second image; and a control unit configured to perform first control for notifying information related to the detection result. A camera system having the above-described processing device also constitutes another aspect of the present disclosure. A processing method corresponding to the above-described processing device also constitutes another aspect of the present disclosure. A storage medium storing a program for causing a computer to execute the above-described processing method also constitutes another aspect of the present disclosure.

[0005] Other features of the present disclosure will become apparent from the following description of embodiments with reference to the accompanying drawings. Brief Description of the Drawings

[0006] Figure 1 is a block diagram of a photographing device.

[0007] Figure 2A and Figure 2B show a part of a light receiving surface of an image sensor.

[0008] Figure 3 This is a flowchart showing the distance information generation process.

[0009] Figure 4 This shows the correlation calculation results in the case where a pair of image signal sequences are highly correlated in an ideal state without noise.

[0010] Figure 5 This shows the correlation calculation results in a small block with noise.

[0011] Figure 6 This shows the imaging system according to the first embodiment.

[0012] Figure 7 This is a block diagram of an image signal processing device.

[0013] Figure 8 This is a flowchart showing the abnormal detection and removal process in the background image of the imaging system according to the first embodiment.

[0014] Figure 9 This shows the subject area and the background area in the image data.

[0015] Figure 10 This is a flowchart showing the abnormal detection and removal process in the background image of the imaging system according to the second embodiment. Detailed implementation

[0016] As used herein, the term "unit" may refer to a software context, a hardware context, or a combination of software and hardware contexts. In a software context, the term "unit" refers to a functionality, an application, a software module, a function, a routine, an instruction set, or a program that can be executed by a programmable processor (such as a microprocessor, a central processing unit (CPU), or a specially designed programmable device or controller). The memory contains instructions or programs that, when executed by the CPU, cause the CPU to perform operations corresponding to the unit or function. In a hardware context, the term "unit" refers to a hardware element, a circuit, an assembly, a physical structure, a system, a module, or a subsystem. According to a specific embodiment, the term "unit" may include mechanical, optical, or electronic components, or any combination thereof. The term "unit" may include active (e.g., transistors) or passive (e.g., capacitors) components. The term "unit" may include a semiconductor device having a substrate and other material layers, where the other material layers have various conductivity concentrations. The term "unit" may include a CPU or a programmable processor that can execute a program stored in the memory to perform a specific function. The term "unit" may include logic elements (e.g., AND, OR) implemented by transistor circuits or any other switching circuits. In a combination of software and hardware contexts, the term "unit" or "circuit" refers to any combination of the software and hardware contexts described above. Additionally, the terms "element", "assembly", "component", or "device" may also refer to a "circuit" that is integrated or not integrated with a packaging material.

[0017] Reference is now made to the accompanying drawings, and a detailed description of embodiments according to the present disclosure will be given. Corresponding elements in the respective drawings will be designated by the same reference numerals, and repeated descriptions thereof will be omitted.

[0018] First Embodiment

[0019] Figure 1 is a block diagram of a imaging device 100. The imaging device 100 can input, output, and record images. Each component connected to the internal bus 101 can communicate data with each other via the internal bus 101.

[0020] The lens unit 106 includes a lens unit including a zoom lens and a focus lens, an aperture mechanism, and a drive motor. The optical image passing through the lens unit 106 is received by the imaging unit 107. The imaging unit 107 uses a CCD or CMOS sensor, etc., and converts the optical signal into an electrical signal.

[0021] The CPU 102 controls each component in the imaging device 100 according to a program stored in the ROM 103, using the RAM 104 as a working memory. The ROM 103 is a non-volatile recording device and records programs for operating the CPU 102, various adjustment parameters, and the like. The RAM 104 is a volatile memory using semiconductor devices, which is generally slower than the frame memory 111 and has a smaller capacity than the frame memory 111. The frame memory 111 is a device that temporarily stores image signals and can read out image signals as needed. Since image signals are a large amount of data, the frame memory 111 will have a high bandwidth and a large capacity. Double Data Rate 4 Synchronous Dynamic RAM (DDR4-SDRAM) or the like has recently been used as the frame memory 111. Using the frame memory 111, processes such as synthesizing images that are different over time and only cutting out a desired area can be performed.

[0022] The image processing unit 105 performs various image processes on data from the imaging unit 107 or image data stored in the frame memory 111 or the recording medium 112 under the control of the CPU 102. The image processes performed by the image processing unit 105 include pixel interpolation of image data, encoding process, compression process, decoding process, enlargement / reduction process (resize), noise reduction process, color conversion process, and the like. The image processing unit 105 performs processes such as correction of performance variations of pixels of the imaging unit 107, correction of defective pixels, white balance correction, brightness correction, and correction of distortion and peripheral light loss caused by lens characteristics. The image processing unit 105 performs a process for generating distance information related to the distance from the imaging device 100 to the subject to be imaged. The image processing unit 105 can be configured as a dedicated circuit block for performing specific image processes. Depending on the type of image process, the CPU 102 can perform image processes without using the image processing unit 105 according to a program.

[0023] The CPU 102 controls the lens unit 106 based on the calculation results obtained by the image processing unit 105, and performs optical image magnification, focal length adjustment, and adjustment of the aperture stop configured to adjust the amount of light. The CPU 102 can perform image stabilization by moving a part of the lens unit in a plane perpendicular to the optical axis.

[0024] The operation unit 113 is an interface with an external device that accepts user operations. The operation unit 113 can use mechanical elements such as buttons and switches, and includes a power switch, a mode switch, and the like.

[0025] The display unit 114 serves as a display device that can be visually confirmed by the user. When the display unit 114 displays, for example, an image processed by the image processing unit 105 and a setting menu, the user can check the operating status of the imaging apparatus 100. Small, low-power devices such as a liquid crystal display (LCD) and an organic electroluminescence (EL) have recently been used as the display unit 114. The display unit 114 may include a resistive or capacitive film device called a touch panel, and may be used as a substitute for the operation unit 113. The CPU 102 generates a character string for informing the user of the setting status of the imaging apparatus 100 and a menu for setting the imaging apparatus 100, and displays them on the display unit 114 by superimposing them on the image processed by the image processing unit 105. In addition to character information, the CPU 102 may superimpose other imaging auxiliary displays such as a histogram, a vectorscope, a waveform monitor, a zebra pattern, peaking, and a false color.

[0026] The image (or video) terminal unit 109 includes a plurality of image terminals. Examples of image terminals are a serial digital interface (SDI), a high-definition multimedia interface (HDMI) (registered trademark), and a display port (DisplayPort, registered trademark). Outputting an image signal via the image terminal unit 109 can display an image in real time on an external monitor not shown. The CPU 102 can process an image input via the image terminal unit 109 in the image processing unit 105 and display the processed image on the display unit 114.

[0027] The network module 108 is an interface for inputting and outputting image and audio signals. The network module 108 can communicate with external devices via the Internet, etc., and can also send and receive files, commands, and various types of data such as image signals and metadata. The transmission method used by the network module 108 can be wireless or wired.

[0028] The recording medium 112 can record image data and various setting data, and uses a large-capacity storage element. For example, a hard disk drive (HDD) or a solid-state drive (SSD) is used as the recording medium 112, and is attached to the recording medium interface (I / F) 110. When the user presses a predetermined button (hereinafter referred to as a record button) on the operation unit 113, the CPU 102 starts recording the image data processed by the image processing unit 105 into the recording medium 112 via the recording medium I / F 110. When the user presses the record button again, the recording is stopped. During recording, a signal indicating that the camera device 100 is recording is output via the network module 108 and the image terminal unit 109. This configuration can notify an external device that recording is in progress, or record the image data output from the camera device 100 in an external device in conjunction with the recording operation on the camera device 100.

[0029] The subject detector 115 uses artificial intelligence such as deep learning using a neural network to detect a subject. In the case of subject detection using deep learning, the CPU 102 transfers programs for processing, network structures, weight parameters, etc. stored in the ROM 103 to the subject detector 115. The subject detector 115 performs processing based on various parameters acquired from the CPU 102 to detect a subject from the image signal, and loads the processing result into the RAM 104.

[0030] The pose (or orientation) detector 116 detects the pose state of the imaging device 100 using, for example, a gyro sensor or an acceleration sensor. This configuration can detect whether the camera is tilted or shaking.

[0031] Figure 2A and Figure 2B FIG. shows a part of the light receiving surface of the image sensor of the imaging unit 107. In the imaging unit 107, in order to achieve on-chip phase difference autofocus (AF), pixel units each having two photodiodes as photoelectric converters (as light receiving units) are arranged in an array for each micro lens. This configuration enables each pixel unit to receive a light beam split by the exit pupil of the lens unit 106.

[0032] Figure 2A is a schematic diagram of a part of the image sensor surface in an example Bayer array of red (R), blue (B), and green (Gb, Gr). Figure 2B FIG. shows Figure 2A example pixel units each having two photodiodes for each micro lens corresponding to the color filter array in.

[0033] The image sensor according to the present embodiment can output two signals for phase difference detection (hereinafter also referred to as A image signal and B image signal) from each pixel unit. The image sensor can also output an imaging signal (A image signal + B image signal) obtained by adding the signals of the two photodiodes. In the case of outputting the added signal, an output equivalent to that of the image sensor of the example Bayer array shown in Figure 2A is output.

[0034] The imaging unit 107 can output a phase difference detection signal for each pixel unit, but can also output a value obtained by adding and averaging the phase difference detection signals of a plurality of adjacent pixel units. Outputting the added and averaged value can reduce the time for reading signals from the imaging unit 107 and the bandwidth used by the internal bus 101. Using such an output signal from the imaging unit 107, the CPU 102 performs correlation calculation of two image signals, and calculates information such as a defocus amount, parallax information, and various types of reliability.

[0035] The defocus amount on the image plane is calculated based on the shift between the A image signal and the B image signal. The defocus amount has a positive or negative value, and whether the defocus amount is positive or negative determines whether the state is a front focus state or a rear focus state. The absolute value of the defocus amount provides information related to the degree of focus (degree of focus shift), and a defocus amount of 0 indicates a focused state. That is, the CPU 102 calculates information related to whether the image is in a front focus or a rear focus based on the positive or negative sign of the defocus amount. The CPU 102 also calculates focus degree information based on the absolute value of the defocus amount. When the defocus amount exceeds a predetermined value, information related to whether the image is in a front focus or a rear focus is output, and when the absolute value of the defocus amount is within the predetermined value, information indicating that the image is in focus is output.

[0036] The CPU 102 controls the lens unit 106 according to the defocus amount for focusing. The CPU 102 also calculates the distance to the subject using the triangulation principle based on the parallax information and the lens information related to the lens unit 106.

[0037] Although Figure 2A and Figure 2B show an example in which pixel units each having two photodiodes are arranged in an array for a single microlens, the present disclosure is not limited to this example. Pixel units each having three or more photodiodes can be arranged in an array for a single microlens. A plurality of pixel units having different opening positions of the light receiving unit with respect to the microlens can be provided. In other words, as long as two signals such as the A image signal and the B image signal that can detect the phase difference can be obtained, configurations other than Figure 2A and Figure 2B the configuration shown can be used.

[0038] A description of the distance information generation process performed by the image processing unit 105 will now be given. Figure 3 is a flowchart showing the distance information generation process.

[0039] In step S301, the image processing unit 105 calculates the B image signal for phase difference detection by calculating the difference between the two signals: the (A image signal + B image signal) for imaging output from the imaging unit 107 and the A image signal for phase difference detection. This embodiment uses the method of outputting the (A image signal + B image signal) for imaging and the A image signal for phase difference detection, but the present disclosure is not limited to this example. In this state, the A image signal and the B image signal can be output from the imaging unit 107. In this case, the (A image signal + B image signal) for imaging can be calculated by adding the A image signal and the B image signal. In the case of providing two sensors such as a stereo camera, the image signals output from the respective image sensors can be the A image signal and the B image signal.

[0040] In step S302, the image processing unit 105 corrects the shading caused by optical factors for each of the A image signal for phase difference detection and the B image signal for phase difference detection.

[0041] In step S303, the image processing unit 105 performs a filtering process on each of the A image signal for phase difference detection and the B image signal for phase difference detection. For example, the filtering process can be performed by a high-pass filter including a finite impulse response (FIR) filter. In this embodiment, the A image signal for phase difference detection and the B image signal for phase difference detection pass through the high-pass filter, but the present disclosure is not limited to this example. The signal can be generated by a band-pass filter or a low-pass filter having different filtering coefficients. Then, the generated A image signal for phase difference detection and the generated B image signal for phase difference detection can be used for the correlation calculation process described later.

[0042] In step S304, the image processing unit 105 performs a correlation calculation using the small blocks obtained by dividing the A image signal for phase difference detection and the B image signal for phase difference detection that have been filtered in step S303. The size or shape of the small blocks is not limited, and adjacent blocks can overlap each other.

[0043] Now, a description of the correlation calculation of a pair of images (A image and B image) will be given. The signal sequence of the A image at the target pixel position is represented as E(1) to E(m), and the signal sequence of the B image at the target pixel position is represented as F(1) to F(m). When the signal sequence F(1) to F(m) of the B image is shifted relative to the signal sequence E(1) to E(m) of the A image, the correlation quantity C(k) at the shift amount k between the two signal sequences is calculated using the following equation (1).

[0044] C(k) = ∑|E(n) - F(n + k)| (1)

[0045] In Equation (1), the Σ operation means an operation for calculating the sum with respect to n. In the Σ operation, the ranges of n and n + k are restricted to the range from 1 to m. The offset k is an integer value and is a relative pixel offset in units of the detection pitch of a pair of data. Hereinafter, the k for which the discrete correlation quantity C(k) is minimum is denoted as kj. In an ideal state without noise, Figure 4 shows the calculation result of Equation (1) in the case where the correlation between a pair of image signal strings is high. As Figure 4 shown, the correlation quantity C(k) is minimum at the offset (k = kj = 0) where the correlation between a pair of image signal strings is high. The point interpolation processing shown in Equations (2) to (4) calculates x that gives the minimum value C(x) for the continuous correlation quantity. The pixel offset x is a real value and the unit is pixel.

[0046]

[0047] SLOP = MAX{C(kj + 1) - C(kj), C(kj - 1) - C(kj)} (4)

[0048] SLOP in Equation (4) represents the slope of the change between the minimum and local minimum correlation quantities and the adjacent correlation quantities. In Figure 4 , as a specific example, C(kj) = C(0) = 1000, C(kj - 1) = C(-1) = 1700, C(kj + 1) = C(1) = 1830. kj = 0. According to Equations (2) to (4), SLOP = 830, x = -0.078 [pixel]. In the in-focus state, the ideal pixel offset x between the signal sequence of Image A and the signal sequence of Image B is 0.00.

[0049] Figure 5 shows the calculation result in the case where Equation (1) is applied to a small block containing noise. As Figure 5 shown, due to the influence of randomly distributed noise, the correlation between the signal sequence of Image A and the signal sequence of Image B is reduced. The minimum value of the correlation quantity C(k) is greater than Figure 4 the minimum value in, and the correlation quantity curve has an overall flat shape (or the absolute difference between the maximum value and the minimum value is small).

[0050] In Figure 5 , as a specific example, C(kj) = C(0) = 1300, C(kj - 1) = C(-1) = 1480, C(kj + 1) = C(1) = 1800. kj = 0. According to Equations (2) to (4), SLOP = 500, x = -0.32 [pixel]. Compared with Figure 4Compared with the calculation result in the state without noise shown, the pixel offset x is far from the ideal value.

[0051] When the correlation between a pair of image signal strings is low, the change amount of the correlation quantity C(k) is small, and the correlation quantity curve is usually flat, so the SLOP value is small. Similarly, when the subject image has low contrast, the correlation between a pair of image signal strings is low, and the correlation quantity curve has a flat shape. Based on this characteristic, the reliability of the calculated pixel offset x can be determined by the SLOP value. That is, when the SLOP value is large, it can be judged that the correlation between a pair of image signal strings is high, and when the SLOP value is small, it can be judged that no significant correlation is obtained between a pair of image signal strings. In this embodiment, the correlation calculation is performed using Equation (1), so at the offset where the correlation between a pair of image signal strings is the highest, the correlation quantity C(k) is the smallest and locally minimum. As another method, a correlation calculation method in which the correlation quantity C(k) is the largest and locally maximum at the offset where the correlation between a pair of image signal strings is the highest can be used.

[0052] In step S305, the image processing unit 105 calculates the reliability. The reliability can be defined by the values of C(kj) and SLOP indicating the degree of agreement between the two images calculated in step S304 as described above.

[0053] In step S306, the image processing unit 105 performs interpolation processing. The pixel offset calculated in step S304 may not be usable because the reliability calculated in step S305 is low. In this case, it is not necessary to perform interpolation using the pixel offsets calculated around it. The interpolation method can apply a median filter or reduce the pixel offset data and then enlarge it again. Color data can be extracted from (A image signal + B image signal) for imaging, and the color data can be used to interpolate the pixel offset.

[0054] In step S307, the image processing unit 105 calculates the defocus amount by referring to the pixel offset x calculated in step S304. More specifically, the defocus amount (denoted as DEF) can be obtained by the following Equation (5):

[0055] DEF = P×x (5)

[0056] In Equation (5), P is a conversion coefficient determined by the detection pitch (pixel arrangement pitch) and the distance between the projection centers of the left and right viewpoints in a pair of parallax images, and is expressed in mm / pixel.

[0057] In step S308, the image processing unit 105 calculates the distance based on the defocus amount calculated in step S307. In the case where Da is the distance to the subject, Db is the focal position, and F is the focal length, the following equation (6) approximately holds:

[0058]

[0059] Therefore, the distance Da to the subject is represented by the following equation (7):

[0060]

[0061] In the case where Db at DEF = 0 is Db0, the absolute distance Da' to the subject is represented by the following equation (8):

[0062]

[0063] The relative distance Da - Da' is represented by the following equation (9) according to equations (7) and (8):

[0064]

[0065] As described above, by processing according to the Figure 3 flow in, it is possible to calculate the pixel offset amount, defocus amount, and distance information based on the A image signal for phase difference detection and the B image signal for phase difference detection.

[0066] Figure 6 FIG. shows a camera system according to the present embodiment. The camera system includes a camera device 100, an image signal processing device 700, and a display device 300. The camera device 100, the image signal processing device 700, and the display device 300 are connected by wire or wirelessly. The display device 300 displays an image signal (first image or video) input via an image input terminal (not shown). A coordinate detection device 601 is attached to the camera device 100, detects the position and orientation of the camera device 100 by referring to marks on the ceiling, floor, etc. and by emitting infrared rays, etc. and detecting the reflected light, and transmits the detected information to the image signal processing device 700. The camera device 100 captures an image (second image or video) of a display area that displays the first image of the display device 300.

[0067] Figure 7 is a block diagram of the image signal processing device 700. The image signal processing device 700 can input, output, and record images. Each component connected to the internal bus 701 can communicate data with each other via the internal bus 701.

[0068] The CPU 702 controls each component of the image signal processing device 700 according to a program stored in the ROM 703, using the RAM 704 as a working memory. The ROM 703 is a non-volatile storage device that stores programs for operating the CPU 702 and various adjustment parameters. The RAM 704 is a volatile memory using semiconductor devices, and is generally slower than the frame memory 709 and has a smaller capacity than the frame memory 709. The frame memory 709 is a device that temporarily stores image signals and can read out image signals as needed. Since image signals are a large amount of data, the frame memory 709 will have a high bandwidth and a large capacity. Recently, DDR4-SDRAM etc. are used as the frame memory 709. Using the frame memory 709, processes such as synthesizing images that are different over time and only cutting out a required area can be performed.

[0069] Under the control of the CPU 702, the image processing unit 705 performs various image processing on the image data stored in the frame memory 709 or the recording medium 712. The image processing unit 705 may include dedicated circuit blocks for performing specific image processing. Depending on the type of image processing, the CPU 702 can perform image processing according to a program without using the image processing unit 705.

[0070] The operation unit 710 is an interface with an external device that accepts user operations. The operation unit 710 includes a mouse, a keyboard, a touch panel, etc.

[0071] The display unit 711 serves as a display device that can be visually confirmed by the user. When the display unit 711 displays, for example, an image processed by the image processing unit 705, a setting menu, etc., the user can check the operation status of the image signal processing device 700. Recently, small and low-power devices such as LCDs and organic ELs have been used as the display unit 711. The display unit 711 may also include a resistive film type or capacitive film element called a touch panel and can serve as an alternative to the operation unit 710. The CPU 702 generates a string for informing the user of the setting status of the image signal processing device 700 and a menu for setting the image signal processing device 700, and displays them on the display unit 711 by superimposing them on the image processed by the image processing unit 705.

[0072] The image terminal unit 707 includes a plurality of image terminals. Examples of image terminals include SDI, HDMI, and DisplayPort. Outputting an image signal via the image terminal unit 707 can display an image in real time on an external monitor (not shown). The CPU 702 can process an image input via the image terminal unit 707 in the image processing unit 705 and display the processed image on the display unit 711.

[0073] The network module 706 is an interface for inputting and outputting image signals and audio signals. The network module 706 can communicate with an external device via the Internet or the like, and can also send and receive files, commands, and various types of data such as image signals and metadata. The transmission method used by the network module 706 can be wireless or wired.

[0074] The recording medium 712 can record image data and various setting data and uses a mass storage element. For example, an HDD or an SSD is used as the recording medium 712 and is attached to the recording medium I / F 708. When the user presses a predetermined button (hereinafter referred to as a recording button) on the operation unit 710, the CPU 702 starts recording the image data processed by the image processing unit 705 or the image data input from the image terminal unit 707 into the recording medium 712 via the recording medium I / F 708. When the user presses the recording button again, the recording is stopped. The CPU 702 can also record the image data into the recording medium 712 according to an in-progress recording indication signal received via the network module 706 or the image terminal unit 707.

[0075] The image signal processing device 700 also acquires lens information such as the focal length of the imaging device 100 and information such as exposure via the network module 706 and the image terminal unit 707. The CPU 702 reads out the three-dimensional model data recorded on the recording medium I / F 708. Next, the CPU 702 re-renders the three-dimensional model data in the image processing unit 705 based on the lens information, exposure, etc. of the imaging device 100 and the position and pose information related to the imaging device 100 obtained from the coordinate detection device 601. Then, the CPU 702 generates a CG image suitable for the viewing angle of the imaging device 100 and outputs it as a background image to the display device 300.

[0076] Now referring to Figure 8 , a description of the abnormality detection and removal processing of the background image according to the present embodiment will be given. Figure 8 is a flowchart showing the abnormality detection and removal processing of the background image of the imaging system according to the present embodiment.

[0077] In step S801, the CPU 702 first reads out the three-dimensional model data recorded in the recording medium I / F 708. Next, based on the lens information of the imaging device 100, information related to exposure, etc., and the position and orientation information related to the imaging device 100 obtained from the coordinate detection device 601, the CPU 702 re-renders the three-dimensional model data in the image processing unit 705. Then, the CPU 702 generates a CG image that matches the viewing angle of the imaging device 100 and outputs it as a background image (first image) to the display device 300. The background image is displayed on the display device 300.

[0078] In step S802, the CPU 102 acquires the image data (second image) including the subject and the background image that has been captured by the imaging unit 107 and processed by the image processing unit 105, as well as the distance information, and stores them in the frame memory 111.

[0079] In step S803, the CPU 102 first reads out the image data and the distance information from the frame memory 111. Next, as Figure 9 shown, the CPU 102 determines the subject area including the subject and the background area (display area) including the background image in the image data according to the distance information. Next, the CPU 102 adds flags indicating the subject area and the background area to the image data and stores the image data in the frame memory 111. The method of distinguishing between the subject area and the background area is: setting a threshold for the distance information, determining that the area is the background area when the distance is equal to or greater than the threshold, and determining that the area is the subject area when the distance is less than the threshold. The threshold can be set manually by the user or can be automatically set based on the aperture value (F value), focal position, etc. of the lens unit 106. Then, the CPU 102 outputs the flags indicating the subject area and the background area, as well as the image data, to the image signal processing device 700 via the image terminal unit 109 and the network module 108.

[0080] In this embodiment, the CPU 102 distinguishes between the subject area and the background area, but the CPU 702 can acquire the image data and the distance information from the imaging device 100 and distinguish between the subject area and the background area.

[0081] In step S804, the CPU 702 acquires, via the image terminal unit 707 and the network module 706, the flags indicating the subject area and the background area output by the imaging device 100, as well as the image data. That is, as a result of distinguishing between the subject area and the background area, the CPU 702 functions as an acquisition unit for acquiring the flags indicating the subject area and the background area. When the resolution or the imaging angle of the background image output in step S801 is different from the resolution or the imaging angle of the image data output by the imaging device in step S803, the CPU 702 performs a resizing process or an angle correction process on the background image to match the resolution and the imaging angle. Then, the CPU 702 compares the image data of the portion corresponding to the background area. In the present embodiment, the CPU 702 functions as a detector configured to compare the image data of the portion corresponding to the background area of the image and detect a difference (first difference).

[0082] In step S805, the CPU 702 determines whether the difference (first difference) between the pixels in the area compared in step S804 is equal to or greater than a predetermined value (equal to or greater than the first predetermined value). In the present embodiment, the CPU 702 determines whether there are pixels whose difference is equal to or greater than the predetermined value. When the CPU 702 determines that the difference between the pixels is equal to or greater than the predetermined value, the CPU 702 performs the process of step S806. When the CPU 702 determines that the difference between the pixels is not equal to or greater than the predetermined value, the CPU 702 ends this process. This embodiment makes this determination based on whether there are pixels whose difference between the pixels in the comparison area is equal to or greater than the predetermined value, but the present disclosure is not limited to this example. For example, when the average value of the differences between the pixels is equal to or greater than the predetermined value, or when the number of pixels whose difference between the pixels is equal to or greater than the predetermined value is equal to or greater than a predetermined number of pixels, the process of step S806 may be performed.

[0083] In step S806, the CPU 702 determines whether image data is being recorded. When the CPU 702 determines that image data is being recorded, the process of step S807 is performed, and when the CPU 702 determines that no image data is being recorded, the process of step S808 is performed.

[0084] In step S807, the image processing unit 705 generates a corrected image of the image data input from the imaging device 100 under the control of the CPU 702. More specifically, the CPU 702 first performs a fast Fourier transform (FFT) on the background image output in step S801 and the portion corresponding to the background area in the image data output by the imaging device in step S803. Thereby, the spatial axis signal can be converted into a frequency axis signal. Then, each frequency component is compared, and the signal of the frequency component that only exists in the image data input from the imaging device 100 is determined as moiré, and signal processing is performed to remove this frequency component (such that the frequency component is less than a first predetermined amount). A method for removing a specific frequency component can use, for example, a filtering process using a notch filter. Then, by performing an inverse FFT on the frequency axis signal of the frequency component from which the moiré has been removed and converting it into a spatial axis signal, the image data from which the moiré has been removed can be obtained.

[0085] In step S808, the CPU 702 generates an alert (or warning) display for notifying the user that moiré or the like has occurred and an unexpected background image has been captured, and outputs the alert to the imaging device 100 and the display device 300 via the image terminal unit 707. Then, the alert is displayed on the display unit 114 of the imaging device 100, the display device 300, and the display unit 711 of the image signal processing device 700, and is conveyed to the user. The alert display input to the imaging device 100 can be output from the image terminal unit 109, and the alert can be displayed on an external monitor (not shown). In the present embodiment, the CPU 702 serves as a control unit configured to perform control to notify the user that the difference in the image data in the portion corresponding to the background area is large, moiré has occurred, and an unexpected background image has been captured.

[0086] Therefore, in the present embodiment, the CPU 702 displays an alert on the display unit, but the present disclosure is not limited to this embodiment, as long as it is possible to notify the user that moiré has occurred and an unexpected background image has been captured. For example, control for notifying the user by vibration or sound can be performed.

[0087] In step S809, when the resolution or shooting angle of the background image output in step S801 is different from the resolution or shooting angle of the corrected image generated in step S807, the CPU 702 performs size adjustment processing and angle correction processing on the background image so that the resolution and shooting angle match. Then, the CPU 702 compares the image data of the part corresponding to the background area. The CPU 702 determines whether correction can be performed based on the difference between the pixels in the comparison area. When the CPU 702 determines that the difference between the pixels in the comparison area is equal to or greater than a predetermined value (equal to or greater than the second predetermined value), that is, when correction cannot be performed, the CPU 702 executes the processing of step S810. When the CPU 702 determines that the difference between the pixels in the comparison area is less than the predetermined value (less than the second predetermined value), that is, when it is determined that correction can be performed, the CPU 702 executes the processing of step S811. In this embodiment, it is determined whether the difference between the pixels in the comparison area is equal to or greater than the predetermined value, but the present disclosure is not limited to this embodiment. For example, when the average value of the differences between the pixels is equal to or greater than the predetermined value, or when the number of pixels whose differences between the pixels are equal to or greater than the predetermined value is equal to or greater than the predetermined number, the processing of step S810 can be performed.

[0088] In step S810, the CPU 702 generates an alarm display for notifying the user that moiré or the like has occurred and an unexpected background image has been captured, and outputs the alarm to the imaging device 100 via the image terminal unit 707. Then, the alarm is displayed on the display unit 114 of the imaging device 100 and the display unit 711 of the image signal processing device 700, and the user is informed that it will not affect the recorded image, and the image data input from the imaging device 100 is recorded in the recording medium I / F 708. The alarm display input to the imaging device 100 can be output from the image terminal unit 109, and the alarm can be displayed on an external monitor (not shown).

[0089] In step S811, the CPU 702 performs processing to record the corrected image generated by the image processing unit 705 in step S807 in the recording medium I / F 708 and output it via the image terminal unit 707.

[0090] In step S812, the CPU 702 performs processing to record, as metadata, the data indicating the corrected area in step S807 and the time code indicating the time or frame when correction is performed in the recording medium I / F 708 and output it via the image terminal unit 707.

[0091] In this embodiment, it is determined whether correction can be performed in step S809, and the corrected image is recorded and the corrected image is output in step S811. However, the present disclosure is not limited to this embodiment. The process of step S810 may also be performed without determining whether correction can be performed in step S809, and the background area may be corrected in a post-processing step after imaging using the captured image that has been recorded, the metadata recorded in step S812, and the background image output in step S801. At this time, the CPU 702 associates the corrected captured image with the metadata indicating that moiré has been corrected and records them in the recording medium 712. In addition, the CPU 702 may output and record in the recording medium 712 the metadata indicating that moiré has appeared in the display area of the display device 300 together with the captured image without performing the correction for removing moiré in step S807, so as to perform correction later.

[0092] In this embodiment, the method for removing moiré converts the image data into a frequency-axis signal and identifies and removes the frequency components of moiré. However, the present disclosure is not limited to this embodiment. For example, moiré can be removed by converting the background image output in step S801 according to the viewing angle conversion of the imaging device 100 so that the resolution and the imaging angle of the image data output by the imaging device 100 match, and by replacing the background area of the image data input from the imaging device 100.

[0093] This embodiment uses the distance information calculated from the A image signal and the B image signal obtained from the imaging unit 107 as a means for separating the subject area and the background area. However, the present disclosure is not limited to this embodiment. The distance to the subject can be obtained by using other measures such as a distance sensor, or the subject area and the background area can be separated without using distance information by using image segmentation technology.

[0094] Performing the above processing can detect and remove moiré that appears during imaging while the background image is being displayed on the display unit. In the case where there is a difference equal to or greater than a predetermined value between the background image output by the image signal processing device 700 to the display device 300 and the background image displayed on the display device 300 and captured by the imaging device 100, not only can moiré be detected and removed, but also changes in brightness, color, distortion, pixel / area defects, etc. can be detected and removed. The method for detecting such changes is not limited to the above method of calculating the difference, and any calculation method can be used as long as the background image output to the display device 300 can be compared with the background image captured by the imaging device 100 for each area and pixel, such as a method of calculating a ratio.

[0095] In addition to the moiré detection method in this embodiment, moiré can also be estimated based on optical design information related to the imaging device 100 (such as the pixel pitch of the image sensor, focusing, zooming, and subject distance, etc.) and pixel pitch information related to the display device 300.

[0096] Second Embodiment

[0097] In the first embodiment, moiré is removed by generating a corrected image of the image data input from the imaging device 100.

[0098] Since interference is displayed as a repeating pattern included in the image that is the background image, moiré may occur during imaging. In this case, moiré can be suppressed by correcting the background image to be displayed. Therefore, in this embodiment, moiré is removed by correcting and outputting the background image output by the image signal processing device 700.

[0099] Figure 10 It is a flowchart showing the abnormal detection and removal process of the background image of the imaging system according to this embodiment. Those elements or steps that are corresponding elements or steps of the first embodiment in this embodiment will be designated by the same reference numerals, and the repeated description thereof will be omitted.

[0100] In step S1001, under the control of the CPU 702, the image processing unit 705 generates a corrected image of the background image output in step S801 and outputs the corrected image to the display device 300. More specifically, for example, an image in which high-frequency components (the high-frequency components are less than a second predetermined amount) are reduced by applying a low-pass filter to the background image output in step S801 is generated, and this image is output to the display device 300.

[0101] In step S1002, when the resolution or shooting angle of the background image output in step S1001 is different from the resolution or shooting angle of the image data captured in step S802, the CPU 702 performs size adjustment processing or angle correction processing on the background image so that the resolution and shooting angle match. Then, the CPU 102 compares the image data of the part corresponding to the background area. When the CPU 702 determines that the difference between the pixels in the comparison area is equal to or greater than a predetermined value (equal to or greater than the third predetermined value), the process of step S810 is executed. When the CPU 702 determines that the difference between the pixels in the comparison area is less than the predetermined value (less than or equal to the third predetermined value), the process of step S1003 is executed. In this embodiment, it is determined whether the difference between the pixels in the comparison area is equal to or greater than the predetermined value, but the present disclosure is not limited to this embodiment. For example, when the average value of the differences between the pixels is equal to or greater than the predetermined value, or when the number of pixels whose differences between the pixels are equal to or greater than the predetermined value is equal to or greater than the predetermined number of pixels, the process of step S810 can be executed.

[0102] In step S1003, the CPU 102 captures the background image and the subject output to the display device 300 in step S1001 and records them in the recording medium I / F 110. The CPU 102 can output the captured image data via the image terminal unit 109 and record the captured image data in the recording medium 712 of the image signal processing device 700.

[0103] Performing the above processing can detect and remove abnormalities such as moiré patterns that occur during shooting while the background image is being displayed on the display unit.

[0104] Other embodiments

[0105] Embodiments of the present invention can also be implemented by the following method, that is, software (including a computer program product including computer programs / instructions) that executes the functions of the above embodiments is provided to a system or device through a network or various storage media, and a computer (central processing unit (CPU), microprocessing unit (MPU)) of the system or device reads and executes the computer programs / instructions.

[0106] Although the present disclosure has been described with reference to the embodiments, it should be understood that the present disclosure is not limited to the disclosed embodiments. The scope of the appended claims is to be accorded the broadest interpretation so as to encompass all such modifications as well as equivalent structures and functions.

[0107] Each embodiment can provide a processing device that can detect abnormalities such as moiré patterns that occur during shooting while the background image is being displayed on the display unit.

Claims

1. A processing device configured to cause a display device to display a first image and to acquire from a camera device a second image generated by photographing at a viewing angle including a display area of ​​the display device, the processing device comprising: an acquisition unit configured to acquire a distinction result for distinguishing between a region where a subject existing between the display region and the imaging device has been photographed and the display region in the second image; a detector configured to detect a change in the second image by comparing the first image to an image of a display area in the second image; as well as A control unit is configured to perform a first control for notifying information related to the detection result.

2. The processing device according to claim 1, characterized in that The distinction result is obtained based on the distance from the imaging device to the object to be imaged.

3. The processing device according to claim 1, characterized in that The detector is configured to perform comparison by detecting a difference between a third image acquired by correcting at least one of a resolution and a photographing angle of the first image and the display area.

4. The processing device according to claim 1, characterized in that The detector is configured to detect a first difference between the first image and the display area, and The control unit is configured to perform the first control when the processing device is not recording an image and the first difference is equal to or greater than a first predetermined value.

5. The processing device according to claim 1, characterized in that The detector is configured as: detecting a first difference between the first image and the display area, and detecting a second difference between a display area in a fourth image and a fifth image, the fourth image being acquired by correcting the second image so that the first difference is less than a first predetermined value, the fifth image being acquired by correcting at least one of a resolution and a camera angle of the first image, and Wherein, the control unit is configured as: in a case where the second difference is equal to or greater than a second predetermined value, performing second control to notify information related to the second difference and record the second image, and In the case where the second difference is smaller than a second predetermined value, the fourth image is recorded.

6. The processing device according to claim 5, characterized in that The control unit is configured to record data indicating at least one of information about a region, a time, and a frame in which correction is performed in the second image.

7. The processing device according to any one of claims 1 to 6, characterized in that The detector is configured as: detecting a first difference between the first image and the display area, and detecting a third difference between a seventh image acquired by correcting at least one of a resolution and a camera angle of a sixth image and a display area in the second image, the sixth image being acquired by correcting the first image so that the first difference is less than a second predetermined value, Wherein, the control unit is configured as: in a case where the third difference is equal to or greater than a third predetermined value, performing third control to notify information related to the third difference and record the second image, and When the third difference is smaller than a third predetermined value, the second image is recorded without performing the third control.

8. The processing device according to claim 7, characterized in that The control unit is configured to record data indicating at least one of information about a region, a time, and a frame in which correction is performed in the first image.

9. A processing device configured to cause a display device to display a first image and to acquire, from a camera device, a second image generated by photographing at a viewing angle including a display area of ​​the display device, the processing device comprising: a detector configured to detect whether moiré has occurred in a display area of ​​the second image; as well as A control unit is configured to record, in a case where the moiré has been detected, an image in which the moiré has been corrected and information about a region in which the moiré has been detected on a recording medium.

10. A camera system, comprising: The processing device according to any one of claims 1 to 9; a display device configured to display a first image; as well as The camera device is configured to capture a second image including a display area of ​​the display device.

11. A processing method configured to cause a display device to display a first image and to acquire from a camera device a second image generated by shooting at a viewing angle including a display area of ​​the display device, the processing method comprising the following steps: acquiring a distinction result for distinguishing between a region where a subject existing between the display region and the imaging device has been photographed and the display region in the second image; detecting a change in the second image by comparing the first image to an image of a display area in the second image; as well as A first control is performed for notifying information related to the detection result. 12 . A non-transitory computer-readable storage medium storing a program for causing a computer to execute the processing method according to claim 11 .

13. A processing method configured to cause a display device to display a first image and to acquire from a camera device a second image generated by shooting at a viewing angle including a display area of ​​the display device, the processing method comprising the following steps: detecting whether moiré patterns have appeared in a display area of ​​the second image; as well as In a case where the moiré has been detected, an image in which the moiré has been corrected and information on a region in which the moiré has been detected are recorded on a recording medium.

14. A non-transitory computer-readable storage medium storing a program for causing a computer to execute the processing method according to claim 13.

15. A computer program product comprising a program for causing a computer to execute the processing method according to claim 11 or 13.

Images