Signal processing device, imaging system, and signal processing method

Abstract

To suppress the generation of an artifact.SOLUTION: A signal processing device comprises: a determination part that compares continuous flames contained in a first video image signal photographed by using an image sensor that can acquires a pixel value for each pixel or for each sharing pixel structure, and determines a motion region and a resting region for each control unit; a frame image generation part that generates a frame image by different processes to be used correspondingly to a motion region and a rating region, respectively; and a video image signal generation part that generates, by using the frame image generated by the frame image generation part, a second video image signal with a frame rate higher than that of the first video image signal. The frame image generation part is configured to: time-divides a frame period required for a display of one frame into a plurality of sub-frame period; with respect to the control unit determined to be the motion region, define a pixel value acquired from the sharing pixel structure to be a pixel value for each sub frame; and with respect to the control unit determined to be a resting region, define the pixel value acquired from the pixel to be a pixel value for each sub frame.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a signal processing device, an imaging system, and a signal processing method. [Background technology]

[0002] Conventionally, an image sensor is a semiconductor chip that converts light focused by an imaging lens into an electric signal using photodiodes arranged in a two-dimensional plane, reads out the photoinduced electric charge accumulated in the photodiodes, and obtains the two-dimensional spatial distribution of light intensity as an image. In addition, an image sensor for capturing moving images periodically reads out the electric charge accumulated in the pixels and captures successive frame images to capture moving images.

[0003] In general, when capturing moving images, if the number of pixels in the horizontal direction is H, the number of pixels in the vertical direction is V, and the frame rate, which is the period for reading out images, is F [fps (frames per second)], then the "pixel readout rate," which is the number of pixels read out per second, is obtained by multiplying H x V x F [pixels / sec]. This "pixel readout rate" value has a significant effect on power consumption, the performance of the A / D conversion circuit, and even the output data rate. In the field of image sensors, in addition to the introduction of advanced semiconductor manufacturing technologies such as microfabrication processes and three-dimensional stacking technology, performance has been improved by improving circuit technology and on-chip signal processing, but even with these efforts, it is not easy to increase the pixel readout rate.

[0004] In a general moving image acquisition method, the spatial resolution (number of pixels in the horizontal and vertical directions) and the temporal resolution (frame rate) of the moving image acquired from the structure of the image sensor are set to be constant during shooting, but in consideration of the nature of the moving image, it is not necessarily required to be constant. In other words, when shooting a stationary object, it is not necessary to maintain a high frame rate, and even if shooting at a low frame rate, it is possible to suppress the deterioration of the subjective image quality. On the other hand, when shooting a moving object, since the spatial frequency is reduced due to motion blur, it is not necessary to maintain a high spatial resolution, and it is possible to suppress the deterioration of the subjective image quality even if shooting at a low spatial resolution.

[0005] Considering the above-mentioned nature of moving images, an image sensor capable of shooting with a high spatial resolution and a low frame rate when the object being shot is stationary, and with a low spatial resolution and a high frame rate when the object is moving, can suppress the degradation of subjective image quality while keeping the pixel read rate low, compared with an image sensor that shoots with both a high spatial resolution and a high frame rate. In addition, since a screen to be shot generally contains various objects with different speeds of movement, it is desirable to configure the screen by combining an area (still area) shot with a high spatial resolution and a low frame rate and an area (moving area) shot with a low spatial resolution and a high frame rate.

[0006] As a technology for realizing such imaging, an image sensor using a pixel parallel structure described in Patent Document 1 is known. This image sensor has a structure in which one A / D conversion circuit is formed for each pixel with the same area as the pixel by utilizing a three-dimensional stacked structure. With such a structure, it is possible to perform a readout operation for each pixel independently, thereby realizing control of spatial resolution and frame rate. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] International Publication No. WO2016 / 009832 [Patent Document 2] JP 2022-123539 A Summary of the Invention [Problem to be solved by the invention]

[0008] In the technology described in the above-mentioned Patent Document 1, since it is difficult to make the pixel smaller than the area of ​​the A / D conversion circuit, there is a limit to the miniaturization of the pixel (photodiode: PD). Therefore, in order to realize the miniaturization, it is considered to reduce the number of transistors per pixel. In order to reduce the number of transistors per pixel, it is common to adopt a shared pixel structure in which multiple pixels share one floating diffusion FD, amplification transistor SF, and selection transistor SL, thereby reducing the number of transistors per pixel. For example, an imaging element has been proposed that has a structure that can more finely control two elements, spatial resolution and time resolution, while miniaturizing the device by reducing the number of additional transistors per shared pixel structure to one. Patent Document 2 can be given as an example of a document that discloses such a technology.

[0009] When adopting a shared pixel structure as described in Patent Document 2, it is possible to select either a mode in which binning is not performed at high spatial resolution and low frame rate for each arbitrary control unit (hereinafter referred to as standard mode), or a mode in which binning is performed at low spatial resolution and high frame rate (hereinafter referred to as high-speed mode). If one frame period in standard mode is T, in high-speed mode, charges for four pixels are simultaneously read out from a common FD, so that readout is possible in a time period of T / 4 (subframe period), and it is possible to shoot at four times the frame rate, albeit at a low resolution.

[0010] On the other hand, in the standard mode, in order to reduce the number of transistors per pixel, there is one signal value readout line and one FD per shared pixel structure, which is common to the first pixel PD1, the second pixel PD2, the third pixel PD3, and the fourth pixel PD4. The signal value that can be read out in a period of T / 4 is limited to any one of the PDs among the first pixel PD1, the second pixel PD2, the third pixel PD3, and the fourth pixel PD4. Therefore, the pixels PD are read out one by one in a period of T / 4, and signal values ​​from all the pixels PD are read out in a period of T.

[0011] This is not a big problem in stationary areas, but if an object moves within a range of one or two pixels per frame on the imaging surface, if the object is captured as a stationary area in standard drive mode despite being moving, artifacts can occur because the readout timing of the signal values ​​of nearby pixels differs significantly.

[0012] SUMMARY OF THE PRESENT EMBODIMENTS The present invention provides a signal processing device, an imaging system, and a signal processing method that are capable of suppressing the occurrence of artifacts. [Means for solving the problem]

[0013] [1] In order to solve the above problem, one aspect of the present invention is a signal processing device that processes a video signal captured using an image sensor having a plurality of shared pixel structures each having a plurality of pixels as components and capable of acquiring pixel values ​​for each of the pixels or for each of the shared pixel structures, the signal processing device comprising: a determination unit that compares consecutive frames included in a first video signal captured using the image sensor and determines, for each control unit including a plurality of pixels, whether the first video signal is a dynamic region, which is a region where pixel values ​​change significantly, or a still region, which is a region where pixel values ​​change little; a frame image generation unit that generates a frame image by different processing depending on whether the first video signal is a dynamic region or a still region; and a video signal generation unit that uses the frame images generated by the frame image generation unit to generate a second video signal having a frame rate higher than that of the first video signal, the frame image generation unit time-dividing a frame period required to display one frame into a plurality of sub-frame periods, and for the control units determined to be the dynamic regions, pixel values ​​acquired from the shared pixel structures are used as pixel values ​​for each sub-frame, and for the control units determined to be the still regions, pixel values ​​acquired from the pixels are used as pixel values ​​for each sub-frame.

[0014] [2] Moreover, in one aspect of the present invention, in the signal processing device described in [1] above, the determination unit further determines whether the still region is a completely still region in which there is zero change in pixel values, or a quasi-still region in which there is a non-zero change in pixel values ​​but below a predetermined threshold, and the frame image generation unit generates a frame image by different processing depending on whether the still region is the completely still region or the quasi-still region.

[0015] [3] Moreover, in one aspect of the present invention, in the signal processing device described in [1] or [2] above, the frame image generation unit generates a frame image for the control unit determined to be the completely still region by combining pixel values ​​obtained from different pixels for each subframe.

[0016] [4] Furthermore, in one aspect of the present invention, in the signal processing device described in any one of [1] to [3] above, the frame image generation unit generates a frame image for the control unit determined to be the quasi-static region by combining pixel values ​​obtained from the pixels that differ for each subframe and pixel values ​​interpolated based on the obtained pixel values.

[0017] [5] Moreover, in one aspect of the present invention, in the signal processing device described in any one of [1] to [4] above, the frame image generation unit generates a frame image by combining pixel values ​​obtained from the pixels arranged in a mosaic pattern for the control unit determined to be the quasi-static region and pixel values ​​interpolated for the pixels for which no pixel values ​​were obtained.

[0018] [6] Moreover, in one aspect of the present invention, in the signal processing device described in [5] above, the frame image generation unit generates a frame image in a first subframe for the control unit determined to be the quasi-static region by combining pixel values ​​obtained from the pixels arranged in a mosaic pattern with pixel values ​​interpolated for the pixels whose pixel values ​​were not obtained, and generates a frame image in a second subframe by combining pixel values ​​of the pixels not read out in the first subframe, the pixel values ​​obtained from the pixels arranged in a mosaic pattern with pixel values ​​interpolated for the pixels whose pixel values ​​were not obtained.

[0019] [7] Another aspect of the present invention is an imaging system comprising an image sensor having a plurality of shared pixel structures each having a plurality of pixels as components and capable of acquiring pixel values ​​for each of the pixels or each of the shared pixel structures, and a signal processing device according to any one of [1] to [4] above that processes a video signal captured using the image sensor.

[0020] [8] Also, one aspect of the present invention is a signal processing method for processing a video signal captured using an image sensor having a plurality of shared pixel structures each having a plurality of pixels as components and capable of acquiring pixel values ​​for each of the pixels or for each of the shared pixel structures, the signal processing method comprising: a determination step of comparing consecutive frames included in a first video signal captured using the image sensor, and determining, for each control unit each including a plurality of pixels, whether the first video signal is a dynamic region in which pixel values ​​change significantly or a still region in which pixel values ​​change little; a frame image generation step of generating a frame image by different processing depending on whether the first video signal is a dynamic region or a still region; and a video signal generation step of generating a second video signal having a frame rate higher than that of the first video signal using the frame images generated by the frame image generation step, wherein the frame image generation step time-divides a frame period required to display one frame into a plurality of subframe periods, and for the control unit determined to be the dynamic region, the pixel value acquired from the shared pixel structures is set as the pixel value for each subframe, and for the control unit determined to be the still region, the pixel value acquired from the pixel is set as the pixel value for each subframe. Effect of the Invention

[0021] According to the present invention, it is possible to provide a signal processing device, an imaging system, and a signal processing method capable of suppressing the occurrence of artifacts. [Brief description of the drawings]

[0022] [Figure 1] 1 is a functional configuration diagram showing an example of the functional configuration of an imaging system according to a first embodiment. [Diagram 2] 2 is a diagram for explaining an example of a pixel configuration of an image sensor having a shared pixel structure according to the first embodiment. FIG. [Diagram 3] 4 is an example of a drive mode discrimination chart according to the first embodiment. [Figure 4] 5 is a timing chart showing an example of pixel value readout timing in a shared pixel structure in a standard driving mode according to the first embodiment. [Diagram 5]4 is a conceptual diagram illustrating an image of pixel value readout in a high-speed drive mode according to the first embodiment. FIG. [Figure 6] FIG. 4 is an image diagram illustrating an image of signal processing in a high-speed drive mode according to the first embodiment. [Figure 7] 4 is a conceptual diagram illustrating an image of pixel value readout in a standard driving mode according to the first embodiment. FIG. [Figure 8] FIG. 4 is an image diagram illustrating an image of signal processing in a completely still area in the standard driving mode according to the first embodiment. [Figure 9] FIG. 4 is a conceptual diagram illustrating signal processing in a quasi-static region in a standard driving mode according to the first embodiment. [Figure 10] 11 is a conceptual diagram illustrating an image of signal processing in a quasi-static region in a standard drive mode in a three-CCD imaging system according to a second embodiment. FIG. [Figure 11] 1 is a block diagram showing an example of an internal configuration of a signal processing device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] [Embodiment] A signal processing device, an imaging system, and a signal processing method according to the aspects of the present invention will be described in detail below with reference to the accompanying drawings, showing preferred embodiments. Note that the aspects of the present invention are not limited to these embodiments, and include various modifications or improvements. In other words, the components described below include those that a person skilled in the art can easily imagine, and those that are substantially the same, and the components described below can be appropriately combined. Also, various omissions, substitutions, or modifications of the components can be made without departing from the gist of the present invention. Also, in the following drawings, the scale and number of each structure may be different from the scale and number of the actual structure in order to make each structure easier to understand.

[0024] [Embodiment 1] 1 is a functional configuration diagram showing an example of the functional configuration of an imaging system according to the first embodiment. First, an example of the functional configuration of the imaging system 1 according to the first embodiment will be described with reference to the diagram. The imaging system 1 includes a signal processing device 10 and an image sensor (imaging element) 20. The signal processing device 10 and the image sensor 20 may be accommodated in a single housing as a single imaging device, or may be configured as different devices. When the signal processing device 10 and the image sensor 20 are configured as different devices, the signal processing device 10 and the image sensor 20 may be connected by a communication network (not shown).

[0025] The image sensor 20 is an imaging element for capturing moving images. The image sensor 20 may be capable of capturing moving images at, for example, 4K60p (i.e., a screen resolution of 3840×2160 [pixels] and a frame rate of 30 [FPS]) or 2K240p (i.e., a screen resolution of 2560×1440 [pixels] and a frame rate of 240 [FPS]). In the image sensor 20, pixel values ​​are read out by a pixel value reading means (not shown), and the read pixel values ​​are provided to the signal processing device 10 as pixel value information PDI. The pixel value reading means may be included in the image sensor 20. The image sensor 20 may be an event trigger type imaging element that outputs an event occurrence trigger signal when there is a change in the amount of light received during a predetermined period of time.

[0026] FIG. 2 is a diagram for explaining an example of a pixel configuration of an imaging element having a shared pixel structure according to the first embodiment. With reference to the same figure, an example of a pixel configuration of an image sensor 20 will be described. As shown in the figure, the image sensor 20 has a plurality of shared pixel structures CP. Each shared pixel structure CP is composed of a plurality of imaging elements PD arranged in a matrix. In the example shown in the figure, four pixels, namely, a first pixel PD1, a second pixel PD2, a third pixel PD3, and a fourth pixel PD4, are arranged in one shared pixel structure CP. It can also be said that the shared pixel structure CP has a plurality of pixels PD as components. The image sensor 20 can acquire a pixel value for each pixel PD or for each shared pixel structure CP. Note that one shared pixel structure CP may include a plurality of pixels, for example, eight pixels or sixteen pixels.

[0027] In this embodiment, the four shared pixel structures CP are treated as one control unit CU, and the driving mode can be set individually. In the figure, a first control unit CU1 and a second control unit CU2 are shown as the control units CU. For example, it is possible to drive each control unit CU in a different driving mode, such as the first control unit CU1 in the high-speed driving mode and the second control unit CU2 in the standard driving mode. Note that one control unit CU may include one or more shared pixel structures CP, and may include, for example, one shared pixel structure CP or four shared pixel structures CP.

[0028] 1, the signal processing device 10 acquires pixel value information PDI from the image sensor 20. The signal processing device 10 processes the video signal based on the acquired pixel value information PDI. The signal processing device 10 generates a high-resolution, high-frame-rate video signal from a video signal captured by the image sensor 20. Specifically, the signal processing device 10 generates a 4K240p video signal (i.e., a screen resolution of 3840×2160 [pixels] and a frame rate of 240 [FPS]) based on a 4K60p or 2K240p video signal generated from the pixel value information PDI.

[0029] The signal processing device 10 includes a camera signal processing unit 11, a determination unit 12, a frame image generating unit 13, a video signal generating unit 14, and a storage unit 15 as functional components. Each of these functional units is realized, for example, by using an electronic circuit. Each functional unit may include a storage means such as a semiconductor memory or a magnetic hard disk device, as necessary. Each function may be realized by a computer having a CPU (Central Processing Unit) and software. All or a part of each functional unit may be realized by hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field-Programmable Gate Array). All or a part of each functional unit may be realized by a combination of software and hardware.

[0030] The camera signal processing unit 11 generates a first video signal VI1 based on pixel value information PDI acquired from the image sensor 20. The first video signal VI1 is a video signal used by the signal processing device 10 to generate a video signal, and is a video signal before conversion to a high resolution and high frame rate. The first video signal VI1 is, for example, a 4K60p or 2K240p video signal. The camera signal processing unit 11 outputs the generated first video signal VI1 to the determination unit 12.

[0031] The determination unit 12 compares consecutive frame images included in the first video signal VI1 captured using the image sensor 20. As a result of the comparison, the determination unit 12 makes a determination on the region for each control unit CU. For example, the determination unit 12 obtains information on whether the region is a moving region or a still region from the image sensor 20 or the camera signal processing unit 11, and determines whether the still region is a completely still region or a quasi-still region. A moving region is a region in which the pixel value changes greatly between consecutive frame images. A still region is a region in which the pixel value changes little between consecutive frame images. The determination of whether a region is a moving region or a still region may be made based on whether the change in pixel value between consecutive frame images is equal to or less than a predetermined value.

[0032] A completely still region is a region where the change in pixel value between consecutive frame images is zero. A quasi-still region is a region where the change in pixel value between consecutive frame images is not zero but is equal to or less than a predetermined threshold. The predetermined threshold may be the same value as the threshold for determining whether the region is a moving region or a still region. The determination unit 12 outputs the result of the determination on the region to the frame image generation unit 13 as region information AI. The region information AI includes information specifying the type of region (i.e., whether it is a moving region, a quasi-still region, or a completely still region) for each region (i.e., control unit CU) that the frame image has. Note that the determination unit 12 may obtain information on whether the region is a moving region or a still region by making a determination itself, instead of obtaining information on whether the region is a moving region or a still region from the image sensor 20 or the camera signal processing unit 11.

[0033] FIG. 3 is an example of a drive mode discrimination chart according to the first embodiment. An example of drive mode determination by the determination unit 12 will be described with reference to the same figure. The determination unit 12 first calculates a motion vector V for each region. The region for which the determination unit 12 performs the determination may be each control unit CU. In the calculation process of the motion vector V, specifically, the difference between frames is calculated for each frame. The motion vector V indicates the amount of change in pixel value between frame images.

[0034] As a result of calculating the motion vector V for each region, if the value of the motion vector V is greater than the threshold value α, the region is determined to be a moving region. In a region determined to be a moving region, the region is driven in the high-speed drive mode. On the other hand, as a result of calculating the motion vector V for each region, if the value of the motion vector V is equal to or less than the threshold value α, the region is determined to be a still region. In a region determined to be a still region, the region is driven in the standard drive mode. That is, according to this embodiment, a single frame image will have a mixture of regions driven in the high-speed drive mode and regions driven in the standard drive mode.

[0035] If the motion vector V of a still region is zero, the determination unit 12 determines that the region is a completely still region. Furthermore, if the motion vector of a still region is not zero but is sufficiently small (below the threshold value α), the determination unit 12 determines that the region is a quasi-still region. The value of the threshold value α can be set arbitrarily. In general, the threshold value α is preferably set so that a region with movement of 2 pixels / frame or more is a moving region, a region below that is a quasi-still region, and a region with almost no movement is a completely still region.

[0036] Returning to FIG. 1, frame image generation unit 13 generates frame images that are components of second video signal VI2 generated by signal processing device 10, using a method according to the area determined by determination unit 12. Frame image generation unit 13 time-divides a frame period required to display one frame into a plurality of subframe periods, and generates a frame image for each subframe period. Frame image generation unit 13 generates frame images by different processes depending on whether the area is a moving area or a still area. Frame image generation unit 13 also generates frame images by different processes depending on whether the area is a completely still area or a quasi-still area. Frame image generation unit 13 outputs the generated frame images to video signal generation unit 14 as frame image information FI.

[0037] The video signal generation unit 14 generates a second video signal VI2 having a frame rate higher than that of the first video signal VI1, using the frame images generated by the frame image generation unit 13. The second video signal VI2 generated by the video signal generation unit 14 is stored in, for example, a storage unit 15. The second video signal VI2 stored in the storage unit 15 may be output in response to, for example, an output command from outside the imaging system 1.

[0038] 4 is a timing chart showing an example of pixel value read timing in a shared pixel structure in the standard drive mode according to the first embodiment. With reference to the same figure, an example of pixel value read timing in the standard drive mode will be described. In this embodiment, one frame period T is made up of four subframe (SF) periods. In other words, the frame image generating unit 13 performs processing by time-dividing the frame period required to display one frame into a plurality of subframe periods. Pixel values ​​are read out from each pixel PD for each subframe (i.e., at intervals of subframe period T / 4).

[0039] More specifically, in one frame period T, a pixel value is read out from the first pixel PD1 after T / 4 has elapsed, a pixel value is read out from the second pixel PD2 after 2T / 4 has elapsed, a pixel value is read out from the third pixel PD3 after 3T / 4 has elapsed, and a pixel value is read out from the fourth pixel PD3 after T. In this way, the exposure period of each pixel PD is T.

[0040] Next, an example of subframe image generation processing in high-speed drive mode will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is an image diagram showing an image of pixel value readout in the high-speed drive mode according to the first embodiment. Fig. 6 is an image diagram showing an image of signal processing in the high-speed drive mode according to the first embodiment. The high-speed drive mode is processing performed in a control unit CU determined to be a dynamic region.

[0041] In the high-speed drive mode, the pixel values ​​acquired from the shared pixel structure CP are used as pixel values ​​for each subframe, and a frame image with low pixel count and high frame rate is generated. In other words, in the high-speed drive mode, the first pixel PD1, the second pixel PD2, the third pixel PD3, and the fourth pixel PD4 are binned. As shown in FIG. 5, in one subframe period, a signal value obtained by integrating all the pixel values ​​of the first pixel PD1, the second pixel PD2, the third pixel PD3, and the fourth pixel PD4 is read out. Also, as shown in FIG. 6, in each subframe period, reading from all the pixels PD of the first pixel PD1, the second pixel PD2, the third pixel PD3, and the fourth pixel PD4 is completed, and the read signal value becomes the output image (frame image) of the subframe as it is.

[0042] 7 is an image diagram showing an image of pixel value readout in the standard drive mode according to the first embodiment. An example of a subframe image generation process in the standard drive mode will be described with reference to the same figure. The standard drive mode is a process performed in a control unit CU determined to be a stationary region. The standard drive mode differs from the high-speed drive mode in that the pixel value acquired from the pixel PD is used as the pixel value for each subframe, in that the pixel value acquired from the shared pixel structure CP is used as the pixel value for each subframe.

[0043] In the standard drive mode, the signal value of the first pixel PD1 is read out during the first sub-frame period SF1, the signal value of the fourth pixel PD4 is read out during the second sub-frame period SF2, the signal value of the second pixel PD2 is read out during the third sub-frame period SF3, and the signal value of the third pixel PD3 is read out during the fourth sub-frame period SF4. That is, in the standard drive mode, all signal values ​​of the pixels PD included in the shared pixel structure CP are read out during one frame period. Therefore, in this embodiment, since it is not possible to read out signal values ​​for all the pixels PD included in the shared pixel structure CP during one sub-frame period SF, the signal values ​​are complemented to generate an output image.

[0044] The signal value complementing method differs between the completely still region and the quasi-still region among the still regions. The signal value complementing method in the completely still region and the quasi-still region will be described with reference to Figs. 8 and 9. In both the completely still region and the quasi-still region, an output image (frame image) is generated based on the pixel value of any pixel PD read out in each sub-frame period. In the description with reference to Figs. 8 and 9, the signal values ​​read out from the first pixel PD1, the second pixel PD2, the third pixel PD3, and the fourth pixel PD4, which are the pixels PD included in the shared pixel structure CP, will be referred to as the first signal value S1, the fourth signal value S4, the second signal value S2, and the third signal value S3, respectively.

[0045] FIG. 8 is an image diagram showing an image of signal processing in a completely still region in the standard drive mode according to the first embodiment. First, an example of a method for complementing signal values ​​in a completely still region will be described with reference to the same figure. As described with reference to FIG. 3, a completely still region is a region in which the motion vector V is zero. That is, in a completely still region, the temporal change in pixel values ​​is almost zero. Therefore, since the temporal change is almost zero in a completely still region, as shown in FIG. 8, it is possible to generate a frame image by synthesizing signal values ​​read out in all subframe periods in the time direction.

[0046] More specifically, in a completely still region in the standard drive mode, the first signal value S1 read in the first subframe period SF1, the fourth signal value S4 read in the second subframe period SF2, the second signal value S2 read in the third subframe period SF3, and the third signal value S3 read in the fourth subframe period SF4 are synthesized to form one frame image. The frame images in each subframe are all synthesized to form one frame image. For a control unit CU determined to be a completely still region, it can also be said that a frame image is generated by synthesizing pixel values ​​acquired from different pixels PD for each subframe.

[0047] FIG. 9 is an image diagram showing an image of signal processing in a quasi-static region in the standard drive mode according to the first embodiment. Next, an example of a method for complementing signal values ​​in a quasi-completely static region will be described with reference to the same figure. As described with reference to FIG. 3, a quasi-static region is a region in which the motion vector V is not zero but is equal to or less than a predetermined threshold. That is, in a quasi-static region, the pixel values ​​change gradually over time. When the above-mentioned processing in a completely static region is applied to such a quasi-static region, artifact problems may occur. Therefore, in this embodiment, a processing method that does not cause artist problems is performed in the quasi-static region.

[0048] Specifically, since only the signal value of the first pixel PD1 exists during the first subframe period SF1, the signal value read from the fourth pixel PD4 is used during the second subframe period SF2, which is a subframe adjacent to the first subframe period SF1, to perform the complementation process. As a result, the first signal value S1 at the position of the first pixel PD1 and the first signal value S1 at the position of the fourth pixel PD4 are determined. Here, the third subframe period SF3 and the fourth subframe period SF4, which are subframes not adjacent to the first subframe period SF1, have exposure timings that are significantly different from the first subframe period SF1. Therefore, the second signal value S2 and the third signal value S3, which are signal values ​​at the positions of the second pixel PD2 and the third pixel PD3 acquired during the third subframe period SF3 and the fourth subframe period SF4, are not used to generate a frame image during the first subframe period SF1.

[0049] A signal value S2' at the second pixel PD2 and a signal value S3' at the position of the third pixel PD are interpolated from a first signal value S1, which is the signal value of the first pixel PD1, and a fourth signal value S4, which is the signal value of the fourth pixel PD4. The frame images obtained as a result of such interpolation processing are the frame images of the first sub-frame period SF1 and the frame images of the second sub-frame period SF2. That is, for a control unit CU determined to be a quasi-static region, a frame image is generated by combining pixel values ​​acquired from different pixels PD for each sub-frame SF and pixel values ​​interpolated based on the acquired pixel values.

[0050] As for the interpolation method, a general demosaicing technique that is used when interpolating a Bayer array used in a single-chip camera can be used. For example, a method of calculating inter-pixel correlation in the vertical and horizontal directions and performing interpolation processing according to the degree of correlation can be used.

[0051] Although explanations of the third subframe period SF3 and the fourth subframe period SF4 will be omitted, it is possible to output image data for each subframe by performing processing similar to that of the first subframe period SF1 and the second subframe period SF2.

[0052] For the control unit CU determined to be a quasi-static region, it is preferable to obtain pixel values ​​from pixels PD (e.g., the first pixel PD1 and the fourth pixel PD4) arranged in a mosaic pattern as shown in the figure, and perform interpolation processing. By obtaining pixel values ​​from the pixels PD arranged in a mosaic pattern and performing interpolation processing on pixels PD (e.g., the second pixel PD2 and the third pixel PD3) whose pixel values ​​were not obtained, it is possible to more effectively prevent the occurrence of artifact problems.

[0053] Furthermore, in this embodiment, among the four subframe periods obtained by dividing one frame period, the same subframe image (for example, a frame image generated based on the first pixel PD1 and the fourth pixel PD4) may be used in two adjacent subframe periods, and the same subframe image (for example, a frame image generated based on the second pixel PD2 and the third pixel PD3) may be used in the other two adjacent subframe periods. That is, in the first subframe, a frame image may be generated by combining pixel values ​​acquired from pixels PD arranged in a mosaic pattern with pixel values ​​supplemented in pixels PD whose pixel values ​​were not acquired, and further, in the second subframe, a frame image may be generated by combining pixel values ​​of pixels PD not read out in the first subframe, which are acquired from pixels PD arranged in a mosaic pattern, with pixel values ​​supplemented in pixels PD whose pixel values ​​were not acquired.

[0054] In the embodiment described above, the frame image in the first sub-frame period SF1 and the frame image in the second sub-frame period SF2 are the same frame image, and the frame image in the third sub-frame period SF3 and the frame image in the fourth sub-frame period SF4 are the same frame image. However, this embodiment is not limited to this example, and different frame images may be generated in each of the first sub-frame period SF1 to the fourth sub-frame period SF4.

[0055] [Embodiment 2] Next, a second embodiment will be described. In the first embodiment described above, it has been assumed that the red (R), green (G), and blue (B) imaging elements operate in the same manner. However, in the case of an R, G, and B three-chip imaging system, it is possible to adopt the processing according to this embodiment by changing the exposure timing and the readout order for each imaging element. The second embodiment is an example of a case where the processing according to this embodiment is adopted for an R, G, and B three-chip imaging system. Hereinafter, the second embodiment will be described with reference to FIG. 10.

[0056] 10 is a conceptual diagram illustrating an image of signal processing in a quasi-static region in the standard drive mode in a three-chip imaging system according to embodiment 2. The diagram shows an example of a three-chip imaging system having G1, G2, B1, and R1 pixels arranged in a Bayer array. In the case of a three-chip imaging system as shown in the figure, for example, during the first sub-frame SF1, the G imaging element reads out the position of the first pixel PD1, the B imaging element reads out the position of the third pixel PD3, and the R imaging element reads out the position of the second pixel PD2, thereby obtaining signal values ​​at different PD positions for R, G, and B.

[0057] For the position of the fourth pixel PD4, the signal value read out during the second subframe SF2 is used. By demosaicing the signal values ​​of G1G2B1R1, it is possible to create an image in the first subframe SF1. Interpolation processing can be similarly performed for the periods of the second subframe SF2, the third subframe SF3, and the fourth subframe SF4. When such correction processing is used, demosaicing makes it possible to generate a video signal that has higher definition and suppresses the occurrence of artifacts than when interpolated in the spatial direction.

[0058] [Internal configuration of signal processing device] FIG. 11 is a block diagram showing an example of the internal configuration of the signal processing device 10 according to the present embodiment. At least some of the functions of the signal processing device 10 can be realized using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions included in a program read from the RAM 902 or the like. The central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations according to each instruction. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. Note that RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with an external input / output device or the like. The input / output devices 904 and 905 are input / output devices. The input / output devices 904 and 905 exchange data with the central processing unit 901 via the input / output port 903. The bus 906 is a common communication path used inside the computer. For example, the central processing unit 901 reads and writes data from the RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port via the bus 906. Also, all or part of each functional unit provided in the signal processing device 10 may be realized using hardware such as an ASIC, a PLD, or an FPGA. Also, all or part of each functional unit may be realized by a combination of software and hardware.

[0059] The effects described in this specification are merely explanatory or exemplary and are not limiting. In other words, the technology according to the present disclosure may have other effects that are apparent to a person skilled in the art from the description of this specification, in addition to or in place of the above effects. Furthermore, the present invention is not limited to these embodiments, and various modifications and substitutions can be made within the scope of the present invention. [Explanation of symbols]

[0060] 1. Imaging system 10 Signal Processing Device 11 Camera signal processing section 12 Judgment section 13 Frame image generator 14 Video signal generator 15 Storage section 20 Image Sensor CU Control Unit CP Shared Pixels PD pixel

Claims

1. A signal processing device that processes a video signal captured using an image sensor that has a plurality of shared pixel structures each having a plurality of pixels as components and is capable of acquiring a pixel value for each of the pixels or each of the shared pixel structures, a determination unit that compares consecutive frames included in a first video signal captured using the image sensor, and determines, for each control unit including a plurality of pixels, a dynamic region in which a change in pixel value is large, and a static region in which a change in pixel value is small; a frame image generating unit that generates a frame image by performing different processing depending on the moving area and the still area; a video signal generation unit that generates a second video signal having a frame rate higher than that of the first video signal, using the frame images generated by the frame image generation unit; Equipped with The frame image generating unit A frame period required to display one frame is time-divided into a plurality of subframe periods; For the control unit determined to be the moving region, a pixel value acquired from the shared pixel structure is set as a pixel value for each subframe; For the control unit determined to be the still region, the pixel value acquired from the pixel is set as the pixel value for each subframe. Signal processing device.

2. the determination unit further determines whether the still region is a completely still region in which a change in pixel value is zero, or a quasi-still region in which a change in pixel value is not zero but is equal to or less than a predetermined threshold, The frame image generating unit generates a frame image by performing different processes depending on the completely still region and the quasi-still region. The signal processing device according to claim 1 .

3. The frame image generating unit generates a frame image by synthesizing pixel values ​​acquired from the pixels that differ for each subframe for the control unit that is determined to be the completely still region. The signal processing device according to claim 2 .

4. The frame image generating unit generates a frame image by combining pixel values ​​acquired from the pixels that differ for each subframe and pixel values ​​that are interpolated based on the acquired pixel values, for the control unit determined to be the quasi-static region. The signal processing device according to claim 2 .

5. The frame image generating unit generates a frame image by synthesizing pixel values ​​acquired from the pixels arranged in a mosaic pattern for the control unit determined to be the quasi-static region and pixel values ​​complemented for the pixels for which pixel values ​​were not acquired. The signal processing device according to claim 4.

6. the frame image generation unit generates a frame image by combining pixel values ​​acquired from the pixels arranged in a mosaic pattern and pixel values ​​interpolated for the pixels for which pixel values ​​were not acquired, for the control unit determined to be the quasi-static region, in a first subframe; In the second subframe, a frame image is generated by combining pixel values ​​of the pixels not read out in the first subframe, the pixel values ​​being acquired from the pixels arranged in a mosaic pattern, and pixel values ​​complemented in the pixels whose pixel values ​​were not acquired. The signal processing device according to claim 5 .

7. an image sensor having a plurality of shared pixel structures each having a plurality of pixels as components, the image sensor being capable of acquiring a pixel value for each of the pixels or each of the shared pixel structures; A signal processing device according to any one of claims 1 to 6, which processes a video signal captured by the image sensor. An imaging system comprising:

8. A signal processing method for processing a video signal captured using an image sensor having a plurality of shared pixel structures each having a plurality of pixels as components, the image sensor being capable of acquiring a pixel value for each of the pixels or each of the shared pixel structures, the method comprising: a determination step of comparing successive frames included in a first video signal captured using the image sensor, and determining, for each control unit including a plurality of pixels, a dynamic region which is a region in which pixel values ​​change significantly, and a static region which is a region in which pixel values ​​change slightly; a frame image generating step of generating a frame image by different processing depending on the moving area and the still area; a video signal generating step of generating a second video signal having a frame rate higher than that of the first video signal, using the frame images generated by the frame image generating step; having The frame image generating step includes: A frame period required to display one frame is time-divided into a plurality of subframe periods; For the control unit determined to be the moving region, a pixel value acquired from the shared pixel structure is set as a pixel value for each subframe; For the control unit determined to be the still region, the pixel value acquired from the pixel is set as the pixel value for each subframe. Signal processing methods.

Images