Display device, signal processing device, and signal processing method

Abstract

The present technology relates to a display device, a signal processing device, and a signal processing method that make it possible to realize interpolation that is more suitable for viewing when interpolating motion between original images. The present invention provides a display device including a signal processing unit that controls an interpolation rate of an interpolated image frame in accordance with a motion in a certain direction between original image frames when generating an interpolated image frame that interpolates between the original image frames along a time axis. The present technology can be applied to, for example, a television receiver.

Description

[Technical field]

[0001] The present technology relates to a display device, a signal processing device, and a signal processing method, and in particular to a display device, a signal processing device, and a signal processing method that are capable of realizing interpolation more suitable for viewing when interpolating movement between original images. [Background technology]

[0002] 2. Description of the Related Art In display devices such as television receivers, frame rate conversion using motion compensation is known as one type of signal processing for improving image quality.

[0003] As an example of this type of frame rate conversion, a method is known in which one or more interpolated image frames are added between adjacent original image frames along the time axis using motion compensation to interpolate the images of the original image frames (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2008 / 102827 Summary of the Invention [Problem to be solved by the invention]

[0005] When interpolating the movement between the original frames using the interpolated frames, it is desired to realize an interpolation that is more suitable for viewing.

[0006] The present technology has been made in consideration of such circumstances, and is intended to realize interpolation that is more suitable for viewing when interpolating motion between original images. [Means for solving the problem]

[0007] According to one aspect of the present technology, there is provided a display device which, when generating an interpolated image frame that interpolates between original image frames along a time axis, determines a target interpolation rate that is a target of an interpolation rate of the interpolated image frame based on an amount of shaking detected from the original image frames, and controls the interpolation rate in accordance with a movement in a fixed direction between the original image frames based on the determined target interpolation rate, and includes a signal processing unit which controls the interpolation rate according to the movement in a fixed direction between the original image frames, When the one or more interpolated image frames are generated by uniformly moving the change positions between the original image frames, the interpolation rate is set to 100%, and when the interpolated image frames are the original image frames themselves, the interpolation rate is set to 0%, so that the lower the interpolation rate, the closer the interpolated image frames become to the original image frames. It is a display device.

[0008] In a display device according to one aspect of the present technology, when generating an interpolated image frame that interpolates between original image frames along a time axis, a target interpolation rate that is a target of the interpolation rate of the interpolated image frame is determined based on an amount of shaking detected from the original image frames, and the interpolation rate is controlled according to a motion in a fixed direction between the original image frames based on the determined target interpolation rate. When the one or more interpolated image frames are generated by uniformly moving the change positions between the original image frames, the interpolation rate is set to 100%, and when the interpolated image frames are the original image frames themselves, the interpolation rate is set to 0%, so that the lower the interpolation rate, the closer the interpolated image frames become to the original image frames. .

[0009] A signal processing device according to one aspect of the present technology includes an interpolation rate control unit that, when generating an interpolated image frame that interpolates between original image frames along a time axis, determines a target interpolation rate that is a target of an interpolation rate of the interpolated image frame based on an amount of shaking detected from the original image frames, and controls the interpolation rate in accordance with a movement in a fixed direction between the original image frames based on the determined target interpolation rate, and the interpolation rate is When the one or more interpolated image frames are generated by uniformly moving the change positions between the original image frames, the interpolation rate is set to 100%, and when the interpolated image frames are the original image frames themselves, the interpolation rate is set to 0%, so that the lower the interpolation rate, the closer the interpolated image frames become to the original image frames. .

[0010] A signal processing method according to one aspect of the present technology includes a signal processing device, which, when generating an interpolated image frame that interpolates between original image frames along a time axis, determines a target interpolation rate that is a target of an interpolation rate of the interpolated image frame based on an amount of shaking detected from the original image frames, and controls the interpolation rate according to a motion in a fixed direction between the original image frames based on the determined target interpolation rate, and When the one or more interpolated image frames are generated by uniformly moving the change positions between the original image frames, the interpolation rate is set to 100%, and when the interpolated image frames are the original image frames themselves, the interpolation rate is set to 0%, so that the lower the interpolation rate, the closer the interpolated image frames become to the original image frames. .

[0011] In a signal processing device and a signal processing method according to an aspect of the present technology, when generating an interpolated image frame that interpolates between original image frames along a time axis, a target interpolation rate that is a target of the interpolation rate of the interpolated image frame is determined based on an amount of shaking detected from the original image frames, and the interpolation rate is controlled according to a motion in a fixed direction between the original image frames based on the determined target interpolation rate. When the one or more interpolated image frames are generated by uniformly moving the change positions between the original image frames, the interpolation rate is set to 100%, and when the interpolated image frames are the original image frames themselves, the interpolation rate is set to 0%, so that the lower the interpolation rate, the closer the interpolated image frames become to the original image frames. .

[0012] It should be noted that the display device and the signal processing device according to one aspect of the present technology may be independent devices, or may be internal blocks constituting a single device. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram showing an example configuration of a signal processing device to which the present technology is applied. [Diagram 2] 11A and 11B are diagrams illustrating examples of a method for detecting motion and a method for determining a target interpolation rate. [Diagram 3] 11 is a flowchart illustrating the flow of a first interpolation rate control process. [Figure 4] FIG. 13 is a diagram showing an example of an interpolation rate range for each type of video content. [Diagram 5] 13A and 13B are diagrams illustrating an example of a transition of an interpolation rate according to an amount of motion. [Figure 6] 13 is a flowchart illustrating a flow of an interpolation process. [Figure 7] 11A to 11C are diagrams illustrating examples of display of interpolated image frames according to an interpolation rate. [Figure 8] 11A and 11B are diagrams illustrating examples of a method for detecting shaking and a method for determining a target interpolation rate. [Figure 9] 13 is a flowchart illustrating the flow of a second interpolation rate control process. [Figure 10] 1 is a block diagram showing a configuration example of a display device to which the present technology is applied. [Figure 11] 1 is a block diagram showing a configuration example of a processing device to which the present technology is applied. [Figure 12]FIG. 1 illustrates an example of the configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present technology will be described with reference to the drawings. The description will be given in the following order.

[0015] 1. First embodiment 2. Second embodiment 3. Variations 4. Computer Configuration

[0016] <1. First embodiment>

[0017] (Example of signal processing device configuration) FIG. 1 is a block diagram showing an example configuration of a signal processing device to which the present technology is applied.

[0018] In FIG. 1, a signal processing device 10 is built into, for example, a television receiver, and performs signal processing on a video signal that has been subjected to processing such as channel selection and decoding by a tuner, a decoder, etc. provided in a preceding stage, and outputs the video signal obtained as a result of the signal processing to a subsequent circuit (a circuit related to video display).

[0019] The signal processing device 10 includes a preprocessing unit 101, a memory control unit 102, a memory 103, a motion vector detection unit 104, an interpolation rate control unit 105, and an interpolation unit .

[0020] In the signal processing device 10, a digital component signal YUV is input as a video signal, and is supplied to a pre-processing unit 101, and is also sequentially written into a memory 103 via a memory control unit .

[0021] Here, the digital component signal YUV is a video signal consisting of a luminance signal Y and color difference signals U and V. In addition, in the signal processing device 10, the digital component signal YUV is processed on a frame-by-frame basis, and an original frame of the video signal is called an original frame to be distinguished from an interpolated image frame, which will be described later.

[0022] The pre-processing unit 101 performs a process of separating a luminance signal Y from a digital component signal YUV. The luminance signal Y separated by the pre-processing unit 101 is sequentially written into a memory 103 via a memory control unit .

[0023] The luminance signal Y written in the memory 103 is sequentially read out by the memory control unit 102 and supplied to the motion vector detection unit 104. The motion vector detection unit 104 performs a motion vector detection process by block matching using the luminance signal Y of the current original image frame and the luminance signals Y of the previous and next original image frames.

[0024] The motion vector mv of each original image frame detected by the motion vector detection unit 104 is written to the memory 103 via the memory control unit 102, and then read out from the memory 103 and returned to the motion vector detection unit 104 for reference in detecting the motion vector of the next original image frame.

[0025] The memory control unit 102 reads out the digital component signal YUV written in the memory 103 in predetermined frame units suitable for the interpolation process by the interpolation unit 106, and supplies the signal to the interpolation unit 106 together with the motion vector mv.

[0026] Here, for example, the memory control unit 102 reads out two systems of digital component signals YUV written in the memory 103. Furthermore, the memory control unit 102 reads out a motion vector mv representing the motion between the two original image frames. The two systems of digital component signals YUV(prev, succ) and the motion vector mv thus read out are supplied to the interpolation unit 106.

[0027] Furthermore, the motion vector detection unit 104 detects the movement of the entire screen or an object based on the detected motion vector mv of each original image frame, and supplies the motion detection result to an interpolation rate control unit 105. However, it is assumed that the motion detection result supplied to the interpolation rate control unit 105 is synchronized with the digital component signal YUV(prev, succ) and the motion vector mv supplied to the interpolation unit 106.

[0028] The interpolation rate control unit 105 is composed of, for example, a CPU (Central Processing Unit) etc. The interpolation rate control unit 105 controls the interpolation rate of the interpolated image frame based on the motion detection result supplied from the motion vector detection unit 104. For example, the interpolation rate control unit 105 determines a target interpolation rate that is a target of the interpolation rate based on the amount of motion detected from the original image frame, and controls the interpolation rate in stages so as to gradually approach the determined target interpolation rate.

[0029] Although details will be described later, the interpolation rate control unit 105 controls the interpolation rate according to the movement in a fixed direction between original image frames. Here, for example, video scenes in which the screen is panning in a fixed direction, the screen is zooming in or out, or the screen is rotating are detected, and the interpolation rate is controlled according to these video scenes.

[0030] Here, the interpolated image frame is a frame for interpolating between original image frames along the time axis. For example, when an original image frame (prev) and an original image frame (succ) along the time axis are input to the interpolation unit 106 as a digital component signal YUV(prev,succ), the interpolation rate of the interpolated image frame that interpolates between the original image frame (prev) and the original image frame (succ) is controlled.

[0031] The interpolation rate is a ratio indicating the degree of interpolation of the interpolated image frames inserted (added) between the original image frames along the time axis. For example, when converting the frame rate from 24 Hz to 120 Hz, four interpolated image frames are required between the original image frames (prev, succ). By changing the interpolation rate, the position of the object in the interpolated image frames inserted (added) between the original image frames can be changed.

[0032] In other words, in this example, when four interpolated image frames are generated by uniformly moving the change positions between the original image frames, the interpolation rate is set to 100%. If the interpolation rate is reduced to 50% or 25%, the movement will not be uniform; the lower the interpolation rate, the closer the interpolated image frames will be to the original image frames, and the original image frames themselves will have an interpolation rate of 0%.

[0033] The interpolation unit 106 receives, in synchronization with each other, the digital component signal YUV (prev, succ) and the motion vector mv from the memory control unit 102 and the interpolation rate from the interpolation rate control unit 105. The interpolation unit 106 generates and interpolates an interpolated image frame that interpolates between the previous and next original image frames (prev, succ) based on the interpolation rate, and outputs the interpolated digital component signal YUV to a downstream circuit.

[0034] Here, for example, an interpolated image frame is generated as follows: That is, addresses of pixels in the previous and next original image frames (prev, succ) used to calculate pixel values ​​of the interpolated image frame are calculated based on parameters of the interpolation position according to the interpolation rate of the video in each interpolated image frame and a motion vector mv between the previous and next original image frames (prev, succ), and then the pixel values ​​of these addresses are weighted according to the interpolation position according to the interpolation rate, thereby performing interpolation and generating the interpolated image frame.

[0035] In the signal processing device 10 configured as described above, when generating (inserting) an interpolated image frame into an original image frame (prev, succ) along the time axis, an interpolation rate of the interpolated image frame is determined according to a scene identified from movement in a certain direction between the original image frames (prev, succ) (e.g., the amount of motion detected from the original image frames), and the interpolated image frame is generated (inserted) into the original image frame (prev, succ) based on the determined interpolation rate.

[0036] (Method of motion detection and interpolation rate determination) FIG. 2 is a diagram showing an example of a method of detecting a motion based on a motion vector and a method of determining a target interpolation rate according to the result of the motion detection.

[0037] As shown in Fig. 2, there are two types of detection items detected from the original image frame: the movement of the entire screen and the movement of an object contained within the screen. Note that, in this case, the area of ​​the entire screen corresponds to the area corresponding to the original image frame.

[0038] The motion of the entire screen can be detected, for example, by the following detection method: The motion vectors of the entire screen are accumulated in eight directions, namely, left, upper left, upper, upper right, right, lower right, lower, and lower left, and the specific direction that has the largest motion vector sum among the motion vector sums in the respective directions obtained as a result is determined to be the motion direction of the entire screen.

[0039] In this case, the interpolation rate control unit 105 determines a target interpolation rate according to the detected total amount of motion vectors for the entire screen. Here, for example, based on the total amount of motion vectors in a specific direction according to the determination result of the direction of motion for the entire screen, a video scene in which the entire screen is panned in a certain direction, the entire screen is zoomed in or out, or the entire screen is rotated is detected, and the target interpolation rate is determined according to the amount of motion in a certain direction (amount of motion for the entire screen) for that video scene.

[0040] Furthermore, the following detection method can be used to detect the movement of the object, for example. That is, the largest motion vector sum S among the motion vector sums in the above eight directions (left, upper left, upward, upper right, right, downward, and downward left) can be set as the motion vector sum S' of the object, and this value is converted (converted) into a value for the entire screen.

[0041] For example, by detecting a still area in the entire screen (e.g., an area of ​​A% of the entire screen), the ratio of the area in the entire screen corresponding to a moving object can be calculated, and the motion vector sum S can be converted to the motion vector sum S' of the object using that ratio. More specifically, the motion vector sum S' of the object satisfies the relationship of the following formula (1).

[0042] S' = S × 100 / (100 - A) ···(1)

[0043] In this case, the interpolation rate control unit 105 determines a target interpolation rate according to the total motion vector amount S' of the detected object. Here, for example, a scene in which an object such as a person or a vehicle is moving with a fixed camera is detected based on the total motion vector amount S' of the object, and the target interpolation rate is determined according to the amount of movement in a certain direction (the amount of movement of the object).

[0044] Here, the detection items are described as the movement of the entire screen and the movement of the object, but the target interpolation rate can be determined according to at least one of the amounts of movement. Also, the movement of the entire screen and the movement of the object are examples of parameters according to the video scene, and other parameters may be used.

[0045] (First Interpolation Rate Control Process) FIG. 3 is a flowchart illustrating the flow of the first interpolation rate control process executed by the interpolation rate control unit 105 of FIG.

[0046] In step S11, the interpolation rate control unit 105 judges whether the interpolation rate (current interpolation rate) set in the interpolation unit 106 has satisfied the holding period. If it is judged in step S11 that the current interpolation rate has satisfied the holding period, the process proceeds to step S12. Note that in this judgment process, for example, the timing of a vertical synchronization signal (V: Vertical Sync) such as 10V is judged.

[0047] In step S12, the interpolation rate control unit 105 calculates a motion vector sum amount based on the motion vector detected by the motion vector detection unit 104. Here, as shown in the detection method of Fig. 2, for example, the motion vector sum amount of the entire screen or the motion vector sum amount of the target object is calculated.

[0048] In step S13, the interpolation rate control unit 105 determines a target interpolation rate according to the calculated total amount of motion vectors. As shown in the determination method in Fig. 2, for example, a target interpolation rate according to the total amount of motion vectors of the entire screen or a target interpolation rate according to the total amount of motion vectors of an object is determined.

[0049] In step S14, the interpolation rate control unit 105 changes the current interpolation rate by one step toward the determined target interpolation rate, and sets the current interpolation rate in the interpolation unit 106. Here, for example, when the target interpolation rate is determined to be 100%, and 10% is set as the unit of change, if the interpolation rate set in the interpolation unit 106 is 60%, it is advanced by one step to 70%.

[0050] In step S15, the interpolation rate control unit 105 sets a retention period of the interpolation rate. Here, the timing of a vertical synchronization signal such as 10V is set as the retention period of the interpolation rate used in the determination process in step S11.

[0051] When the process of step S15 is completed, the process proceeds to END. If it is determined in step S11 that the holding period of the timing of 10V or the like is not satisfied, steps S12 to S15 are skipped and the process proceeds to END. Here, the process from START to END is performed within 1V, and the process is started again from START at the next V.

[0052] That is, for example, at a predetermined timing such as 1V, the processing of steps S11 to S15 is repeated, and when a predetermined holding period such as 10V is satisfied ("YES" in S11), a target interpolation rate according to the total amount of motion vectors is determined (S13), and the current interpolation rate is gradually changed by one step so as to approach the target interpolation rate, and is set in the interpolation unit 106 (S14).

[0053] For example, when the target interpolation rate is determined to be 100%, if the current interpolation rate is 60%, it is set to 70%, and then, for each 10V timing, the current interpolation rate is changed stepwise (for example, in 10% increments) to successively approach the target interpolation rate of 100%, such as 80%, 90%, and so on, until the interpolation rate is finally set to 100%.

[0054] In addition, the target interpolation rate is not limited to a fixed value, but can be a variable value. For example, in the above example, if the video scene changes and the target interpolation rate is changed to 70% at the timing when the current interpolation rate is changed to 90% toward the target interpolation rate of 100%, the current interpolation rate is changed stepwise (for example, in 10% increments) toward the changed target interpolation rate of 70%, such as 90%, 80%, and 70%.

[0055] The flow of the first interpolation rate control process has been described above. In this first interpolation rate control process, when generating an interpolated image frame for an original image frame along a time axis, the interpolation rate of the interpolated image frame is controlled according to a certain directional movement between the original image frames. More specifically, when controlling the interpolation rate, a target interpolation rate is determined based on the amount of movement detected from the original image frames, and the interpolation rate is controlled based on the determined target interpolation rate.

[0056] 4 shows an example of the interpolation rate range for each type of video content. For example, when the type of video content is a movie, the interpolation rate control unit 105 performs a first interpolation rate control process to control the interpolation rate of the interpolated image frame to be changed within an interpolation rate range of 40 to 90%. In other words, this interpolation rate range is a range within which the interpolation rate determined for each type of video content can be changed.

[0057] Also, in FIG. 4, for example, the interpolation rate is controlled within the range of 50 to 90% when the type of video content is a movie animation, 80 to 100% when the type is a TV animation, 70 to 90% when the type is a drama, 90 to 100% when the type is sports, and 60 to 95% when the type is other.

[0058] For example, in the case of movies, in order to maintain the texture of low frame rates, the range of changeable interpolation rates is from a low interpolation rate such as 40% to an interpolation rate less than 100%, such as 90%, but in the case of more realistic video content such as sports, the range of changeable interpolation rates is allowed to be higher. In particular, sports are often shot at a frame rate such as 60Hz, and the fewer number of interpolated frames reduces the visual discomfort caused by the interpolation and the errors that occur when they are created, so it is desirable to set the changeable range higher, such as an interpolation rate range of 90-100%.

[0059] Fig. 5 shows an example of the transition of the interpolation rate according to the amount of motion. In Fig. 5, the horizontal axis represents the amount of motion, such as the movement in a certain direction across the screen (detected amount of panning). Here, for example, a control can be performed such that the interpolation rate is increased as the amount of motion shown on the horizontal axis increases, based on an interpolation rate of 60% or the like.

[0060] Here, when the interpolation rate is high, the artificial creation of the interpolated image frame can cause problems such as an uncomfortable appearance for the user and errors when the image is created, while when the interpolation rate is low, there is a problem that judder (a phenomenon in which the image appears jittery) becomes noticeable. In particular, scenes in which judder is noticeable include, for example, when the entire screen is panned monotonously in a certain direction, when the entire screen is zoomed in or out, and even when the entire screen is rotating.

[0061] Therefore, in this technology, for example, an interpolation rate at which the user's sense of incongruity or errors are relatively unnoticeable (for example, an interpolation rate of 60%) is used as a reference, and scenes where judder is noticeable, such as panning scenes, are detected, and control is performed to raise the interpolation rate (towards a target interpolation rate). As a result, in the case of movies, for example, it is possible to display images that are easier for the user to watch by reducing judder in scenes where judder is noticeable and stressful, while retaining the impression of a film signal.

[0062] (Interpolation) FIG. 6 is a flowchart illustrating the flow of the interpolation process executed by the interpolation unit 106 in FIG.

[0063] In step S31, the interpolation unit 106 determines whether or not the interpolation rate set by the interpolation rate control unit 105 has been changed. If it is determined in step S31 that the interpolation rate has been changed, the process proceeds to step S32.

[0064] In step S32, the interpolation unit 106 changes the current interpolation rate in accordance with the control from the interpolation rate control unit 105. Here, corresponding to the processing in step S14 in Fig. 3, when a predetermined holding period of, for example, 10 V is satisfied, the current interpolation rate is changed by one step toward the target interpolation rate and set in the interpolation unit 106.

[0065] When the process of step S32 ends, the process proceeds to step S33. If it is determined in step S31 that the interpolation rate has not been changed, step S32 is skipped and the process proceeds to step S33.

[0066] In step S33, the interpolation unit 106 generates an interpolated image frame according to the interpolation rate set by the interpolation rate control unit 105, based on the digital component signal YUV(prev, succ) from the memory control unit 102 and the motion vector mv.

[0067] Here, a number of interpolated image frames are inserted between the original image frames (prev, succ) according to the frame rate conversion, and based on the interpolation rate set by the interpolation rate control unit 105, the lower the interpolation rate, the closer the interpolated image frames are to the original image frames.

[0068] When the process of step S33 ends, the process proceeds to step S34, where it is determined whether or not to end the process.

[0069] If it is determined in step S34 that the process should not end, the process returns to step S31, and the processes of steps S31 to S34 are repeated. By repeating the processes of steps S31 to S34, the interpolation rate set in the interpolation unit 106 is changed stepwise so as to successively approach the target interpolation rate, and an interpolated image frame according to the changed interpolation rate is generated.

[0070] If it is determined in step S34 that the process is to be ended, the interpolation process is ended.

[0071] The flow of the interpolation process has been described above. In this interpolation process, when generating an interpolated image frame for an original image frame along the time axis, the interpolated image frame is generated according to the interpolation rate set by the above-mentioned first interpolation rate control process (the process of step S14 in FIG. 3).

[0072] Here, Fig. 7 shows an example of displaying an interpolated image frame according to an interpolation rate. Note that Fig. 7A shows a case where the interpolation rate is 100%, while Fig. 7B shows a case where the interpolation rate is 50%. In Fig. 7, the direction of time is from the left side to the right side.

[0073] For example, when converting the frame rate from 24 Hz to 120 Hz, it is necessary to insert four interpolated image frames between the original image frames along the time axis, so in Figures 7A and 7B, interpolated image frames #1 to #4 are inserted between original image frames #1 and #2, respectively.

[0074] In FIG. 7A, when the interpolation rate is 100%, the change position between the original image frame #1 and the original image frame #2 is uniformly moved, and the interpolated image frame #1 is A , Interpolated Frame #2 A , Interpolated frame #3 A , Interpolated frame #4 A More specifically, in A of FIG. 7, when the object is moving from right to left in the screen when the original image frames #1 and #2 are focused on, the interpolated image frame #1 is generated between the original image frames #1 and #2. A ~#4 A If we pay attention to this, we can see that the object on the screen moves evenly from right to left within the screen for each interpolated image frame.

[0075] On the other hand, in FIG. 7B, when the interpolation rate is 50%, the movement is not uniform as in the case of the above-mentioned interpolation rate of 100%. In other words, when the interpolation rate is 50%, the four interpolated image frames #1 and #2 are inserted between the original image frames #1 and #2. B~#4 B Of these, two interpolated frames #1 from original frame #1 B ,#2 B In the case of 100% interpolation, the image is 50% closer to the original frame #1, while the two interpolated frames #3 from the original frame #2 are B ,#4 B In this case, the image is shifted 50% toward original frame #2 compared to when the interpolation rate is 100%.

[0076] Furthermore, although not shown in the figure, when the interpolation rate is 25%, the two interpolated image frames #1 and #2 from original image frame #1 become images that are 50% more dependent on original image frame #1 than when the interpolation rate is 50%, while the two interpolated image frames #3 and #4 from original image frame #2 become images that are 50% more dependent on original image frame #2 than when the interpolation rate is 50%.

[0077] When the interpolation rate becomes 0%, the two interpolated image frames #1 and #2 from the original image frame #1 have the same image as the original image frame #1, while the two interpolated image frames #3 and #4 from the original image frame #2 have the same image as the original image frame #2. In other words, when the interpolation rate is 0%, three original image frames #1 and three original image frames #2 are displayed in order (the original image frames are displayed in the order #1, #1, #1, #2, #2, #2).

[0078] As described above, in the first embodiment, when generating interpolated image frames (e.g., four interpolated image frames #1, #2, #3, #4) for original image frames (e.g., original image frames #1 and #2) along the time axis, a target interpolation rate is determined based on the amount of motion (e.g., the movement of the entire screen or the movement of an object) detected from the original image frames, and the interpolation rate of the interpolated image frames is controlled based on the determined target interpolation rate.

[0079] For example, in the case of a film signal such as a movie, if the interpolation is performed at an interpolation position that divides the magnitude of the image movement between the original frames equally, the judder is significantly reduced and the image movement becomes very smooth. As a result, users who are accustomed to the judder of film signals may feel that it is not like a film signal and that the sense of speed and tension is reduced. On the other hand, in a panning scene where the entire screen moves, the judder is more noticeable, and some users may feel stressed.

[0080] In contrast, with this technology, by controlling the interpolation rate to change depending on the scene in the video, it is possible to realize an interpolation that is easier to view when interpolating the movement of the original frames.For example, in the case of a movie, while retaining the impression of a film signal, in scenes where judder is noticeable and stressful, it is possible to reduce judder and provide an image that is easier to view.

[0081] Furthermore, when controlling the interpolation rate to change depending on the video content, if the interpolation rate for each video content is fixed (for example, the interpolation rate for movies is fixed at X% and the interpolation rate for animation is fixed at Y%), it will be impossible to control the rate in response to scenes where it is better to have judder or scenes where it is better to have no judder.

[0082] In contrast, with the present technology, when control is performed to change the interpolation rate depending on the video scene, it is possible to control to change the movable range of the interpolation rate depending on the type of video content, so that it is possible to set an interpolation rate suitable for each video content, and by changing the interpolation rate depending on the scene, it is possible to provide video that is easier to watch.

[0083] In the above-mentioned Patent Document 1, the amount of processing during imaging blur suppression processing is changed according to the reliability of the detection of the motion vector, or the gain multiplied by the motion vector is changed according to the reliability. On the other hand, in the present technology, regardless of the reliability, a case where the screen faces a certain direction or the main object faces a certain direction is detected and the interpolation rate is made variable.

[0084] <2. Second embodiment>

[0085] However, even with a video signal having a low frame rate, there are scenes in which the entire screen shakes, giving a sense of speed or tension, or scenes in which the main object shakes, and when controlling the interpolation rate in such scenes, it is necessary to control the interpolation rate so that it does not become too high. Next, a case will be described in which the interpolation rate of an interpolated image frame is controlled based on the amount of shaking detected from an original image frame.

[0086] Note that the second embodiment differs from the first embodiment in that in the signal processing device 10 of FIG. 1, the interpolation rate control unit 105 determines a target interpolation rate based on the amount of shaking detected from an original image frame. However, since the other configurations are similar, a description of overlapping parts will be omitted as appropriate.

[0087] (Method of shaking detection and interpolation rate determination) FIG. 8 is a diagram showing an example of a method of detecting a shaking based on a motion vector and a method of determining a target interpolation rate according to the detection result of the shaking.

[0088] As shown in FIG. 8, there are two types of detection items detected from the original image frame: shaking of the entire screen and shaking of the target object.

[0089] The shaking of the entire screen can be detected, for example, by the following detection method: That is, in the four combinations of diagonal directions, namely left and right, upper left and lower right, upper and lower, and upper right and lower left, if there is a combination in which the magnitude of the motion vector amount of the entire screen switches every few V (Vertical Sync), it is determined that the entire screen is shaking.

[0090] In addition, the amount of shaking of the entire screen can be obtained from the amount of motion vectors of the combinations that are determined to be shaking. That is, the amount of shaking, which is the sum of the shaking in a specific combination of diagonal directions, is calculated based on the amount of motion vectors obtained for each combination of predetermined diagonal directions on the entire screen.

[0091] In this case, the interpolation rate control unit 105 determines a target interpolation rate according to the detected amount of shaking of the entire screen.

[0092] In addition, the shaking of the object can be detected, for example, by the following detection method. That is, among the shaking amounts of the above four combinations (combinations of left and right, upper left and lower right, upper and lower, and upper right and lower left), the largest shaking amount T can be set as the shaking amount T' of the object, and here, this value is converted (converted) into a value for the entire screen.

[0093] As in the case of the sum of the vectors of the object's movement described above, for example, by detecting an area of ​​A% of the entire screen area, the ratio of the area corresponding to the object (moving object) to the entire screen can be calculated, and the shaking amount T can be converted to the shaking amount T' of the object using this ratio. More specifically, the shaking amount T' of the object satisfies the relationship of the following formula (2).

[0094] T' = T × 100 / (100 - A) ···(2)

[0095] In this case, the interpolation rate control unit 105 determines a target interpolation rate according to the detected shaking amount T' of the object.

[0096] Here, the shaking of the entire screen and the shaking of the object are described as detection items, but the target interpolation rate can be determined according to at least one of the shaking amounts. The shaking of the entire screen and the shaking of the object are examples of parameters according to the video scene, and other parameters may be used.

[0097] (Second Interpolation Rate Control Process) FIG. 9 is a flowchart illustrating the flow of the second interpolation rate control process executed by the interpolation rate control unit 105 of FIG.

[0098] In steps S51 to S55, a process for controlling the interpolation rate is executed similarly to steps S11 to S15 in FIG. 3. However, a difference is that in step S53, when determining a target interpolation rate, a shaking amount obtained from a motion vector is used instead of a motion vector sum amount obtained from a motion vector.

[0099] That is, in the second interpolation rate control process, the processes of steps S51 to S55 are repeated at a predetermined timing such as 1V, and when a predetermined holding period such as 10V is satisfied ("YES" in S51), a target interpolation rate according to the amount of shaking is determined (S53), and the current interpolation rate is gradually changed by one step so as to approach the target interpolation rate, and is set in the interpolation unit 106 (S54).

[0100] The flow of the second interpolation rate control process has been described above. In this second interpolation rate control process, when generating an interpolated image frame for an original image frame along a time axis, the interpolation rate of the interpolated image frame is controlled according to the movement between the original image frames. More specifically, when controlling the interpolation rate, a target interpolation rate is determined based on the amount of shaking detected from the original image frames, and the interpolation rate is controlled based on the determined target interpolation rate.

[0101] As described above, in the second embodiment, it is possible to detect not only the movement of the entire screen or the movement of the object, but also the shaking of the entire screen or the shaking of the object, and to control changing the interpolation rate in order to maintain the impression of the speed and tension of the scene.

[0102] <3. Modifications>

[0103] In the above description, the signal processing device 10 has been described as an independent device, but it may be configured to be included in a device such as a display device.

[0104] (Example of display device configuration) FIG. 10 is a block diagram showing a configuration example of a display device to which the present technology is applied.

[0105] The display device 1 is a device having a display unit, such as a television receiver, a display device, a personal computer, a tablet terminal, a smartphone, a mobile phone, a head mounted display, or a game machine.

[0106] In FIG. 10, the display device 1 includes a signal input unit 11, a signal processing unit 12, and a display unit 13.

[0107] The signal input unit 11 is configured as, for example, a tuner, and supplies a video signal obtained by demodulating a broadcast signal received via an antenna to the signal processing unit 12. The broadcast signal here includes, for example, a video signal of broadcast content transmitted according to a predetermined broadcasting method such as terrestrial broadcasting or satellite broadcasting.

[0108] In addition, the signal input unit 11 is not limited to a tuner, and may be configured, for example, as a communication module so that a video signal of communication content distributed via a communication network such as the Internet is input to the signal processing unit 12.

[0109] In addition, the signal input unit 11 may be configured as an interface conforming to a predetermined standard such as HDMI (High Definition Multimedia Interface) (registered trademark) or USB (Universal Serial Bus), so that video signals of recorded content recorded on a recorder or recorded content recorded on a recording medium such as an optical disc or semiconductor memory are input to the signal processing unit 12.

[0110] The above-mentioned broadcast content, communication content, and recorded content are merely examples of content, and the video signals of various types of content can be converted into a YUV digital component signal and input to the signal processing unit 12.

[0111] The signal processing unit 12 has a configuration corresponding to the signal processing device 10 of Fig. 1. The signal processing unit 12 executes an interpolation rate control process (Fig. 3 or Fig. 9) and an interpolation process (Fig. 6) on a video signal (digital component signal YUV) supplied from the signal input unit 11. The video signal in which an interpolated image frame is inserted into an original image frame by the interpolation rate control process or the interpolation process is supplied to a display unit 13.

[0112] The display unit 13 is composed of, for example, a liquid crystal display unit such as an LCD (Liquid Crystal Display) or a self-luminous display unit such as an OLED (Organic Light Emitting Diode), etc. The display unit 13 is driven by converting the video signal supplied from the signal processing unit 12 into an RGB signal as necessary, and displays a video (image) according to the video signal.

[0113] In addition, in the display device 1 of Figure 10, for the sake of simplicity, only a series of video signals is shown, but a series of audio signals may also be included so that audio is output in synchronization with the video of the content.

[0114] (Example of processing device configuration) FIG. 11 is a block diagram showing an example configuration of a display system to which the present technology is applied.

[0115] 11, the display system includes a processing device 2 and a display device 3. The processing device 2 is, for example, a device such as a set-top box, a recorder, a playback device (player), or a storage device. The display device 3 is, for example, a device such as a display device, a television receiver, a tablet terminal, or a personal computer.

[0116] The processing device 2 and the display device 3 are connected by a cable or the like via an interface that complies with a predetermined standard.

[0117] In FIG. 11, the processing device 2 includes a signal input unit 21 and a signal processing unit 22.

[0118] The signal input unit 21 is configured as, for example, a tuner, a communication module, or an interface conforming to a predetermined standard such as HDMI (registered trademark) or USB.

[0119] The signal processing unit 22 has a configuration corresponding to the signal processing device 10 of Fig. 1. The signal processing unit 22 executes an interpolation rate control process (Fig. 3 or Fig. 9) and an interpolation process (Fig. 6) on the video signal (digital component signal YUV) supplied from the signal input unit 21. The video signal in which an interpolated image frame is inserted into an original image frame by the interpolation rate control process and the interpolation process is output to the display device 3 via a predetermined interface such as a cable. In this case, the signal processing unit 22 has a configuration including a processing part that performs signal conversion of the digital component signal YUV suitable for the predetermined interface.

[0120] In addition, in FIG. 11, the display device 3 includes a display unit 31.

[0121] The display unit 31 is composed of, for example, a liquid crystal display unit such as an LCD, a self-luminous display unit such as an OLED, etc. The display unit 31 is driven based on a video signal input from the processing device 2, and displays a video (image) according to the video signal.

[0122] In FIG. 11, the processing device 2 and the display device 3 are connected via a specified interface such as a cable, but this is not limiting, and for example, video signals may be exchanged according to a communication method that complies with a specified standard.

[0123] (Other configurations) In the above description, the first embodiment describes a case where the target interpolation rate is determined based on the amount of movement of the entire screen or an object detected from the motion vector of the original image frame, and the second embodiment describes a case where the target interpolation rate is determined based on the amount of shaking of the entire screen or an object detected from the motion vector of the original image frame, but the target interpolation rate may be determined based on the amount of movement and amount of shaking of the entire screen or an object. That is, the interpolation rate control unit 105 can determine the target interpolation rate based on at least one of the amount of motion and the amount of shaking detected from the original image frame, and control the interpolation rate in stages so as to gradually approach the determined target interpolation rate.

[0124] <4. Computer configuration>

[0125] The above-mentioned series of processes (for example, the interpolation rate control process shown in FIG. 3 or FIG. 9) can be executed by hardware or software. When the series of processes is executed by software, a program constituting the software is installed in a computer of each device. FIG. 12 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes by a program.

[0126] In the computer 1000, a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, and a RAM (Random Access Memory) 1003 are interconnected by a bus 1004. An input / output interface 1005 is further connected to the bus 1004. An input unit 1006, an output unit 1007, a recording unit 1008, a communication unit 1009, and a drive 1010 are connected to the input / output interface 1005.

[0127] The input unit 1006 includes a microphone, a keyboard, a mouse, etc. The output unit 1007 includes a speaker, a display, etc. The recording unit 1008 includes a hard disk, a non-volatile memory, etc. The communication unit 1009 includes a network interface, etc. The drive 1010 drives a removable recording medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0128] In the computer 1000 configured as described above, the CPU 1001 loads a program recorded in the ROM 1002 or the recording unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004, and executes it, thereby performing the series of processes described above.

[0129] The program executed by the computer 1000 (CPU 1001) can be provided by being recorded on a removable recording medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0130] In the computer 1000, the program can be installed in the recording unit 1008 via the input / output interface 1005 by mounting the removable recording medium 1011 in the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the recording unit 1008. Alternatively, the program can be installed in the ROM 1002 or the recording unit 1008 in advance.

[0131] Here, in this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing executed in parallel or individually (for example, parallel processing or object-based processing). In addition, the program may be processed by one computer (processor), or may be processed in a distributed manner by multiple computers.

[0132] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0133] Also, each step of the interpolation rate control process shown in Fig. 3 or 9 can be executed by one device or can be shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes included in the one step can be executed by one device or can be shared and executed by multiple devices.

[0134] The present technology can be configured as follows.

[0135] (1) A signal processing unit is provided for controlling an interpolation rate of the interpolated image frame in response to a motion in a certain direction between the original image frames when generating an interpolated image frame that interpolates between the original image frames along a time axis. Display device. (2) The signal processing unit includes: determining a target interpolation rate that is a target of the interpolation rate based on an amount of motion detected from the original image frame; The interpolation rate is controlled based on the determined target interpolation rate. The display device according to (1) above. (3) The signal processing unit changes the interpolation rate stepwise so as to gradually approach the determined target interpolation rate. The display device according to (2) above. (4) The signal processing unit includes: changing the target interpolation rate based on an amount of motion detected from the original frame; The interpolation rate is changed stepwise so as to gradually approach the changed target interpolation rate. The display device according to (3) above. (5) The signal processing unit controls the generation of the interpolated image frame using the changed interpolation rate so as to be continued for a predetermined period. The display device according to any one of (2) to (4). (6) The signal processing unit changes an interpolation rate range, which is a range within which the interpolation rate can be changed, according to a type of content including the original image frame. The display device according to any one of (2) to (5). (7) The signal processing unit includes: Calculating a total amount of motion vectors in a specific direction based on the motion vectors obtained for each predetermined direction in the entire screen corresponding to the original image frame; The target interpolation rate is determined according to the calculated total amount of the motion vectors. The display device according to (2) above. (8) The signal processing unit includes: calculating a total sum of motion vectors of an object in a specific direction within a predetermined area corresponding to the entire screen based on the motion vectors obtained for each predetermined direction in the entire screen corresponding to the original image frame; The target interpolation rate is determined according to the calculated total amount of the motion vectors of the object. The display device according to (2) or (7). (9) The signal processing unit includes: determining a target interpolation rate that is a target of the interpolation rate based on an amount of shaking detected from the original image frame; The interpolation rate is controlled based on the determined target interpolation rate. The display device according to any one of (1) to (8). (10) The signal processing unit includes: Calculating a fluctuation amount, which is a sum of fluctuations in a specific combination of diagonal directions, based on a motion vector amount obtained for each combination of predetermined diagonal directions in the entire screen corresponding to the original image frame; The target interpolation rate is determined according to the calculated amount of shaking. The display device according to (9) above. (11) The signal processing unit includes: Calculating a shaking amount, which is a sum of shaking of an object in a specific combination of diagonal directions in a predetermined area corresponding to the entire screen, based on a motion vector amount obtained for each combination of predetermined diagonal directions in the entire screen corresponding to the original image frame; The target interpolation rate is determined according to the calculated amount of shaking of the object. The display device according to (9) or (10). (12) The interpolation rate is a rate indicating the degree of interpolation of the interpolated image frame. The display device according to any one of (1) to (11). (13) The interpolation rate is set to 100% when one or more of the interpolated frames are generated by uniformly moving the change positions between the original frames, and set to 0% when the interpolated frames are the original frames themselves. The lower the interpolation rate, the closer the interpolated frames become to the original frames. The display device according to (12) above. (14) The signal processing unit includes: an interpolation rate control unit for controlling the interpolation rate; an interpolation unit that generates the interpolated image frame using the interpolation rate and interpolates between the original image frames; have The display device according to any one of (1) to (13). (15) The signal processing unit includes: A motion vector detection unit detects a motion vector from the original image frame, The interpolation rate control unit controls the interpolation rate based on the detected motion vector. The display device according to (14) above. (16) The apparatus further includes a display unit for displaying a video of a content including the original image frame. The display device according to any one of (1) to (15). (17) and an interpolation rate control unit that controls an interpolation rate of the interpolated image frame in response to a motion in a certain direction between the original image frames when generating an interpolated image frame that interpolates between the original image frames along a time axis. Signal processing device. (18) A signal processing device, When generating an interpolated image frame that interpolates between original image frames along a time axis, an interpolation rate of the interpolated image frame is controlled in accordance with a motion in a certain direction between the original image frames. Signal processing methods. [Explanation of symbols]

[0136] 1 display device, 2 processing device, 3 display device, 10 signal processing device, 11 signal input section, 12 signal processing section, 13 display section, 21 signal input section, 22 signal processing section, 31 display section, 101 preprocessing section, 102 memory control section, 103 memory, 104 motion vector detection section, 105 interpolation rate control section, 106 interpolation section, 1000 computer, 1001 CPU

Claims

1. When generating an interpolated image frame that interpolates between original image frames along a time axis, determining a target interpolation rate that is a target of the interpolation rate of the interpolated image frame based on the amount of shaking detected from the original image frame; Based on the determined target interpolation rate, the interpolation rate is controlled in response to a motion in a certain direction between the original frames. A signal processing unit is provided, The interpolation rate is set to 100% when one or more of the interpolated frames are generated by uniformly moving the change positions between the original frames, and set to 0% when the interpolated frames are the original frames themselves, so that the lower the interpolation rate, the closer the interpolated frames become to the original frames. Display device.

2. The signal processing unit includes: determining the target interpolation rate based on an amount of motion detected from the original frame; The interpolation rate is controlled based on the determined target interpolation rate. The display device according to claim 1 .

3. The signal processing unit changes the interpolation rate so as to gradually approach the determined target interpolation rate. Make incremental changes The display device according to claim 2 .

4. The signal processing unit includes: changing the target interpolation rate based on an amount of motion detected from the original frame; The interpolation rate is changed stepwise so as to gradually approach the changed target interpolation rate. The display device according to claim 3 .

5. The signal processing unit controls the generation of the interpolated image frame using the changed interpolation rate so as to be continued for a predetermined period. The display device according to claim 3 .

6. The signal processing unit changes an interpolation rate range, which is a range within which the interpolation rate can be changed, according to a type of content including the original image frame. The display device according to claim 1 .

7. The signal processing unit includes: Calculating a total amount of motion vectors in a specific direction based on the motion vectors obtained for each predetermined direction in the entire screen corresponding to the original image frame; The target interpolation rate is determined according to the calculated total amount of the motion vectors. The display device according to claim 2 .

8. The signal processing unit includes: calculating a total motion vector amount in a specific direction in an area corresponding to an object that is a moving body based on the motion vectors obtained for each predetermined direction in the entire screen corresponding to the original image frame, and converting the calculated total motion vector amount into a value for the entire screen to obtain the total motion vector amount of the object; The target interpolation rate is determined according to the calculated total motion vector of the object. The display device according to claim 2 .

9. The signal processing unit includes: Calculating a fluctuation amount, which is a sum of fluctuations in a specific combination of diagonal directions, based on a motion vector amount obtained for each combination of predetermined diagonal directions in the entire screen corresponding to the original image frame; The target interpolation rate is determined according to the calculated amount of shaking. The display device according to claim 1 .

10. The signal processing unit includes: a shaking amount is calculated based on the motion vector amount obtained for each combination of predetermined diagonal directions in the entire screen corresponding to the original image frame, the shaking amount being the sum of the shaking in a specific combination of diagonal directions in an area corresponding to the object which is a moving body, and the calculated shaking amount is converted into a value for the entire screen to obtain the shaking amount of the object; The target interpolation rate is determined according to the calculated amount of shaking of the object. The display device according to claim 1 .

11. The interpolation rate is a rate indicating the degree of interpolation of the interpolated image frame. The display device according to claim 1 .

12. The signal processing unit includes: an interpolation rate control unit for controlling the interpolation rate; an interpolation unit that generates the interpolated image frame using the interpolation rate and interpolates between the original image frames; have The display device according to claim 1 .

13. The signal processing unit includes: A motion vector detection unit detects a motion vector from the original image frame, The interpolation rate control unit controls the interpolation rate based on the detected motion vector. The display device according to claim 12.

14. The apparatus further includes a display unit for displaying a video of a content including the original image frame. The display device according to claim 1 .

15. When generating an interpolated image frame that interpolates between original image frames along a time axis, determining a target interpolation rate that is a target of the interpolation rate of the interpolated image frame based on the amount of shaking detected from the original image frame; Based on the determined target interpolation rate, a motion in a certain direction between the original frames is Accordingly, an interpolation rate control unit is provided for controlling the interpolation rate, The interpolation rate is set to 100% when one or more of the interpolated frames are generated by uniformly moving the change positions between the original frames, and set to 0% when the interpolated frames are the original frames themselves, so that the lower the interpolation rate, the closer the interpolated frames become to the original frames. Signal processing device.

16. A signal processing device, When generating an interpolated image frame that interpolates between original image frames along a time axis, determining a target interpolation rate that is a target of the interpolation rate of the interpolated image frame based on the amount of shaking detected from the original image frame; controlling the interpolation rate according to a motion in a certain direction between the original frames based on the determined target interpolation rate; The interpolation rate is set to 100% when one or more of the interpolated frames are generated by uniformly moving the change positions between the original frames, and set to 0% when the interpolated frames are the original frames themselves, so that the lower the interpolation rate, the closer the interpolated frames become to the original frames. Signal processing methods.

Images