Video processing method, program, and video processing system

Abstract

This video processing method makes it possible to more easily detect a pattern image superimposed on a video in order to correct the deviation of the display position of the video without the pattern image being recognized by a viewer. In this video processing method, a pattern image (a third pattern image) is acquired from the acquired one or more superimposed subframes. In the video processing method, the deviation of the display position of a video projected on a display surface by comparing a feature point of the acquired pattern image with a standard feature point. In the video processing method, increase / decrease processing for increasing or decreasing the degree of acquiring the pattern image from the acquired one or more superimposed subframes according to the acquired result of the pattern image is executed.

Description

Image processing method, program, and image processing system

[0001] The present disclosure relates to a video processing method, a program, and a video processing system.

[0002] Patent Literature 1 discloses an image processing device that applies a homography transformation to a pattern captured image obtained by an imaging unit capturing a predetermined structured light pattern projected by a projection unit. The image processing device then uses the pattern captured image to which the homography transformation has been applied to detect corresponding points between the projected image projected by the projection unit and the captured image captured by the imaging unit.

[0003] International Publication No. 2018 / 225531

[0004] The present disclosure provides a video processing method and the like that makes it easy to detect a pattern image that is superimposed on a video to correct a shift in the display position of the video without being recognized by the viewer.

[0005] A video processing method according to one aspect of the present disclosure acquires a plurality of subframes obtained by temporally dividing a frame included in video data, performs an output process to output one or more superimposed subframes obtained by superimposing a pattern image on one or more subframes based on the plurality of subframes so as to be displayed on a display surface, acquires the one or more superimposed subframes displayed on the display surface by imaging, acquires the pattern image from the one or more acquired superimposed subframes, compares feature points of the acquired pattern image with reference feature points to detect a deviation in the display position of the image projected on the display surface, and performs an increase / decrease process to increase / decrease the degree to which the pattern image is acquired from the one or more acquired superimposed subframes depending on the acquisition result of the pattern image.

[0006] The present disclosure has an advantage in that a pattern image superimposed on a video to correct a shift in the display position of the video can be easily detected without being recognized by the viewer.

[0007] FIG. 1 is a schematic diagram showing an overall configuration including a video processing system according to an embodiment. FIG. 2 is a block diagram showing the configuration of a projection device according to an embodiment. FIG. 3 is a flowchart showing an example of embedding determination processing. FIG. 4 is a flowchart showing an example of processing for generating a plurality of subframes. FIG. 5 is a diagram showing an example of a pattern image. FIG. 6 is an explanatory diagram of an example of operation of a video selection unit of a projection device according to an embodiment. FIG. 7 is a schematic diagram of a video projection unit of a projection device according to an embodiment. FIG. 8 is a diagram showing correlation between a control signal applied to a light path shift element and a video signal. FIG. 9 is a block diagram showing the configuration of an imaging device according to an embodiment. FIG. 10 is a flowchart showing an example of detection processing of a pattern image. FIG. 11 is a block diagram showing the configuration of a control device according to an embodiment. FIG. 12 is a flowchart showing an example of initialization of misalignment correction processing. FIG. 13 is a diagram showing an example of feature points of a third pattern image. FIG. 14 is a flowchart showing an example of misalignment correction processing. FIG. 15 is a flowchart showing an example of operation for misalignment correction of a video processing system according to an embodiment. FIG. 16 is an explanatory diagram of a first example of increase / decrease processing. FIG. 17 is an explanatory diagram of a second example of increase / decrease processing. FIG. 18 is an explanatory diagram of a third example of increase / decrease processing. Fig. 19 is a diagram showing an example of a mask image used in a third example of increase / decrease processing. Fig. 20 is a sequence diagram showing an example of operation of a video processing system according to an embodiment. Fig. 21 is a schematic diagram showing an overall configuration including a video processing system according to a first modified example of an embodiment. Fig. 22 is a sequence diagram showing an example of operation of a video processing system according to a first modified example of an embodiment. Fig. 23 is a sequence diagram showing an example of operation of a video processing system according to a second modified example of an embodiment.

[0008] [1. Findings that Form the Basis of the Present Disclosure] First, the inventor's viewpoint will be explained below.

[0009] Conventionally, a known image processing method involves capturing an image of a projection image using an imaging device and performing geometric correction of the projection image using the captured image to correct distortion of the projection image projected onto a display surface such as a screen by a projection device (projector), i.e., to correct a misalignment in the display position of the projection image. Misalignment in the display position of the projection image can occur due to disturbances such as vibrations that cause the projection device to shift in position. The inventors of the present application have been studying a method for performing geometric correction of a projection image while the viewer is watching the image, i.e., while the image is being projected onto a display surface by the projection device, without the viewer noticing. Hereinafter, this image processing method will be referred to as a "comparative image processing method."

[0010] In the video processing method of the comparative example, a frame included in video data is temporally divided into a plurality of subframes (four in this example) and projected onto a display surface. A first pattern image and a second pattern image are superimposed on two of the subframes. The first pattern image is an image including a predetermined pattern binarized in black and white. The second pattern image is an image in which the luminance values ​​of each pixel of the first pattern image are inverted, i.e., the black and white of the predetermined pattern included in the first pattern image are inverted.

[0011] Hereinafter, a subframe on which the first pattern image is superimposed will be referred to as a "first superimposed subframe," and a subframe on which the second pattern image is superimposed will be referred to as a "second superimposed subframe." Here, assuming that the two subframes are the same original image, the first superimposed subframe will be an image in which the first pattern image is superimposed on the original image, and the second superimposed subframe will be an image in which the second pattern image is superimposed on the original image. The original image is an image included in video data on which no pattern image is superimposed.

[0012] In the video processing method of the comparative example, the first and second superimposed subframes are captured by an imaging device, and a difference image is obtained by calculating the difference between the captured image of the first superimposed subframe and the captured image of the second superimposed subframe. The pattern shape of the difference image and the pattern shape of the first pattern image generally match. This is because the original image can be removed by calculating the difference between the first and second superimposed subframes. Hereinafter, this difference image will also be referred to as a "third pattern image."

[0013] The image processing method of the comparative example then compares the feature points of the third pattern image with the reference feature points to detect a deviation in the display position of the image projected onto the display surface, and corrects the deviation in the display position of the image based on the detection result. Note that the detection of the deviation in the display position of the image and the correction of the deviation in the display position of the image will be described in detail in [2. Configuration] below.

[0014] Here, in order to accurately detect the deviation in the display position of the video, it is necessary to accurately detect the pattern image from the first superimposed sub-frame and the second superimposed sub-frame captured by the imaging device. Accurate detection of the pattern image can be achieved, for example, by increasing the pixel values ​​of the pattern images (here, the first pattern image and the second pattern image). However, simply increasing the pixel values ​​of the pattern images poses a problem in that the first superimposed sub-frame and the second superimposed sub-frame projected onto the display surface are likely to deviate from the original image, making the pattern image more noticeable to the viewer.

[0015] In view of the above, the inventors have come up with the present disclosure.

[0016] Hereinafter, embodiments will be described with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection forms, steps, step order, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not described in independent claims will be described as optional components.

[0017] It should be noted that the drawings are schematic diagrams and are not necessarily strict illustrations. In addition, in the drawings, substantially the same components are denoted by the same reference numerals, and overlapping descriptions may be omitted or simplified.

[0018] (Embodiment) [2. Configuration] [2-1. Overall Configuration] First, the overall configuration including a video processing system 100 according to an embodiment will be described. FIG. 1 is a block diagram showing the overall configuration including a video processing system 100 according to an embodiment. The video processing system 100 includes two projection devices 1A and 1B, two image capture devices 2A and 2B, and a control device 3. The video processing system 100 is a system that processes video data transmitted from a playback device 4. Hereinafter, when there is no need to distinguish between the two projection devices 1A and 1B, they will be referred to as "projection device 1." Furthermore, when there is no need to distinguish between the two image capture devices 2A and 2B, they will be referred to as "image capture device 2."

[0019] The projection device 1 is a device having a projector function, and projects an image onto the display surface 50 of the screen 5 based on the image data included in the image signal transmitted from the reproduction device 4. The projection device 1 is not limited to projecting an image onto the display surface 50 of the screen 5, but may also project an image onto the surface of a structure other than a screen, such as a wall, as the display surface 50. In this embodiment, two projection devices 1A and 1B each project an image onto the display surface 50, thereby projecting a composite image onto the display surface 50.

[0020] The imaging device 2 is a device with a camera function, and captures an image projected onto the display surface 50. In the embodiment, the imaging device 2A captures an image projected onto the display surface 50 by the projection device 1A. The imaging device 2B captures an image projected onto the display surface 50 by the projection device 1B. In the embodiment, the imaging device 2 is a device separate from the projection device 1, but may be built into the projection device 1. Specifically, the imaging devices 2A and 2B may be built into the projection devices 1A and 1B, respectively.

[0021] The control device 3 is an information terminal such as a desktop or laptop personal computer, and controls the projection device 1 and the imaging device 2 by communicating with them via a network N1 such as a LAN (Local Area Network). The communication between the projection device 1, the imaging device 2 and the control device 3 is performed in accordance with a known network protocol such as HTTP (Hypertext Transfer Protocol), FTP (File Transfer Protocol), or TCP (Transmission Control Protocol).

[0022] In the embodiment, the control device 3 is realized by installing software dedicated to the video processing system 100 on a general-purpose information terminal. Note that the control device 3 is not limited to a general-purpose information terminal, but may also be an information terminal dedicated to the video processing system 100. Furthermore, the information terminal is not limited to a personal computer, but may also be realized by, for example, a smartphone or a tablet terminal.

[0023] The playback device 4 is a device that has a function of playing back video recorded on optical media such as a DVD (Digital Versatile Disc, registered trademark) or a BD (Blu-ray (registered trademark) Disc). Note that the playback device 4 may also be a device that has a function of playing back video recorded on a storage device such as an HDD (Hard Disc Drive).

[0024] [2-2. Projection Device] Next, the configuration of the projection device 1 will be described in detail. Fig. 2 is a block diagram showing the configuration of the projection device 1 according to the embodiment. As shown in Fig. 2, the projection device 1 includes an image input unit 11, an image generation unit 12, a synchronization signal extraction unit 13, an image selection unit 14, an image projection unit 15, a synchronization signal output unit 16, a communication unit 17, a parameter storage unit 18, and a superimposition pattern storage unit 19. The image input unit 11, the image generation unit 12, the synchronization signal extraction unit 13, the image selection unit 14, the image projection unit 15, the synchronization signal output unit 16, and the communication unit 17 may each be realized by a dedicated circuit, or may be realized by a processor executing a corresponding computer program stored in memory.

[0025] The video input unit 11 acquires a video signal input from the outside (here, the playback device 4) and converts the acquired video signal into an internal video signal. Here, the resolution and frame rate of the video signal are not particularly limited. In other words, video signals having various resolutions or frame rates are input to the video input unit 11 from the playback device 4. In this embodiment, the internal video signal has a 4K resolution and a first frame rate (e.g., 60 fps).

[0026] The video generation unit 12 performs various processes on the internal video signal from the video input unit 11. First, the video generation unit 12 performs an embedding determination process to determine whether a pattern image can be embedded (superimposed) in the internal video signal. In this embodiment, the pattern image is embedded in the blue signal of the internal video signal, which has a relatively low brightness sensitivity for humans. In other words, both pattern images (a first pattern image PP1 and a second pattern image PP2, which will be described later) are superimposed on the video signal of the blue component. Therefore, in this embodiment, the video generation unit 12 performs the embedding determination process on the blue signal of the internal video signal.

[0027] The embedding determination process will be described below with reference to Fig. 3. Fig. 3 is a flowchart showing an example of the embedding determination process. The embedding determination process described below is executed for each frame F1 (see Fig. 6, which will be described later).

[0028] First, the video generator 12 counts the number N of pixels whose signal value (pixel value) of the blue signal among the internal video signals is within a predetermined range (S101). Here, the predetermined range is the range between the upper and lower limit values ​​of the blue signal value, which is a parameter stored in the parameter storage unit 18. If the signal value of the blue signal is within the predetermined range, it is possible to embed a pattern image by increasing or decreasing the signal value of the blue signal. On the other hand, if the signal value of the blue signal is outside the predetermined range, the blue signal will be saturated when the signal value of the blue signal is increased or decreased, making it impossible to embed a pattern image.

[0029] Next, the image generation unit 12 compares the counted pixel number N with a value obtained by multiplying the total number of pixels included in the frame F1 by the validity ratio (S102). Here, the validity ratio is a parameter stored in the parameter storage unit 18, and represents the percentage of all pixels in the frame F1 into which a pattern image can be embedded. If the pixel number N is equal to or greater than the value obtained by multiplying the total number of pixels by the validity ratio (S102: Yes), the image generation unit 12 determines that a pattern image can be embedded in the frame F1 (S103). On the other hand, if the pixel number N is less than the value obtained by multiplying the total number of pixels by the validity ratio (S102: No), the image generation unit 12 determines that a pattern image cannot be embedded in the frame F1 (S104).

[0030] If the embedding mode is "enabled," the video generation unit 12 executes steps S101 to S104. If the embedding mode is "disabled," the video generation unit 12 executes step S104 without executing steps S101 and S102. If the embedding mode is "forced," the video generation unit 12 executes step S103 without executing steps S101 and S102. Here, the embedding mode is a parameter held in the parameter holding unit 18.

[0031] Second, the image generation unit 12 performs geometric correction on the internal image signal in accordance with a geometric correction look-up table (LUT). This process corrects the misalignment of the display position of the image projected onto the display surface 50 from the projection device 1. The LUT for geometric correction is a parameter stored in the parameter storage unit 18.

[0032] Third, the video generator 12 executes a generation process for generating a plurality of sub-frames SF1 (see FIG. 6 described later) by temporally dividing the frame F1. The generation process will be described below with reference to FIG. 4. FIG. 4 is a flowchart showing an example of the generation process for a plurality of sub-frames SF1. The generation process described below is executed for each frame F1.

[0033] First, the video generation unit 12 generates subframe "A" (i.e., subframe SF11) by extracting odd-numbered pixels from among the pixels in the X direction (horizontal direction) of frame F1 and odd-numbered pixels from among the pixels in the Y direction (vertical direction) of frame F1 (S201). The video generation unit 12 also generates subframe "B" (i.e., subframe SF12) by extracting even-numbered pixels from among the pixels in the X direction of frame F1 and odd-numbered pixels from among the pixels in the Y direction of frame F1 (S202). The video generation unit 12 also generates subframe "C" (i.e., subframe SF13) by extracting even-numbered pixels from among the pixels in the X direction of frame F1 and even-numbered pixels from among the pixels in the Y direction of frame F1 (S203). In addition, the video generation unit 12 generates subframe "D" (i.e., subframe SF14) by extracting odd-numbered pixels from the pixels in the X direction of frame F1 and even-numbered pixels from the pixels in the Y direction of frame F1 (S204).

[0034] Each of these multiple subframes SF1 is an image composed only of subpixels of the same phase in each pixel of frame F1. For example, if each pixel of frame F1 is composed of four subpixels "A," "B," "C," and "D," each pixel of subframe "A" is composed only of the subpixel "A" of the corresponding pixel in frame F1.

[0035] Next, the video generation unit 12 refers to the result of the embedding determination process for frame F1 (S205). If the result of the embedding determination process indicates that the pattern image cannot be embedded (S205: No), the video generation unit 12 ends the generation process. On the other hand, if the result of the embedding determination process indicates that the pattern image can be embedded (S205: Yes), the video generation unit 12 then executes a process to determine the type of pattern image to embed in frame F1.

[0036] Fig. 5 shows examples of pattern images. (a) to (c) of Fig. 5 each represent a first pattern image PP1, and (d) to (f) of Fig. 5 each represent a second pattern image PP2. Specifically, (a) of Fig. 5 represents a first pattern image PP11 for the R (red) channel, (b) of Fig. 5 represents a first pattern image PP21 for the G (green) channel, and (c) of Fig. 5 represents a first pattern image PP31 for the B (blue) channel. (d) of Fig. 5 represents a second pattern image PP12 for the R channel, (e) of Fig. 5 represents a second pattern image PP22 for the G channel, and (f) of Fig. 5 represents a second pattern image PP32 for the B channel.

[0037] In the embodiment, the video generation unit 12 sequentially embeds a first pattern image PP11 and a second pattern image PP12 for the R channel, a first pattern image PP21 and a second pattern image PP22 for the G channel, and a first pattern image PP31 and a second pattern image PP32 for the B channel for each frame F1.

[0038] 4 , the video generation unit 12 refers to the result of the embedding determination process for the frame preceding frame F1 (S206). If a pattern image can be embedded in the previous frame (S206: Yes), the video generation unit 12 updates the type of pattern image to be embedded (S207). For example, if a first pattern image PP11 and a second pattern image PP12 for the R channel were embedded in the previous frame, the video generation unit 12 determines that the pattern images to be embedded in frame F1 are the first pattern image PP11 and the second pattern image PP12 for the G channel.

[0039] On the other hand, if it is not possible to embed a pattern image in the previous frame (S206: No), the video generation unit 12 initializes the type of pattern image to be embedded (S208). Here, initialization refers to determining that the pattern images to be embedded in frame F1 are the first pattern image PP11 and the second pattern image PP12 for the R channel.

[0040] In this way, starting from frame F1 where a pattern image has changed from not being embeddable to being embeddable, the video generation unit 12 embeds the first pattern image PP11 and the second pattern image PP12 for the R channel into that frame F1. Then, as long as the determination result that embedding is possible continues, the video generation unit 12 sequentially embeds the first pattern image PP11 and the second pattern image PP12 for the R channel, the first pattern image PP11 and the second pattern image PP12 for the G channel, and the first pattern image PP11 and the second pattern image PP12 for the B channel for each frame F1.

[0041] Next, the video generation unit 12 generates subframe "B'" (S209). Here, subframe "B'" is an image in which the first pattern image PP1 is embedded (superimposed) on a composite image obtained by combining subframes "B" and "D." Specifically, for each pixel of the composite image, the video generation unit 12 adds the embedding signal value α to the signal value of the blue signal of a pixel corresponding to white in the first pattern image PP1, and subtracts the embedding signal value α from the signal value of the blue signal of a pixel corresponding to black in the first pattern image PP1, thereby generating subframe "B'." Here, the embedding signal value α is a parameter held in the parameter holding unit 18.

[0042] The video generation unit 12 also generates subframe "D'" (S210). Here, subframe "D'" is an image in which the second pattern image PP2 is embedded (superimposed) on a composite image obtained by combining subframes "B" and "D." Specifically, for each pixel of the composite image, the video generation unit 12 adds the embedding signal value α to the signal value of the blue signal of a pixel corresponding to white in the second pattern image PP2, and subtracts the embedding signal value α from the signal value of the blue signal of a pixel corresponding to black in the second pattern image PP2, thereby generating subframe "D'."

[0043] The above-mentioned subframe "B'" corresponds to the first superimposed subframe SF21 (see FIG. 6 described later), and the subframe "D'" corresponds to the second superimposed subframe SF22 (see FIG. 6 described later). The composite image obtained by combining subframes "B" and "D" corresponds to both the "first subframe" and the "second subframe."

[0044] As described above, in the embodiment, the first superimposed subframe SF21 is an image obtained by superimposing the first pattern image PP1 on a first subframe (here, the composite image) based on a plurality of subframes SF1. The second superimposed subframe SF22 is an image obtained by superimposing the second pattern image PP2 on a second subframe (here, the composite image) based on a plurality of subframes SF1. The first and second subframes are both images obtained by combining two subframes (here, subframes "B" and "D") from the plurality of subframes SF1, and are the same image. On the other hand, subframes SF11 and SF13, on which neither the first pattern image PP1 nor the second pattern image PP2 is superimposed, are both images different from both the first superimposed subframe SF21 and the second superimposed subframe SF22.

[0045] The synchronization signal extraction unit 13 generates an internal synchronization signal having the same frame rate (60 fps in this case) as the frame rate of the internal video signal based on a synchronization signal input together with the video signal from outside (the playback device 4 in this case). The internal synchronization signal is provided to the video generation unit 12, the video selection unit 14, and the video projection unit 15. The video generation unit 12, the video selection unit 14, and the video projection unit 15 operate for each frame based on the internal synchronization signal.

[0046] The video selection unit 14 selects a subframe set of video to be projected onto the display surface 50 from the video projection unit 15, depending on the result of the embedding determination process for the frame F1 performed by the video generation unit 12. Here, the subframe set is made up of a plurality of subframes SF1 corresponding to the frame F1.

[0047] 6 is an explanatory diagram of an example of the operation of the video selection unit 14 of the projection device 1 according to the embodiment. As shown in Fig. 6, when the result of the embedding determination process for frame F1 indicates that embedding of a pattern image is not possible, the video selection unit 14 selects a subframe set consisting of subframes "A," "B," "C," and "D." Here, subframes "A," "B," "C," and "D" correspond to subframes SF11, SF12, SF13, and SF14, respectively.

[0048] 6, if the result of the embedding determination process for frame F1 indicates that a pattern image can be embedded, the video selector 14 selects a subframe set consisting of subframes "A," "B'," "C," and "D'." Here, subframes "A," "B'," "C," and "D'" correspond to subframe SF11, first superimposed subframe SF21, subframe SF13, and second superimposed subframe SF22, respectively.

[0049] As described above, in the embodiment, video selection unit 14 selects a subframe set to be output to display surface 50 in accordance with the result of the embedding determination process in video generation unit 12. Also, as already described, in the embedding determination process, for each frame F1, it is determined whether or not it is possible to embed a pattern image by referring to the signal value of the blue signal in the internal video signal. In other words, video processing system 100 according to the embodiment determines whether or not to output first superimposed subframe SF21 and second superimposed subframe SF22 to display surface 50 based on the pixel value of the video signal in frame F1 (here, the signal value of the blue signal).

[0050] The image projection unit 15 projects an image onto the display surface 50 in accordance with the image signal of the subframe set selected by the image selection unit 14. The specific configuration and operation of the image projection unit 15 will be described below with reference to Figures 7 and 8. Figure 7 is a schematic diagram of the image projection unit 15 of the projection device 1 according to the embodiment. Figure 8 is a diagram showing the correlation between the control signal provided to the light path shift element 153 and the image signal.

[0051] As shown in FIG. 7, the image projection unit 15 includes a light source 151 , a modulation device 152 , a light path shift element 153 , and a projection lens 154 .

[0052] The light source 151 has, for example, an extra-high pressure mercury lamp or a metal halide lamp, and outputs parallel light to the modulation device 152 .

[0053] The modulation device 152 modulates the light output from the light source 151 in accordance with the input video signal, and outputs the modulated light to the light path shift element 153 .

[0054] The light path shift element 153 is made of, for example, a transparent parallel plate glass, and tilts in accordance with the signal voltage of the control signal. The optical path of light incident on the light path shift element 153 shifts in accordance with the tilt of the light path shift element 153. In the embodiment, the control signal includes a horizontal control signal and a vertical control signal. Therefore, the light path shift element 153 can tilt in either the horizontal direction or the vertical direction in accordance with the signal voltage of the control signal.

[0055] The projection lens 154 condenses the light output from the light path shift element 153 and outputs it to the display surface 50 , thereby forming an image on the display surface 50 according to the light output from the light path shift element 153 .

[0056] In this embodiment, pixel shifting technology is used to sequentially project each subframe SF1 included in the subframe set selected by the video selector 14 onto the display surface 50 while shifting it by half a pixel at a second frame rate (here, 240 fps). As shown in FIG. 8 , the horizontal and vertical control signals are both rectangular wave signals that alternate between high and low levels at a first period Td1 (here, 1 / 120 seconds). The horizontal and vertical control signals are out of phase with each other by ¼ of the first period Td1. Therefore, the combination of the signal voltages of the horizontal and vertical control signals changes at a second period Td2 (here, 1 / 240 seconds).

[0057] For example, when the horizontal control signal and the vertical control signal are both at high level, the image projection unit 15 projects light corresponding to subframe "A" onto the display surface 50. As a result, the subframe "A" is projected onto the display surface 50.

[0058] Furthermore, when the horizontal control signal becomes low level and the vertical control signal becomes high level, video projection unit 15 projects light corresponding to subframe "B" or subframe "B'" onto display surface 50. As a result, subframe "B" or subframe "B'" is projected onto display surface 50 at a position that is shifted by half a pixel in the horizontal direction from the display position of subframe "A."

[0059] Furthermore, when the horizontal control signal and the vertical control signal become low level, video projection unit 15 projects light corresponding to subframe "C" onto display surface 50. As a result, subframe "C" is projected onto display surface 50 at a position shifted by half a pixel horizontally and half a pixel vertically from the display position of subframe "A."

[0060] Furthermore, when the horizontal control signal becomes high level and the vertical control signal becomes low level, video projection unit 15 projects light corresponding to subframe "D" or subframe "D'" onto display surface 50. As a result, subframe "D" or subframe "D'" is projected onto display surface 50 at a position that is shifted by half a pixel in the vertical direction from the display position of subframe "A."

[0061] In this way, the image processing system 100 according to the embodiment uses pixel shifting technology to sequentially project a plurality of subframes SF1 (here, subframe "A", subframe "B" (or "B'"), subframe "C", and subframe "D" (or "D'")) onto the display surface 50. As a result, the image processing system 100 according to the embodiment projects an image onto the display surface 50 at a higher resolution (here, 4K resolution) than the resolution that the modulation device 152 of the projection device 1 can support (here, 2K resolution).

[0062] The synchronization signal output unit 16 outputs a synchronization signal to the imaging device 2. The synchronization signal is a pulse signal that goes high at the timing when the first superimposed sub-frame SF21 and the second superimposed sub-frame SF22 are projected onto the display surface 50. Note that if the first superimposed sub-frame SF21 and the second superimposed sub-frame SF22 are not included in the sub-frame set selected by the video selection unit 14, the synchronization signal output unit 16 does not output a synchronization signal to the imaging device 2.

[0063] The communication unit 17 is a communication interface for communicating with the control device 3 via the network N1. The communication unit 17 receives a parameter setting command transmitted from the control device 3 and changes various parameters stored in the parameter storage unit 18 in accordance with the content of the received parameter setting command. Note that communication between the communication unit 17 and the control device 3 may be wired communication or wireless communication.

[0064] The parameter storage unit 18 is a semiconductor memory or the like, and stores various parameters referenced when the projection device 1 operates. In this embodiment, the parameter storage unit 18 stores the upper and lower limit values ​​of the signal value of the blue signal, which are parameters referenced in the embedding determination process already described, as well as the validity rate, the embedding signal value α, and the embedding mode. The parameter storage unit 18 also stores the LUT for geometric correction already described. Note that these parameters are merely examples, and the parameter storage unit 18 may store other parameters as well.

[0065] The superimposition pattern storage unit 19 is a semiconductor memory or the like that stores bitmap data of the pattern images (first pattern image PP1 and second pattern image PP2) to be superimposed on the sub-frame SF1. The parameter storage unit 18 and the superimposition pattern storage unit 19 may be realized by the same semiconductor memory. The superimposition pattern storage unit 19 may be realized by a programmable logic circuit such as an FPGA that generates the bitmap data of the pattern images (first pattern image PP1 and second pattern image PP2) to be superimposed on the sub-frame SF1 each time.

[0066] [2-3. Imaging Device] Next, the configuration of the imaging device 2 will be described in detail. Fig. 9 is a block diagram showing the configuration of the imaging device 2 according to the embodiment. As shown in Fig. 9, the imaging device 2 includes a communication unit 21, a screen generation unit 22, a synchronization signal input unit 23, an imaging unit 24, a pattern detection unit 25, a parameter storage unit 26, and a superimposition pattern storage unit 27. The communication unit 21, the screen generation unit 22, the synchronization signal input unit 23, the imaging unit 24, and the pattern detection unit 25 may each be realized by a dedicated circuit, or may be realized by a processor executing a corresponding computer program stored in a memory.

[0067] The communication unit 21 is a communication interface for communicating with the control device 3 via the network N1. The communication unit 21 receives commands transmitted from the control device 3 and relays the received commands to the screen generation unit 22. The communication unit 21 also transmits the results of processing executed by the screen generation unit 22 to the control device 3. Note that communication between the communication unit 21 and the control device 3 may be wired communication or wireless communication.

[0068] The screen generation unit 22 generates a screen to be displayed on a display attached to the control device 3 by a screen display unit 32 (described later) of the control device 3. In the embodiment, the screen generation unit 22 generates an HTML page in response to a command from the control device 3. For example, the screen generation unit 22 generates an HTML page including various current parameters of the imaging device 2 and icons for accepting changes to the various parameters in response to a command from the control device 3. Furthermore, for example, the screen generation unit 22 executes processing to change various parameters of the imaging device 2 or processing to start or end imaging by the imaging unit 24 in response to a command from the control device 3, and generates an HTML page including the processing results.

[0069] The synchronization signal input unit 23 receives a synchronization signal transmitted from the projection device 1 and provides the received synchronization signal to the imaging unit 24 .

[0070] The imaging unit 24 captures the image projected on the display surface 50. In this embodiment, the imaging unit 24 starts exposure at a timing according to the trigger mode. Here, the trigger mode is a parameter stored in the parameter storage unit 26. When the trigger mode is "synchronization signal," the imaging unit 24 starts exposure at the timing when a pulse of the synchronization signal from the projection device 1 rises. In other words, in this case, the imaging unit 24 captures only the first superimposed sub-frame SF21 and the second superimposed sub-frame SF22 of the image projected on the display surface 50. Furthermore, when the trigger mode is "program," the imaging unit 24 starts exposure upon receiving a command to start imaging from the control device 3.

[0071] The time from when the imaging unit 24 starts to when it finishes exposure is determined by the exposure time (here, in milliseconds) stored in the parameter storage unit 26. Furthermore, if the trigger delay amount (here, in microseconds) stored in the parameter storage unit 26 is not zero, the imaging unit 24 starts exposure with a delay of the trigger delay amount from when the synchronization signal pulse rises.

[0072] The pattern detection unit 25 executes a detection process to detect a pattern image from the first superimposed sub-frame SF21 and the second superimposed sub-frame SF22 captured by the imaging unit 24. The detection process will be described below with reference to FIG. 10 . FIG. 10 is a flowchart showing an example of the pattern image detection process. The detection process described below is executed each time the first superimposed sub-frame SF21 and the second superimposed sub-frame SF22 are captured by the imaging unit 24.

[0073] First, the pattern detection unit 25 obtains a difference image by calculating the difference between the first superimposed sub-frame SF21 and the second superimposed sub-frame SF22 captured by the imaging unit 24 (S301). Note that the display positions of the first superimposed sub-frame SF21 and the second superimposed sub-frame SF22 on the display surface 50 are shifted from each other by the shift amount caused by the optical path shift element 153 because pixel shift technology is used in the image projection unit 15 of the projection device 1. Therefore, the pattern detection unit 25 shifts either the first superimposed sub-frame SF21 or the second superimposed sub-frame SF22 by the shift amount before calculating the difference.

[0074] Here, the difference image acquired in step S301 is any one of a pattern image for the R channel, a pattern image for the G channel, and a pattern image for the B channel. For example, in the projection device 1, if a first pattern image PP11 and a second pattern image PP12 for the R channel are embedded in frame F1, the pattern detection unit 25 will acquire the pattern image for the R channel when the video corresponding to frame F1 is projected onto the display surface 50.

[0075] Next, the pattern detection unit 25 averages the multiple difference images (S302). Here, for each frame F1, the pattern detection unit 25 sequentially acquires a difference image corresponding to the pattern image for the R channel, a difference image corresponding to the pattern image for the G channel, and a difference image corresponding to the pattern image for the B channel. Therefore, as long as a pattern image is embedded in each frame F1 in the projection device 1, the pattern detection unit 25 can acquire a difference image corresponding to the pattern image for the same channel every three frames. Therefore, once the pattern detection unit 25 has acquired a predetermined number of difference images (e.g., 10) for each of the R channel, G channel, and B channel, it averages these multiple difference images. This makes it possible to reduce noise contained in the averaged difference images.

[0076] Next, the pattern detection unit 25 binarizes the averaged difference image (S303). Here, because the first superimposed sub-frame SF21 and the second superimposed sub-frame SF22 captured by the imaging unit 24 are both color images, the averaged difference image is also a color image. Therefore, the pattern detection unit 25 binarizes the averaged difference image to obtain a black and white binarized difference image.

[0077] Next, the pattern detection unit 25 determines the type of pattern image by pattern matching the black and white binarized difference image with the R channel pattern image template, the G channel pattern image template, and the B channel pattern image template stored in the superimposition pattern storage unit 27 (S304). For example, if the black and white binarized difference image and the G channel pattern image template generally match, the pattern detection unit 25 determines that the difference image is a G channel pattern image.

[0078] Then, the pattern detection unit 25 writes the difference image, the type of which has been determined, into memory as a third pattern image PP3 (S305). By repeating the above steps S301 to S305, the imaging device 2 acquires a third pattern image PP3 for the R channel, a third pattern image PP3 for the G channel, and a third pattern image PP3 for the B channel.

[0079] The parameter storage unit 26 is a semiconductor memory or the like, and stores various parameters that are referenced when the imaging device 2 operates. In the embodiment, the parameter storage unit 26 stores the trigger mode, exposure time, and trigger delay amount already described. Note that these parameters are merely examples, and the parameter storage unit 26 may store other parameters as well.

[0080] The superimposition pattern storage unit 27 is a semiconductor memory or the like, and stores bitmap data of the pattern image template for the R channel, the pattern image template for the G channel, and the pattern image template for the B channel, which are used in the detection process described above. Note that the parameter storage unit 26 and the superimposition pattern storage unit 27 may be realized by the same semiconductor memory.

[0081] [2-4. Control Device] Next, the configuration of the control device 3 will be described in detail. Fig. 11 is a block diagram showing the configuration of the control device 3 according to the embodiment. As shown in Fig. 11, the control device 3 includes an input unit 31, a screen display unit 32, a communication unit 33, a deviation correction unit 34, and a data storage unit 35. The input unit 31, the screen display unit 32, the communication unit 33, and the deviation correction unit 34 may each be realized by a dedicated circuit, or may be realized by a processor executing a corresponding computer program stored in a memory.

[0082] The input unit 31 accepts input from a user using, for example, a keyboard or a pointing device such as a mouse. The input unit 31 provides control commands corresponding to the input from the user to the projection device 1 or the imaging device 2. The control commands include, for example, an instruction to change various parameters of the imaging device 2, an instruction to transmit various parameters of the imaging device 2, an instruction to transmit a third pattern image PP3 from the imaging device 2, an instruction to initialize the misalignment correction process by the misalignment correction unit 34 (described later), or an instruction to start or end the misalignment correction process by the misalignment correction unit 34.

[0083] The screen display unit 32 displays a UI (User Interface) screen for operating the control device 3 on a display provided in the control device 3. For example, the screen display unit 32 displays an HTML page or the like generated by the screen generation unit 22 of the imaging device 2 on the display.

[0084] The communication unit 33 is a communication interface for communicating with each of the projection device 1 and the imaging device 2 via the network N1. The communication unit 33 transmits control commands to the projection device 1 or the imaging device 2. The communication unit 33 also transmits LUT data for geometric correction after correction by a misalignment correction process, which will be described later, to the projection device 1. Note that communication between the communication unit 33 and the projection device 1 and communication between the communication unit 33 and the imaging device 2 may be wired communication or wireless communication.

[0085] The deviation correction unit 34 has a function of executing initialization of the deviation correction process. The initialization of the deviation correction process will be described below with reference to FIG. 12 . FIG. 12 is a flowchart showing an example of the initialization of the deviation correction process. The initialization of the deviation correction process may be executed once in response to a user input received by the input unit 31 before the deviation correction process is executed, for example, when the image processing system 100 is first used.

[0086] First, the misalignment correction unit 34 acquires LUT data for geometric correction from the projection device 1 (S401). Next, the misalignment correction unit 34 acquires a third pattern image PP3 for the R channel, a third pattern image PP3 for the G channel, and a third pattern image PP3 for the B channel from the imaging device 2, and detects feature points SP1 of the third pattern image PP3 from these images (S402).

[0087] A method for detecting feature points SP1 in the third pattern image PP3 will be described below with reference to FIG. 13 . FIG. 13 is a diagram showing an example of feature points SP1 in the third pattern image PP3. FIG. 13 shows a third pattern image PP3 obtained by combining a third pattern image PP3 for the R channel, a third pattern image PP3 for the G channel, and a third pattern image PP3 for the B channel. Here, the third pattern images PP3 for each channel are combined by coloring white pixels in the third pattern image PP3 for the R channel red, white pixels in the third pattern image PP3 for the G channel green, and white pixels in the third pattern image PP3 for the B channel blue.

[0088] 13, each pixel is color-coded according to the type of hatching. Here, the feature point SP1 is the intersection of four regions, where the colors of the upper region, lower region, right region, and left region are all different from one another. In particular, in the synthesized third pattern image PP3, there is only one point where the colors of the upper region, lower region, right region, and left region form a specific combination. In this embodiment, the misalignment correction unit 34 detects the intersection of the four regions where the color combination forms the specific combination as the feature point SP1.

[0089] Returning to FIG. 12, the misalignment correction unit 34 stores data linking each point of the LUT for geometric correction with the detected feature point SP1 of the third pattern image PP3 in the data storage unit 35 as initial data (S403).

[0090] The misalignment correction unit 34 also has a function of executing misalignment correction processing. The misalignment correction processing will be described below with reference to Fig. 14. Fig. 14 is a flowchart showing an example of the misalignment correction processing.

[0091] First, the misalignment correction unit 34 waits until the pattern image (third pattern image PP3 for each channel) acquired from the imaging device 2 is updated (S501: No). Then, when the pattern image acquired from the imaging device 2 is updated (S501: Yes), the misalignment correction unit 34 detects the feature point SP1 based on the acquired third pattern image PP3 for each channel (S502). The method for detecting the feature point SP1 has already been described, so its description will be omitted here.

[0092] Next, the deviation correction unit 34 compares the detected feature point SP1 with the feature point SP1 included in the initial data (S503). Here, the deviation correction unit 34 compares the XY plane coordinates of the detected feature point SP1 with the XY plane coordinates of the feature point SP1 included in the initial data.

[0093] If the comparison shows that there is no deviation in the position of the feature point SP1 (S504: No), the deviation correction unit 34 does not update the geometric correction LUT or the initial data. On the other hand, if there is deviation in the position of the feature point SP1 (S504: Yes), the deviation correction unit 34 generates a geometric correction LUT that reduces the deviation to zero and updates the geometric correction LUT (S505). The deviation correction unit 34 also updates the initial data using the updated geometric correction LUT (S506). Specifically, the deviation correction unit 34 updates the detected feature point SP1 as the feature point SP1 included in the initial data.

[0094] At this time, the misalignment correction unit 34 transmits the updated (corrected) geometric correction LUT data to the projection device 1 via the communication unit 33 and the network N1. The projection device 1 then performs geometric correction on the internal video signal in accordance with the acquired corrected geometric correction LUT. This makes it possible to correct the misalignment of the display position of the image on the display surface 50.

[0095] The data storage unit 35 is a semiconductor memory or the like, and stores initial data including LUT data for geometric correction acquired from the imaging device 2, and the third pattern image PP3 for each channel acquired from the imaging device 2, etc.

[0096] 3. Operation The overall operation of the video processing system 100 according to the embodiment, that is, the video processing method according to the embodiment, will be described below.

[0097] [3-1. Misalignment Correction] First, an example of the operation of misalignment correction in the video processing system 100 according to the embodiment will be described with reference to Fig. 15. Fig. 15 is a flowchart showing an example of the operation of misalignment correction in the video processing system 100 according to the embodiment.

[0098] First, the image processing system 100 acquires a plurality of sub-frames SF1 obtained by temporally dividing a frame F1 included in the image data (S601). In the embodiment, the image generation unit 12 of the projection device 1 executes step S601.

[0099] Next, the video processing system 100 outputs the first superimposed sub-frame SF21 and the second superimposed sub-frame SF22 to be displayed on the display surface 50 (S602). The first superimposed sub-frame SF21 is an image obtained by superimposing a first pattern image PP1 on a first sub-frame based on a plurality of sub-frames SF1. The second superimposed sub-frame SF22 is an image obtained by superimposing a second pattern image PP2, which is obtained by inverting the pixel values ​​of the first pattern image PP1, on a second sub-frame based on a plurality of sub-frames SF1. In this embodiment, step S602 is performed by the video generation unit 12, video selection unit 14, and video projection unit 15 of the projection device 1.

[0100] Next, the video processing system 100 captures the first superimposed sub-frame SF21 and the second superimposed sub-frame SF22 displayed on the display surface 50 (step S603). In this embodiment, step S603 is executed by the imaging unit 24 and the pattern detection unit 25 of the imaging device 2.

[0101] Next, the video processing system 100 acquires a third pattern image PP3 from the difference between the acquired first superimposed sub-frame SF21 and second superimposed sub-frame SF22 (step S604). In this embodiment, step S604 is executed by the pattern detection unit 25 of the imaging device 2.

[0102] The image processing system 100 then compares the feature point SP1 of the acquired third pattern image PP3 with the reference feature point to detect a deviation in the display position of the image projected on the display surface 50 (S605). Here, the reference feature point is the feature point SP1 of the third pattern image PP3 included in the initial data, as already described. In this embodiment, step S605 is executed by the deviation correction unit 34 of the control device 3.

[0103] In the embodiment, the video processing system 100 executes the process of updating the LUT for geometric correction in order to correct the detected deviation of the display position, but this process does not have to be executed.

[0104] [3-2. Increase / Decrease Processing] Next, an example of the operation of the increase / decrease processing of the video processing system 100 according to the embodiment will be described. The increase / decrease processing is processing for increasing / decreasing the degree (hereinafter simply referred to as "degree") of acquiring a pattern image from one or more acquired superimposed subframes (here, the first superimposed subframe SF21 and the second superimposed subframe SF22) according to the acquisition result of the pattern image (here, the third pattern image PP3). In the embodiment, the increase / decrease processing is executed mainly by the deviation correction unit 34 of the control device 3.

[0105] In the increase / decrease process, first, it is determined whether to increase or decrease the degree depending on the result of acquiring the pattern image (third pattern image PP3). Specifically, the misalignment correction unit 34 detects one or more feature points SP1 included in the third pattern image PP3 and counts the detected one or more feature points SP1. Then, the misalignment correction unit 34 determines whether the ratio of the number of the detected one or more feature points SP1 to the total number of feature points SP1 included in the third pattern image PP3 is equal to or greater than a threshold value (e.g., 80%). If the ratio is equal to or greater than the threshold value, the misalignment correction unit 34 determines to maintain the degree without increasing or decreasing it. On the other hand, if the ratio is less than the threshold value, the misalignment correction unit 34 determines to increase or decrease the degree (here, to increase the degree).

[0106] Below, we will list first to third examples of the increase / decrease process for increasing or decreasing the degree. Note that any one of the first to third examples listed below may be used as the increase / decrease process, or two or more examples may be combined and used as the increase / decrease process.

[0107] 16 is an explanatory diagram of a first example of the increase / decrease process. (a) of FIG. 16 shows a case where the degree of increase / decrease is relatively weak, and (b) of FIG. 16 shows a case where the degree of increase / decrease is relatively strong. (a) of FIG. 16 and (b) of FIG. 16 show, from top to bottom, pixel values ​​of the blue signal for one horizontal line of the original image, pixel values ​​of the blue signal for that one horizontal line of the first pattern image PP1, pixel values ​​of the blue signal for that one horizontal line of the second pattern image PP2, pixel values ​​of the blue signal for that one horizontal line of the first superimposed sub-frame SF21, pixel values ​​of the blue signal for that one horizontal line of the second superimposed sub-frame SF22, and pixel values ​​of the blue signal for that one horizontal line of the third pattern image PP3.

[0108] In the first example, as shown in Fig. 16 , the degree is increased or decreased by increasing or decreasing the pixel values ​​of the pattern images (first pattern image PP1 and second pattern image PP2) superimposed on one or more sub-frames SF1. Specifically, as shown in Fig. 16 (a), the pixel values ​​of the blue signal in each line of the first pattern image PP1 and the second pattern image PP2 are decreased, thereby decreasing the pixel values ​​of the blue signal in each line of the third pattern image PP3. On the other hand, as shown in Fig. 16 (b), the pixel values ​​of the blue signal in each line of the first pattern image PP1 and the second pattern image PP2 are increased, thereby increasing the pixel values ​​of the blue signal in each line of the third pattern image PP3.

[0109] As described above, in the first example, the degree (here, the pixel value of the third pattern image PP3) is increased or decreased by increasing or decreasing the pixel values ​​of the pattern images (first pattern image PP1 and second pattern image PP2) superimposed on one or more sub-frames SF1. Increasing the pixel value of the third pattern image PP3 improves the signal-to-noise ratio, making it easier for the misalignment correction unit 34 of the control device 3 to detect one or more feature points SP1 included in the third pattern image PP3. However, in this case, the first pattern image PP1 and the second pattern image PP2 become more noticeable in the first superimposed sub-frame SF21 and the second superimposed sub-frame SF22, respectively, and are more easily recognized by the viewer.

[0110] On the other hand, by reducing the pixel values ​​of the third pattern image PP3, the first pattern image PP1 and the second pattern image PP2 become less noticeable in the first superimposed sub-frame SF21 and the second superimposed sub-frame SF22, respectively, and are less likely to be recognized by the viewer. However, in this case, it becomes more difficult for the misalignment correction unit 34 of the control device 3 to detect one or more feature points SP1 included in the third pattern image PP3.

[0111] 17A and 17B are explanatory diagrams of a second example of the increase / decrease process. Fig. 17A shows a case where the degree of increase / decrease is relatively weak, and Fig. 17B shows a case where the degree of increase / decrease is relatively strong. Fig. 17A shows the pixel values ​​of blue signals in one horizontal line of one frame of the third pattern image PP3, and Fig. 17B shows the pixel values ​​of blue signals in the same line of N frames of the third pattern image PP3 (N is a natural number greater than or equal to 2).

[0112] In the second example, as shown in Fig. 17 , the degree of blue reflection is increased or decreased by increasing or decreasing the number of pattern images (third pattern image PP3) acquired from one or more sub-frames SF1. Specifically, as shown in Fig. 17 (a), when a small number of third pattern images PP3 are added (here, one frame's worth), the pixel values ​​of the blue signals in each line of the third pattern image PP3 after addition become smaller. On the other hand, as shown in Fig. 17 (b), when a large number of third pattern images PP3 are added (here, N frames' worth), the pixel values ​​of the blue signals in each line of the third pattern image PP3 after addition become larger.

[0113] As described above, in the second example, the degree (here, the pixel values ​​of the third pattern image PP3 after addition) is increased or decreased by increasing or decreasing the number of pattern images (third pattern image PP3) acquired from one or more sub-frames SF1. Increasing the number of third pattern images PP3 to be added improves the signal-to-noise ratio, making it easier for the misalignment correction unit 34 of the control device 3 to detect one or more feature points SP1 included in the third pattern image PP3 after addition. However, in this case, it is necessary to wait until multiple third pattern images PP3 are acquired, which can easily increase the time required for processing.

[0114] On the other hand, the time required for the above process can be shortened by reducing the number of third pattern images PP3 to be added. However, in this case, the pixel values ​​of the third pattern image PP3 after addition become smaller, making it difficult for the misalignment correction unit 34 of the control device 3 to detect one or more feature points SP1 included in the third pattern image PP3.

[0115] FIG. 18 is an explanatory diagram of a third example of the increase / decrease process. (a) of FIG. 18 shows a case where the degree is relatively weak, and (b) of FIG. 18 shows a case where the degree is relatively strong. Also, (a) of FIG. 18 and (b) of FIG. 18 show, from top to bottom, an original image, pattern images (first pattern image PP1 and second pattern image PP2), a mask image, and an image in which the pattern image and mask image are superimposed on the original image. In the mask image, white areas represent areas where the pattern image is transparent, and black areas represent areas where the pattern image is blocked. Therefore, in the image in which the pattern image and mask image are superimposed on the original image, the pattern image is superimposed on the original image in areas corresponding to the white areas of the mask image, and the original image remains unchanged in areas corresponding to the black areas of the mask image.

[0116] In the third example, as shown in Fig. 18 , the degree is increased or decreased by increasing or decreasing the area of ​​the pattern images (first pattern image PP1 and second pattern image PP2) superimposed on one or more sub-frames SF1. Specifically, as shown in Fig. 18 (a), the area of ​​the pattern image superimposed on sub-frame SF1 is reduced by increasing the area of ​​the black region of the mask image. On the other hand, as shown in Fig. 18 (b), the area of ​​the pattern image superimposed on sub-frame SF1 is increased by decreasing the area of ​​the black region of the mask image.

[0117] As described above, in the third example, the degree (here, the area of ​​the first pattern image PP1 or the second pattern image PP2 in the superimposed sub-frames) is increased or decreased by increasing or decreasing the area of ​​the pattern images (first pattern image PP1 and second pattern image PP2) superimposed on one or more sub-frames SF1. Increasing the area of ​​the first pattern image PP1 or the second pattern image PP2 in the superimposed sub-frames improves the signal-to-noise ratio, making it easier for the misalignment correction unit 34 of the control device 3 to detect one or more feature points SP1 included in the third pattern image PP3. However, in this case, the first pattern image PP1 and the second pattern image PP2 become more noticeable in the first superimposed sub-frame SF21 and the second superimposed sub-frame SF22, respectively, and are more easily recognized by the viewer.

[0118] On the other hand, by reducing the area of ​​the first pattern image PP1 or the second pattern image PP2 in the superimposed sub-frames, the first pattern image PP1 and the second pattern image PP2 become less noticeable in the first superimposed sub-frame SF21 and the second superimposed sub-frame SF22, respectively, and are less likely to be recognized by the viewer. However, in this case, it becomes more difficult for the misalignment correction unit 34 of the control device 3 to detect one or more feature points SP1 included in the third pattern image PP3.

[0119] As already described, in the embodiment, the two projectors 1A and 1B each project an image onto the display surface 50, thereby projecting a composite image onto the display surface 50. Therefore, in the embodiment, the two projectors 1A and 1B each project a divided mask image onto the display surface 50, thereby projecting a composite mask image onto the display surface 50. FIG. 19 is a diagram showing an example of a mask image used in the third example of the increase / decrease processing. In the example shown in FIG. 19, the right half of the mask image is the mask image projected from the projector 1A, and the left half of the mask image is the mask image projected from the projector 1B. These mask images are projected onto the display surface 50 as a single composite mask image.

[0120] [3-3. Main Operation] Next, a description will be given of an example of the main operation of the video processing system 100 according to the embodiment. In the embodiment, the video processing system 100 determines whether or not to superimpose pattern images (first pattern image PP1 and second pattern image PP2) on frame F1, in other words, whether or not to display the first superimposed sub-frame SF21 and the second superimposed sub-frame SF22 on the display surface 50, based on the detection result of the sensor 6 (see FIG. 1 ).

[0121] The sensor 6 is, for example, a human presence sensor that detects people around the display surface 50 of the screen 5. The sensor 6 transmits a signal including the detection result (hereinafter also referred to as a "detection signal") to the control device 3. If the detection result of the sensor 6 indicates the presence of people around the display surface 50, that is, if it is estimated that there is a viewer of the video projected on the display surface 50, the control device 3 determines not to display the first superimposed sub-frame SF21 and the second superimposed sub-frame SF22 on the display surface 50. On the other hand, if the detection result of the sensor 6 indicates that there is no person around the display surface 50, that is, if it is estimated that there is no viewer, the control device 3 determines to display the first superimposed sub-frame SF21 and the second superimposed sub-frame SF22 on the display surface 50.

[0122] Note that instead of the sensor 6, for example, a camera that captures an image of a person around the display surface 50 of the screen 5 may be used. In this case, by performing appropriate image analysis processing on the image captured by the camera, it is possible to estimate the presence or absence of a person around the display surface 50, similar to the case where the detection result of the sensor 6 is referenced.

[0123] 20 is a sequence diagram showing an example of the operation of the video processing system 100 according to the embodiment. First, if there is a viewer of the video projected on the display surface 50, the sensor 6 transmits a detection signal including a detection result indicating the presence of a person around the display surface 50 to the control device 3 (S701). Then, the control device 3 stops the execution of the output process for outputting one or more superimposed subframes (here, the first superimposed subframe SF21 and the second superimposed subframe SF22) to be displayed on the display surface 50 (S702). Step S702 corresponds to setting the degree in the increase / decrease process to zero.

[0124] Thereafter, when there are no viewers around the display surface 50, the sensor 6 transmits a detection signal including a detection result indicating that there are no people around the display surface 50 to the control device 3 (S703). Then, the control device 3 resumes execution of the output process (S704). Step S704 corresponds to returning the degree to the default in the increase / decrease process.

[0125] When the control device 3 resumes execution of the output process, the projection device 1 transmits a synchronization signal to the imaging device 2 (S705). The imaging device 2 captures the image projected on the display surface 50 based on the synchronization signal transmitted from the projection device 1 (S706). The imaging device 2 then executes a detection process to detect a pattern image (third pattern image PP3) from one or more superimposed subframes (first superimposed subframe SF21 and second superimposed subframe SF22) obtained by capturing the image (S707). The imaging device 2 then transmits the detected third pattern image PP3 to the control device 3 (S708). The third pattern image PP3 includes a third pattern image PP3 for the R channel, a third pattern image PP3 for the G channel, and a third pattern image PP3 for the B channel. Steps S705 to S708 are repeated until the control device 3 receives a detection signal including a detection result indicating that no person is present around the display surface 50.

[0126] When the control device 3 acquires the third pattern image PP3 transmitted from the imaging device 2, it executes an increase / decrease process. First, the control device 3 detects one or more feature points SP1 included in the third pattern image PP3 (S709). Next, the control device 3 determines whether the ratio of the number of detected feature points SP1 to the total number of feature points SP1 included in the third pattern image PP3 (hereinafter also referred to as the "detection rate") is equal to or greater than a threshold value (e.g., 80%) (S710). If the determination fails (S710: No), in other words, if the detection rate is less than the threshold value, the control device 3 executes a process to increase the degree (S711). For example, when the first example of the increase / decrease process is adopted, the control device 3 instructs the projection device 1 to increase the pixel values ​​of the pattern images (first pattern image PP1 and second pattern image PP2) to be superimposed on one or more subframes SF1.

[0127] On the other hand, if the determination is successful (S710: Yes), in other words, if the detection rate is equal to or greater than the threshold, the control device 3 compares the detected feature point SP1 of the third pattern image PP3 with the reference feature point to detect a misalignment in the display position of the image projected on the display surface 50 (S712). If there is no misalignment in the position of the feature point SP1 (S713: No), the control device 3 does not update the geometric correction LUT or initial data. On the other hand, if there is a misalignment in the position of the feature point SP1 (S713: Yes), the control device 3 updates the geometric correction LUT to zero the misalignment and transmits the updated geometric correction LUT data (misalignment correction data) to the projection device 1 (S714). The projection device 1 then geometrically corrects the internal video signal according to the acquired geometric correction LUT data (S715). Steps S709 to S715 are repeated until the control device 3 receives a detection signal including a detection result indicating that no person is present around the display surface 50.

[0128] 4. Advantages, etc. Advantages of the video processing system 100 (video processing method) according to the embodiment will be described below. As described above, the video processing system 100 according to the embodiment performs an increase / decrease process to increase / decrease the degree to which a pattern image is acquired from one or more acquired superimposed sub-frames (first superimposed sub-frame SF21 and second superimposed sub-frame SF22) depending on the acquisition result of the pattern image (third pattern image PP3).

[0129] For this reason, in the video processing system 100 according to the embodiment, the degree of pattern image acquisition can be adjusted to a degree that allows the pattern image to be sufficiently detected, and therefore, compared to a case in which the degree of pattern image acquisition is uniformly increased, it is easier to superimpose the pattern image on one or more sub-frames SF1 in a manner that is less noticeable to the viewer. Therefore, the video processing system 100 according to the embodiment has the advantage that the pattern image superimposed on the video to correct a shift in the display position of the video can be easily detected without being noticed by the viewer.

[0130] Furthermore, if the environment around the screen 5 makes it difficult for the imaging device 2 to capture an image, or if the superimposed pattern images (first pattern image PP1 and second pattern image PP2) are difficult to detect, there is a possibility that detection of the pattern image (third pattern image PP3) will fail. Even in such cases, the video processing system 100 according to the embodiment has the advantage that it is possible to adjust the degree to which the pattern image is acquired, making it easier to successfully detect the pattern image.

[0131] Furthermore, the video processing system 100 according to the embodiment acquires the detection result of the sensor 6, i.e., presence / absence information indicating the presence or absence of a person viewing the display surface 50, and stops execution of the increase / decrease process when the acquired presence / absence information indicates the presence of a person, and executes the increase / decrease process when it indicates the absence of a person. Therefore, the video processing system 100 according to the embodiment does not increase or decrease the degree of pattern image acquisition when a viewer is present, which has the advantage that the pattern image is less likely to be recognized by the viewer.

[0132] Furthermore, in the video processing system 100 according to the embodiment, when the presence / absence information indicates the presence of a person, execution of the output process is stopped. Therefore, in the video processing system 100 according to the embodiment, when a viewer is present, the pattern image is not superimposed on one or more sub-frames SF1, which has the advantage that the pattern image is not perceived by the viewer.

[0133] 5. Other Embodiments Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

[0134] 21 is a schematic diagram showing an overall configuration including a video processing system 100 according to a first modification of the embodiment. The first modification differs from the video processing system 100 according to the embodiment in that a scheduler 7 is used instead of the sensor 6.

[0135] The scheduler 7 is an information terminal such as a desktop or laptop personal computer that works in conjunction with a video screening system and manages the screening schedule. The scheduler 7 may be configured integrally with the control device 3.

[0136] The scheduler 7 transmits a signal to the control device 3 in accordance with the screening schedule. Then, when the signal indicates that a screening is about to start, that is, when it is estimated that there is an audience watching the video projected on the display surface 50, the control device 3 decides not to display the first superimposed sub-frame SF21 and the second superimposed sub-frame SF22 on the display surface 50. On the other hand, when the signal indicates that a screening is about to end, that is, when it is estimated that there will be no audience watching until the next screening due to a change in audience, etc., the control device 3 decides to display the first superimposed sub-frame SF21 and the second superimposed sub-frame SF22 on the display surface 50.

[0137] 22 is a sequence diagram showing an example of the operation of the video processing system 100 according to the first modified example of the embodiment. In the following explanation, steps S702 and S704 to S715 are the same as those in the embodiment, and therefore detailed explanations thereof will be omitted.

[0138] First, the scheduler 7 transmits a signal indicating that the screening is to begin to the control device 3 (S801). The control device 3 then executes step S702. Thereafter, the scheduler 7 transmits a signal indicating that the screening is to end to the control device 3 (S802). The control device 3 then executes step S704. Thereafter, steps S705 to S708 are repeated until the control device 3 receives a signal indicating that the screening is to end. Furthermore, thereafter, steps S709 to S715 are repeated until the control device 3 receives a signal indicating that the screening is to end.

[0139] In the first variant, the execution of the output process is stopped during the screening, that is, when it is estimated that there is an audience watching the image projected on the display surface 50 of the screen 5, so as with the embodiment, there is an advantage that the pattern image is not recognized by the audience.

[0140] [5-2. Second Modification] The second modification differs from the image processing system 100 according to the embodiment in that the output process is always executed regardless of the detection result of the sensor 6, i.e., regardless of whether a viewer is present or not. In the second modification, when the detection result of the sensor 6 indicates the presence of a person around the display surface 50, i.e., when it is estimated that a viewer of the image projected on the display surface 50 is present, the control device 3 weakens the degree and executes the increase / decrease process. Note that the degree to which the degree is weakened may be set as appropriate by the user. On the other hand, when the detection result of the sensor 6 indicates that no person is present around the display surface 50, i.e., when it is estimated that no viewer is present, the control device 3 strengthens the degree and stops executing the increase / decrease process. Note that the degree to which the degree is strengthened may be set as appropriate by the user.

[0141] 23 is a sequence diagram showing an example of the operation of the video processing system 100 according to the second modification of the embodiment. In the following explanation, steps S705 to S715 are the same as those in the embodiment, and therefore detailed explanations will be omitted. In addition, the following explanation will be given assuming that the control device 3 is always executing output processing.

[0142] First, if there is a viewer of the image projected on the display surface 50, the sensor 6 transmits a detection signal including a detection result indicating the presence of a person around the display surface 50 to the control device 3 (S901). Then, the control device 3 weakens the degree of the increase / decrease process (S902). Thereafter, steps S705 to S708 are repeated until the control device 3 receives a detection signal including a detection result indicating that there is no person around the display surface 50. Furthermore, thereafter, steps S709 to S715 are repeated until the control device 3 receives a detection signal including a detection result indicating that there is no person around the display surface 50.

[0143] Thereafter, when there are no viewers around the display surface 50, the sensor 6 transmits a detection signal including a detection result indicating that there are no people around the display surface 50 to the control device 3 (S903). The control device 3 then increases the level of the increase / decrease process (S904). Thereafter, the control device 3 stops executing the increase / decrease process until it receives a detection signal including a detection result indicating that there are people around the display surface 50.

[0144] In the second modified example, the degree of increase / decrease processing is weakened when it is estimated that a viewer is present around the display surface 50 of the screen 5, which has the advantage that the pattern image is less likely to be recognized by the viewer.

[0145] In the second modified example, in steps S709 to S711, the degree of the increase / decrease process is increased until the detection rate becomes equal to or greater than the threshold, but the degree of the increase / decrease process in step S904 is set as the upper limit.

[0146] In the second modification, the scheduler 7 may be used instead of the sensor 6. In this case, when the control device 3 receives a signal from the scheduler 7 indicating the start of a screening, it may weaken the degree and then execute the increase / decrease process. When the control device 3 receives a signal from the scheduler 7 indicating the end of a screening, it may strengthen the degree and then stop the execution of the increase / decrease process.

[0147] 5-3. Other Modifications For example, in the above embodiment, the control device 3 may decrease the degree of the increase / decrease process if the detection rate is equal to or greater than a predetermined value (for example, 90%) that is greater than the threshold value.

[0148] For example, in the above embodiment, the control device 3 may increase the degree of the increase / decrease process when the feature point detection rate is less than the threshold value for a predetermined number of consecutive times, thereby increasing the success rate of feature point detection even when, for example, scenes in which detection is difficult continue.

[0149] For example, in the above embodiment, the first pattern image PP1 and the second pattern image PP2 are superimposed on the first subframe and the second subframe, which are the same image based on multiple subframes SF1, respectively. However, this is not limited to this. For example, the first subframe and the second subframe may be different images. Furthermore, the method of superimposing a pattern image on one or more subframes SF1 is not limited to the method in the above embodiment, as long as a pattern image can be obtained by capturing a superimposed subframe in which a pattern image is superimposed on one or more subframes SF1.

[0150] For example, in the above embodiment, the video processing system 100 is realized by a plurality of devices, but this is not limiting, and the video processing system 100 may be realized by a single device.

[0151] In the above-described embodiment, the processing performed by a specific processing unit may be performed by another processing unit. The order of multiple processing operations may be changed, or multiple processing operations may be performed in parallel.

[0152] In the above-described embodiments, each component may be realized by executing a software program suitable for that component, or by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0153] Furthermore, each component may be realized by hardware. Each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0154] Furthermore, the general or specific aspects of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0155] The present disclosure may also be realized as a video processing method executed by a computer such as the video processing system of the above-described embodiment. The present disclosure may also be realized as a program (computer program product) for causing a computer to execute such a video processing method, or as a computer-readable non-transitory recording medium on which such a program is recorded.

[0156] In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would think of, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the intent of this disclosure.

[0157] (Summary) As described above, the video processing method according to the first aspect acquires a plurality of subframes SF1 obtained by temporally dividing a frame F1 included in video data. This video processing method also executes an output process for outputting one or more superimposed subframes, each of which is obtained by superimposing a pattern image on one or more subframes SF1 based on the plurality of subframes SF1, for display on the display surface 50. This video processing method also acquires the one or more superimposed subframes displayed on the display surface 50 by imaging. This video processing method also acquires a pattern image from the one or more acquired superimposed subframes. This video processing method also detects a deviation in the display position of the image projected on the display surface 50 by comparing a feature point SP1 of the acquired pattern image with a reference feature point. This video processing method also executes an increase / decrease process for increasing / decreasing the degree to which the pattern image is acquired from the one or more acquired superimposed subframes, depending on the pattern image acquisition result.

[0158] This type of video processing method can adjust the degree of pattern image acquisition to a degree that allows the pattern image to be sufficiently detected, and therefore, compared to a case where the degree of pattern image acquisition is uniformly increased, it is easier to superimpose a pattern image on one or more subframes SF1 in a manner that is less noticeable to the viewer. Therefore, this type of video processing method has the advantage that it is easy to detect a pattern image that is superimposed on video to correct a shift in the display position of the video without the viewer noticing it.

[0159] Also, for example, in the video processing method according to the second aspect, in the first aspect, the increase / decrease processing increases or decreases the degree by increasing or decreasing the pixel values ​​of a pattern image superimposed on one or more subframes SF1.

[0160] Such an image processing method has the advantage that the pixel values ​​of the acquired pattern image can be increased or decreased, making it easy to adjust the S / N ratio.

[0161] Also, for example, in the video processing method according to the third aspect, in the first or second aspect, the increase / decrease process increases or decreases the degree by increasing or decreasing the number of pattern images obtained from one or more superimposed subframes.

[0162] This type of image processing method has the advantage that the number of pattern images to be added can be increased or decreased, making it easy to adjust the S / N ratio.

[0163] Also, for example, in the video processing method according to the fourth aspect, in any one of the first to third aspects, the increase / decrease process increases or decreases the degree by increasing or decreasing the area of ​​the pattern image to be superimposed on one or more superimposed subframes.

[0164] Such an image processing method has the advantage that the area of ​​the acquired pattern image can be increased or decreased, making it easy to adjust the S / N ratio.

[0165] Furthermore, for example, in a video processing method according to a fifth aspect, in any one of the first to fourth aspects, presence / absence information indicating the presence or absence of a person viewing the display surface 50 is acquired. Furthermore, in this video processing method, if the acquired presence / absence information indicates the presence of a person, execution of the increase / decrease process is stopped, and if it indicates the absence of a person, execution of the increase / decrease process is executed.

[0166] This type of video processing method has the advantage that the degree to which pattern images are acquired does not increase or decrease when a viewer is present, making it difficult for the viewer to perceive the pattern images.

[0167] Furthermore, for example, in the video processing method according to the sixth aspect, when the presence / absence information indicates that a person is present viewing the display surface 50 in the fifth aspect, the execution of the output process is stopped.

[0168] This video processing method has the advantage that the pattern image is not superimposed on one or more sub-frames SF1 when a viewer is present, and therefore the pattern image is not perceived by the viewer.

[0169] Furthermore, for example, in a video processing method according to a seventh aspect, in any one of the first to sixth aspects, the one or more superimposed subframes include a first superimposed subframe SF21 in which a first pattern image PP1 is superimposed on a first subframe based on a plurality of subframes SF1, and a second superimposed subframe SF22 in which a second pattern image PP2, obtained by inverting pixel values ​​of the first pattern image PP1, is superimposed on a second subframe based on the plurality of subframes SF1. Furthermore, in this video processing method, the first superimposed subframe SF21 and the second superimposed subframe SF22 displayed on the display surface 50 are acquired by imaging. Furthermore, in this video processing method, a third pattern image PP3, which is a pattern image, is acquired from the difference between the acquired first superimposed subframe SF21 and the acquired second superimposed subframe SF22. Furthermore, in this video processing method, a feature point SP1 of the acquired third pattern image PP3 is compared with a reference feature point to detect a deviation in the display position of the image projected on the display surface 50.

[0170] Such an image processing method can adjust the degree of acquisition of the third pattern image PP3 to a degree that allows the third pattern image PP3 to be sufficiently detected, and therefore has the advantage that it is easier to superimpose the first pattern image PP1 and the second pattern image PP2 on one or more sub-frames SF1 in a manner that is less noticeable to the viewer, compared to when the degree of acquisition of the third pattern image PP3 is uniformly increased.

[0171] Furthermore, for example, a program according to an eighth aspect causes one or more processors to execute the video processing method according to any one of the first to seventh aspects.

[0172] Such a program can adjust the degree of pattern image acquisition to a degree that allows the pattern image to be sufficiently detected, and therefore, compared to a case where the degree of pattern image acquisition is uniformly increased, it is easier to superimpose a pattern image on one or more subframes SF1 in a manner that is less noticeable to the viewer. Therefore, such a program has the advantage that it is easy to detect a pattern image that is superimposed on an image to correct a shift in the display position of the image without the viewer noticing it.

[0173] Furthermore, for example, a video processing system 100 according to a ninth aspect includes a first acquisition unit (video generation unit 12 of the projection device 1), an output unit (video generation unit 12, video selection unit 14, and video projection unit 15 of the projection device 1), a second acquisition unit (imaging unit 24 and pattern detection unit 25 of the imaging device 2), a third acquisition unit (pattern detection unit 25 of the imaging device 2), a detection unit (deviation correction unit 34 of the control device 3), and an increase / decrease unit (deviation correction unit 34 of the control device 3). The first acquisition unit acquires multiple subframes SF1 obtained by temporally dividing a frame F1 included in the video data. The output unit executes output processing to output one or more superimposed subframes, each of which has a pattern image superimposed on one or more subframes SF1 based on the multiple subframes SF1, for display on the display surface 50. The second acquisition unit acquires the one or more superimposed subframes displayed on the display surface 50 by imaging. The third acquisition unit acquires a pattern image from the acquired one or more superimposed sub-frames. The detection unit compares feature point SP1 of the acquired pattern image with the reference feature point to detect a deviation in the display position of the image projected on the display surface 50. The increase / decrease unit executes an increase / decrease process to increase / decrease the degree to which the pattern image is acquired from the acquired one or more superimposed sub-frames, depending on the pattern image acquisition result.

[0174] Since this type of video processing system 100 can adjust the degree of pattern image acquisition to a degree that allows the pattern image to be sufficiently detected, it is easier to superimpose a pattern image on one or more sub-frames SF1 in a manner that is less noticeable to the viewer than when the degree of pattern image acquisition is uniformly increased. Therefore, this type of video processing system 100 has the advantage that it is easy to detect a pattern image that is superimposed on video to correct a shift in the display position of the video without the viewer noticing it.

[0175] The image processing method and the like of the present disclosure can be used in a system that corrects the deviation of the display position of an image projected on a display surface such as a screen.

Claims

1. Obtain a plurality of sub-frames obtained by temporally dividing frames included in video data, and output an output process for causing one or more superimposed sub-frames obtained by superimposing a pattern image on one or more sub-frames based on the plurality of sub-frames to be displayed on a display surface. Obtain the one or more superimposed sub-frames displayed on the display surface by imaging, obtain the pattern image from the one or more obtained superimposed sub-frames, and compare the feature points of the obtained pattern image with the reference feature points to detect a deviation in the display position of the video projected on the display surface. Execute an increase / decrease process for increasing or decreasing the degree of obtaining the pattern image from the one or more obtained superimposed sub-frames according to the acquisition result of the pattern image. A video processing method.

2. In the increase / decrease process, the degree is increased or decreased by increasing or decreasing the pixel value of the pattern image superimposed on the one or more sub-frames. The video processing method according to claim 1.

3. In the increase / decrease process, the degree is increased or decreased by increasing or decreasing the number of pattern images obtained from the one or more superimposed sub-frames. The video processing method according to claim 1.

4. In the increase / decrease process, the degree is increased or decreased by increasing or decreasing the area of the pattern image superimposed on the one or more superimposed sub-frames. The video processing method according to claim 1.

5. Obtain presence / absence information indicating the presence or absence of a person viewing the display surface. When the obtained presence / absence information indicates that the person is present, stop the execution of the increase / decrease process. When the obtained presence / absence information indicates that the person is not present, execute the increase / decrease process. The video processing method according to any one of claims 1 to 4.

6. When the presence / absence information indicates that a person viewing the display surface is present, stop the execution of the output process. The video processing method according to claim 5.

7. The one or more superimposed sub-frames include a first superimposed sub-frame in which a first pattern image is superimposed on a first sub-frame based on the plurality of sub-frames, and a second superimposed sub-frame in which a second pattern image obtained by inverting pixel values of the first pattern image is superimposed on a second sub-frame based on the plurality of sub-frames. The first superimposed sub-frame and the second superimposed sub-frame displayed on the display surface are acquired by imaging. A third pattern image that is the pattern image is acquired from the difference between the acquired first superimposed sub-frame and the second superimposed sub-frame. By comparing feature points of the acquired third pattern image with reference feature points, a deviation in the display position of the video projected on the display surface is detected. The video processing method according to any one of claims 1 to 4.

8. A program for causing one or more processors to execute the video processing method according to any one of claims 1 to 4.

9. A first acquisition unit that acquires a plurality of sub-frames obtained by temporally dividing frames included in video data, an output unit that executes an output process for causing one or more superimposed sub-frames in which a pattern image is superimposed on one or more sub-frames based on the plurality of sub-frames to be displayed on a display surface, a second acquisition unit that acquires the one or more superimposed sub-frames displayed on the display surface by imaging, a third acquisition unit that acquires the pattern image from the acquired one or more superimposed sub-frames, a detection unit that detects a deviation in the display position of the video projected on the display surface by comparing feature points of the acquired pattern image with reference feature points, and an increase / decrease unit that executes an increase / decrease process for increasing or decreasing the degree of acquiring the pattern image from the acquired one or more superimposed sub-frames according to the acquisition result of the pattern image. A video processing system comprising:

Images