Display systems and information processing devices
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- SEIKO EPSON CORP
- Filing Date
- 2025-01-24
- Publication Date
- 2026-08-05
Smart Images
Figure 2026126640000001_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a display system including a projection device and an information processing device, and to an information processing device of the display system.
Background Art
[0002] For example, Patent Document 1 discloses a display system having a video display device and a mobile terminal. According to this document, the video display device includes a projection unit that projects a plurality of measurement patterns, and the mobile terminal includes a camera that captures the plurality of measurement patterns projected by the projection unit, and a control unit that generates information regarding distortion correction of a video based on the plurality of measurement patterns captured by the camera.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] However, the display system of Patent Document 1 has a problem that the roll angle of the corrected projection image becomes large. Specifically, when performing distortion correction of a projection image based on a captured image by a camera of a mobile terminal, compared with the case of performing distortion correction of a projection image based on a captured image by a camera of a video display device, since the relative positional relationship between the video display device and the camera is not determined in advance, the roll angle of the corrected projection image has become large. The roll angle means that the bottom side of the corrected projection image has an inclination with respect to a line segment parallel to the horizontal plane. That is, there has been a demand for a display system that can obtain a projection image with a small roll angle.
Means for Solving the Problems
[0005] A display system according to one aspect of the present invention includes a projection device for projecting an image, and an information processing device having an imaging unit and an inertial sensor, wherein the projection device projects a first image including a first image pattern onto a projection surface, and the information processing device, after acquiring first image capture data of the first image captured by the imaging unit, causes the projection device to project a second image on the first image which has undergone geometric correction based on the data of the first image, the first image capture data, and the detected value of the inertial sensor, accepts an angle adjustment operation by the user, and, based on the angle adjustment operation, causes the projection device to project a third image which has been obtained by rotating the second image about the normal vector of the projection surface as an axis.
[0006] An information processing device according to one aspect of the present invention is an information processing device for a display system including a projection device for projecting an image and an information processing device having a communication unit capable of communicating with the projection device, wherein the information processing device further comprises an imaging unit and an inertial sensor, and after acquiring first image data captured by the imaging unit, a first image including a first image pattern projected by the projection device onto a projection surface, the device causes the projection device to project a second image on the first image which has undergone geometric correction based on the data of the first image, the first image data, and the detected value of the inertial sensor, accepts an angle adjustment operation by the user, and, based on the angle adjustment operation, causes the projection device to project a third image which has been obtained by rotating the second image about the normal vector of the projection surface as an axis. [Brief explanation of the drawing]
[0007] [Figure 1] A schematic diagram of the display system according to Embodiment 1. [Figure 2] A block diagram showing the general configuration of the display system. [Figure 3] A flowchart illustrating the flow of the first adjustment method. [Figure 4] A figure showing an example of the first image data. [Figure 5] A diagram showing one aspect of the projected image during the adjustment process. [Figure 6] A diagram showing an example of the adjustment screen. [Figure 7] A diagram showing one aspect of the projected image after adjustment. [Figure 8] A flowchart illustrating the flow of the second adjustment method according to Embodiment 2. [Figure 9] A schematic diagram of the display system configuration. [Figure 10] A diagram showing an example of the operation screen. [Figure 11] A diagram showing an example of the second image data. [Figure 12] A diagram showing an example of an adjustment screen related to a modified example. [Figure 13] A diagram showing an example of an adjustment screen related to a modified example. [Modes for carrying out the invention]
[0008] Embodiment 1 ***Overview of the display system*** Figure 1 is a schematic diagram of the display system according to Embodiment 1. Embodiments of the present invention will be described below with reference to the drawings. The following embodiments illustrate just one example of the present invention and are not limited to the embodiments described below. Various modifications that can be implemented without changing the gist of the present invention are also included in the present invention. In addition, in the following drawings, dimensions and scales may differ from those of the actual objects for the sake of clarity in the explanation.
[0009] As shown in Figure 1, the display system 100 consists of a projection device 40, an information processing device 80, and the like. The projection device 40 is, for example, a stationary projector that projects an image defined in the image data. In Figure 1, the projection device 40 projects the first image 11 onto a rectangular screen SC installed on the wall of a conference room. The wall is a substantially vertical surface, and the screen SC mounted along the wall is also substantially vertical. In each figure, including Figure 1, the three mutually orthogonal axes, the X-axis, Y-axis, and Z-axis, are illustrated. In Figure 1, the vertical direction of the screen SC is defined as the Z-axis, and the direction of extension of the Z-axis is defined as the Z-direction. The X-axis runs along the long side of the screen SC, and its extension direction is defined as the X-direction. The direction perpendicular to the Z-direction and the X-direction is defined as the Y-direction. In a preferred example, the X-direction runs horizontally, and the Z-direction runs vertically. The screen SC corresponds to the projection surface, and its bottom edge runs horizontally. The projection device 40 may also be a ceiling-mounted type. On the X-axis, the tip of the arrow is called the X-plus direction, and the opposite side is called the X-minus direction. The same applies to the Y-axis and Z-axis. The Z-plus direction is also called upward, and the Z-plus direction is also called downward.
[0010] The information processing device 80 is a smartphone and includes a display unit 83, an imaging unit 84, and the like. The display unit 83 is rectangular in shape and is equipped with a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display. The display unit 83 has a touch panel and also functions as an operation unit 88. An operation button 88b is provided next to the display unit 83. The operation button 88b is part of the operation unit 88 and is the power button for starting up the information processing device 80. The operation button 88b may also have a fingerprint authentication function. The information processing device 80 is held by the user in a horizontal orientation with the long side of the display unit 83 aligned with the X direction. The information processing device 80 is not limited to a smartphone, but may be any portable information processing device with similar functions, such as a notebook computer or a tablet computer. In a preferred example, the projection device 40 and the information processing device 80 are capable of bidirectional communication via a wireless LAN (Local Area Network).
[0011] FIG. 1 shows one aspect when an adjustment program for correcting the distortion of the projection screen is being executed. Although it will be described in detail later, the adjustment program is a program for correcting the distortion of the projection screen so that the projection screen becomes a similar shape to the screen SC. As shown in FIG. 1, the first image 11 has a trapezoidal shape that is wide at the top and narrow at the bottom, and is in a state where the X plus side is inclined downward in the screen SC. Inside the first image 11, a first image pattern Pa1 whose outer shape is a similar shape to the first image 11 is displayed. In FIG. 1, the information processing device 80, according to the adjustment program, the imaging unit 84 captures the first image 11, and the captured image is displayed on the display unit 83. The entire first image pattern Pa1 fits within the captured image. The details of the adjustment program will be described later.
[0012] ***Configuration of the projection device*** FIG. 2 is a block diagram showing the schematic configuration of the display system. As shown in FIG. 2, the projection device 40 is composed of a control unit 41, a storage unit 42, a communication unit 43, an operation unit 44, an IF unit 45, an image processing unit 46, a projection unit 47, etc. The control unit 41 is connected to each of the above units via a system bus 49. The control unit 41 is configured to include one or more processors, and operates according to a control program stored in the storage unit 42 to comprehensively control the operation of the projection device 40.
[0013] <## The storage unit 42 is configured to include a RAM (Random Access Memory) and a ROM (Read Only Memory). The RAM is used for temporary storage of various data, etc., and the ROM stores a control program and control data for controlling the operation of the projection device 40. The control data includes coordinate data of the four corners of the projection image. The communication unit 43 is configured to include a wireless communication device for performing wireless communication by means of a wireless LAN, Bluetooth (registered trademark), or the like. Based on the control of the control unit 41, the communication unit 43 transmits and receives information through a wireless connection with the information processing device 80.
[0014] The operation unit 44 includes a plurality of operation keys for the user to give various instructions to the projection device 40. The operation keys included in the operation unit 44 include a "power key" for switching on and off the power, a "menu key" for displaying a menu for performing various settings, and a "direction key" for selecting menu items. Note that a remote controller (not shown) capable of remote operation may be used as the input operation unit. In this case, the remote controller transmits an infrared operation signal according to the operation content of the operator, and a remote control signal receiving unit (not shown) receives this and transmits the operation signal to the control unit 41.
[0015] The IF unit 45 is an interface part with external devices and includes a plurality of connection terminals including HDMI (registered trademark, High-Definition Multimedia Interface) terminals. The plurality of connection terminals may further include, for example, USB (Universal Serial Bus) terminals, VGA (Video Graphics Array) terminals, and the like.
[0016] Based on the control of the control unit 41, the image processing unit 46 performs necessary image processing on the input image information and outputs the processed image information to the projection unit 47. The image processing includes image processing for correcting the distortion of the projected image according to the correction parameters received from the information processing device 80.
[0017] The projection unit 47 consists of a light source, a light modulator, and a projection optical system (none of which are shown). The projection unit 47 modulates the light emitted from the light source with the light modulator to form image light as defined in the image information, and projects this image light from the projection optical system. The light modulator may be, for example, a liquid crystal panel or a digital micromirror device. The light modulator may be a single liquid crystal panel or digital mirror device, or it may include multiple liquid crystal panels or digital mirror devices. The projection optical system includes one or more optical elements that adjust the magnification and imaging position of the light output from the light modulator.
[0018] The control unit 41 and the image processing unit 46 may be composed of one or more processors, or they may be composed of dedicated processing devices such as ASICs (Application Specific Integrated Circuits) or FPGAs (Field Programmable Gate Arrays).
[0019] ***Configuration of the Information Processing Device*** As shown in Figure 2, the information processing device 80 consists of a control unit 81, a storage unit 82, a display unit 83, an imaging unit 84, a sensor unit 85, a first communication unit 86, a second communication unit 87, an operation unit 88, and the like. The control unit 81 is connected to each of the above units via a system bus 89.
[0020] The control unit 81 is comprised of one or more processors. The control unit 81 operates according to the OS (Operating System) and control programs stored in the memory unit 82, thereby providing overall control over the operation of the information processing device 80. The storage unit 82 consists of memory such as RAM and ROM. The storage unit 82 stores the OS, control programs, and various data. The control programs include a first adjustment program 82a and a second adjustment program 82b. The various data include test image data 82c and operation screen data 82d.
[0021] The display unit 83 is configured with a display device such as a liquid crystal display or an organic EL display, and displays an image based on image information. The imaging unit 84 is a camera equipped with an image sensor such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The imaging unit 84 performs imaging based on the control of the control unit 81, and the captured image data is stored in the storage unit 82.
[0022] The operation unit 88 includes a touch panel integrated with the display unit 83 and operation buttons 88b, and accepts user input. When a user operates the operation unit 88, the operation unit 88 outputs an operation signal corresponding to the operation to the control unit 81 via the system bus 89. The sensor unit 85 is an inertial sensor equipped with a 3-axis accelerometer and a 3-axis gyroscope (angular velocity) sensor, and detects the attitude of the information processing device 80. The sensor unit 85 outputs a detection signal to the control unit 81 via the system bus 89 according to a predetermined sampling frequency.
[0023] The first communication unit 86 is a communication unit that transmits and receives data to and from the nearest base station (not shown) via an antenna, based on a fourth-generation mobile communication system compliant with the IMT-Advance standard and / or a fifth-generation mobile communication system compliant with the IMT-2020 standard.
[0024] The second communication unit 87 is configured to include wireless communication devices for wireless communication using wireless LAN, Bluetooth (registered trademark), etc. Based on the control of the control unit 81, the second communication unit 87 transmits and receives information wirelessly with the projection device 40. In other words, the display system 100 comprises a projection device 40 for projecting images and an information processing device 80 having an imaging unit 84 and a sensor unit 85 consisting of an inertial sensor. The information processing device 80 also has a second communication unit 87 that can communicate with the projection device 40.
[0025] ***First adjustment method*** Figure 3 is a flowchart showing the flow of the first adjustment method. Figure 4 is a diagram showing an example of the first image data. Figure 5 is a diagram showing one aspect of the projected image during the adjustment process, and corresponds to Figure 1. Figure 6 is a diagram showing an example of the adjustment screen. Figure 7 is a diagram showing one aspect of the projected image after adjustment, and corresponds to Figure 5. Here, we will explain the first adjustment method for correcting distortion in the projected image, primarily using Figure 3, and incorporating other diagrams as appropriate. Each of the following steps is performed by the information processing device 80 executing the first adjustment program 82a. Specifically, the first adjustment program 82a is executed when the user operates the distortion adjustment program icon displayed on the display unit 83 of the information processing device 80. When the first adjustment program 82a is executed, the information processing device 80 and the projection device 40 are wirelessly connected, and the projection device 40 operates according to commands from the information processing device 80. In the following description, the main entities executing each step are the control unit 81 of the information processing device 80 and the control unit 41 of the projection device 40.
[0026] In step S10, the information processing device 80 accepts an adjustment start operation from the user. Specifically, when the adjustment start icon (not shown) on the display unit 83 is tapped, the process proceeds to step S11.
[0027] In step S11, the information processing device 80 reads the first image data from the test image data 82c in the storage unit 82 and transmits it to the projection device 40. As shown in Figure 4, the first image data 51 is image data in which the first image pattern Pa1, consisting of a chessboard pattern similar in shape to the outline, is placed within the rectangular outline shown by the dotted line. The first image pattern Pa1 is a checkerboard pattern in which black squares and white squares are alternately arranged in the X and Z directions. Each vertex of the multiple squares is called a detection point or checker corner.
[0028] In step S30, the projection device 40 determines whether or not it has received data from the information processing device 80. If data has been received, it proceeds to step S31. If data has not been received, it continues to wait for data to be received. Note that the determination of whether or not data has been received from the information processing device 80 is also performed in subsequent flows, but the description of the subsequent determination process is omitted in Figure 3.
[0029] In step S31, the projection device 40 projects the image specified in the received first image data 51. Figure 1 shows a first image 11, which is an example of a projected image. The first image 11 has a trapezoidal shape that is wider at the top and narrower at the bottom, and the X-plus side is tilted downwards on the screen SC. The first image 11 displays a first image pattern Pa1, which is a chessboard pattern. In other words, the projection device 40 projects the first image 11, which includes the first image pattern Pa1, onto the screen SC, which serves as the projection surface.
[0030] In step S12, the information processing device 80 performs geometric correction processing. Specifically, the following processes (1) to (4) are performed. (1) The information processing device 80 causes the imaging unit 84 to capture the first image 11. Specifically, as shown in Figure 1, when the user presses the operation button 88b to start imaging and then moves the information processing device 80 left and right in the X direction relative to the screen SC while holding it in their hand, the imaging unit 84 captures multiple images at multiple different imaging positions. At this time, the display unit 83 displays a guide message that reads, "Images will be automatically captured by moving left and right. Please capture the entire pattern in the camera."
[0031] (2) In this step, the information processing device 80 detects a chessboard pattern from each of the captured images by analyzing each of the multiple captured images. Specifically, it detects the points of the chessboard pattern for each captured image. In (3), the information processing device 80 determines whether the entire chessboard pattern was detected from two or more captured images in (2). If it is detected from two or more captured images, the device proceeds to process (4). If it is not detected, the device returns to process (1).
[0032] (4) In this step, the information processing device 80 calculates correction parameters. In this step, the control unit 81 functions as the calculation unit 81a. First, the calculation unit 81a calculates a projection transformation matrix based on the detection points of the reference first image data 51 and the detection points of the captured image data. The projection transformation matrix is a matrix that transforms the coordinate system in the captured image to the coordinate system in the projection device 40. In other words, the projection transformation matrix is a matrix that transforms the camera image coordinate system to the projector coordinate system.
[0033] Next, the calculation unit 81a calculates the normal vector of the screen SC based on the projection transformation matrix and calculates the vertical vector of the screen SC based on the detection data from the sensor unit 85. Then, based on the normal vector and vertical vector of the screen SC, it calculates the horizontal vector of the screen SC and estimates the two-dimensional coordinate system of the screen SC. Next, the calculation unit 81a calculates a conversion matrix between the projector coordinate system and the screen coordinate system. Specifically, it determines the coordinates of the four corners of the corrected projected screen so that the projected image is rectangular on the screen SC, and converts these coordinates to coordinates in the projector coordinate system using the conversion matrix. Then, the calculation unit 81a calculates correction parameters for distortion correction from the coordinates of the four corners in the projector coordinate system and the coordinates of the four corners of the projected screen. In other words, the information processing device 80 acquires first image data obtained by the imaging unit 84, which captures the first image 11.
[0034] In step S13, the information processing device 80 transmits the calculated correction parameters to the projection device 40. In other words, the information processing device 80 causes the projection device 40 to project the second image 12, which has undergone geometric correction to the first image 11 based on the reference first image data 51, the first captured image data, and the detected value of the sensor unit 85.
[0035] In step S32, the projection device 40 generates a second image data using the correction parameters received by the image processing unit 46 and projects the image defined in the second image data. Figure 5 shows a second image 12, which is an example of a projected second image. The second image 12 is a rectangular image, but it has a roll angle with respect to the screen SC. Specifically, it has a roll angle of angle θ with respect to the base of the screen SC. As a result, the second image 12 is tilted upward to the right with respect to the screen SC in the X direction. Also, the size of the second image 12 with respect to the screen SC is relatively small. Note that in Figure 5, the normal vector of the screen SC is defined as the center line 60. The center line 60 is a line segment extending in the Y direction from the center of the screen SC, as shown in Figure 2.
[0036] In step S14, the projection image adjustment screen is displayed on the display unit 83 of the information processing device 80. Specifically, the adjustment screen data is read from the operation screen data 82d of the storage unit 42 and displayed on the OSD (On Screen Display) by the image processing unit 46. As shown in Figure 6, the adjustment screen 70 displays operation icons 71a and 71b for angle adjustment, operation icon 72 for position adjustment, and operation icons 73a and 73b for zooming in and out. Tapping the angle adjustment icon 71a once rotates the projected image 3° counterclockwise around the center line 60 (Figure 5). Each subsequent tap of the operation icon 71a rotates the projected image 3° counterclockwise. Similarly, tapping the operation icon 71b once rotates the projected image 3° clockwise around the center line 60. Each subsequent tap of the operation icon 71b rotates the projected image 3° clockwise. Note that the angle is not limited to 3°.
[0037] The position adjustment operation icon 72 has triangular arrow icons 72a, 72b, 72c, and 72d arranged in a cross shape. Tapping arrow icon 72a once moves the projected image a predetermined distance in the X-positive direction. Similarly, tapping arrow icon 72c once moves the projected image a predetermined distance in the X-minus direction. Tapping arrow icon 72b once moves the projected image a predetermined distance in the Z-minus direction. Similarly, tapping arrow icon 72d once moves the projected image a predetermined distance in the Z-positive direction. By manipulating arrow icons 72a, 72b, 72c, and 72d, you can move the projected image up, down, left, and right. Tapping the enlargement icon 73a once will increase the size of the projected image. Similarly, tapping the reduction icon 73b once will decrease the size of the projected image. By operating icons 73a and 73b, you can change the size of the projected image.
[0038] In step S15, the information processing device 80 accepts an operation on the adjustment screen 70. For example, if the user operates the angle adjustment operation icon 71b and the zoom operation icon 73a on the second image 12 in Figure 5, correction parameters corresponding to that operation are generated.
[0039] In step S16, the information processing device 80 transmits correction parameters corresponding to the operation to the projection device 40. In other words, the information processing device 80 accepts the angle adjustment operation by the user and, based on the angle adjustment operation, causes the projection device 40 to project a third image 13, which is obtained by rotating the second image 12 around the center line 60 as the normal vector of the projection surface.
[0040] In step S33, the projection device 40 generates a third image data using the correction parameters received by the image processing unit 46 and projects the image defined in the third image data. Figure 7 shows a third image 13, which is an example of a projected third image. The projection device 40 stores the parameters, including the coordinates of the four corners of the third image data, in the storage unit 42. The third image 13 has a rectangular shape with its base aligned with the base of screen SC, and is appropriately sized for screen SC.
[0041] In step S17, an operation screen (not shown) is displayed on the display unit 83 of the information processing device 80 to confirm whether or not to end the adjustment using the adjustment screen 70. If the end icon is pressed, the process ends. If the continue icon is pressed, the process returns to step S14 and the adjustment using the adjustment screen 70 continues. Alternatively, if a certain amount of time has elapsed since the operation screen was displayed, the correction process may be considered complete and the process may be terminated.
[0042] As described above, the display system 100 and information processing device 80 of this embodiment provide the following advantages. The display system 100 includes a projection device 40 for projecting images and an information processing device 80 having an imaging unit 84 and a sensor unit 85 consisting of an inertial sensor. The projection device 40 projects a first image 11 including a first image pattern Pa1 onto a screen SC which serves as a projection surface. The information processing device 80 acquires first image data obtained by the imaging unit 84 of the first image 11, then projects a second image 12 onto the projection device 40, which has undergone geometric correction to the first image 11 based on a reference first image data 51, the first image data, and the detected value of the sensor unit 85. The information processing device 80 accepts an angle adjustment operation from the user and, based on the angle adjustment operation, projects a third image 13 onto the projection device 40, which is obtained by rotating the second image 12 around a center line 60 which serves as the normal vector of the projection surface.
[0043] According to this, geometric correction is performed on the first image 11 based on the reference first image data 51, the first captured image data, and the detected value of the sensor unit 85. By rotating the geometrically corrected second image 12 around the center line 60, which is the normal vector of the screen SC as the projection surface, the roll angle of the corrected third image 13 relative to the screen SC can be reduced. Therefore, a display system 100 can be provided that can obtain a projected image with a small roll angle.
[0044] Furthermore, the information processing device 80 transmits the data of the first image to the projection device 40. Specifically, it reads the first image data from the test image data 82c in the storage unit 82 and transmits it to the projection device 40. In other words, the first image data, including the first image pattern Pa1, is stored in the information processing device 80. Therefore, when updating the first adjustment program 82a, which includes the first image data, it is not necessary to update both the program stored in the projection device 40 and the program stored in the information processing device 80; only the first adjustment program 82a needs to be updated. Thus, the user's workload can be reduced. The same applies to the second adjustment program 82b.
[0045] The information processing device 80 is an information processing device for a display system that includes a projection device 40 for projecting images and a second communication unit 87 capable of communicating with the projection device 40. The information processing device 80 further includes an imaging unit 84 and a sensor unit 85 consisting of an inertial sensor. After acquiring first image data captured by the imaging unit 84 of a first image 11 including a first image pattern Pa1 projected by the projection device 40 onto a screen SC, the device causes the projection device 40 to project a second image 12, which has undergone geometric correction to the first image 11 based on a reference first image data 51, the first image data, and the detected value of the sensor unit 85. The device accepts an angle adjustment operation from the user and, based on the angle adjustment operation, causes the projection device 40 to project a third image 13, which is obtained by rotating the second image 12 around a center line 60 as the normal vector of the projection surface.
[0046] According to this, it is possible to provide an information processing device 80 suitable for a display system 100 that can obtain a projected image with a small roll angle.
[0047] Embodiment 2 ***Different forms of adjustment*** Figure 8 is a flowchart showing the flow of the second adjustment method according to Embodiment 2, and corresponds to Figure 3. Figure 9 is a schematic configuration diagram of the display system, and corresponds to Figure 1. In the above embodiment, the shape of the projected image was described as being corrected by performing geometric correction and roll angle correction to correct distortion, but the invention is not limited to this, and the unevenness of the projected image may also be corrected.
[0048] In the following, the same parts as in the above embodiment will be numbered, and redundant explanations will be omitted. As shown in Figure 9, the display system 110 of this embodiment consists of a projection device 48, an information processing device 80, and the like. The projection device 48 of this embodiment is a short-throw projector and is installed approximately directly below the screen SC which is installed along the wall. The configuration of the projection device 48 is basically the same as that of the projection device 40 in Figure 2, but the projection unit 47 is equipped with an optical system that supports short-throw projection, including a wide-angle lens. The information processing device 80 is the same as the information processing device 80 in Figure 2, but in this embodiment, the second adjustment program 82b of the storage unit 82 is executed. Aside from these points, it is the same as described in Embodiment 1.
[0049] As shown in Figure 9, the projection angle (elevation angle) of the short-throw projector 48 is larger than that of the stationary projector 40 in Figure 1. When the projection angle is large, slight irregularities (wavy surfaces) in the screen that do not cause problems with a normal projector will affect the projected image, causing distortion. According to the second adjustment method of this embodiment, in addition to shape correction by geometric correction and roll angle correction, irregularity correction can be performed. The projection device 48 may also be installed above the screen SC. In this case, an installation device is provided above the screen SC, extending from the wall surface in the Y-minus direction, and the projection device 48 is fixed to this installation device, projecting the image from diagonally above the screen SC.
[0050] Each of the following steps is performed by the information processing device 80 executing the second adjustment program 82b. Specifically, the second adjustment program 82b is executed by the user operating the distortion adjustment program icon (not shown) displayed on the display unit 83 of the information processing device 80. Since the second adjustment program 82b performs unevenness correction following the first adjustment program 82a, the following explanation will describe the information processing device 80 from step S17 onwards, and the projection device 48 from step S33 onwards. The content prior to that is the same as the explanation in Figure 3.
[0051] ***Second adjustment method*** Step S17 in Figure 8 is the same as step S17 in Figure 3. More specifically, in step S17, an operation screen (not shown) is displayed on the display unit 83 of the information processing device 80 to confirm whether or not to terminate the adjustment using the adjustment screen 70 (Figure 6). If the continue icon is operated, the process returns to step S14 and the adjustment using the adjustment screen 70 is continued. Here, we will explain assuming that the exit icon has been activated. At this time, the projection device 48 is projecting the third image 13 (Figure 7) using the third image data.
[0052] Figure 10 shows an example of the operation screen and corresponds to Figure 6. When the termination operation is performed, the display unit 83 of the information processing device 80 displays the operation screen 74 shown in Figure 10. Specifically, the operation screen data is read from the operation screen data 82d of the storage unit 42 and displayed on the OSD by the image processing unit 46. The operation screen 74 displays the guide message "Shape correction completed," and operation icons 74a for continuing surface correction and 74b for not performing surface correction. In this explanation, we assume that the operation icon 74a for performing surface unevenness correction was operated in step S19. If the operation icon 74b for not performing surface unevenness correction was operated, the adjustment will be terminated.
[0053] Figure 11 shows an example of the second image data and corresponds to Figure 4. In step S20, the information processing device 80 reads the fourth image data from the test image data 82c in the storage unit 82 and transmits it to the projection device 40. As shown in Figure 11, the fourth image data 52 is image data in which a second image pattern Pa2, consisting of a dot matrix pattern similar in shape to the contour line, is placed within a rectangular contour line indicated by a dotted line. The second image pattern Pa2 is a dot matrix pattern in which multiple dots 3 are arranged at equal intervals in the X and Y directions. Each dot 3 is circular, and its center is the detection point. The proportion occupied by the second image pattern Pa2 within the rectangular contour line is larger than that of the first image pattern Pa1 (Figure 4). In other words, the margin around the second image pattern Pa2 in the fourth image data 52 is narrower than that in the first image data 51 (Figure 4). This allows for unevenness correction to be applied even to the periphery of the screen SC.
[0054] In step S34, the projection device 48 projects the image specified in the received fourth image data 52.
[0055] In step S21, the information processing device 80 performs unevenness correction processing. Specifically, the processes (11) to (14) below are performed. (11) In this step, the information processing device 80 causes the imaging unit 84 to capture a fourth image (not shown). At this time, the display unit 83 displays a guide message, for example, "Please capture the entire pattern in front of the projected image with the camera."
[0056] In (12), the information processing device 80 detects a dot matrix pattern from the captured image by analyzing the captured image. More specifically, it detects the detection points of the dot matrix pattern. The captured image corresponds to the second captured image data. In (13), the information processing device 80 determines whether the entire dot matrix pattern has been detected from the captured image in (12). If it has been detected, it proceeds to process (14). If it has not been detected, it returns to process (11).
[0057] In (14), the information processing device 80 calculates correction parameters. At this time, the control unit 81 functions as a calculation unit 81a. The calculation unit 81a calculates a correction value for each dot 3 based on the position of each dot 3 in the reference second image pattern Pa2 and the position of each dot 3 in the captured image, and generates correction parameters. Alternatively, multiple images may be captured in (11), and then each captured image and the detection and determination of the entire dot matrix pattern may be performed in (12) and (13).
[0058] In step S22, the information processing device 80 transmits the calculated correction parameters to the projection device 48.
[0059] In step S35, the projection device 48 generates a fifth image data using the correction parameters received by the image processing unit 46, and projects a fifth image (not shown) using the fifth image data. The fifth image is a projected image in which both shape correction and surface unevenness correction have been applied. The projection device 48 stores the parameters, including the surface unevenness correction of the fifth image data, in the storage unit 42. In other words, the projection device 48 projects the third image, then projects the fourth image, which includes the second image pattern Pa2, onto the screen SC. The information processing device 80 acquires the second image data, which captures the fourth image, and based on the data of the fourth image and the second image data, causes the projection device 48 to project a fifth image, which has been corrected for unevenness relative to the fourth image.
[0060] In step S23, an operation screen (not shown) is displayed on the display unit 83 of the information processing device 80 to confirm whether or not to terminate the surface unevenness correction. If the continue icon is pressed, the process returns to step S21 and the surface unevenness correction continues. If the terminate icon is pressed, the surface unevenness correction is terminated.
[0061] As described above, the display system 110 and information processing device 80 of this embodiment can provide the following effects in addition to those of the above embodiment. The projection device 48 of the display system 110 projects the third image, then projects the fourth image, which includes the second image pattern Pa2, onto the screen SC. The information processing device 80 acquires the second image data, which captures the fourth image, and based on the data of the fourth image and the second image data, causes the projection device 48 to project a fifth image, which has been corrected for unevenness relative to the fourth image.
[0062] According to this, in addition to shape correction by geometric correction and roll angle correction, surface irregularity correction can also be applied. Therefore, in addition to shape correction of the projected image, distortion of the projected image caused by slight irregularities in the screen SC, which are likely to occur in short-throw projectors 48, can be corrected. Furthermore, since surface irregularity correction is performed on the fourth image that has already undergone shape correction, the adjustment procedure for the projected image can be optimized. Therefore, it is possible to provide a display system 110 that can obtain a projected image with a small roll angle and reduced surface distortion.
[0063] The information processing device 80 projects the third image onto the projection device 48, then projects the fourth image, which includes the second image pattern Pa2, onto the screen SC onto the projection device 48, acquires the second image capture data of the fourth image, and then projects the fifth image, which has undergone surface contour correction on the fourth image, onto the projection device 48 based on the data of the fourth image and the second image capture data. According to this, it is possible to provide an information processing device 80 suitable for a display system 110 that can obtain a projected image with a small roll angle and reduced surface distortion. Furthermore, the information processing device 80 can optimize the adjustment procedure for the projected image.
[0064] ***Modification*** Figure 12 shows an example of the adjustment screen related to a modified example, and corresponds to Figure 6. In the above description, the adjustment screen 75 (Figure 6) is described as having two angle adjustment operation icons 71a and 71b, but it is not limited to this, and may also have additional angle adjustment operation icons. Hereafter, the same parts as in the above embodiment will be numbered, and redundant explanations will be omitted.
[0065] The adjustment screen 75 of this modified example shown in Figure 12 is provided with four angle adjustment operation icons 71a, 71b, 71c, and 71d. Aside from this, the explanation is the same as in Figure 6. In this modified example, tapping the angle adjustment icon 71a once causes the second image 12 (Figure 5), as the second image, to rotate 5° counterclockwise around the center line 60. The operation icon 71c is located next to the operation icon 71a and is slightly smaller than the operation icon 71a. Tapping the operation icon 71c once rotates the projected image 1° counterclockwise around the center line 60. This allows for a large rotation using the operation icon 71a, followed by fine adjustments using the operation icon 71c. The operation performed by operation icon 71c is the first angle adjustment operation, and the operation performed by operation icon 71a is the second angle adjustment operation. In other words, the angle adjustment operation includes the first angle adjustment operation performed by operation icon 71c and the second angle adjustment operation performed by operation icon 71a. The direction in which the second image is rotated about the center line 60, which is the normal vector, by the second angle adjustment operation is the same as the direction in which the second image is rotated about the center line 60 by the first angle adjustment operation, and the angle by which the second image is rotated about the center line 60 by the second angle adjustment operation is greater than the angle by which the second image is rotated about the center line 60 by the first angle adjustment operation.
[0066] In this modified version, tapping the angle adjustment icon 71b once rotates the projected image 5° clockwise around the center line 60. The operation icon 71d is located next to the operation icon 71b and is slightly smaller than the operation icon 71b. Tapping the operation icon 71d once rotates the projected image 1° clockwise around the center line 60. This allows for a large rotation using the operation icon 71b, followed by fine adjustments using the operation icon 71d. Furthermore, by combining the operation icons 71a, 71b, 71c, and 71d, the roll angle of the second image 12 can be adjusted efficiently.
[0067] Return to Figure 6. In the above, tapping the operation icon 71a on the adjustment screen 70 causes the second image 12 to rotate 3° counterclockwise. However, the rotation angle may vary depending on the operation method. Specifically, tapping the operation icon 71a may cause a 1° rotation, and pressing and holding the operation icon 71a for, for example, 2 seconds or more may cause a 5° rotation. In this case, tapping the operation icon 71a is the first angle adjustment operation, and pressing and holding is the second angle adjustment operation. Similarly, tapping the operation icon 71b may cause a 1° rotation, and pressing and holding the operation icon 71b for, for example, 2 seconds or more may cause a 5° rotation.
[0068] As described above, this modified version provides the following effects in addition to those of the above embodiment. Angle adjustment operations include a first angle adjustment operation using operation icon 71c and a second angle adjustment operation using operation icon 71a. The direction in which the second image is rotated about the center line 60, which is the normal vector, by the second angle adjustment operation is the same as the direction in which the second image is rotated about the center line 60 by the first angle adjustment operation. The angle by which the second image is rotated about the center line 60 by the second angle adjustment operation is greater than the angle by which the second image is rotated about the center line 60 by the first angle adjustment operation. According to this, the angle adjustment operation includes a first angle adjustment operation with the same rotation direction but different rotation angles, and a second angle adjustment operation. Therefore, the user can select the degree of adjustment, such as coarse adjustment or fine adjustment, of the angle of the second image, and the angle adjustment operation can be performed efficiently.
[0069] Figure 13 shows an example of the adjustment screen related to a modified example, and corresponds to Figure 6. In the above description, the adjustment screen 70 (Figure 6) was described as being provided with operation icons 71a and 71b for angle adjustment, operation icon 72 for position adjustment, and operation icons 73a and 73b for zooming in and out. However, it is not limited to this, and an image capture screen for the projected image may also be provided. Hereafter, the same parts as in the above embodiment will be numbered, and redundant explanations will be omitted.
[0070] In the modified example shown in Figure 13, the adjustment screen 76 displays two angle adjustment operation icons 71a and 71b, as well as the image of the projected screen. Aside from these points, the explanation is the same as in Figure 6. On the adjustment screen 76, the captured image of the projected screen captured by the imaging unit 84 is displayed on the X-minus side of the operation icons 71a and 71b. In Figure 13, the second image 12 is displayed as the second image before roll angle adjustment. According to this, the user can adjust the roll angle by operating the operation icons 71a and 71b while viewing the captured screen on the adjustment screen 76. In other words, the information processing device 80 displays the adjustment screen 76 which accepts angle adjustment operations, and the adjustment screen 76 includes the operation icons 71a and 71b as angle adjustment operation icons and the captured image captured by the imaging unit 84.
[0071] As described above, this modified version provides the following effects in addition to those of the above embodiment. The information processing device 80 displays an adjustment screen 76 for accepting angle adjustment operations. The adjustment screen 76 includes operation icons 71a and 71b as angle adjustment operation icons, and an image captured by the imaging unit 84. According to this, the adjustment screen 76 displays the captured image captured by the imaging unit 84 on the operation screen that accepts angle adjustment operations, allowing the user to perform angle adjustment operations while checking the captured image.
[0072] Return to Figure 1. The above explanation assumes that the screen SC, which serves as the projection surface, is installed on the wall of a conference room or similar space. However, this is not the only option; the wall itself may also be used as the projection surface. Furthermore, although the second adjustment method was described as being applied to a short-throw projector 48, the second adjustment method may also be performed on a standard stationary projector 40. This allows for the acquisition of a projected image with a small roll angle and reduced surface distortion.
[0073] ***Summary of this disclosure*** A summary of this disclosure is provided below. (Note 1) A display system including a projection device for projecting images, and an information processing device having an imaging unit and an inertial sensor, The projection device projects a first image including a first image pattern onto the projection surface. After the information processing device acquires the first image image data obtained by the imaging unit, A second image, which has undergone geometric correction to the first image based on the data of the first image, the first captured image data, and the detected value of the inertial sensor, is projected onto the projection device. It accepts angle adjustment operations from the user. Based on the angle adjustment operation, a third image is projected onto the projection device by rotating the second image about the normal vector of the projection surface as the axis. Display system. According to this method, geometric correction is performed on the first image based on the reference first image data, the first captured image data, and the detected value from the inertial sensor. By rotating the geometrically corrected second image around the center line 60, which is the normal vector of the projection surface, the roll angle of the corrected projection image relative to the projection surface can be reduced. Therefore, it is possible to provide a display system that can produce projected images with a small roll angle.
[0074] (Note 2) The angle adjustment operation includes a first angle adjustment operation and a second angle adjustment operation. The direction in which the second image is rotated about the normal vector by the second angle adjustment operation is the same as the direction in which the second image is rotated about the normal vector by the first angle adjustment operation. The angle by which the second image is rotated about the normal vector by the second angle adjustment operation is greater than the angle by which the second image is rotated about the normal vector by the first angle adjustment operation. The display system described in Appendix 1. According to this, the angle adjustment operation includes a first angle adjustment operation with the same rotation direction but different rotation angles, and a second angle adjustment operation. Therefore, the user can select the degree of adjustment, such as coarse adjustment or fine adjustment, of the angle of the second image, and the angle adjustment operation can be performed efficiently.
[0075] (Note 3) After projecting the third image, the projection device projects a fourth image, including the second image pattern, onto the projection surface. The information processing device acquires the second image data obtained by capturing the fourth image, Based on the data of the fourth image and the second image capture data, a fifth image, which has undergone surface contour correction for the fourth image, is projected onto the projection device. The display system described in Appendix 1 or 2. According to this, in addition to shape correction by geometric correction and roll angle correction, surface irregularity correction can also be applied. Therefore, in addition to shape correction of the projected image, distortion of the projected image caused by slight irregularities in the screen SC, which are likely to occur in short-throw projectors 48, can be corrected. Furthermore, since surface irregularity correction is performed on the fourth image that has already undergone shape correction, the adjustment procedure for the projected image can be optimized. Therefore, it is possible to provide a display system that can produce projected images with a small roll angle and reduced surface distortion.
[0076] (Note 4) The information processing device transmits the first image data to the projection device. The display system described in any one of the following appendices 1 to 3. According to this, when updating the first adjustment program 82a, which includes the first image data, it is not necessary to update both the program stored in the projection device 40 and the program stored in the information processing device 80; only the first adjustment program 82a needs to be updated. Therefore, the user's workload can be reduced. The same applies to the second adjustment program 82b.
[0077] (Note 5) The information processing device displays an adjustment screen that accepts the angle adjustment operation, The adjustment screen includes an angle adjustment operation icon and an image captured by the imaging unit. The display system described in any one of the following appendices 1 to 4. According to this, the adjustment screen 76 displays the captured image captured by the imaging unit 84 on the operation screen that accepts angle adjustment operations, allowing the user to perform angle adjustment operations while checking the captured image.
[0078] (Note 6) An information processing device for a display system, comprising a projection device for projecting images and an information processing device having a communication unit capable of communicating with the projection device, The information processing device further comprises an imaging unit and an inertial sensor, After acquiring first image data captured by the imaging unit, the projection device projects a first image pattern onto the projection surface, A second image, which has undergone geometric correction to the first image based on the data of the first image, the first captured image data, and the detected value of the inertial sensor, is projected onto the projection device. It accepts angle adjustment operations from the user. Based on the angle adjustment operation, a third image is projected onto the projection device by rotating the second image about the normal vector of the projection surface as the axis. Information processing device. This makes it possible to provide an information processing device suitable for display systems that can obtain projected images with a small roll angle.
[0079] (Note 7) The angle adjustment operation includes a first angle adjustment operation and a second angle adjustment operation. The direction in which the second image is rotated about the normal vector by the second angle adjustment operation is the same as the direction in which the second image is rotated about the normal vector by the first angle adjustment operation. The angle by which the second image is rotated about the normal vector by the second angle adjustment operation is greater than the angle by which the second image is rotated about the normal vector by the first angle adjustment operation. The information processing device described in Appendix 6. This allows for efficient angle adjustment of the second image.
[0080] (Note 8) After projecting the third image onto the projection device, project a fourth image including the second image pattern onto the projection surface onto the projection device. The second image data obtained by capturing the fourth image described above is acquired. Based on the data of the fourth image and the second image capture data, a fifth image, which has undergone surface contour correction for the fourth image, is projected onto the projection device. The information processing device described in Appendix 6 or 7. This makes it possible to provide an information processing device suitable for display systems that can obtain projected images with a small roll angle and reduced surface distortion. Furthermore, the information processing device can optimize the adjustment procedure for projected images.
[0081] (Note 9) The data of the first image is transmitted to the projection device. An information processing device as described in any one of the items 6 to 8 of the appendix. According to this, when updating the first adjustment program 82a, which includes the first image data, it is not necessary to update both the program stored in the projection device 40 and the program stored in the information processing device 80; only the first adjustment program 82a needs to be updated. Therefore, the user's workload can be reduced. The same applies to the second adjustment program 82b.
[0082] (Note 10) The adjustment screen that accepts the aforementioned angle adjustment operation is displayed, The adjustment screen includes an angle adjustment operation icon and an image captured by the imaging unit. An information processing device as described in any one of the items 6 to 9 of the appendix. According to this, users can efficiently adjust the angle while checking the captured image. [Explanation of Symbols]
[0083] 3...dot, 11...first image, 12...second image, 13...third image, 40...projection device, 41...control unit, 42...storage unit, 43...communication unit, 44...operation unit, 45...IF unit, 46...image processing unit, 47...projection unit, 48...projection device, 49...system bus, 51...first image data, 52...fourth image data, 60...centerline, 70...adjustment screen, 71a...operation icon, 71b...operation icon, 71c...operation icon, 71d...operation icon, 72...operation icon for position adjustment, 72a...arrow icon, 72b...arrow icon, 72c...arrow icon, 72d...arrow icon, 73a...operation icon for zooming in, 73b...for zooming out 74...Operation screen, 74a...Operation icon for performing surface unevenness correction, 74b...Operation icon for not performing surface unevenness correction, 75...Adjustment screen, 76...Adjustment screen, 80...Information processing device, 81...Control unit, 81a...Calculation unit, 82...Storage unit, 82a...First adjustment program, 82b...Second adjustment program, 82c...Test image data, 82d...Operation screen data, 83...Display unit, 84...Imaging unit, 85...Sensor unit, 86...First communication unit, 87...Second communication unit, 88...Operation unit, 88b...Operation buttons, 89...System bus, 100...Display system, 110...Display system, Pa1...First image pattern, Pa2...Second image pattern.
Claims
1. A display system including a projection device for projecting images, and an information processing device having an imaging unit and an inertial sensor, The projection device projects a first image including a first image pattern onto the projection surface. After the information processing device acquires the first image data obtained by the imaging unit, A second image, which has undergone geometric correction to the first image based on the data of the first image, the first captured image data, and the detected value of the inertial sensor, is projected onto the projection device. It accepts angle adjustment operations from the user. Based on the angle adjustment operation, a third image is projected onto the projection device by rotating the second image about the normal vector of the projection surface as the axis. Display system.
2. The angle adjustment operation includes a first angle adjustment operation and a second angle adjustment operation. The direction in which the second image is rotated about the normal vector by the second angle adjustment operation is the same as the direction in which the second image is rotated about the normal vector by the first angle adjustment operation. The angle by which the second image is rotated about the normal vector by the second angle adjustment operation is greater than the angle by which the second image is rotated about the normal vector by the first angle adjustment operation. The display system according to claim 1.
3. After projecting the third image, the projection device projects a fourth image, including the second image pattern, onto the projection surface. The information processing device acquires the second image data obtained by capturing the fourth image, Based on the data of the fourth image and the second image capture data, a fifth image, which has undergone surface contour correction on the fourth image, is projected onto the projection device. The display system according to claim 1.
4. The information processing device transmits the data of the first image to the projection device. The display system according to claim 1.
5. The information processing device displays an adjustment screen that accepts the angle adjustment operation, The adjustment screen includes an angle adjustment operation icon and an image captured by the imaging unit. The display system according to claim 1.
6. An information processing device for a display system, comprising a projection device for projecting images and an information processing device having a communication unit capable of communicating with the projection device, The information processing device further comprises an imaging unit and an inertial sensor, After acquiring first image data captured by the imaging unit, the projection device projects a first image pattern onto the projection surface, A second image, which has undergone geometric correction to the first image based on the data of the first image, the first captured image data, and the detected value of the inertial sensor, is projected onto the projection device. It accepts angle adjustment operations from the user. Based on the angle adjustment operation, a third image is projected onto the projection device by rotating the second image about the normal vector of the projection surface as the axis. Information processing device.
7. The angle adjustment operation includes a first angle adjustment operation and a second angle adjustment operation. The direction in which the second image is rotated about the normal vector by the second angle adjustment operation is the same as the direction in which the second image is rotated about the normal vector by the first angle adjustment operation. The angle by which the second image is rotated about the normal vector by the second angle adjustment operation is greater than the angle by which the second image is rotated about the normal vector by the first angle adjustment operation. The information processing apparatus according to claim 6.
8. After projecting the third image onto the projection device, project a fourth image including the second image pattern onto the projection surface onto the projection device. The second image data obtained by capturing the fourth image described above is acquired, Based on the data of the fourth image and the second image capture data, a fifth image, which has undergone surface contour correction on the fourth image, is projected onto the projection device. The information processing apparatus according to claim 6.
9. The data of the first image is transmitted to the projection device. The information processing apparatus according to claim 6.
10. The adjustment screen that accepts the aforementioned angle adjustment operation is displayed, The adjustment screen includes an angle adjustment operation icon and an image captured by the imaging unit. The information processing apparatus according to claim 6.