Display system and information processing apparatus

Abstract

Provided are a display system and an information processing apparatus, the display system being capable of obtaining a projected image with a small roll angle. The display system includes a projection apparatus that projects an image and an information processing apparatus that has an imaging section and an inertial sensor, wherein the projection apparatus projects a first image including a first image pattern toward a projection surface, the information processing apparatus, after acquiring first captured image data obtained by the imaging section capturing the first image, causes the projection apparatus to project a second image, receives a user's angle adjustment operation, and causes the projection apparatus to project a third image, wherein the second image is obtained by performing geometric correction on the first image based on data of the first image, the first captured image data, and a detection value of the inertial sensor, and the third image is obtained by rotating the second image about a normal vector of the projection surface based on the angle adjustment operation.

Description

Technical Field

[0001] The present invention relates to a display system comprising a projection device and an information processing device, and the information processing device of the display system. Background Technology

[0002] For example, Patent Document 1 discloses a display system having an image display device and a portable terminal. According to this document, the image display device has a projection unit that projects multiple measurement patterns, and the portable terminal has a camera that captures the multiple measurement patterns projected by the projection unit; and a control unit that generates information related to image distortion correction based on the multiple measurement patterns captured by the camera.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: International Publication No. 2017 / 179111 Summary of the Invention

[0006] However, in the display system of Patent Document 1, there is a problem that the tilt angle of the corrected projected image becomes larger. Specifically, compared to the case where distortion correction of the projected image is performed based on images captured by a camera on an image display device, the case where distortion correction of the projected image is performed based on images captured by a camera on a portable terminal results in a larger tilt angle of the corrected projected image because the relative positional relationship between the image display device and the camera is not predetermined. The tilt angle refers to the degree of inclination of the bottom edge of the corrected projected image relative to a line segment parallel to the horizontal plane.

[0007] That is, a display system that can produce a projected image with a small tilt angle is required.

[0008] One aspect of the display system of this application includes: a projection device that projects an image; and an information processing device having a camera unit and an inertial sensor, wherein the projection device projects a first image containing a first image pattern onto a projection surface, and after obtaining first captured image data obtained by the camera unit capturing the first image, the information processing device causes the projection device to project a second image, and upon accepting an angle adjustment operation by a user, causes the projection device to project a third image, wherein the second image is obtained by geometrically correcting the first image based on the data of the first image, the first captured image data, and the detection value of the inertial sensor, and the third image is obtained by rotating the second image about the normal vector of the projection surface as an axis based on the angle adjustment operation.

[0009] One aspect of the information processing apparatus of this application is an information processing apparatus for a display system, the display system comprising: a projection device that projects an image; and an information processing apparatus having a communication unit capable of communicating with the projection device, wherein the information processing apparatus further comprises a camera unit and an inertial sensor, and after acquiring first captured image data obtained by the camera unit capturing a first image, causes the projection device to project a second image, and accepts an angle adjustment operation by the user, causes the projection device to project a third image, wherein the first image comprises a first image pattern and is projected by the projection device onto a projection surface, the second image is obtained by geometrically correcting the first image based on the data of the first image, the first captured image data, and the detection value of the inertial sensor, and the third image is obtained by rotating the second image about the normal vector of the projection surface based on the angle adjustment operation. Attached Figure Description

[0010] Figure 1 This is a schematic structural diagram of the display system according to Implementation Method 1.

[0011] Figure 2 It is a block diagram that represents the general structure of the display system.

[0012] Figure 3 This is a flowchart representing the process of the first adjustment method.

[0013] Figure 4 This is a diagram representing an example of the first image data.

[0014] Figure 5 It is a diagram representing a shape of the projected image during the adjustment process.

[0015] Figure 6 This is an example of adjusting the screen.

[0016] Figure 7 It is a diagram representing a shape of the adjusted projected image.

[0017] Figure 8 This is a flowchart illustrating the process of the second adjustment method in Implementation Method 2.

[0018] Figure 9 This is a schematic diagram of the display system.

[0019] Figure 10 This is an example of a user interface.

[0020] Figure 11 This is a diagram representing an example of fourth image data.

[0021] Figure 12 This is an example of an adjustment screen for a modified version.

[0022] Figure 13 This is an example of an adjustment screen for a modified version.

[0023] Label Explanation

[0024] 3: Dot; 11: First image; 12: Second image; 13: Third image; 40: Projection device; 41: Control unit; 42: Storage unit; 43: Communication unit; 44: Operation unit; 45: IF unit; 46: Image processing unit; 47: Projection unit; 48: Projection device; 49: System bus; 51: First image data; 52: Fourth image data; 60: Center line; 70: Adjust screen; 71a: Operation icon; 71b: Operation icon; 71c: Operation icon; 71d: Operation icon; 72: Operation icon for position adjustment; 72a: Arrow icon; 72b: Arrow icon; 72c: Arrow icon; 72d: Arrow icon; 73a: Operation icon for zooming in; 73b: Operation icon for zooming out. Operation icons used; 74: Operation screen; 74a: Operation icon for performing concavity / convexity correction; 74b: Operation icon for not performing concavity / convexity correction; 75: Adjustment screen; 76: Adjustment screen; 80: Information processing device; 81: Control unit; 81a: Calculation unit; 82: Storage unit; 82a: First adjustment program; 82b: Second adjustment program; 82c: Test image data; 82d: Operation screen data; 83: Display unit; 84: Camera unit; 85: Sensor unit; 86: First communication unit; 87: Second communication unit; 88: Operation unit; 88b: Operation button; 89: System bus; 100: Display system; 110: Display system; Pa1: First image pattern; Pa2: Second image pattern. Detailed Implementation

[0025] Implementation Method 1 Overview of the display system

[0026] Figure 1 This is a schematic structural diagram of the display system according to Implementation Method 1.

[0027] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. The following embodiments are for illustrative purposes only and are examples of the present invention; however, the present invention is not limited to these embodiments, and various modifications that can be implemented without altering the spirit of the invention are also included. Furthermore, in the following figures, dimensions and scales may sometimes be used for ease of understanding and explanation, differing from actual dimensions.

[0028] like Figure 1 As shown, the display system 100 consists of a projection device 40, an information processing device 80, etc.

[0029] The projection device 40 is, for example, a fixed projector that projects an image defined by image data. Figure 1 In this context, the projection device 40 projects a first image 11 onto a rectangular screen SC mounted on the wall of the conference room. The wall is substantially vertical, and the screen SC, mounted along the wall, is also substantially vertical. Additionally, in the context of... Figure 1 The figures within illustrate the X-axis, Y-axis, and Z-axis, which are three mutually orthogonal axes. Figure 1 In this design, the vertical orientation of the screen SC is defined as the Z-axis, and the extension direction of the Z-axis is defined as the Z-direction. The X-axis extends along the long side of the screen SC, and its extension direction is defined as the X-direction. Furthermore, the direction orthogonal to both the Z-direction and the X-direction is defined as the Y-direction. In a preferred embodiment, the X-direction is horizontal, and the Z-direction is vertical. The screen SC serves as the projection surface, with its bottom edge horizontal. Alternatively, the projection device 40 can be a ceiling-mounted type. On the X-axis, the front side of the arrow is referred to as the positive X-direction, and the opposite side is referred to as the negative X-direction. The same applies to the Y-axis and Z-axis. Additionally, the positive Z-direction is referred to as upward, and the negative Z-direction as downward.

[0030] The information processing device 80 is a smartphone, including a display unit 83, a camera unit 84, etc. The display unit 83 is rectangular and configured to include a liquid crystal display (LCD), an organic EL (Electroluminescence) display, or other display devices. The display unit 83 has a touch panel and also functions as an operation unit 88. An operation button 88b is located next to the display unit 83. The operation button 88b is part of the operation unit 88 and serves as the start button for activating the information processing device 80. The operation button 88b may also have fingerprint authentication functionality. The information processing device 80 is held horizontally by the user with the long side of the display unit 83 along the X-direction. The information processing device 80 is not limited to a smartphone; any portable information processing device with similar functions is acceptable, such as a laptop computer, a tablet computer, etc.

[0031] In a preferred embodiment, the projection device 40 and the information processing device 80 can communicate bidirectionally via a wireless LAN (Local Area Network).

[0032] Figure 1 This illustrates one method of performing an adjustment procedure to correct distortion in the projected image. The adjustment procedure is a process that corrects distortion in the projected image to make the projected image similar to the screen SC; details are explained later.

[0033] like Figure 1As shown, the first image 11 is trapezoidal in shape, wider at the top and narrower at the bottom, and is tilted downwards in the X-direction on the screen SC. A first image pattern Pa1, with a similar shape to the first image 11, is also displayed in the first image 11.

[0034] exist Figure 1 In this process, the camera unit 84 of the information processing device 80 captures the first image 11 according to the adjustment procedure and displays the captured image on the display unit 83. The captured image includes the entire first image pattern Pa1. Details of the adjustment procedure will be explained later.

[0035] Structure of projection device

[0036] Figure 2 It is a block diagram that represents the general structure of the display system.

[0037] like Figure 2 As shown, the projection device 40 comprises a control unit 41, a storage unit 42, a communication unit 43, an operation unit 44, an information processing unit 45, an image processing unit 46, and a projection unit 47. The control unit 41 is connected to the aforementioned units via a system bus 49.

[0038] The control unit 41 is configured to have one or more processors and to operate according to the control program stored in the storage unit 42, thereby uniformly controlling the operation of the projection device 40.

[0039] The storage unit 42 is configured to include RAM (Random Access Memory) and ROM (Read Only Memory). The RAM is used for temporary storage of various data, while the ROM stores control programs and control data used to control the operation of the projection device 40. The control data includes the coordinate data of the four corners of the projected image.

[0040] The communication unit 43 is configured as a wireless communication device for wireless communication via wireless LAN, Bluetooth (registered trademark), etc. Based on the control of the control unit 41, the communication unit 43 transmits and receives information with the information processing device 80 via a wireless connection.

[0041] The operation unit 44 is equipped with multiple operation keys for the user to give various instructions to the projection device 40. These operation keys include a "power button" for switching the power on and off, a "menu button" for displaying menus for various settings, and "direction keys" for selecting menu items. Alternatively, a remote control (not shown) capable of remote operation can be used as the input operation unit. In this case, the remote control sends an infrared operation signal corresponding to the operator's operation, and a remote control signal receiving unit (not shown) receives the operation signal and sends it to the control unit 41.

[0042] The IF section 45 is the interface section for external devices and has multiple connection terminals, including an HDMI (High-Definition Multimedia Interface) terminal. Other possible connection terminals include, for example, a USB (Universal Serial Bus) terminal and a VGA (Video Graphics Array) terminal.

[0043] Based on the control of the control unit 41, the image processing unit 46 performs necessary image processing on the input image information and outputs the processed image information to the projection unit 47. The image processing includes image processing to correct distortion of the projected image based on correction parameters received from the information processing device 80.

[0044] The projection unit 47 comprises a light source, a light modulation device, and a projection optical system (none shown). The projection unit 47 uses the light modulation device to modulate light emitted from the light source to form image light defined by image information, and projects this image light through the projection optical system. The light modulation device can be, for example, a liquid crystal panel or a digital micromirror device. The light modulation device can be a single liquid crystal panel or a digital micromirror device, or it can include multiple liquid crystal panels or digital micromirror devices. The projection optical system includes one or more optical elements that adjust the magnification and imaging position of the light output from the light modulation device.

[0045] In addition, the control unit 41 and the image processing unit 46 may be composed of one or more processors, or they may be composed of dedicated processing devices such as ASIC (Application Specific Integrated Circuit) and FPGA (Field Programmable Gate Array).

[0046] Structure of information processing device

[0047] like Figure 2 As shown, the information processing device 80 comprises a control unit 81, a storage unit 82, a display unit 83, a camera unit 84, a sensor unit 85, a first communication unit 86, a second communication unit 87, and an operation unit 88. The control unit 81 is connected to the above-mentioned units via a system bus 89.

[0048] The control unit 81 is configured to include one or more processors. The control unit 81 operates according to the OS (Operating System) and control program stored in the storage unit 82, thereby uniformly controlling the operation of the information processing device 80.

[0049] The storage unit 82 is composed of RAM and ROM, among other memories. The storage unit 82 stores the operating system, control programs, and various data. The control programs include a first adjustment program 82a and a second adjustment program 82b. The various data include test image data 82c and operation screen data 82d.

[0050] The display unit 83 is configured to include a display device such as a liquid crystal display or an organic EL display, and to display images based on image information.

[0051] The camera unit 84 is a camera equipped with imaging elements such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The camera unit 84 captures images under the control of the control unit 81 and stores the captured image data in the storage unit 82.

[0052] The operation unit 88 is configured to include a touch panel integrated with the display unit 83 and operation buttons 88b, and accepts user input operations. When the user operates the operation unit 88, the operation unit 88 outputs an operation signal corresponding to the operation content to the control unit 81 via the system bus 89.

[0053] The sensor unit 85 is an inertial sensor equipped with a three-axis accelerometer and a three-axis gyroscope (angular velocity) sensor, which detects the attitude of the information processing device 80. The sensor unit 85 outputs a detection signal to the control unit 81 via the system bus 89 according to a pre-specified sampling frequency.

[0054] The first communication unit 86 is a communication unit that performs data transmission and reception between an antenna and the nearest base station (not shown) based on a fourth-generation mobile communication system according to the IMT-Advance standard and / or a fifth-generation mobile communication system according to the IMT-2020 standard.

[0055] The second communication unit 87 is configured as a wireless communication device for wireless communication via wireless LAN, Bluetooth (registered trademark), etc. Based on the control of the control unit 81, the second communication unit 87 transmits and receives information with the projection device 40 via a wireless connection.

[0056] In other words, the display system 100 is configured to include: a projection device 40 that projects images; and an information processing device 80 that has an image camera 84 and a sensor unit 85 composed of an inertial sensor. Furthermore, the information processing device 80 has a second communication unit 87 capable of communicating with the projection device 40.

[0057] First adjustment method

[0058] Figure 3 This is a flowchart representing the process of the first adjustment method. Figure 4 This is a diagram representing an example of the first image data. Figure 5 This is a diagram illustrating one way of representing the projected image during the adjustment process, and... Figure 1 correspond. Figure 6 This is an example of adjusting the screen. Figure 7 This is a diagram representing one way of showing the adjusted projected image, and... Figure 5 correspond.

[0059] Here, with Figure 3 Using this as the main body, and appropriately combined with other accompanying drawings, the first adjustment method for correcting distortion in the projected image will be explained.

[0060] The following steps are performed by executing the first adjustment procedure 82a through the information processing device 80. Specifically, the first adjustment procedure 82a is executed by the user operating the distortion adjustment procedure icon displayed on the display unit 83 of the information processing device 80. When the first adjustment procedure 82a is executed, the information processing device 80 and the projection device 40 are wirelessly connected, and the projection device 40 operates according to the instructions from the information processing device 80. In the following description, the execution entities for each step are the control unit 81 of the information processing device 80 and the control unit 41 of the projection device 40.

[0061] In step S10, the information processing device 80 begins operation based on the adjustment initiated by the user. Specifically, when the adjustment start icon (not shown) on the display unit 83 is clicked, step S11 is initiated.

[0062] In step S11, the information processing device 80 reads the first image data from the test image data 82c in the storage unit 82 and sends it to the projection device 40. For example... Figure 4As shown, the first image data 51 is image data formed by arranging a first image pattern Pa1, which is composed of a checkerboard pattern similar in shape to the outline, within a rectangular outline shown by dashed lines. The first image pattern Pa1 is a checkerboard pattern in which black and white quadrilaterals are arranged alternately along the X and Z directions. Each vertex of the quadrilaterals is called a detection point or a checkerboard corner point.

[0063] In step S30, the projection device 40 determines whether it has received data from the information processing device 80. If data has been received, the process proceeds to step S31. If no data has been received, the process continues to wait for data reception. Furthermore, the determination of whether data has been received from the information processing device 80 is also performed in subsequent processes, but... Figure 3 The record of subsequent judgment and processing is omitted.

[0064] In step S31, the projection device 40 projects the image defined by the received first image data 51. Figure 1 The image shown is a first image 11, which is an example of a projected image. The first image 11 is trapezoidal in shape, wider at the top and narrower at the bottom, and is tilted downwards in the X-direction on the screen SC. The first image 11 displays a first image pattern Pa1 as a chessboard pattern. In other words, the projection device 40 projects the first image 11, which includes the first image pattern Pa1, onto the screen SC, which serves as the projection surface.

[0065] In step S12, the information processing device 80 performs geometric correction processing. Specifically, the following processes (1) to (4) are performed.

[0066] In (1), the information processing device 80 causes the camera unit 84 to capture the first image 11. More specifically, as... Figure 1 As shown, when the user presses the start shooting button 88b and moves left and right relative to the screen SC in the X direction while holding the handheld information processing device 80, multiple images are captured by the camera unit 84 at multiple different shooting positions. At this time, the display unit 83 displays the guidance text such as "Please move left and right to take an automatic picture. Please include the entire picture in the camera."

[0067] In (2), the information processing device 80 detects a chessboard pattern from each of the multiple captured images by analyzing them separately. Specifically, it detects detection points of the chessboard pattern for each captured image.

[0068] In (3), the information processing device 80 determines whether the entire chessboard pattern is detected from two or more captured images in (2). If the entire chessboard pattern is detected from two or more captured images, it proceeds to (4) processing. If it is not detected, it returns to (1) processing.

[0069] In step (4), the information processing device 80 calculates the correction parameters. At this time, the control unit 81 functions as the calculation unit 81a. First, the calculation unit 81a calculates the projection transformation matrix based on the detection points of the first image data 51, which serves as a reference, and the detection points of the captured image data. The projection transformation matrix is ​​a matrix that transforms the coordinate system in the captured image to the coordinate system in the projection device 40. In other words, the projection transformation matrix is ​​a matrix that transforms the camera image coordinate system to the projector coordinate system.

[0070] Next, the computation unit 81a calculates the normal vector of the screen SC based on the projection transformation matrix, and calculates the vertical vector of the screen SC based on the detection data of the sensor unit 85. Furthermore, based on the normal vector and vertical vector of the screen SC, the horizontal vector of the screen SC is calculated, and the two-dimensional coordinate system of the screen SC is estimated.

[0071] Next, the calculation unit 81a calculates the transformation matrix between the projector coordinate system and the screen coordinate system. Specifically, to determine the coordinates of the four corners of the corrected projected image in a way that makes the projected image a rectangle on the screen SC, the transformation matrix is ​​used to transform these coordinates into coordinates in the projector coordinate system. Furthermore, the calculation unit 81a calculates the correction parameters for distortion correction based on the coordinates of the four corners in the projector coordinate system and the coordinates of the four corners of the projected image.

[0072] In other words, the information processing device 80 acquires the first captured image data obtained by the camera unit 84 capturing the first image 11.

[0073] In step S13, the information processing device 80 sends the calculated correction parameters to the projection device 40. In other words, the information processing device 80 causes the projection device 40 to project a second image 12, which is obtained by geometrically correcting the first image 11 based on the first image data 51, the first captured image data, and the detection values ​​of the sensor unit 85.

[0074] In step S32, the projection device 40 generates second image data using the correction parameters received from the image processing unit 46, and projects an image defined by the second image data. Figure 5 The image shown is a second image 12, which is an example of a projected second image. The second image 12 is a rectangular image, but it has a tilt angle relative to the screen SC. Specifically, it has a tilt angle θ relative to the bottom edge of the screen SC. Therefore, the second image 12 becomes an image tilted to the upper right relative to the screen SC in the X direction. Furthermore, the size of the second image 12 relative to the screen SC is smaller. Additionally, in... Figure 5 In this case, the normal vector of the screen SC is taken as the center line 60. For example... Figure 2As shown, center line 60 is a line segment extending from the center of screen SC along the Y direction.

[0075] In step S14, an adjustment screen for the projected image is displayed on the display unit 83 of the information processing device 80. Specifically, the control unit 81 reads adjustment screen data from the operation screen data 82d in the storage unit 82 and causes the display unit 83 to display the adjustment screen. For example... Figure 6 As shown, the adjustment screen 70 displays operation icons 71a and 71b for angle adjustment, operation icon 72 for position adjustment, and operation icons 73a and 73b for zooming in and out.

[0076] When the angle adjustment icon 71a is clicked once, the projected image is aligned with the centerline 60 ( Figure 5 The projected image rotates 3° counterclockwise around the center line 60. Each click of operation icon 71a rotates the projected image 3° counterclockwise. Similarly, each click of operation icon 71b rotates the projected image 3° clockwise around the center line 60. The angle is not limited to 3°.

[0077] The operation icons 72 used for position adjustment are arranged in a cross shape with triangular arrow icons 72a, 72b, 72c, and 72d.

[0078] Clicking arrow icon 72a once moves the projected image a specified distance in the positive X direction. Similarly, clicking arrow icon 72c once moves the projected image a specified distance in the negative X direction. Clicking arrow icon 72b once moves the projected image a specified distance in the negative Z direction. Similarly, clicking arrow icon 72d once moves the projected image a specified distance in the positive Z direction. By manipulating arrow icons 72a, 72b, 72c, and 72d, the projected image can be moved up, down, left, and right.

[0079] When the zoom-in icon 73a is clicked once, the projected image size increases by one size. Similarly, when the zoom-out icon 73b is clicked once, the projected image size decreases by one size. By operating the icons 73a and 73b, the size of the projected image can be changed.

[0080] In step S15, the information processing device 80 accepts an operation on the adjustment screen 70. For example, when the user... Figure 5 If the second image 12 operates on the angle adjustment operation icon 71b and the zoom operation icon 73a, then the correction parameters corresponding to the operation are generated.

[0081] In step S16, the information processing device 80 sends the correction parameters corresponding to the operation content to the projection device 40. In other words, the information processing device 80 accepts the user's angle adjustment operation and causes the projection device 40 to project a third image 13, which is obtained by rotating the second image 12 about the center line 60, which is the normal vector of the projection surface, based on the angle adjustment operation.

[0082] In step S33, the projection device 40 generates third image data using the correction parameters received from the image processing unit 46, and projects an image defined by the third image data. Figure 7 The image shown is a third image 13, which is an example of a projected third image. The projection device 40 stores parameters containing the coordinates of the four corners of the third image data in the storage unit 42.

[0083] The third image 13 is a rectangular shape with its bottom edge along the bottom edge of the screen SC, making it the size suitable for the screen SC.

[0084] In step S17, an operation screen (not shown) is displayed on the display unit 83 of the information processing device 80 to confirm whether to end the adjustment based on the adjustment screen 70. If the end icon is operated, the process ends. If the continue icon is operated, the process returns to step S14 and continues the adjustment based on the adjustment screen 70. Furthermore, if a certain amount of time has elapsed since the operation screen was displayed, the correction process can also be considered complete and the process ends.

[0085] As described above, the display system 100 and information processing device 80 according to this embodiment can achieve the following effects.

[0086] The display system 100 includes: a projection device 40 that projects images; and an information processing device 80 having a camera unit 84 and a sensor unit 85 composed of an inertial sensor. The projection device 40 projects a first image 11 containing a first image pattern Pa1 onto a screen SC, which serves as the projection surface. After acquiring first captured image data obtained by the camera unit 84 capturing the first image 11, the information processing device 80 causes the projection device 40 to project a second image 12. Upon accepting the user's angle adjustment operation, the projection device 40 projects a third image 13. The second image 12 is obtained by geometrically correcting the first image 11 based on the first image data 51, the first captured image data, and the detection value of the sensor unit 85. The third image 13 is obtained by rotating the second image 12 about the center line 60, which serves as the normal vector of the projection surface, based on the angle adjustment operation.

[0087] Therefore, the first image 11 is geometrically corrected based on the first image data 51, the first captured image data, and the detection value of the sensor unit 85, and the second image 12 obtained by geometric correction is rotated about the center line 60, which is the normal vector of the screen SC, which is the projection surface, thereby reducing the tilt angle of the corrected third image 13 relative to the screen SC.

[0088] Therefore, a display system 100 is provided that can produce a projected image with a small tilt angle.

[0089] Furthermore, the information processing device 80 sends the data of the first image to the projection device 40. Specifically, the first image data is read from the test image data 82c in the storage unit 82 and sent to the projection device 40. That is, the first image data containing the first image pattern Pa1 is stored in the information processing device 80.

[0090] Therefore, when updating the first adjustment program 82a containing the first image data, it is not necessary to update both the program stored in the projection device 40 and the program stored in the information processing device 80; only the first adjustment program 82a needs to be updated. This reduces the user's workload. The same applies to the second adjustment program 82b.

[0091] The information processing device 80 is an information processing device for a display system. The display system includes a projection device 40 for projecting images and an information processing device 80 having a second communication unit 87 capable of communicating with the projection device 40. The information processing device 80 also includes a camera unit 84 and a sensor unit 85 composed of an inertial sensor. After acquiring first captured image data obtained by the camera unit 84 capturing a first image 11, the projection device 40 projects a second image 12, and the projection device 40 projects a third image 13 in response to the user's angle adjustment operation. The first image 11 is projected by the projection device 40 onto the screen SC and includes a first image pattern Pa1. The second image 12 is obtained by geometrically correcting the first image 11 based on the first image data 51, the first captured image data, and the detection value of the sensor unit 85. The third image 13 is obtained by rotating the second image 12 about the center line 60, which is the normal vector of the projection surface, based on the angle adjustment operation.

[0092] Thus, an information processing device 80 is provided that is suitable for a display system 100 that can produce a projected image with a small tilt angle.

[0093] Implementation Method 2 Different ways of adjusting

[0094] Figure 8This is a flowchart illustrating the process of the second adjustment method in Implementation Method 2, and... Figure 3 correspond. Figure 9 It is a schematic diagram of the display system, and Figure 1 correspond.

[0095] In the above embodiments, it is explained that the shape of the projected image is corrected by performing geometric correction and tilt angle correction to correct distortion, but it is not limited to this, and the unevenness of the projected image can also be further corrected.

[0096] Hereinafter, the same symbols will be used to mark the same parts as in the above embodiments, and repeated descriptions will be omitted.

[0097] like Figure 9 As shown, the display system 110 of this embodiment comprises a projection device 48, an information processing device 80, etc. The projection device 48 of this embodiment is a short-focus projector, positioned approximately directly below the screen SC, which is mounted along the wall. The structure of the projection device 48 is similar to... Figure 2 The projection device 40 is basically the same, but the projection section 47 includes an optical system corresponding to short-focus projection, which includes a wide-angle lens. Additionally, the information processing device 80 and... Figure 2 The information processing device 80 is the same, but in this embodiment, the second adjustment program 82b of the storage unit 82 is executed. Apart from these points, everything else is the same as described in Embodiment 1.

[0098] like Figure 9 As shown, in the short-focus projection device 48, and Figure 1 Compared to a fixed projection device 40, the projection angle (elevation angle) of the image is larger. When the projection angle is larger, minor unevenness (undulation) of the screen, which is not usually an obstacle for a projector, will affect the projected image, causing distortion. According to the second adjustment method of this embodiment, in addition to shape correction based on geometric correction and tilt angle correction, unevenness correction can also be performed. Alternatively, the projection device 48 can also be positioned above the screen SC. In this case, a mounting device extending from the wall towards the negative Y direction is provided above the screen SC, and the projection device 48 is fixed to this mounting device, projecting the image from an obliquely above the screen SC.

[0099] The following steps are performed by executing the second adjustment procedure 82b through the information processing device 80. Specifically, the second adjustment procedure 82b is executed by the user operating the distortion adjustment procedure icon (not shown) displayed on the display unit 83 of the information processing device 80. The second adjustment procedure 82b is a process of concavity / convexity correction following the first adjustment procedure 82a; therefore, the following explanation will focus on step S17 in the information processing device 80, and on step S33 in the projection device 48. The preceding content is consistent with... Figure 3The explanation is the same as in the previous text.

[0100] Second adjustment method

[0101] Figure 8 Step S17 and Figure 3 The same applies to step S17. Specifically, in step S17, the display unit 83 of the information processing device 80 displays a message indicating whether the adjustment screen 70 has ended. Figure 6 The operation screen for confirming the adjustment is shown (not illustrated). If the continue icon is selected, return to step S14 and continue the adjustment based on the adjustment screen 70.

[0102] Here, the explanation assumes the operation of the end icon has been performed. Additionally, at this time, the projection device 48 projects a third image 13 based on the third image data. Figure 7 ).

[0103] Figure 10 This is an example of a user interface.

[0104] When the operation is completed, the display unit 83 of the information processing device 80 displays... Figure 10 The operation screen 74 is shown. Specifically, the control unit 81 reads the operation screen data 82d from the storage unit 82 and causes the display unit 83 to display the operation screen. The operation screen 74 displays guidance text such as "Shape correction completed," and also displays an operation icon 74a to continue bump correction and an operation icon 74b to stop bump correction.

[0105] Here, we will explain the case where operation icon 74a for performing bump correction was executed in step S19. Furthermore, if operation icon 74b for not performing bump correction was executed, the adjustment ends.

[0106] Figure 11 This is a diagram representing an example of fourth image data.

[0107] In step S20, the information processing device 80 reads the fourth image data from the test image data 82c in the storage unit 82 and sends it to the projection device 48.

[0108] like Figure 11 As shown, the fourth image data 52 is image data in which a second image pattern Pa2, composed of a dot matrix pattern similar in shape to the contour line shown by the dashed line, is arranged within the rectangular contour line. The second image pattern Pa2 is a dot matrix pattern in which multiple points 3 are arranged at equal intervals in the X and Z directions. The points 3 are circular, and their centers are the detection points. The proportion of the second image pattern Pa2 in the rectangular contour line is larger than that of the first image pattern Pa1. Figure 4The blank area around the second image pattern Pa2 in the fourth image data 52 is larger than that in the first image data 51. Figure 4 Narrow. Therefore, bump correction can also be implemented at the periphery of the screen SC.

[0109] In step S34, the projection device 48 projects the image defined by the received fourth image data 52.

[0110] In step S21, the information processing device 80 performs a bump correction process. Specifically, the following processes (11) to (14) are performed.

[0111] In (11), the information processing device 80 causes the camera unit 84 to capture a fourth image (not shown). At this time, the display unit 83 displays guidance text such as "Please take a picture of the entire pattern from the front of the projected image into the camera."

[0112] In (12), the information processing device 80 analyzes the captured image to detect a dot pattern from the captured image. Specifically, it detects the detection points of the dot pattern. The captured image is equivalent to second captured image data.

[0113] In (13), the information processing device 80 determines whether the entire dot matrix pattern is detected from the image captured in (12). If it is detected, it proceeds to (14) for processing. If it is not detected, it returns to (11) for processing.

[0114] In (14), the information processing device 80 calculates the correction parameters. At this time, the control unit 81 functions as the calculation unit 81a. The calculation unit 81a calculates the correction value for each point 3 and generates the correction parameters based on the position of each point 3 in the second image pattern Pa2, which serves as a reference, and the position of each point 3 in the captured image.

[0115] In addition, multiple images can be captured in (11), and the analysis of each captured image and the detection and determination of the overall dot matrix pattern can be performed in (12) and (13).

[0116] In step S22, the information processing device 80 sends the calculated correction parameters to the projection device 48.

[0117] In step S35, the projection device 48 generates fifth image data using the correction parameters received from the image processing unit 46, and projects a fifth image (not shown) based on the fifth image data. The fifth image becomes a projected image that has undergone both shape correction and bump correction. The projection device 48 stores the bump correction parameters, including the fifth image data, in the storage unit 42.

[0118] In other words, after the projection device 48 projects the third image, it projects the fourth image containing the second image pattern Pa2 onto the screen SC. The information processing device 80 obtains the second captured image data obtained by capturing the fourth image, and causes the projection device 48 to project the fifth image. The fifth image is obtained by performing concavity and convexity correction on the fourth image based on the data of the fourth image and the second captured image data.

[0119] In step S23, an operation screen (not shown) is displayed on the display unit 83 of the information processing device 80 to confirm whether to end the bump correction. If the continue icon is selected, the process returns to step S21 to continue the bump correction. If the end icon is selected, the bump correction ends.

[0120] As described above, in addition to the effects described in the above embodiments, the display system 110 and information processing device 80 according to this embodiment can also achieve the following effects.

[0121] After the projection device 48 of the display system 110 projects the third image, it projects the fourth image containing the second image pattern Pa2 onto the screen SC. The information processing device 80 obtains the second captured image data obtained by capturing the fourth image, and causes the projection device 48 to project the fifth image. The fifth image is obtained by performing concavity and convexity correction on the fourth image based on the data of the fourth image and the second captured image data.

[0122] Therefore, in addition to shape correction based on geometric correction and tilt angle correction, bump correction can also be performed. Thus, besides shape correction of the projected image, distortion of the projected image caused by minute bumps in the screen SC, which are easily generated in short-focus projection devices 48, can also be corrected. Furthermore, since bump correction is performed on the fourth image after shape correction, the adjustment steps of the projected image can be optimized.

[0123] Therefore, a display system 110 is able to provide a projected image with a small tilt angle and reduced concave-convex distortion.

[0124] After the information processing device 80 causes the projection device 48 to project the third image, it causes the projection device 48 to project the fourth image containing the second image pattern Pa2 onto the screen SC, obtains the second captured image data obtained by capturing the fourth image, and causes the projection device 48 to project the fifth image, which is obtained by performing concavity and convexity correction on the fourth image based on the data of the fourth image and the second captured image data.

[0125] Therefore, an information processing device 80 can be provided for a display system 110 that produces a projected image with a small tilt angle and reduced distortion. Furthermore, the information processing device 80 can optimize the adjustment steps of the projected image.

[0126] Variations

[0127] Figure 12 This is an example of an adjustment screen for a modified version, and... Figure 6 correspond.

[0128] In the above, regarding the adjustment of the screen to 70 ( Figure 6 The example described above has two operation icons 71a and 71b for angle adjustment, but it is not limited to this; it may also include operation icons for angle adjustment. Hereinafter, the same symbols will be used to mark the same parts as in the above embodiment, and repeated descriptions will be omitted.

[0129] exist Figure 12 The adjustment screen 75 shown in this modified example has four operation icons 71a, 71b, 71c, and 71d for angle adjustment. Apart from these, other icons are similar to... Figure 6 The explanation is the same as in the previous text.

[0130] In this modified example, when the angle adjustment operation icon 71a is clicked once, the second image 12 (which serves as the second image) is generated. Figure 5 Rotate 5° counterclockwise around the center line 60.

[0131] Operation icon 71c is positioned next to operation icon 71a and is slightly smaller than it. When operation icon 71c is clicked once, the projected image rotates 1° counterclockwise around the center line 60. Therefore, after a significant rotation using operation icon 71a, fine adjustments can be made using operation icon 71c.

[0132] The operation based on operation icon 71c is a first angle adjustment operation, and the operation based on operation icon 71a is a second angle adjustment operation. In other words, the angle adjustment operation includes a first angle adjustment operation based on operation icon 71c and a second angle adjustment operation based on operation icon 71a. The second angle adjustment operation rotates the second image about the center line 60, which is the normal vector, in the same direction as the first angle adjustment operation. However, the angle of rotation of the second image about the center line 60 is greater than the angle of rotation of the second image about the center line 60 by the first angle adjustment operation.

[0133] In this modified example, when the operation icon 71b for angle adjustment is clicked once, the projected image rotates 5° clockwise around the center line 60.

[0134] Operation icon 71d is positioned next to operation icon 71b and is slightly smaller than it. When operation icon 71d is clicked once, the projected image rotates 1° clockwise around the center line 60. Therefore, after a significant rotation via operation icon 71b, fine adjustments can be made via operation icon 71d.

[0135] Furthermore, by combining the operation icons 71a, 71b, 71c, and 71d, the tilt angle of the second image 12 can be adjusted efficiently.

[0136] return Figure 6 .

[0137] In the above description, clicking the operation icon 71a on the adjustment screen 70 rotates the second image 12 counterclockwise by 3°, but the rotation angle can vary depending on the operation method. Specifically, clicking the operation icon 71a can rotate it by 1°, and pressing and holding the operation icon 71a for, for example, more than 2 seconds can rotate it by 5°. In this case, clicking the operation icon 71a is a first angle adjustment operation, and pressing and holding is a second angle adjustment operation. Similarly, clicking the operation icon 71b can rotate it by 1°, and pressing and holding the operation icon 71b for, for example, more than 2 seconds can rotate it by 5°.

[0138] As described above, according to this modified example, in addition to the effects of the above embodiments, the following effects can also be obtained.

[0139] The angle adjustment operation includes a first angle adjustment operation based on operation icon 71c and a second angle adjustment operation based on operation icon 71a. The second angle adjustment operation rotates the second image about the center line 60, which is the normal vector, in the same direction as the first angle adjustment operation. The angle of rotation of the second image about the center line 60 is greater than the angle of rotation of the second image about the center line 60 by the first angle adjustment operation.

[0140] Therefore, the angle adjustment operation includes a first angle adjustment operation and a second angle adjustment operation with the same rotation direction but different rotation angles. Thus, users can select the degree of adjustment, such as coarse adjustment or fine adjustment, of the angle of the second image, and can perform angle adjustment operations efficiently.

[0141] Figure 13 This is an example of an adjustment screen for a modified version, and... Figure 6 correspond.

[0142] The above explains how to adjust the screen to 70 ( Figure 6The image may include operation icons 71a and 71b for angle adjustment, operation icon 72 for position adjustment, and operation icons 73a and 73b for zooming in and out. However, it is not limited to this; a camera image of the projected image may also be included. Hereinafter, the same symbols will be used to mark the same parts as in the above embodiment, and repeated descriptions will be omitted.

[0143] exist Figure 13 The adjustment screen 76 shown in this modified example displays two angle adjustment operation icons 71a and 71b, and the image of the projected screen. Apart from these points, other... Figure 6 The explanation is the same as in the previous text.

[0144] In adjustment screen 76, the captured image obtained by the camera unit 84 capturing the projected image is displayed on the negative X side of operation icons 71a and 71b. Figure 13 The second image 12, which is a second image before the tilt angle adjustment, is displayed on the adjustment screen 76. Thus, the user can adjust the tilt angle by operating the operation icons 71a and 71b while observing the captured image on the adjustment screen 76. In other words, the information processing device 80 displays the adjustment screen 76 for accepting angle adjustment operations, and the adjustment screen 76 includes the operation icons 71a and 71b as angle adjustment operation icons and the captured image taken by the camera unit 84.

[0145] As described above, according to this modified example, in addition to the effects of the above embodiments, the following effects can also be obtained.

[0146] The information processing device 80 displays an adjustment screen 76 for accepting angle adjustment operations. The adjustment screen 76 includes operation icons 71a and 71b, which serve as icons for angle adjustment operations, and a captured image taken by the camera unit 84.

[0147] Therefore, in the adjustment screen 76, by displaying the captured image taken by the camera unit 84 on the operation screen for accepting angle adjustment operations, the user can perform angle adjustment operations while confirming the captured image.

[0148] return Figure 1 .

[0149] The above describes the case where a screen SC is installed on the wall of a conference room or similar space as a projection surface, but it is not limited to this case; the wall itself can also be used as a projection surface.

[0150] Furthermore, while the second adjustment method was described as being applied to a short-focus projection device 48, it can also be performed by a conventional fixed-type projection device 40. This results in a projected image with a small tilt angle and reduced distortion.

[0151] This is a summary of the disclosure.

[0152] The following is a summary published in this note.

[0153] (Postscript 1)

[0154] A display system includes: a projection device that projects an image; and an information processing device having a camera unit and an inertial sensor, wherein the projection device projects a first image containing a first image pattern onto a projection surface, and after acquiring first captured image data obtained by the camera unit from the first image, the information processing device causes the projection device to project a second image, and upon accepting an angle adjustment operation by a user, causes the projection device to project a third image, wherein the second image is obtained by geometrically correcting the first image based on the data of the first image, the first captured image data, and the detection value of the inertial sensor, and the third image is obtained by rotating the second image about the normal vector of the projection surface based on the angle adjustment operation.

[0155] Therefore, the first image is geometrically corrected based on the first image data, the first captured image data, and the detection value of the inertial sensor, and the second image obtained by geometric correction is rotated about the center line 60, which is the normal vector of the projection surface, thereby reducing the tilt angle of the corrected projection image relative to the projection surface.

[0156] Therefore, a display system that can provide a projected image with a small tilt angle can be provided.

[0157] (Postscript 2)

[0158] According to the display system described in Appendix 1, the angle adjustment operation includes a first angle adjustment operation and a second angle adjustment operation. The direction in which the second image is rotated about the normal vector axis by the second angle adjustment operation is the same as the direction in which the second image is rotated about the normal vector axis by the first angle adjustment operation. The angle in which the second image is rotated about the normal vector axis by the second angle adjustment operation is greater than the angle in which the second image is rotated about the normal vector axis by the first angle adjustment operation.

[0159] Therefore, the angle adjustment operation includes a first angle adjustment operation and a second angle adjustment operation with the same rotation direction but different rotation angles. Thus, users can select the degree of adjustment, such as coarse adjustment or fine adjustment, of the angle of the second image, and can perform angle adjustment operations efficiently.

[0160] (Note 3)

[0161] According to the display system described in Appendix 1 or 2, after the projection device projects the third image, it projects a fourth image containing a second image pattern onto the projection surface. The information processing device obtains second captured image data obtained by capturing the fourth image, and causes the projection device to project a fifth image, which is obtained by performing concavity and convexity correction on the fourth image based on the data of the fourth image and the second captured image data.

[0162] Therefore, in addition to shape correction based on geometric correction and tilt angle correction, bump correction can also be performed. Thus, in addition to shape correction of the projected image, distortion of the projected image caused by minor bumps and depressions in the screen SC, which are easily generated in short-focus projection devices 48, can also be corrected. Furthermore, since bump correction is performed on the fourth image obtained after shape correction, the adjustment steps of the projected image can be optimized.

[0163] Therefore, a display system can be provided that produces a projected image with a small tilt angle and reduced convexity distortion.

[0164] (Postscript 4)

[0165] According to any one of Appendices 1 to 3, in the display system, the information processing device sends the first image data to the projection device.

[0166] Therefore, when updating the first adjustment program 82a containing the first image data, it is not necessary to update both the program stored in the projection device 40 and the program stored in the information processing device 80; only the first adjustment program 82a needs to be updated. This reduces the user's workload. The same applies to the second adjustment program 82b.

[0167] (Note 5)

[0168] According to any one of Appendices 1 to 4, the display system wherein the information processing device displays an adjustment screen that accepts the angle adjustment operation, the adjustment screen including an angle adjustment operation icon and a captured image taken by the camera unit.

[0169] Therefore, in the adjustment screen 76, by displaying the captured image taken by the camera unit 84 on the operation screen for accepting angle adjustment operations, the user can perform angle adjustment operations while confirming the captured image.

[0170] (Note 6)

[0171] An information processing apparatus is provided for a display system, the display system comprising: a projection device that projects an image; and an information processing apparatus having a communication unit capable of communicating with the projection device, wherein the information processing apparatus further comprises a camera unit and an inertial sensor, and after acquiring first captured image data obtained by the camera unit capturing a first image, causes the projection device to project a second image, and, upon receiving an angle adjustment operation from a user, causes the projection device to project a third image, wherein the first image comprises a first image pattern and is projected by the projection device onto a projection surface, the second image is obtained by geometrically correcting the first image based on the data of the first image, the first captured image data, and the detection value of the inertial sensor, and the third image is obtained by rotating the second image about the normal vector of the projection surface based on the angle adjustment operation.

[0172] Therefore, an information processing device can be provided that is suitable for display systems that can produce projected images with small tilt angles.

[0173] (Note 7)

[0174] According to the information processing apparatus described in Appendix 6, the angle adjustment operation includes a first angle adjustment operation and a second angle adjustment operation. The direction in which the second image is rotated about the normal vector axis by the second angle adjustment operation is the same as the direction in which the second image is rotated about the normal vector axis by the first angle adjustment operation. The angle in which the second image is rotated about the normal vector axis by the second angle adjustment operation is greater than the angle in which the second image is rotated about the normal vector axis by the first angle adjustment operation.

[0175] Therefore, the angle adjustment operation of the second image can be performed efficiently.

[0176] (Postscript 8)

[0177] According to the information processing apparatus described in Appendix 6 or 7, after the projection device projects the third image, the projection device projects a fourth image containing a second image pattern onto the projection surface, obtains second captured image data obtained by capturing the fourth image, and then the projection device projects a fifth image, which is obtained by performing concavity and convexity correction on the fourth image based on the data of the fourth image and the second captured image data.

[0178] Therefore, an information processing device can be provided that is suitable for display systems that produce projected images with small tilt angles and reduced distortion. Furthermore, the information processing device can optimize the adjustment steps of the projected image.

[0179] (Note 9)

[0180] According to any one of Appendices 6 to 8, the information processing apparatus transmits data of the first image to the projection device.

[0181] Therefore, when updating the first adjustment program 82a containing the first image data, it is not necessary to update both the program stored in the projection device 40 and the program stored in the information processing device 80; only the first adjustment program 82a needs to be updated. This reduces the user's workload. The same applies to the second adjustment program 82b.

[0182] (Postscript 10)

[0183] According to any one of Appendices 6 to 9, the information processing apparatus displays an adjustment screen that accepts the angle adjustment operation, the adjustment screen including an angle adjustment operation icon and a captured image taken by the camera unit.

[0184] This allows users to efficiently adjust the angle while simultaneously reviewing the captured image.

Claims

1. A display system comprising: a projection device for projecting an image; and an information processing device having a camera unit and an inertial sensor, wherein, The projection device projects a first image containing a first image pattern onto the projection surface. After the information processing device obtains the first captured image data obtained by the camera unit capturing the first image, The projection device projects a second image, which is obtained by geometrically correcting the first image based on data from the first image, data from the first captured image, and detection values ​​from the inertial sensor. The user's perspective can be adjusted. The projection device projects a third image, which is obtained by rotating the second image about the normal vector of the projection surface based on the angle adjustment operation.

2. The display system according to claim 1, wherein, The angle adjustment operation includes a first angle adjustment operation and a second angle adjustment operation. The second angle adjustment operation rotates the second image about the normal vector in the same direction as the first angle adjustment operation. The second angle adjustment operation causes the second image to rotate about the normal vector by a greater angle than the first angle adjustment operation caused the second image to rotate about the normal vector.

3. The display system according to claim 1, wherein, After projecting the third image, the projection device projects a fourth image containing the pattern of the second image onto the projection surface. The information processing device acquires the second captured image data obtained from capturing the fourth image. The projection device projects a fifth image, which is obtained by performing concavity and convexity correction on the fourth image based on the data of the fourth image and the data of the second captured image.

4. The display system according to claim 1, wherein, The information processing device sends the data of the first image to the projection device.

5. The display system according to claim 1, wherein, The information processing device displays an adjustment screen indicating whether the angle adjustment operation is being accepted. The adjustment screen includes an angle adjustment operation icon and a captured image from the camera unit.

6. An information processing apparatus for a display system, the display system comprising: a projection device for projecting an image; and an information processing apparatus having a communication unit capable of communicating with the projection device, wherein... The information processing device also includes a camera unit and an inertial sensor. After the information processing device obtains the first captured image data obtained by the camera unit capturing the first image, The projection device projects a second image. Adjusting the user's perspective during the process. The projection device projects a third image. The first image includes a first image pattern and is projected onto the projection surface by the projection device. The second image is obtained by geometrically correcting the first image based on the data of the first image, the data of the first captured image, and the detection value of the inertial sensor. The third image is obtained by rotating the second image about the normal vector of the projection surface based on the angle adjustment operation.

7. The information processing apparatus according to claim 6, wherein, The angle adjustment operation includes a first angle adjustment operation and a second angle adjustment operation. The second angle adjustment operation rotates the second image about the normal vector in the same direction as the first angle adjustment operation. The second angle adjustment operation causes the second image to rotate about the normal vector by a greater angle than the first angle adjustment operation caused the second image to rotate about the normal vector.

8. The information processing apparatus according to claim 6, wherein, After the information processing device projects the third image, it causes the projection device to project a fourth image containing the pattern of the second image onto the projection surface. The second captured image data obtained by capturing the fourth image is acquired. The projection device projects a fifth image, which is obtained by performing concavity and convexity correction on the fourth image based on the data of the fourth image and the data of the second captured image.

9. The information processing apparatus according to claim 6, wherein, The information processing device sends the data of the first image to the projection device.

10. The information processing apparatus according to claim 6, wherein, The information processing device displays an adjustment screen indicating whether the angle adjustment operation is being accepted. The adjustment screen includes an angle adjustment operation icon and a captured image from the camera unit.

Images