Terminal device

Abstract

To enable an input position shift resulting from a display image shift between a terminal device and a display device to be easily corrected in accordance with a function of the display device.SOLUTION: The terminal device is a smartphone, for example, and displays a first image on a display and outputs a second image partly including the first image to a display device such as a navigation device. In addition, the terminal device acquires calibration information indicating relative positional information on the first image with respect to the second image from an external server. When a user makes a touch input on the second image displayed on the display device, its input position is transmitted from the display device to the terminal device. The terminal device acquires conversion necessity / unnecessity information indicating whether or not the input position on the second image needs to be converted on the basis of the calibration information, and in a case where conversion is necessary, converts the input position on the second image into an input position on the first image on the basis of the calibration information.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to calibration of touch input between a terminal device and a display device connected to each other. [Background technology]

[0002] 2. Description of the Related Art In recent years, it has become common to connect an in-vehicle device such as a car navigation device to a terminal device such as a smartphone for use.

[0003] For example, Patent Document 1 describes a system in which a mobile terminal and an in-vehicle device are linked together to operate an application executed on the mobile terminal from the in-vehicle device.

[0004] Non-patent document 1 proposes a technology in which a dedicated module is incorporated into an application installed on a smartphone, thereby transmitting the display image of the smartphone to an in-vehicle device and displaying it on the display unit of the in-vehicle device, and operating the smartphone application by touch input to the display unit of the in-vehicle device.

[0005] As in Non-Patent Document 1, when a display image of a smartphone is transmitted to a display device and displayed, the number of pixels and the aspect ratio of the display image of the smartphone and the display device do not necessarily match. Therefore, depending on the model of the smartphone, black frame areas may be added to the top, bottom, left and right of the output image in order to ensure the aspect ratio of the output image. In other words, the image displayed on the touch panel of the display device is an image with black frame areas added to the top, bottom, left and right of the display image of the smartphone.

[0006] In this case, since the black framed area is within the display image, the display device cannot determine whether the black framed area is displayed or not. Therefore, when the black framed area is touched on the touch panel of the display device, the smartphone may determine that the area within the display image has been touched, even though the area does not have buttons on the smartphone. In addition, since the area of ​​the image displayed on the touch panel of the display device does not match the actual touch area, the touch coordinates by the user may be shifted even in the area inside the black framed area.

[0007] Therefore, Patent Document 2 describes a method of preparing calibration information for each combination of a smartphone and a display device, and converting a user's input coordinates on the display device into input coordinates on the smartphone based on the calibration information. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2010-130670 A [Patent Document 2] International Publication WO2013 / 150637 [Non-patent literature]

[0009] [Non-Patent Document 1] AppRadio Pioneer Electronics USA http: / / www.pioneerelectronics.com / PUSA / Car / AppRadio Summary of the Invention [Problem to be solved by the invention]

[0010] However, some display devices mounted on vehicles have a function that independently handles the display of the black frame area as described above. For display devices having such a function, the coordinate conversion process as in Patent Document 2 may not be necessary. In addition, when the display device independently processes the black frame area, if the smartphone further performs the process as in Patent Document 2, a problem may occur in which the coordinates recognized by the smartphone are shifted.

[0011] The above are examples of problems that the present invention aims to solve. The present invention aims to easily correct the misalignment of an input position caused by the misalignment of a display image between a terminal device and a display device according to the function of the display device. [Means for solving the problem]

[0012] The invention described in claim 1 is a terminal device comprising: a display unit that displays a first image; an image output unit that outputs a second image including the first image as a part of the first image to a display device; an input position acquisition unit that acquires from the display device an input position on the second image indicating the position of a touch input made by a user to the second image displayed on the display device; a relative position information acquisition unit that acquires from an external server relative position information indicating a relative positional relationship of the first image to the second image; a conversion necessity information acquisition unit that acquires from the display device conversion necessity information indicating whether the input position on the second image needs to be converted based on the relative position information; and a conversion unit that converts the input position on the second image to an input position on the first image based on the relative position information when the conversion necessity information acquisition unit acquires the conversion necessity information indicating that conversion is necessary.

[0013] The invention described in claim 4 is a calibration method executed by a terminal device having a display unit, comprising: a display step of displaying a first image on the display unit; an image output step of outputting a second image including the first image as a part thereof to a display device; an input position acquisition step of acquiring from the display device an input position on the second image indicating a position of a touch input made by a user to the second image displayed on the display device; a relative position information acquisition step of acquiring from an external server relative position information indicating a relative positional relationship of the first image to the second image; a conversion necessity information acquisition step of acquiring from the display device conversion necessity information indicating whether or not the input position on the second image needs to be converted based on the relative position information; and a conversion step of converting the input position on the second image to an input position on the first image based on the relative position information when the conversion necessity information acquisition step acquires the conversion necessity information indicating that conversion is necessary.

[0014] The invention described in claim 5 is a calibration program executed by a terminal device having a display unit and a computer, characterized in that it causes the computer to execute the following steps: a display step of displaying a first image on the display unit; an image output step of outputting a second image including the first image as a part of the first image to the display device; an input position acquisition step of acquiring from the display device an input position on the second image indicating the position of a touch input made by a user to the second image displayed on the display device; a relative position information acquisition step of acquiring from an external server relative position information indicating the relative positional relationship of the first image to the second image; a conversion necessity information acquisition step of acquiring from the display device conversion necessity information indicating whether the input position on the second image needs to be converted based on the relative position information; and a conversion step of converting the input position on the second image to an input position on the first image based on the relative position information when the conversion necessity information acquisition step acquires the conversion necessity information indicating that conversion is necessary. [Brief description of the drawings]

[0015] [Figure 1]1 shows a configuration of a system according to an embodiment of the present invention. [Diagram 2] FIG. 11 is a block diagram showing the functional configuration of a system when a display device without an image size adjustment function is used. [Diagram 3] Here is an example of an image with a black border added to adjust the aspect ratio. [Figure 4] An example of a method for converting an input position will be described. [Diagram 5] FIG. 11 is a block diagram showing the functional configuration of a system when a display device having an image size adjustment function is used. [Figure 6] 13 is a flowchart of a conversion process of an input position. [Figure 7] 1 shows an example of a calibration image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] In a preferred embodiment of the present invention, a terminal device includes a display unit that displays a first image, an image output unit that outputs a second image including the first image as a part thereof to a display device, an input position acquisition unit that acquires from the display device an input position on the second image indicating a position of a touch input made by a user to the second image displayed on the display device, a relative position information acquisition unit that acquires from an external server relative position information indicating a relative positional relationship of the first image to the second image, a conversion necessity information acquisition unit that acquires from the display device conversion necessity information indicating whether the input position on the second image needs to be converted based on the relative position information, and a conversion unit that converts the input position on the second image to an input position on the first image based on the relative position information when the conversion necessity information acquisition unit acquires the conversion necessity information indicating that conversion is necessary.

[0017] The terminal device is, for example, a smartphone, and displays a first image on a display unit, and outputs a second image including the first image as a part of the first image to a display unit such as a navigation device. The terminal device also acquires relative position information indicating the relative positional relationship of the first image with respect to the second image from an external server. When a user performs a touch input on the second image displayed on the display unit, the input position is transmitted from the display unit to the terminal device. The terminal device acquires conversion necessity information indicating whether or not the input position on the second image needs to be converted based on the relative position information, and converts the input position on the second image to the input position on the first image based on the relative position information if conversion is necessary. This makes it possible to correctly convert the input position on the display unit to the input position on the terminal device when the specifications of the display unit of the terminal device and the display unit of the display unit are different and coordinate conversion is required on the terminal device side. On the other hand, when the display unit has a function of coordinate conversion and coordinate conversion on the terminal device side is unnecessary, coordinate conversion is not performed, so that it is possible to prevent the input position from being shifted by performing unnecessary coordinate conversion.

[0018] In one aspect of the terminal device, the relative position information acquisition unit stores the relative position information acquired from the outside in a storage unit within the terminal device, and when the relative position information acquired in the past is stored in the storage unit, the conversion unit uses the relative position information stored in the storage unit. This eliminates the need to acquire the relative position information from an external server every time, thereby reducing the amount of communication.

[0019] Another aspect of the above-mentioned terminal device includes a calibration image output unit that supplies a calibration image to the display device, a calibration input position acquisition unit that acquires from the display device a calibration input position indicating the position of a user's touch input relative to the calibration image displayed on the display device, a relative position information generation unit that generates relative position information based on the acquired calibration input position, and a communication unit that transmits the generated relative position information to an external server.

[0020] In this aspect, when the relative position information cannot be obtained from an external server, the relative position information can be created based on the user's input to the calibration image. Also, by uploading the created relative position information to the server, it becomes possible to share the information among many users.

[0021] In another preferred embodiment of the present invention, a calibration method executed by a terminal device having a display unit includes a display step of displaying a first image on the display unit, an image output step of outputting a second image including the first image as a part thereof to a display device, an input position acquisition step of acquiring an input position on the second image, which indicates a position of a touch input made by a user on the second image displayed on the display device, from the display device, a relative position information acquisition step of acquiring from an external server relative position information indicating a relative positional relationship of the first image with respect to the second image, a conversion necessity information acquisition step of acquiring from the display device conversion necessity information indicating whether or not the input position on the second image needs to be converted based on the relative position information, and a conversion step of converting the input position on the second image to the input position on the first image based on the relative position information when the conversion necessity information acquisition step acquires the conversion necessity information indicating the necessity of conversion. This makes it possible to correctly convert the input position on the display device to the input position on the terminal device when the specifications of the display unit of the terminal device and the display unit of the display device are different and coordinate conversion is required on the terminal device side.

[0022] In another preferred embodiment of the present invention, a calibration program executed by a terminal device including a display unit and a computer causes the computer to execute the following steps: a display step of displaying a first image on the display unit; an image output step of outputting a second image including the first image as a part of the first image to a display device; an input position acquisition step of acquiring an input position on the second image, which indicates a position of a touch input made by a user on the second image displayed on the display device, from the display device; a relative position information acquisition step of acquiring from an external server relative position information indicating a relative positional relationship of the first image with respect to the second image; a conversion necessity information acquisition step of acquiring from the display device conversion necessity information indicating whether or not the input position on the second image needs to be converted based on the relative position information; and a conversion step of converting the input position on the second image to the input position on the first image based on the relative position information when the conversion necessity information acquisition step acquires the conversion necessity information indicating the necessity of conversion. This makes it possible to correctly convert the input position on the display device to the input position on the terminal device when the specifications of the display unit of the terminal device and the display unit of the display device are different and coordinate conversion is required on the terminal device side. EXAMPLES

[0023] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. [Basic configuration] 1 shows the configuration of a system including a terminal device according to an embodiment of the present invention. As shown in FIG. 1, the system includes a server 1, a smartphone 10, and a display device 20.

[0024] The smartphone 10 and the display device 20 are configured to be able to communicate with each other. The smartphone 10 corresponds to a terminal device of the present invention. The display device 20 is a device mounted on a vehicle, and examples of the device include a navigation device and an in-vehicle AV device.

[0025] In this system, a display image of the smartphone 10 is transmitted to the display device 20 and displayed on the display device 20. When a user performs an operation input on the display device 20, the display device 20 transmits a signal corresponding to the operation input to the smartphone 10. This allows the user to operate the smartphone 10 by performing an operation input on the display device 20.

[0026] The smartphone 10 displays an image on a touch panel (display unit) 14 using a liquid crystal display or the like. The smartphone 10 transmits an image D1 displayed on the touch panel 14 to the display device 20, for example, via a High Definition Multimedia Interface (HDMI).

[0027] The display device 20 includes a touch panel (display unit) 21 using a liquid crystal display or the like. The display device 20 receives an image D1 from the smartphone 10 and displays it on the touch panel 21. The display device 20 also detects an operation input performed by a user on the touch panel 21, and transmits the input position (position coordinates) D2 to the smartphone 10. The input position D2 is transmitted to the smartphone 10 by, for example, SPP (Serial Port Profile) of Bluetooth (registered trademark).

[0028] The server 1 includes a database 2. As described later, relative position information D3 (hereinafter, also referred to as calibration information D3) is stored in the database 2. The smartphone 10 can access the server 1 by using a wireless communication function or the like and download the calibration information.

[0029] [If the display device does not have an image size adjustment function] Fig. 2 is a block diagram showing the functional configuration of the system shown in Fig. 1 when a display device 20A without an image size adjustment function is used. In the following description, the display device without an image size adjustment function is referred to as "display device 20A", the display device with an image size adjustment function is referred to as "display device 20B", and the display device without being limited to either is simply referred to as "display device 20".

[0030] The image size adjustment function is, for example, a function in which a user arbitrarily cuts out a part of an image displayed on a display device, and stretches (enlarges) the cut-out image to fill the entire screen and displays it. In the example of Fig. 2, since the display device 20A does not have the image size adjustment function, the smartphone 10 performs the coordinate conversion described below to match the image size displayed on the smartphone 10 with the image size displayed on the display device 20A.

[0031] The smartphone 10 includes an application 11, an image output unit 12, a control unit 13, a touch panel 14, and communication units 15 and 16.

[0032] The application 11 is an application executed on the smartphone 10, and in this embodiment, is for operating the smartphone 10 by the display device 20 as described above. Note that the application 11 is actually realized by a computer such as a CPU executing a program prepared in advance. Note that the application 11 functions as an input position acquisition unit, a calibration information acquisition unit, a calibration image output unit, a calibration position information acquisition unit, a conversion necessity information acquisition unit, and a calibration information generation unit of the present invention.

[0033] The image output unit 12 receives the display image of the smartphone 10 from the application 11, and transmits it to the display device 20 as an image D1.

[0034] The control unit 13 controls the entire smartphone 10, and is actually realized by an OS and basic applications installed in the smartphone 10. The control unit 13 displays an image D12 generated by various applications running on the smartphone 10 on the touch panel 14 and supplies the image D12 to the application 11.

[0035] The communication unit 15 has a communication function for performing wireless communication with the server 1 and receiving calibration information D3 from the server 1. The communication unit 16 has a communication function for receiving, from the display device 20, an input position D2 corresponding to a user's operation input.

[0036] On the other hand, the display device 20A includes a touch panel 21, a communication unit 22, an input unit 23, and a control unit 25. The touch panel 21 is used when a user performs a desired operation input. The input unit 23 detects an operation input performed by the user on the touch panel 21, and sends the input position D2 to the communication unit 22. The communication unit 22 transmits the input position D2 to the smartphone 10.

[0037] The control unit 25 controls the entire display device 20A. The control unit 25 also includes a memory 26 therein. The memory 26 stores information indicating whether the display device itself has an image size adjustment function. The information on whether the image size adjustment function is present can be set, for example, as a flag (hereinafter also referred to as an "adjustment function flag"). In one example, a display device having an image size adjustment function stores "adjustment function flag = 1" in its memory 26. On the other hand, a display device not having an image size adjustment function stores "adjustment function flag = 0" in its memory 26. In the example of FIG. 2, since the display device 20A does not have an image size adjustment function, "adjustment function flag = 0" is stored in the memory 2.

[0038] When the display device 20A is connected to the smartphone 10, the control unit 25 notifies the smartphone 10 of whether or not coordinate conversion is required on the smartphone 10 side, based on the value of the adjustment function flag stored in the memory 26. In the example of Fig. 2, the display device 20A does not have an image size adjustment function, and it is necessary for the smartphone 10 side to execute coordinate conversion to absorb the difference between the image size of the smartphone 10 and the image size of the display device 20. Therefore, the control unit 25 notifies the smartphone 10 that coordinate conversion is required on the smartphone 10 side ("Coordinate conversion: required").

[0039] Next, the process performed by the application 11 will be described. The application 11 receives an image D12 generated by various applications executed on the smartphone 10 from the control unit 13, and sends it to the image output unit 12. The image output unit 12 transmits the image D12 to the display device 20A as an image D1. The display device 20A displays the received image D1 on the touch panel 21. As a result, the display image of the smartphone 10 is displayed on the touch panel 21 of the display device 20A.

[0040] The user looks at the display image of the display device 20A and performs an operation input to the touch panel 21 of the display device 20A. Specifically, the user performs the operation input by touching a button in the image displayed on the touch panel 21. This operation input is detected by the input unit 23, and the corresponding input position D2 is transmitted to the smartphone 10 via the communication unit 22.

[0041] In the smartphone 10, the communication unit 16 receives the input position D2 and supplies it to the application 11. The application 11 acquires the input position of the user's operation input to the smartphone 10 and supplies the input position to the control unit 13. The control unit 13 executes a process corresponding to the operation input performed by the user, based on the display image D12 and the input position received from the application 11. In this way, it becomes possible to operate the smartphone 10 from the display device 20A.

[0042] (How to convert input positions) Now, when an image of the smartphone 10 is transmitted to the display device 20A and displayed in this way, the specifications of the touch panel 14 of the smartphone 10 and the touch panel 21 of the display device 20, specifically, the number of pixels, the aspect ratio, etc., do not necessarily match. Therefore, when transmitting an image to the display device 20A, the smartphone 10 may add black borders to the top, bottom, left, and right of the image in order to adjust the number of pixels, the aspect ratio, etc. For example, in the above-mentioned HDMI standard, the aspect ratio of the image to be transmitted is fixed to 16:9 or 4:3, but the aspect ratio of the touch panel 14 of the smartphone 10 varies depending on the model. Therefore, when outputting an image by HDMI, the smartphone 10 may adjust the aspect ratio by adding a black border that differs depending on the model using the image output unit 12.

[0043] FIG. 3 shows an example of an image to which a black frame has been added to adjust the aspect ratio. An image IM is displayed on the touch panel 14 of the smartphone 10. This image IM includes a list of multiple songs belonging to an album and a "BACK" button for returning to the previous screen. When transmitting this image IM to the display device 20A for display, the smartphone 10 transmits an image IMx, whose aspect ratio has been adjusted to conform to the HDMI standard, to the display device 20A as an image D2. The image IMx displayed on the touch panel 21 of the display device 20A is an image IM displayed on the smartphone 10 with a black frame area BK added to its outer periphery.

[0044] In this case, since the black frame area BK is also displayed on the touch panel 21, the user can also perform touch input into the black frame area BK. However, since the black frame area BK does not exist in the image IM of the smartphone 10 and is not an operation input area, touch input into the black frame area BK must be processed as an inappropriate input. In addition, since the scale of the image IM displayed on the smartphone 10 and the image IMx displayed on the display device 20A are different, the position coordinates of touch input into the area inside the black frame area BK must also be converted into position coordinates on the image IM of the smartphone 10.

[0045] Here, when the display device 20 has an image size adjustment function, the input position D2 transmitted from the display device 20 is already adapted to the image size of the smartphone 10, and therefore the smartphone 10 does not need to perform a conversion process of the position coordinates. On the other hand, as in the example of FIG. 2, when the display device 20A does not have an image size adjustment function, the input position D2 transmitted from the display device 20A corresponds to the image size of the display device 20A and does not match the image size of the smartphone 10. Therefore, in this embodiment, when the display device 20A does not have an image size adjustment function and a notification of "coordinate conversion: necessary" is received from the display device 20A as coordinate conversion necessity information, the application 11 of the smartphone 10 functions as a conversion unit 11x and performs this conversion process.

[0046] Specifically, the conversion unit 11x receives the input position D2 on the touch panel 21 (i.e., the input position in the image IMx) from the display device 20A, and converts it into an input position on the touch panel 14 of the smartphone 10 (i.e., the input position in the image IM). At this time, the conversion unit 11x mainly performs the following two processes.

[0047] First, if the touch input to the black frame area BK is a simple touch (i.e., an input of the type in which the touch is made at approximately one point and then released without moving the touch position), the conversion unit 11x simply ignores the touch input. On the other hand, if the touch input to the black frame area BK is an input that involves movement of the touch position, such as a drag, the conversion unit 11x converts the input position of the touch input to the black frame area BK to the position of the area inside the black frame area BK that is closest to that position (hereinafter referred to as the "regular area"). Using either of these methods, the conversion unit 11x processes the touch input made to the black frame area BK.

[0048] Secondly, the conversion unit 11x converts the input position of the touch input on the normal area into a corresponding position on the touch panel 14 of the smartphone 10. As a result, even if the number of pixels of the touch panel 14 of the smartphone 10 and the touch panel 21 of the display device 20A differ, the input position on the touch panel 21 of the display device 20A can be correctly converted into an input position on the touch panel 14 of the smartphone 10.

[0049] 4 shows an example of the above conversion method. First, consider the horizontal direction. The number of dots in the horizontal direction of the touch panel 21 of the display device 20A is "W", and the left and right widths of the black frame area BK displayed on the touch panel 21 are "Wa" and "Wb", respectively. Also, the number of dots in the horizontal direction of the touch panel 14 of the smartphone 10 is "Ws".

[0050] In this case, the formula for converting the input position "X" on the touch panel 21 of the display device 20A into the input position "xs" on the touch panel 14 of the smartphone 10 is as follows:

[0051] xs=0 :X <Waのとき xs=Ws*(X-Wa) / {W-(Wa+Wb)} :Wa <X<Wbのとき xs=Ws : When X>(W-Wb) The same is true for the vertical direction. Let the number of dots in the vertical direction of touch panel 21 of display device 20A be "H", the top and bottom widths of black frame area BK displayed on touch panel 21 be "Hb" and "Ha", respectively, and the number of dots in the vertical direction of touch panel 14 of smartphone 10 be "Hs".

[0052] In this case, the equation for converting the input position "Y" on the touch panel 21 of the display device 20A into the input position "ys" on the touch panel 14 of the smartphone 10 is as follows:

[0053] ys=0 :Y <Haのとき ys=Hs*(Y-Ha) / {H-(Ha+Hb)} :Ha <Y<Hbのとき When ys=Hs:Y>(H-Hb) When the touch input to the black frame area BK is simply a touch, the conversion unit 11x can ignore the touch input as described above.

[0054] According to the above conversion method, even when an image IMx including a black frame area BK is displayed on the touch panel 21 of the display device 20A, the input position on the touch panel 21 can be correctly converted into an input position on the touch panel 14 of the smartphone 10.

[0055] Now, in order to execute the above conversion, the number of dots (number of dots in the vertical direction: Hs, number of dots in the horizontal direction: Ws) of the touch panel 14 of the smartphone 10, the number of dots (number of dots in the vertical direction: H, number of dots in the horizontal direction: W) of the touch panel 21 of the display device 20A, and the top, bottom, left and right widths (Wa, Wb, Ha, Hb) of the black frame area BK are required. Of these, the application 11 can obtain the number of dots of the touch panel 14 of the smartphone 10 from the control unit 13 of the smartphone 10, etc., and can obtain the number of dots of the touch panel 21 of the display device 20A from the display device 20A. On the other hand, the top, bottom, left and right widths of the black frame area BK are not information that the smartphone 10 and the display device 20 normally possess.

[0056] Therefore, the application 11 acquires the top, bottom, left and right widths of the black frame area BK from the server 1 as calibration information D3. Specifically, the top, bottom, left and right widths of the black frame area BK are determined based on the specifications of the touch panel 14 of the smartphone 10 and the specifications of the touch panel 21 of the display device. Therefore, the server 1 stores the top, bottom, left and right widths of the black frame area BK as calibration information D3 in advance, in association with a combination of the smartphone 10 and the display device 20A. For example, the server 1 stores calibration information D3 indicating the top, bottom, left and right widths of the black frame area BK in association with a combination of an ID indicating the model name, etc. of the smartphone 10 and an ID indicating the model name, etc. of the display device 20A. When the smartphone 10 is connected to the display device 20A, the smartphone 10 acquires the ID of the display device 20A, transmits the ID of the smartphone 10 and the ID of the display device 20A to the server 1, and acquires the calibration information D3 corresponding to the combination from the server 1. Then, the smartphone 10 converts the input position on the touch panel 21 of the display device 20A into an input position on the touch panel 14 of the smartphone 10, as described above, using the acquired calibration information D3.

[0057] [If the display device has an image size adjustment function] FIG. 5 is a block diagram showing a functional configuration of the system shown in FIG. 1 when a display device 20B having an image size adjustment function is used. The image size adjustment function is, for example, a function in which a user arbitrarily cuts out a part of an image displayed on a display device, and stretches (enlarges) the cut-out image to the entire screen and displays it. As can be understood by comparing with FIG. 2, the display device 20B in FIG. 5 is different from the display device 20A shown in FIG. 2 in that it has a conversion unit 27 and converts input coordinates output from the input unit 23 and supplies them to the communication unit 22 as input coordinates D2, but is otherwise similar to the display device 20A shown in FIG. 2. In addition, in FIG. 5, the server 1 and the smartphone 10 are similar to those in FIG. 2.

[0058] As described above, when transmitting an image to the display device 20B, the smartphone 10 may add black borders to the top, bottom, left and right of the image in order to adjust the number of pixels, aspect ratio, and the like. In this case, the black borders added by the smartphone 10 are part of the image data, and are displayed as is on the display unit 21 of the display device 20B. Therefore, the user uses the image size adjustment function of the display device 20B to stretch (enlarge) and display the image D1 transmitted from the smartphone 10 so that the black border is not displayed (by cropping out the image area other than the black border). As a result, the black borders added by the smartphone 10 are no longer displayed on the display unit 21 of the display device 20B.

[0059] Furthermore, the display device 20B knows how much the user has enlarged the display image by using an image size adjustment function, and the conversion unit 27 converts the user's input position in the enlarged display image into the coordinates of the input position in the display image before enlargement (i.e., the input position corresponding to the image size of the smartphone 10). As a result, even if the user enlarges the display image to an arbitrary size, the input position (input coordinates) of the user's touch input can be correctly converted into an input position in the image size of the smartphone 10. The communication unit 22 transmits the input position D2 converted by the conversion unit 27 to the smartphone 10 so as to fit the image size of the smartphone 10. As a result, coordinate conversion on the smartphone 10 side is not necessary.

[0060] 5 has an image size adjustment function, the memory 26 of the control unit 25 stores "adjustment function flag=1." Therefore, the control unit 25 notifies the smartphone 10 of coordinate conversion necessity information ("coordinate conversion: not required") indicating that coordinate conversion is not required on the smartphone 10 side. As a result, coordinate conversion is not performed on the smartphone 10 side, and therefore it is possible to prevent the smartphone 10 from performing unnecessary coordinate conversion even if the display device 20B side transmits an input position D2 that correctly matches the image size of the smartphone 10.

[0061] Furthermore, the display device 20B may acquire calibration information D3 from the smartphone 10 using, for example, the communication unit 22, and the conversion unit 27 may perform the above-mentioned conversion process using the acquired calibration information D3. With this configuration, even if an image area in which a black frame remains is cropped and enlarged by, for example, a user image size adjustment function, appropriate coordinate conversion can be performed between the display device 20B and the smartphone 10.

[0062] [Input location conversion process] 6 is a flowchart of the input position conversion process. This process is executed by the application 11 in the smartphone 10. Note that it is assumed that the smartphone 10 and the display device 20 are in a state in which they can communicate with each other. In addition, this process is executed when the application 11 is started on the smartphone 10.

[0063] First, when the application 11 is started, the application 11 requests coordinate conversion necessity information from the display device 20 and receives the coordinate conversion necessity information from the control unit 25 of the display device 20 (step S9). The coordinate conversion necessity information is determined based on whether the display device 20 has an image size adjustment function, as described above. If the coordinate conversion necessity information received from the display device 20 is "coordinate conversion: not required" (step S10: No), coordinate conversion on the smartphone 10 side is not required, and the application 11 ends the process.

[0064] On the other hand, if the coordinate conversion necessity information received from the display device 20 is "coordinate conversion: necessary" (step S10: Yes), the application 11 judges whether it is the first time it is started (step S11). Note that this judgment is made based on, for example, processing history information stored in the application 11.

[0065] When the application 11 is started for the first time, normally there is no calibration information yet in the application 11. Therefore, the application 11 identifies the ID of the smartphone 10 and the ID of the display device 20, requests the server 1 for calibration information corresponding to the combination of these, obtains the corresponding calibration information, and stores it in a storage unit or the like in the smartphone 10 (step S12). Then, the process proceeds to step S14.

[0066] On the other hand, if it is not the first startup (step S11; No), the application 11 determines whether or not there is calibration information in the memory unit of the smartphone 10 or the like (step S13). Note that the calibration information in this case means calibration information corresponding to the combination of the smartphone 10 and the display device 20 currently being used. If there is no calibration information (step S13; No), the application 11 acquires the calibration information from the server 1 (step S12). On the other hand, if there is calibration information (step S13; Yes), the process proceeds to step S14.

[0067] In step S14, the application 11 determines whether or not the acquisition of the calibration information has been successful. That is, the application 11 determines whether or not the calibration information has been prepared by the process of step S12 or S13. If the application 11 has succeeded in acquiring the calibration information (step S14: Yes), the process proceeds to step S15.

[0068] On the other hand, if the application 11 does not succeed in acquiring the calibration information (step S14; No), the application 11 performs a process of creating the calibration information by itself using the display device 20. Specifically, the application 11 transmits a calibration image to the display device 20 and causes the touch panel 21 of the display device 20 to display the calibration image (step S16). Possible reasons for not being successful in acquiring the calibration information include the case where the server 1 does not have calibration information corresponding to the combination of the smartphone 10 and the display device 20, or the case where communication with the server 1 has not been performed normally.

[0069] An example of the calibration image is shown in FIG. 7. In the example of FIG. 7, the application 11 transmits an image in which the normal area is expressed in a light color or the like so that the normal area can be distinguished from the black frame area BK as a calibration image to the display device 20. In this case, the calibration image includes the black frame area BK based on the difference in the number of pixels, aspect ratio, etc. between the touch panel 14 of the smartphone 10 and the touch panel of the display device 20. Furthermore, the application 11 indicates the positions of the four inner corners of the black frame area BK with marks C1 to C4, and has the user touch the four inner corners of the black frame area BK. When the user touches the positions of the marks C1 to C4, the display device 20 acquires the coordinates of each position and transmits them to the smartphone 10.

[0070] The application 11 acquires position coordinates corresponding to the four inner corners of the black border area BK, and calculates the top, bottom, left, and right widths of the black border area BK to generate calibration information, and stores the information in a storage unit or the like of the smartphone 10 (step S17). Furthermore, the application 11 uploads the generated calibration information to the server 1 together with the smartphone ID and the display device ID (step S18). As a result, the server 1 is provided with calibration information corresponding to the combination of the smartphone and the display device, which can then be used by another user. Then, the process proceeds to step S15.

[0071] In step S15, the application 11 converts the input position using the prepared calibration information (step S15). That is, the application 11 acquires the input position on the touch panel 21 of the display device 20 from the display device 20 and converts it into an input position on the touch panel 14 of the smartphone 10. Then, the application 11 supplies the input position D13 obtained by the conversion to the control unit 13 of the smartphone 10, and ends the conversion process. The control unit 13 of the smartphone 10 processes the operation input by the user based on the input position D13.

[0072] In this way, even if the specifications such as the number of pixels and the aspect ratio differ between the touch panel 14 of the smartphone 10 and the touch panel 21 of the display device 20 and a black frame area BK is displayed in the display image of the display device 20, the input position on the touch panel 21 of the display device 20 can be correctly converted to the input position on the touch panel 14 of the smartphone 10.

[0073] When the calibration information used for converting the input position is stored in the smartphone 10, the application 11 uses the calibration information, and therefore does not need to download it from the server 1. When the calibration information cannot be obtained from the server 1, the application 11 generates the calibration information based on the user's input using the calibration image exemplified in FIG. 7, so that the conversion process can be performed even when a new model of smartphone or display device for which the server 1 does not have information is used. Note that the task of creating the calibration information using the calibration image is not limited to when the calibration information cannot be obtained as described above. For example, the process of creating the calibration information may be registered in the menu of the smartphone 10, and can be executed when the user desires.

[0074] Moreover, the application 11 uploads the created calibration information to the server 1, so that other users can use the calibration information thereafter. When the server 1 receives calibration information from multiple smartphones, the server 1 can analyze the accuracy of the information and perform averaging processing to hold calibration information with higher accuracy.

[0075] [Variations] In the above embodiment, the control unit 25 of the display device 20 transmits coordinate conversion necessity information, “Coordinate conversion: not required”, to the smartphone 10 when the display device 20 has an image size adjustment function, and transmits coordinate conversion necessity information, “Coordinate conversion: required”, to the smartphone 10 when the display device 20 does not have an image size adjustment function.

[0076] However, even if the display device 20 has an image size adjustment function, in cases where the user does not actually use the image size adjustment function (for example, by setting the image size adjustment function to off) and the image D1 transmitted from the smartphone 10 is displayed as is on the display unit 21, the control unit 25 may transmit coordinate conversion necessity information, such as “Coordinate conversion: required”, to the smartphone 10 and perform the coordinate conversion on the smartphone 10 side.

[0077] Furthermore, for example, when the image size adjustment function on the display device 20 side is capable of enlarging not only the entire image but also enlarging only the vertical direction or only the horizontal direction, the control unit 25 may notify the smartphone 10 of the necessity of coordinate conversion for each of the vertical direction and the horizontal direction as coordinate conversion necessity information. For example, when the user uses the image size adjustment function of the display device 20 to adjust the size only in the vertical direction and not in the horizontal direction, the control unit 25 may transmit coordinate conversion necessity information, such as "vertical coordinate conversion: not required, horizontal coordinate conversion: required," to the smartphone 10 based on the setting. [Explanation of symbols]

[0078] 1 Server 10. Smartphones 11 Applications 11x, 24 conversion unit 12 Image output section 13 Control section 14, 21 Touch panel 20 Display device

Claims

[Claim 1] an image output unit that outputs a second image including the first image as a part thereof to a display device; a relative position information acquisition unit that acquires relative position information indicating a relative positional relationship of the first image with respect to the second image; a conversion information acquisition unit that acquires, from the display device, conversion information indicating whether or not to convert an input position on the second image based on the relative position information; a conversion unit that converts an input position on the second image to an input position on the first image based on the relative position information when the conversion information indicates that conversion is necessary; A terminal device comprising:

Images