Display control system, method, display device, and controller

Abstract

The present disclosure relates to the technical field of controlling an image by linking a controller and a display device. The controller wirelessly transmits orientation and operation data, and relative position data with respect to the display device. The display device specifies an instruction position on the basis of said data, generates an overlay image related to the instruction position, and displays the overlay image on a display screen.

Description

[Technical Field]

[0001] This disclosure belongs to the technical field of display control, and more particularly to a system and method for controlling images and information on a display using a user-operated controller. [Background technology]

[0002] In conventional display control systems, it was common practice to install sensor bars or markers on the display side to achieve a precise interface between the user-operated controller and the display.

[0003] For example, the Nintendo Wii® Remote uses a sensor bar placed at the top of the TV to detect infrared light emitted from the remote, enabling pointing operations. VR devices, on the other hand, employ a method where a camera sensor is placed on the display, and infrared LEDs or other markers are positioned on the controller.

[0004] These methods often required additional hardware for the controller and display, leading to complex system configurations and cumbersome installation processes. [Overview of the project]

[0005] This disclosure relates to a technology aimed at improving the operational accuracy and usability of prior art. In particular, it focuses on providing accurate pointing operations on a display based on real-time communication between a controller and a display device.

[0006] This disclosure includes: (1) The controller wirelessly transmits posture and motion data, as well as relative position data to the display device. (2) The display device identifies the indicated position based on the data received from the controller, (3) A technique for displaying an overlay image generated based on the indicated position on a display. This provides an intuitive and precise user experience throughout the entire system.

[0007] This disclosure enables users to operate on the display with a simple configuration without adding special hardware to the display side. Furthermore, the integrated sensor group in the controller enables high-precision acquisition of attitude and motion data, as well as relative position data, realizing accurate identification of the indicated position on the display. In addition, the controller side configuration is also simple; for example, by utilizing a combination of a 9-axis sensor and a laser distance sensor, high-precision data acquisition and communication can be achieved without requiring any special additional hardware.

[0008] Further aspects and advantages of the present disclosure will be readily apparent to those skilled in the art from the following detailed description, which shows and describes only exemplary embodiments of the present disclosure. As will be understood, other different embodiments are possible, and some of their details can be modified in various obvious ways without departing from the present disclosure. Accordingly, the drawings and description should be considered illustrative and not limiting in nature. [Brief explanation of the drawing]

[0009] [Figure 1] This block diagram shows the configuration of a controller according to one embodiment. [Figure 2] This figure shows the specific shape of a controller according to one embodiment, where (A) is a front view and (B) is a rear view. [Figure 3] This is a block diagram showing the configuration of a display device according to one embodiment. [Figure 4] This flowchart shows the process flow between a controller and a display device according to one embodiment, and explains a series of processes related to the display of a base image, position reset, and real-time image manipulation. [Figure 5A]This is a schematic diagram illustrating a position reset process according to one embodiment, showing a scene where the controller is directed towards a reset cursor displayed on the screen of a display device. [Figure 5B] This is a schematic diagram illustrating a real-time operation process according to one embodiment, showing a scene in which an overlay image displayed on the screen of a display device is displayed according to the position indicated by the controller. [Figure 6A] This is a schematic diagram illustrating a position reset process in multiple display devices according to one embodiment. It illustrates a state in which a reset cursor is displayed on a specific display device and a controller is being operated towards that cursor. [Figure 6B] This is a schematic diagram illustrating a real-time operation process in multiple display devices according to one embodiment. It illustrates a state in which a controller specifies a position and an overlay image is displayed on the display device. [Figure 7] This flowchart shows the process flow between a controller and a display device according to one embodiment, and explains a series of processes related to the display of a base image, position reset, and real-time image manipulation. [Figure 8A] This is a schematic diagram illustrating a position reset process according to one embodiment, illustrating how the controller is directed towards a display device and how the reset cursor is displayed on the circuit board image. [Figure 8B] This is a schematic diagram illustrating real-time operation when the controller is located close to the display device, according to one embodiment, and illustrates the generation and display of an overlay image during close-range operation. [Figure 8C] This is a schematic diagram illustrating real-time operation when the controller is located at a distance from the display device, according to one embodiment, and illustrates how an overlay image adjusted according to the distance is displayed. [Figure 8D]It is a schematic diagram showing a scene where the magnification of an overlay image is changed using an operation ring of a controller according to an embodiment, and illustrates the state of adjustment of the overlay image by user operation. [Figure 9] It is a flowchart of calculation processing performed between a pointing device and a display device according to an embodiment. [Figure 10A] This is an explanation about a certain embodiment. [Figure 10B] This is an explanation about a certain embodiment. [Figure 10C] This is an explanation about a certain embodiment. [Figure 10D] This is an explanation about a certain embodiment. [Figure 10E] This is an explanation about a certain embodiment. [Figure 10F] This is an explanation about a certain embodiment. [Figure 10G] This is an explanation about a certain embodiment. [Figure 10H] This is an explanation about a certain embodiment. [Figure 10I] This is an explanation about a certain embodiment. [Figure 10J] This is an explanation about a certain embodiment. [Figure 10K] This is an explanation about a certain embodiment. [Figure 10L] This is an explanation about a certain embodiment. [Figure 10M] This is an explanation about a certain embodiment. [Figure 10N] This is an explanation about a certain embodiment. [Figure 10O] This is an explanation about a certain embodiment. [Figure 10P] This is an explanation about a certain embodiment. Detailed description of the invention

[0010] As used herein, the term "controller" refers to a user-operated device that has the function of transmitting control signals to a display device via wireless communication. A controller may include a sensor module that generates attitude and motion data and relative position data, an input interface that accepts user input, and a communication unit. In this specification, the terms "controller" and "pointing device" are used interchangeably.

[0011] As used herein, the term "display device" refers to a device equipped with a display screen for presenting image information to a user, and having the function of controlling the displayed content based on data received wirelessly from a controller. A display device may include a communication unit, a processor, and a display control unit.

[0012] As used herein, the term "Base Image" refers to an image displayed as the initial or background image in a display device. This image may be static (still image) or dynamic (moving image). Furthermore, the base image serves as the basis for generating overlay images and provides the user with a visual reference.

[0013] In some embodiments, the underlying image may include natural scenery, urban landscapes, outer space, or abstract patterns. This allows the display device to function beyond that of an information display device, acting like an artificial window. For example, by displaying a landscape image on the display device, a wall-mounted display can provide a visual effect that makes it appear as if the landscape is actually present outside.

[0014] The underlying image may be pre-stored in the display device's storage (e.g., non-temporary storage media) or it may be retrieved from an external source. Retrieval from an external source may include downloading via the internet, streaming, or using cloud-based services. Dynamic content that is updated in real time (e.g., live video, interactive information) may also be used as the underlying image.

[0015] It is also possible to coordinate and operate multiple display devices in a unified manner. In this configuration, multiple display devices understand their relative positions and display a series of images. In one embodiment, a method is employed in which placement information is manually entered into each display device. In another embodiment, the relative positions can be automatically determined through communication between the display devices.

[0016] By linking multiple display devices, vast landscape images are displayed as if they were a single continuous screen. This continuity allows users to experience a visual sensation similar to, for example, looking out a window. Furthermore, the underlying images are not limited to landscape images; a wide range of content can be used, including educational content (e.g., maps, science videos), entertainment (e.g., movies, games), or interactive information (e.g., news, weather forecasts).

[0017] The generation and selection of background images may be performed by the display device's control processor or a cloud-based service. This allows users to select or customize background images based on specific themes or needs.

[0018] By integrating multiple display devices, users can enjoy an immersive and interactive experience. This configuration is applicable to a wide range of uses, including entertainment, education, presentations, and advertising, and enhances the user experience.

[0019] As used herein, the term "overlay image" refers to an image that is enlarged in part or in whole from a substrate image and displayed superimposed on the substrate image on a display device.

[0020] Overlay images come in a variety of formats and uses, and include, but are not limited to, the following examples:

[0021] 1. Cursor The cursor, as an overlay image, is a display that indicates a specific position on the base image, and its position can be moved by the user operating the controller.

[0022] 2. Enlarged view of a portion of the circuit board image Overlay images, which enlarge and display a portion of the base image, are used to provide details about a location indicated by the user. For example, this includes providing more detailed information by enlarging a portion of a map image.

[0023] 3. Supplementary information associated with the base image It is possible to display information related to specific elements on the base image as an overlay image. For example, if a bird is specified in the base image, the bird's species name, ecological information, or related images and text may be displayed.

[0024] 4. Menu or Operation Guide Menus or operation guides may be displayed as overlay images to assist the user. For example, these may include options for changing display device settings or selecting specific functions.

[0025] 5. Real-time information Overlay images may display real-time information generated in response to the content of the base image or user actions. For example, if a video is used as the base image, data related to its timeline will be displayed as an overlay.

[0026] 6. Presentation of multi-layered information Overlay images are also used to display different types of information simultaneously as multiple layers on top of a base image. For example, temperature data, traffic information, or detailed information related to a user-selected point may be overlaid on a map image.

[0027] 7. Interactive elements When a user selects or points to a specific element on a base image, a responsive overlay image based on that selection may be generated. For example, this could include displaying the price or specifications of a product when the user selects a product in a product catalog image.

[0028] 8. Highlight Overlay images may be used to highlight specific areas or elements on a base image. Examples include highlighting specific keywords in text or coloring specific locations on a map.

[0029] 9.3D information or depth information If the base image is capable of 3D display, depth information or a 3D model may be added as an overlay image.

[0030] 10. Annotation Overlay images may be used to display annotations or notes that the user adds to the base image. In this case, the text or shapes entered by the user are displayed in association with the base image.

[0031] In addition to these examples, overlay images may also be used in other forms depending on the specific application and embodiment.

[0032] As used herein, the term "indicated position" refers to a specific location on the display screen that the user directs to by operating the controller. This position is determined within the display coordinate system based on posture and motion data and relative position data.

[0033] As used herein, the term "display coordinate system" refers to a reference coordinate system for defining position and / or orientation within a display device. The origin and the orientation of each axis are determined based on the geometric structure of the display screen.

[0034] As used herein, the term "display control unit" refers to a component that has the function of displaying an image on a display device or overlaying an image onto a substrate image based on instructions from a processor.

[0035] As used herein, the term "attitude and motion data" refers collectively to information relating to the attitude (e.g., tilt, rotation, etc.) and / or motion (e.g., acceleration, movement, etc.) of the controller. In some embodiments, this data is acquired using one or more of the modules exemplified below.

[0036] In some embodiments, a 9-axis sensor may be used. The 9-axis sensor is a measurement module combining a 3-axis accelerometer, a 3-axis gyroscope, and a 3-axis geomagnetic sensor, and is capable of detecting the controller's acceleration, rotational speed, and geomagnetic information. This allows for the comprehensive acquisition of the controller's attitude and motion data. The attitude and motion data may include the controller's Euler angles (pitch (θ): rotation angle around the X axis, roll (φ): rotation angle around the y axis, yaw (ψ): rotation angle around the z axis), quaternions (quantion, q=(q0,q1,q2,q3)), and rotation matrix (R=r_ij).

[0037] Furthermore, in other embodiments, the following modules may be used. Accelerometer: Detects acceleration in the X, Y, and Z axis directions and measures motion and vibration. Gyroscope: Detects angular velocity and measures rotational motion and changes in attitude. Geomagnetic sensor: Detects the controller's orientation and measures its direction and orientation. IMU (Inertial Measurement Unit): A unit that combines an accelerometer, gyroscope, and possibly a geomagnetic sensor, making it suitable for comprehensive acquisition of attitude and motion data. Optical motion capture sensor: Uses infrared or visible light to track the controller's movement and posture. Ultrasonic sensor: Uses ultrasonic reflection to measure the distance and direction of movement of the controller. LiDAR: Uses laser light to measure the position and movement of a controller with high precision. Sound wave tracking system: Measures the position and movement of a controller by utilizing the time lag between the propagation and reception of sound waves. Pressure sensor: Detects the pressure or vibration the controller experiences and supplements the operational data.

[0038] As used herein, the term "sensor module" refers to a configuration that includes one or more of the aforementioned types of sensors, and includes not only cases where these modules are used individually, but also configurations that combine multiple sensors. For example, an accelerometer and a gyroscope may be combined to measure both motion and rotation. For example, an optical sensor and a geomagnetic sensor may be used in combination to measure position and orientation with high precision.

[0039] These modules may be located inside the controller or used in combination with externally installed sensors, and are used to acquire posture and motion data in response to user operations to control the operation of the display device and related systems.

[0040] As used herein, the term “relative position data” refers to data indicating the distance and / or positional relationship between the controller and the display device. In some embodiments, this data is measured using a laser distance sensor, an ultrasonic sensor, an infrared sensor, or other suitable technique. These techniques are used to detect with high accuracy the distance to a reflector (obstacle) in the direction the controller is pointing. For example, laser distance sensors are suitable for high-precision measurements at long distances, while ultrasonic and infrared sensors are suitable for measurements at medium and short distances.

[0041] As used herein, the term "laser distance sensor" refers to a module that uses laser light to measure the distance between a controller and a display device or object. This sensor includes a laser emitter and a photodetector and may employ a time-of-flight (ToF), phase-difference, or other appropriate measurement method. Furthermore, the laser distance sensor may be configured to detect reflectors (obstacles) in the direction pointed by the controller and may be used for positional information of a display device or for specifying an object to be operated.

[0042] The laser distance sensor measures the distance from the controller to the reflector, so if the target is not the display screen, the distance to the display device may not be obtained. Even in this case, the user is prompted to point the controller towards the reset cursor through the position reset process, so this does not pose a major problem in normal use. By accurately pointing the controller based on the position reset cursor, the initial position of the controller and the indicated position on the display device are synchronized, and accurate position information is obtained. Furthermore, if the user feels any unnatural behavior during operation, it is easy to perform another position reset, and the system is designed to maintain a smooth user experience.

[0043] Similarly, if the user moves significantly relative to the display device, or if the controller's operating range changes, and the indicated position is not set correctly, the user can resolve the issue by re-performing a position reset. In this case, the display device has a function to assist the user by performing default correction operations.

[0044] The term "relative position data" as used herein is obtained by using these sensor technologies individually or in combination. This allows for accurate determination of the positional relationship between the controller and the display device, providing an environment in which the user can operate intuitively. This design is a crucial element in supporting high-precision operation during position reset and real-time operation.

[0045] As used herein, the term "operating ring" refers to a component located on the outer circumference of a controller that is used to detect rotational operations performed by the user and to control the operation of the display device or related system based on those operations.

[0046] In some embodiments, the operating ring may be used to perform a variety of functions, such as those listed below. Zoom control: Enlarges or reduces content within a screen or window on a display device. Volume adjustment: The volume of the display device or external system is increased or decreased according to the amount of rotation of the control ring. Movement within the screen or virtual space: By rotating the control ring, you can move in a specific direction within the screen or virtual space. Menu selection or switching of operation modes: Use the rotation of the control ring to select a specific item from multiple options or switch operation modes.

[0047] The operating ring has the function of detecting the direction of rotation (e.g., clockwise or counterclockwise) and the amount of rotation, and its operation is controlled based on the user's intuitive operation. Specifically, the operating ring is provided with convex and concave parts arranged at equal intervals, and a configuration may be adopted in which changes in these are detected using optical, magnetic, or mechanical encoders or sensors. For example, when using an optical encoder, the rotational movement can be obtained as 0 and 1 signals by detecting the difference in light reflection between the convex and concave parts.

[0048] As used herein, the term "control ring" is not limited to a specific application or sensing technology, but is used to control zoom, volume adjustment, navigation, menu selection, or other appropriate functions. Sensing technologies also include optical, magnetic, or mechanical encoders, as well as other appropriate technologies.

[0049] As used herein, the term "position reset button" refers to a part of the input interface provided on the controller, which is used to reset the controller's initial position and set the initial attitude, motion data, and initial relative position data when pressed.

[0050] As used herein, the term "cylindrical controller" refers to a controller having an elongated shape and configured to be used with its longitudinal direction facing a display device. This controller may include a hollow structure or a form with open ends.

[0051] In some embodiments, the controller incorporates a 9-axis sensor and a laser distance sensor, providing users with an intuitive and multi-functional experience in a variety of applications. The availability of this controller is described below.

[0052] Firstly, this controller is suitable for applications in the gaming and entertainment fields. For example, it can be used as a virtual tool such as a lightsaber (product name, trademark, Disney), a fishing rod, or a gun, providing a realistic feel in action and simulation games. This controller can accurately detect user movements using a 9-axis sensor and reflect them in the movement of a pointer or character on the screen. Furthermore, it can utilize a laser distance sensor to reflect the user's distance information in in-game interactions.

[0053] Secondly, this controller can also be used for measurement purposes. For example, when measuring the length of a room or the dimensions of furniture, the controller can be pointed at a wall or object to measure the distance with high accuracy. Also, by holding the controller vertically and pointing it from the floor to the ceiling, it is possible to measure the height of a room. Similarly, by pointing the controller from the user's feet to their head, it is possible to measure their height. Such measurements are based on high-precision distance measurement technology using a laser distance sensor.

[0054] Thirdly, this controller is also useful for educational, presentation, and professional applications. For example, when displaying maps or scientific data, the controller can be used to zoom in on specific areas. Furthermore, by supporting virtual annotation or selection operations on the display device, it becomes possible to share or present information efficiently.

[0055] This controller achieves both operational precision and flexibility by combining a 9-axis sensor and a laser distance sensor. Even without a gyroscope sensor, the 9-axis accelerometer and geomagnetic sensor can adequately detect the user's posture and movement. Furthermore, the laser distance sensor contributes to supplementing positional information and measuring distance, enabling a wide range of application scenarios.

[0056] Furthermore, by utilizing interfaces such as the control ring and buttons, the controller can compensate for the often insufficient operational precision in game control and measurement applications. For example, adjusting the magnification using the control ring enhances visual feedback and improves the user's experience. In this way, this controller can be used for a wide range of applications as an interface device that combines multifunctionality and flexibility.

[0057] Figure 1 is a block diagram showing the configuration of the controller 100 according to the present invention. The controller 100 is equipped with multiple user interfaces, sensors, and control modules for communication with and operation of a display device. These components are designed to enable intuitive and efficient operation.

[0058] The controller 100 includes multiple components. The user interface 120 has multiple buttons or control rings for user operation and generates operation signals. These include a power button 121 for turning the controller on or off, a cross-shaped button 122 for directional operation (up, down, left, and right), a center button 123 for selection and confirmation, an option button 124 for displaying menus and accessing settings screens, and a menu button 125 for specific screen switching and shortcut functions. It also includes a trigger button 126 for reset and auxiliary operations, and a control ring 127 for zoom and volume adjustment and menu item selection.

[0059] The network interface 130 is a module that transmits and receives data with the display device via wireless communication, and establishes communication between the controller 100 and the display device via the antenna 131.

[0060] The sensor module is equipped with a 9-axis sensor 132 that measures the attitude and motion data of the controller, and a laser distance sensor 133 that measures the distance to the reflector in the direction the controller is pointing. These sensors provide a high-precision interface based on user input.

[0061] Storage 113 is a non-volatile storage medium that stores software or programs for controlling the operation of controller 100. On the other hand, memory 112 is a volatile storage medium that temporarily stores data in real time, and the central processing unit 111 comprehensively manages the operation of each component within the controller. This generates signals that are transmitted to the display device based on user input.

[0062] These components are interconnected via the system bus 140 and are designed to enable efficient data flow and synchronized operation. This allows the controller 100 to provide highly accurate and rapid responses to user operations.

[0063] Figure 2 shows the specific shape of the controller 100 shown in Figure 1, where (A) is a front view and (B) is a rear view.

[0064] The controller 100 has a cylindrical shape, and its longitudinal direction is oriented toward the display device. This cylindrical structure is hollow, allowing the user to view the display device or object through the cavity. This design provides the user with an intuitive and immersive operating experience.

[0065] The front of the controller 100 has several buttons arranged along its length, including a power button 121, a cross-shaped button 122, a center button 123, an option button 124, and a menu button 125. A trigger button 126 is provided on the back, enabling reset and auxiliary operations.

[0066] An operating ring 127 is installed at one end, which allows for zoom or volume adjustment and menu item selection via rotation. The ring is positioned where the user's finger can naturally reach it, providing comfortable operation. Because it is accompanied by tactile feedback, the user can operate it without needing visual confirmation.

[0067] Furthermore, the cylindrical, hollow structure provides users with a new operating experience. By looking through the controller at the display device, users can get the feeling of using a telescope. This interaction is particularly effective in enhancing enjoyment and practicality in applications such as interactive games and learning.

[0068] Furthermore, the user interface components are arranged longitudinally along the cylindrical structure, fitting the shape of the hand and making one-handed operation easy. The uniform spacing between buttons prevents accidental operation, enhances the overall design coherence, and refines the product's appearance.

[0069] The cylindrical shape of the controller 100 shown in Figure 2, the arrangement of the user interface, and the position of the operating ring 127 are optimized to allow users to operate the display device intuitively and efficiently. This configuration enhances the user experience, especially in applications requiring interactive operation.

[0070] Figure 3 is a block diagram showing the configuration of a display device 300 according to one embodiment. This display device includes a plurality of components for displaying images or information in response to user operation.

[0071] The display device 300 includes a display panel 320 for visually outputting images or information. This display panel 320 is based on a liquid crystal display (LCD), organic light-emitting diode (OLED), or other suitable display technology and is designed to display visual information at high resolution. Furthermore, this display panel 320 has the capability to display still images, moving images, and interactive graphical user interfaces (GUIs).

[0072] The display device 300 is equipped with a network interface module 330 for sending and receiving data with an external device via wireless communication. This module communicates with a controller or other external device via an antenna 331.

[0073] Furthermore, the display device 300 includes a central processing unit 311 for performing data processing, a memory 312 for real-time data processing, and a non-volatile storage 313 for storing the system-wide operating logic or firmware. This storage 313 is composed of a non-temporary storage medium and stores software and programs for controlling the operation of the display device. This software may include image processing algorithms, user interface operation logic, network communication protocols, etc. This storage also enables system firmware updates and data log storage.

[0074] The display device 300 includes a graphics processing unit 314 for optimizing image processing and achieving high-speed rendering. This GPU 314 controls the display of images on the display panel 320 and plays a role in improving the responsiveness of the user interface.

[0075] These components are interconnected via the system bus 340 and are designed to enable efficient data flow and synchronized operation. This allows the display device 300 to process signals from the controller in response to user operations with high precision and speed.

[0076] With the above configuration, the display device 300 according to one embodiment provides an interactive operation experience through cooperation with a controller, and realizes an intuitive and user-friendly interface. This device is applicable to a wide range of uses, including home entertainment, education, presentations, and more.

[0077] Figure 4 is a flowchart showing the process flow between a controller and a display device according to one embodiment. This process is mainly divided into a position reset process (S410) and a real-time image manipulation process (S420).

[0078] In the position reset process (S410), when the controller detects a user's operation to reset the position (S411), it transmits this information to the display device. The display device displays a reset cursor on the screen (S412), and the user points the controller towards this cursor.

[0079] The display device may, at this time, output (request) to the user to point the controller towards the reset cursor in the form of video, audio, or controller vibration. Alternatively, this may be stated in the instruction manual that the user reads when purchasing the display device.

[0080] The controller acquires initial data using its built-in 9-axis sensor and laser distance sensor (S413), and the controller's initial position is calculated based on this data (S414). Here, the laser distance sensor measures the distance D0 in the direction the controller is pointing, but this measurement result is not used to determine the accuracy of the orientation towards the cursor. The reset cursor is provided to the user as a reference point, and the controller is presumed to be facing the cursor based on user operation.

[0081] In the real-time process (S420), the controller acquires current sensor data (S421), and the display device calculates the instruction coordinates based on this data (S422). Next, the display device generates an overlay image based on the instruction coordinates (S423) and displays it superimposed on that position (S424). This allows the user to operate the display device intuitively and in real time.

[0082] Figures 5A and 5B are schematic diagrams illustrating position reset and real-time operation. Figure 5A shows a reset cursor 522 displayed on the screen 511 of the display device 510, with the controller 550 directed towards it. The controller 550 measures the pitch angle θ0, yaw angle ψ0, and roll angle φ0 as part of the attitude and motion data acquired by the 9-axis sensor. This data provides detailed operation information for the controller and is used to identify the instruction coordinates.

[0083] Figure 5B shows a real-time operation scenario, where an overlay image 523 is displayed on the screen 511 of the display device 510. The overlay image 523 consists of, for example, a zoomed-in portion of a map, a cursor, or text and / or icons related to an object. This image is intuitively displayed on the screen in response to user actions, improving usability.

[0084] Data transmission between the controller 550 and the display device 510 is performed via wireless communication. High-precision, low-latency communication allows the controller's sensor data to be transmitted to the display device 510, which then performs necessary calculations based on this data. This configuration enables the user to experience smooth, real-time operation.

[0085] Figures 6A and 6B show schematic diagrams of position reset and real-time operation when multiple display devices operate in coordination. In this embodiment, the display device set 610 consists of three display devices 610a, 610b, and 610c. Each display device in this set can either have its position information manually entered by the installer after installation, or communicate with each other to automatically understand their positional relationships. This makes it possible for the display device set 610 to operate multiple display devices as if they were a single, integrated screen.

[0086] Figure 6A shows a scene from the position reset process, illustrating the state when the controller 650 sets the initial position. A reset cursor 622 is displayed on screen 611b of display device 610b, one of the display device sets 610, and the user operates the controller 650 towards it. The controller 650 acquires attitude and motion data using a 9-axis sensor, and the initial position is calculated based on this data. The 9-axis sensor here integrates an accelerometer, gyroscope, and geomagnetic sensor, and the pitch angle θ0, yaw angle ψ0, and roll angle φ0 are illustrated exemplarily, but these are only a part of the acquired data.

[0087] Figure 6B illustrates the real-time operation process, showing the state after the controller 650 has moved and rotated. An overlay image 623 is displayed on screen 611c of display device 610c, one of the display device sets 610. In this scene, the controller 650 acquires current attitude and motion data (e.g., pitch angle θ1, yaw angle ψ1, roll angle φ1), and the display device set 610 calculates the instruction coordinates based on this data. During this process, real-time data is transmitted and received between the controller 650 and the display device 610, enabling rapid and intuitive operation.

[0088] Figure 7 is a flowchart showing the process flow between a controller and a display device according to one embodiment. This process consists of three main parts: display of a base image (S701), a position reset process (S710), and a real-time image manipulation process (S720).

[0089] First, the display device displays a base image on the screen (S701). The base image is an image used as a background and may include still images, videos, or user interface elements. This base image is displayed in combination with overlay images generated in subsequent processes and serves as the basis for user interactive operations.

[0090] In the position reset process (S710), the display device displays a reset cursor at a predetermined position on the screen (S712). The user operates the controller toward this reset cursor and acquires initial data using sensors within the controller (e.g., a 9-axis sensor and a laser distance sensor) (S713). Based on this data, the initial position of the controller is calculated (S714). The reset cursor functions as a reference point for the user, and by accurately pointing the controller, the initial position of the controller is synchronized with the reference point on the display screen. This process enables the calculation of highly accurate instruction coordinates in subsequent operations.

[0091] In the real-time process (S720), the controller acquires current sensor data (S721), and the display device calculates the instruction coordinates based on that data (S722). Next, an overlay image is generated according to the distance between the instruction coordinates and the controller and the magnification factor by the control ring (S723), and is displayed superimposed on the base image (S724). The design is such that the size of the overlay image increases as the distance increases, and further fine adjustments can be made using the control ring. This enables dynamic interaction based on distance or operation.

[0092] By combining the base image and the overlay image, users can obtain an intuitive and interactive operating experience. In this embodiment, as the user holding the controller actually moves their body or hands, the indicated position on the screen changes dynamically, and the size of the overlay image is adjusted in real time accordingly. With this system configuration, users can intuitively feel the correspondence between physical actions and visual results, resulting in a seamless operating experience.

[0093] For example, when a user moves closer to the display device while holding the controller, the overlay image shrinks, and detailed information is enlarged on the screen. Conversely, when the user moves back, the overlay image enlarges, and the overall image expands on the screen. Such actions give the user the feeling that they are "zooming in" or "zooming out" on the image or information themselves. This experience, where physical movement and changes on the screen are linked, provides the user with a sense of immersion and achieves a more natural feel than conventional button-operated control. In some embodiments, the size of the overlay image may be determined according to the distance between the controller and the display. For example, the size of the overlay image may decrease as the distance increases, and vice versa. In some embodiments, the magnification of the overlay image may be determined according to the distance between the controller and the display. For example, the magnification of the overlay image may increase as the distance decreases, and vice versa. In some embodiments, both the size and magnification of the overlay image may be determined according to the distance between the controller and the display.

[0094] Furthermore, when the user moves their hand to point the controller at a different part of the display screen, the indicated position also moves in real time, and the range of the target information or image instantly switches. This allows the user to freely explore the content on the display through their own movements. Such operation is particularly useful when dealing with a wide range of information or large images, and provides the user with the satisfaction of directly controlling the information through their own movements.

[0095] Another key feature of this system is that users can complete operations using their own body movements without needing to use control rings or buttons. This design not only provides intuitive operation but also offers users a new interactive experience that incorporates kinesthetic sense. For example, in educational settings, users can move closer to the display to examine image details, or in presentations, they can survey the entire screen from a distance, resulting in a more natural operating environment.

[0096] Figures 8A to 8D are schematic diagrams illustrating the interaction between a controller and a display device in one embodiment. This interaction illustrates a step-by-step process from position reset to real-time operation and further to adjustment of the overlay image's magnification.

[0097] Figure 8A shows the position reset process. A base image 821 is displayed on the screen 811 of the display device 810, and a reset cursor 822 is displayed on top of it. In this process, the user operates the controller 850 towards the reset cursor and acquires attitude and motion data using sensors (e.g., a 9-axis sensor) within the controller. This data includes pitch angle θ0, yaw angle ψ0, and roll angle φ0, and the initial position of the controller is calculated. The reset cursor functions as a reference point for the user's operation, and the initial position is set when the user accurately points the controller.

[0098] Figure 8B illustrates close-range operation in a real-time operation process. When the controller 850 is close to the display device (distance D1), the size of the overlay image 823a displayed on the display screen 811 is adjusted to a smaller size. In this scenario, the controller acquires current attitude and motion data (e.g., pitch angle θ1, yaw angle ψ1, roll angle φ1), and the display device 810 calculates the instruction coordinates based on this data. This real-time data transmission and reception allows the user to perform precise operations.

[0099] Figure 8C illustrates the operation when the controller 850 is located at a distance from the display device (distance D2). In this case, the size of the overlay image 823b on the display screen 811 is significantly adjusted to account for the distance. In this scenario, a magnification factor M1 is applied, and the size of the instruction coordinates and the overlay image are automatically adjusted based on the data transmitted between the controller and the display device. This makes it easy for the user to visually recognize the information even from a distance, providing a comfortable user experience.

[0100] Figure 8D shows a scenario where the magnification of the overlay image 823c is changed using the control ring 851 of the controller 850 while maintaining the distance D2. In this scenario, the magnification is adjusted from M1 to M2 by rotating the control ring 851, and the overlay image 823c is further magnified in detail. This adjustment allows the user to zoom in and examine specific parts of the image. Since the magnification is changed in real time based on the rotation of the control ring, the user can operate it intuitively with visual feedback.

[0101] As shown in Figures 8A to 8D, the indicated position changes in response to the movement and rotation of the controller, and the size and magnification of the overlay image are further adjusted in response to distance or rotation of the control ring. This configuration provides users with a dynamic and interactive operating experience and achieves intuitive operation. Furthermore, this design maximizes the functionality of the display device in entertainment, education, presentations, and other applications.

[0102] The following describes a non-limiting example of the calculation method shown in this disclosure. In this embodiment, Figure 9 shows a processing flowchart that integrates two calculation processes (calculation of the position of the pointer cursor on the screen and correction of the distance sensor). These processes can also be performed separately.

[0103] Calculation example 1: Calculation of the position of the pointer cursor on the screen A communication and processing method between a controller (pointing device) and a display device according to one embodiment will be described. In this process, the controller communicates wirelessly with the display device (S912) and calculates the position of the pointer cursor based on the position indicated by the controller.

[0104] First, acceleration and angular velocity values ​​are acquired by sensors (e.g., acceleration sensor and angular velocity sensor) mounted in the controller (S911), and the sensor data is transmitted to the display device via wireless communication (S912). The display device calculates unit quaternions based on this data (S921). If unit quaternions are output directly from the sensor, these values ​​can be used as is. As a method for calculating unit quaternions, known filters such as the Madgwick filter can be used.

[0105] Next, the rotation matrix is ​​calculated from the unit quaternion (S922). If the unit quaternion is Q(qr,qi,qj,qk), the rotation matrix C is expressed by the following formula.

number

[0106] Furthermore, the rotation angles (roll (φ), pitch (θ), yaw (ψ)) of the pointing device are calculated based on this rotation matrix. These rotation angles are calculated based on the following formulas.

number

[0107] Furthermore, the position coordinates of the pointer cursor are calculated based on the distance to the display device (d), the screen size (width (W), height (H)), and the number of pixels (horizontal (Ph), vertical (Pv)) (S941). First, the number of pixels per unit length is calculated using the following formula.

number

[0108] Next, taking into account the correction based on the distance (d), the difference distance (dx, dy) of the pointer cursor is calculated using the following formula.

number

[0109] These processes are executed repeatedly in real time. The distance d is acquired by a distance sensor in the controller, and if the value fluctuates significantly, it is appropriately corrected. This configuration allows the cursor to move intuitively and smoothly according to the position indicated by the controller, providing the user with a natural operating experience.

[0110] Calculation process example 2: Correction of distance sensor shake using acceleration In this embodiment, a method for correcting deviations in distance sensor measurements using the output of an acceleration sensor is described. This method improves the accuracy of distance measurement between the pointing device and the display device, enabling more stable operation.

[0111] First, the values ​​from the accelerometer installed in the pointing device are transmitted to the computer. The accelerometer measures the acceleration value along the axis aligned with the display direction (hereinafter referred to as the "display axis"), and the change in this value is used as the criterion for the correction process described later.

[0112] The change in the acceleration value of the display axis obtained from the accelerometer is evaluated based on a statistical index (e.g., variance) or a set threshold (S931). Depending on the result of this evaluation, the measurement value from the distance sensor is processed as follows. 1. When the acceleration along the display axis changes significantly: If the acceleration along the display axis changes significantly, it is highly likely that the pointing device is moving significantly. In this case, the distance sensor value is not corrected and is used as is (S932). 2. When the acceleration change along the display axis is small: If the acceleration change along the display axis is small, the pointing device is judged to be stable. In this case, the distance sensor value is corrected to suppress time-series changes by smoothing or other processing (S933). This correction reduces errors in distance measurement and improves the reliability of pointer cursor position determination.

[0113] This flowchart (Figure 9) shows an example that integrates calculation process example 1 and calculation process example 2, but it can also be applied when each process is executed independently. This flexibility allows for various applications and enables advanced system control that improves the user experience.

[0114] This disclosure includes the following embodiments: A001. A method for wirelessly controlling an image on a display using a controller, (b0) Wirelessly acquiring the attitude and motion data of the controller and relative position data with respect to the display from the controller; (c0) Determining the indicated position within the display coordinate system based on the aforementioned posture and motion data and the aforementioned relative position data; (d0) generating an overlay image; and (e0) Display the generated overlay image at the indicated position on the display, A method for providing this. B001. A display device that is wirelessly controlled by a controller, (A0) A display screen for showing images; (B0) A communication unit that wirelessly receives controller orientation and operation data and the controller's relative position data with respect to the display device from the controller; (C0) processor, (i) to identify the indicated position in the display coordinate system based on the received attitude and motion data and relative position data; and (ii) generating an overlay image; A processor configured to perform the following actions; and (D0) A display control unit configured to display the overlay image at the indicated position on the display; A display device equipped with the following features. C001. A controller for controlling the image output of a display device, (1) An input interface for detecting user input; (2) A sensor module that generates attitude and operation data and relative position data of the controller relative to the display device based on user operations detected by the input interface; and (3) A communication unit that wirelessly transmits the generated posture and motion data and relative position data to the display device; A controller equipped with the following features. A101. A method for wirelessly controlling an image on a display using a controller, (a) Displaying a circuit board image on the display; (b) Wirelessly obtaining attitude and motion data of the controller and relative position data of the display from the controller; (c) Determining the indicated position within the display coordinate system based on the posture and motion data and the relative position data; (d) generating an overlay image, The size of the overlay image is determined based on the relative position data of the controller, and a portion of the base image is enlarged based on the indicated position to generate the overlay image including the enlarged portion; and (e) Displaying the generated overlay image superimposed on the indicated position of the substrate image on the display, A method for providing this. B101. A display device that is wirelessly controlled by a controller, (A) A display screen for showing images; (B) A communication unit that wirelessly receives controller orientation and operation data and the controller's relative position data with respect to the display device from the controller; (C) Processor, (i) to identify the indicated position in the display coordinate system based on the received attitude and motion data and relative position data; and (ii) Determining the size of the overlay image based on the relative position data, and generating the overlay image by enlarging the portion of the base image corresponding to the indicated position; A processor configured to perform the following actions; and (D) A display control unit configured to superimpose the generated overlay image onto the indicated position on the base image; A display device equipped with the following features. C011. A controller described in C001 or any of the embodiments thereof, The aforementioned sensor module is A posture and motion measurement module for measuring the posture and motion data of the controller; and A distance sensor module for measuring the relative position of the controller with respect to the display device; Equipped with, controller. A controller described in C012. C011 or any embodiment thereof, The aforementioned attitude and motion measurement module includes a 9-axis sensor, controller. C013. A controller described in C011 or any embodiment thereof, The distance measurement module includes a laser distance sensor. controller. C021. A controller described in C001 or any of the embodiments thereof, The controller has an elongated shape and is configured to be used with its longitudinal direction facing the display device. controller. A controller according to C022. C021 or any of the embodiments thereof, The controller has a cylindrical outer shape, is hollow in the longitudinal direction, and is open at both ends. controller. A controller according to C023. C022 or any embodiment thereof, A controller further comprising an operating ring on the outer circumference of the cylindrical controller, configured to control the operation of the display device by rotational operation. C031. A controller according to C001 or any of the embodiments, The input interface further comprises a controller having a position reset button configured to, when operated, display a target (reset) cursor on the screen of the display device and reset the initial position of the controller. A011. A method of A001 or any embodiment thereof, The above (b) is, (b0) Includes obtaining initial attitude and operation data of the controller and initial relative position data with respect to the display; The above (c) is, (c0) Includes determining the initial indicated position in the display coordinate system based on the initial posture and motion data and the initial relative position data; method. A012. A method described in A011 or any embodiment thereof, The above (b) is, The system further comprises displaying a target (reset) cursor on the display and requesting the user to hold the controller pointed towards the target (reset) cursor. method. A013. A method according to A012 or any embodiment thereof, The above (b) is, The attitude and motion data and relative position data of the controller while the controller is held toward the target (reset) cursor are acquired as the initial attitude and motion data and the initial relative position data. Equipped with, method. A015. A method according to A011 or any embodiment thereof, The above (b) is, (b1) After determining the initial instruction position, the current posture and motion data of the controller and the current relative position data with respect to the display are acquired in real time, The above (c) is, (c1) The process includes determining the current indicated position in the display coordinate system based on the initial posture and motion data, the initial relative position data, the current posture and motion data, and the current relative position data; The above (d) is, (d1) generating an overlay image, The process includes determining the size of the overlay image based on the current relative position data of the controller, further enlarging a portion of the base image based on the current indicated position, and generating the overlay image including the enlarged portion. The above (e) is, (e0) The generated overlay image is superimposed in real time onto the indicated position of the substrate image on the display and displayed, method.

[0115] This disclosure provides, as another embodiment, a method for wirelessly controlling an image on a display using a controller, To detect the distance between the controller and a predetermined area which is a portion of the image on the display; Depending on the detected distance, the image of the predetermined region is zoomed in (magnification increased) or zoomed out (magnification decreased). This includes forms of control methods that include the following: Zooming in and out may be performed using the overlay processing described above or a zoomable GUI (Graphical User Interface). When scanning and displaying an image, a method may be used to display only a predetermined area (e.g., an overlay image including the indicated position) at a magnification corresponding to the detected distance. The predetermined area is, for example, the area corresponding to the circular area of ​​the overlay image 823a in Figure 8B. In addition to the information from the operation ring, the magnification may also be changed according to the distance detected by the distance sensor built into the controller. Furthermore, when the user aligns the controller's pointer with the display and captures an image of the predetermined area, the captured image may be displayed on the display, or the captured image may be transferred to an external electronic device.

[0116] This disclosure also provides, as another embodiment, a controller or pointing device for controlling the image output of a display device, (1) An input interface for detecting user input; (2) A sensor module for detecting the distance between the controller and a predetermined area of ​​the display; (3) The controller or pointing device includes a communication unit that wirelessly transmits data to the display device to zoom in (enlarge) or zoom out (reduce) the image of the predetermined area according to the distance detected by the sensor module.

[0117] This disclosure also represents another implementation form, A display device that is wirelessly controlled by a controller, (1) A display screen for displaying images; (2) The controller includes a sensor module for detecting the distance between the controller and a predetermined area of ​​the display device, and a communication unit for wirelessly receiving the distance detected by the sensor module from the controller; (3) A display control unit that zooms in (enlarges) or zooms out (reduces) the image of the predetermined area according to the distance detected by the sensor module transmitted from the communication unit; This includes the form of a display device equipped with the following:

[0118] This disclosure also represents another implementation form, An image system comprising a display device and a controller for wirelessly controlling the image output of the display device, (1) The controller An input interface that detects user actions; A sensor module for detecting the distance between the controller and a predetermined area of ​​the display; A communication unit that transmits distance data detected by the sensor module to the display device; Equipped with, (2) The display device is An interface for receiving the distance detected by the sensor module from the communication unit, This includes an image system configuration that includes a processing module that zooms in (enlarges) or zooms out (reduces) the image of a predetermined region according to the detection distance received by the interface.

[0119] This disclosure also includes embodiments and examples shown in Figures 10A to 10P. [Summary of the Invention] (Advantages, Effects, Differences from the Prior Art, Applicable Fields) This invention relates to a sensor-equipped pointing device (remote controller) that functions as a remote controller for a display. This device achieves high-precision pointing operations through a combination of built-in accelerometer, angular velocity sensor, and distance sensor. The distance sensor measures the distance between the device and the display, and the angular velocity sensor detects the device's angle. Based on this information, pointing operations corresponding to pixel coordinates on the display are possible. Furthermore, operability is improved by detecting the device's forward and backward movement using the accelerometer and adjusting the sensitivity of the distance sensor. One of the advantages of this invention is that it allows for more precise control of the display than conventional remote controllers. Furthermore, the correction of spatial coordinates using a group of sensors provides a more intuitive and smoother operating experience. This technology is expected to have applications in various fields, including television, presentations, and interactive exhibits. Compared to conventional remote controllers, this invention offers superior advantages in both operability and precision, and is expected to significantly contribute to improving the user interface. [Conventional technology and its disadvantages] Traditional pointing operations required sensor mechanisms on both the display and the remote control device. For example, the Nintendo Wii remote used a sensor bar attached to the TV. VR devices also employ methods such as placing a camera sensor on the main unit and an infrared LED on the controller. However, achieving accurate pointing on a display with a pointing device alone was difficult. Furthermore, with conventional, simple pointing devices that only use acceleration and angular velocity, the degree of movement on the screen in response to the tilt of the pointing device did not change even when the distance between the device and the display changed. For a user to have a natural experience, the coordinates on the screen pointed to by pointing should respond less to changes in angle when the pointing device is close to the screen, and more when it is farther away. In conventional devices, the change in coordinates due to pointing operations did not correlate with the distance to the screen, which caused discomfort to the user. This was a major challenge in terms of device usability and user experience. This invention enables distance-correlated pointing operations using only a pointing device. Since the receiving end does not require sensors or markers, the number of components is reduced, leading to lower manufacturing costs. [Method of Implementation - Invention 1] Figure 10A shows an overview of this disclosure. Figure 10B shows the internal blocks of the pointing device. Figure 10C shows the internal blocks of the display. Figure 10D shows the overall process flow. 10.1 is a pointing device, and 10.2 is a display with a built-in computer. The internal block of the pointing device is shown in Figure 10B, and the block diagram of the display is shown in Figure 10C. The overall processing flow is shown in Figure 10D, and the detailed procedure is shown below. 1. Connect the pointing device (10.1) to a computer (10.2), etc., wirelessly or otherwise. 2. Calculate unit quaternions from the acceleration and angular velocity values ​​of the sensors within the pointing device, or use unit quaternions if they are output from the sensors. There are several known methods for calculating unit quaternions from sensor angular velocity and acceleration, such as the Madgwick filter, and these can be used. 3. Calculate the rotation matrix from the unit quaternion. The unit quaternion is Q(q r, q i , q j , q k If we set it as such, the rotation matrix C can be calculated by the following formula [Number] 4. Convert from the quaternion rotation matrix to the rotation angles (roll φ, pitch θ, yaw ψ) of the pointing device. (Describe the flow diagram of the overall signal flow.) The conversion formula is as follows. [Number] 5. Calculate the differential distances (d h , P v ) from the origin based on the rotation angles, the distance d between the pointing device and the display, the screen size (W, H), and the number of pixels (P x ,d y ). When calculating the differential distances, the issue regarding discomfort is solved by correlating the distance d in a proportional relationship First, the number of pixels per unit length (p x , p y ) can be calculated as p x = P h / W (1) p y = P v / H (2) From these, the differential distances can be calculated as follows. d x = p x * d * tan(ψ) (3) d y = p y * d * tan(θ) (4) 6. At this time, the distance d is obtained from the distance sensor in the pointing device. If the variation of its value is large, correct the value (the correction method is shown below) 7. Set the center of the screen, etc. as the origin (0,0), and at the origin, (d x ,d yThe value obtained by adding () to the pointer cursor coordinates (x,y) pointed to by the pointing device is used. 8. Repeat these processes (include a flowchart of the overall signal flow). The method for correcting the shake of the distance sensor using acceleration is as follows: (A flowchart of the overall signal flow is included.) 1. Send the values ​​from the accelerometer in the pointing device to the computer. 2. Determine the degree of change in the accelerometer's value along the display direction axis using variance, its threshold, etc. 3. If the acceleration in the display direction changes significantly, the distance sensor value will not be corrected. 4. If the change in acceleration in the display direction is small, the distance sensor values ​​are corrected to reduce the time-series change by processing such as smoothing. (The display's operation is explained using diagrams and flowcharts.) [Further development ideas] Based on the above invention, we believe the following developments are also possible. Invention 2. Support for multiple displays, Figure 10E It's also possible to control multiple displays with a single pointing device. As long as the computer on the reference display knows the relative positions of the multiple displays, the pointer can move across the displays by calculating the distance and angle. Invention 3. Zoomed drawing of the drawing range, Figures 10F, 10G The pointing device has a zoom button, which allows you to zoom in and out of the drawing relative to the pointer coordinates displayed on the screen. The zoomed-in area takes the shape of a circle, and its radius and area scale according to the distance d. (This provides an experience similar to using a telescope or a camera's zoom lens.)

[0120] Additional notes 1 A pointing device for controlling an image on an external display, wherein the pointing device is A sensor that detects the pointing device moving away from the external display, A unit that compresses the three-dimensional movement of the pointer of the pointing device as it moves away from the external display based on the signal from the sensor, A pointing device having a correction unit for correcting the movement of the pointer due to displacement at a long distance. Additional note 2 The pointing device described in Appendix 1, wherein the correction unit is configured to prevent hypersensitive movement due to displacement of the pointing at long distances. Additional note 3 A pointing device for controlling an image on an external display, wherein the pointing device is A distance sensor for measuring the distance between the pointing device and the external display, An angular velocity sensor for detecting the angle of the pointing device, The device is characterized by having an acceleration sensor for detecting the forward and backward movement of the pointing device and adjusting the sensitivity of the distance sensor. Additional note 4 The pointing device described in Appendix 3 is characterized in that it enables pointing-based instruction operations corresponding to pixel coordinates on the display based on detection signals from the distance sensor, the angular velocity sensor, and the acceleration sensor. Additional note 5 The pointing device according to any one of the appendices 1 to 4, wherein the external display and the pointing device each have transmitting and receiving antennas and transmit and receive signals wirelessly. Additional note 6 A unit that receives a sensor signal from the pointing device described in any of the appendices 1 to 4, An external display having a computer that calculates and obtains a unit quaternion based on data received from the receiving unit to calculate the rotation angle, and then calculates the difference distance from the rotation angle, distance, screen size, and number of pixels. Additional note 7 A display system having multiple external displays as described in Appendix 6, wherein the pointing device controls images across the multiple external displays. Additional note 8 A pointing device according to any one of the appendices 1 to 4 having a zoom button, and a display system having a unit that enlarges or reduces a drawing based on the pointer coordinates displayed on the external display by operating the zoom button. Additional note 9 A method for controlling the image on the external display using the pointing device described in any of the appendices 1 to 4, The steps include detecting the movement of the pointing device away from the external display using the aforementioned sensor, The steps include compressing the three-dimensional movement of the pointer of the pointing device as it moves away from the external display based on the detection signal from the sensor, A step of correcting the movement of the pointer due to displacement at a long distance. It is characterized by having the following features. Additional note 10 A method for controlling the image on the external display using the pointing device described in any of the appendices 1 to 4, The steps include measuring the distance between the pointing device and the external display using the distance sensor, The steps include detecting the angle of the pointing device using the angular velocity sensor, The steps include detecting the forward and backward movement of the pointing device using the acceleration sensor and adjusting the sensitivity of the distance sensor. It is characterized by having the following features.

[0121] Unless otherwise specified, the terms and expressions used in this specification and claims should be interpreted openly and not restrictively. For example, terms such as “equipped with” and “include” mean “include without limitation” or “include but not limited to,” and do not exclude additional elements or undescribed configurations. Expressions such as “have” should be interpreted as “have at least.”

[0122] Where multiple elements and configurations listed herein are joined by "and," it is interpreted to mean configurations that include any one or more of them, as well as configurations that include any one or more of them. Similarly, where they are joined by "or," it is interpreted to mean configurations that include any one, any one or more of them, or all of them.

[0123] As used herein, singular terms may be interpreted as including plurals depending on the context. Conversely, plural terms may also include singulars depending on the context. Expressions such as "one" or "a" do not exclude the possibility of including multiple components unless otherwise specified.

[0124] The embodiments and examples disclosed herein are provided as illustrative examples of applicable configurations of the invention and are not intended to limit them. Multiple embodiments can be used independently and in combination with one another. Features or components of different embodiments can be combined partially or entirely.

[0125] Preferred embodiments of the present invention have been shown and described herein, but it will be apparent to those skilled in the art that such embodiments are provided only as examples. The present invention is not intended to be limited by any particular example provided herein. Furthermore, the present invention is not limited to any particular description, configuration, or proportion described in the examples, and is applicable to a wide range of conditions and variables. The scope of the present invention is defined by the claims, and the methods and structures within these claims, as well as their equivalents, are intended to be covered.

Claims

1. A method for wirelessly controlling images on a display screen using a controller, (a) Displaying a circuit board image on the display screen; (b) Wirelessly obtaining attitude and motion data of the controller and relative position data with respect to the display screen from the controller; (c) Determining the indicated position in the display coordinate system based on the posture and motion data and the relative position data; (d) Generating an overlay image; (e) displaying the generated overlay image at the indicated position of the substrate image on the display screen; and (f) In response to the detection of a reset operation on the controller, and in synchronization with the determination of the instruction position, the difference distance from the origin of the display screen is calculated based on the posture and operation data at the time of the reset operation, and based on the difference distance, the position where the longitudinal extension of the controller intersects the display screen at the time of the reset operation is set as a new initial position, and the new initial position is stored in the storage unit of the controller. A method for providing this.

2. A method for wirelessly controlling images on a display screen using a controller, (a) Displaying a circuit board image on the display screen; (b) Wirelessly obtaining attitude and motion data of the controller and relative position data with respect to the display screen from the controller; (c) Determining the indicated position within the display coordinate system based on the posture and motion data and the relative position data; (d) generating an overlay image, The process involves determining the size of the overlay image based on the relative position data of the controller, further enlarging a portion of the base image based on the indicated position, and generating the overlay image including the enlarged portion; and (e) displaying the generated overlay image superimposed on the indicated position of the substrate image on the display screen; and (f) During the zoom execution period in which the magnification of the partial image included in the overlay image is dynamically changed based on the change in the relative position data of the controller or the input to the operation unit provided on the controller, control is performed to reduce the motion detection sensitivity in the X-axis and Y-axis directions for the movement of the indicated position compared to normal. A method for providing this.

3. The method according to claim 1, The above (b) is, (b0) Includes obtaining the initial posture and operation data of the controller and the initial relative position data with respect to the display screen; The above (c) is, (c0) Includes determining an initial indicated position in the display coordinate system based on the initial posture and motion data and the initial relative position data; method.

4. The method according to claim 3, The above (b) is, The system further includes displaying a reset cursor on the display screen and requesting the user to hold the controller pointed towards the reset cursor. method.

5. The method according to claim 4, The above (b) is, The attitude and motion data and relative position data of the controller while the controller is held facing the reset cursor are acquired as the initial attitude and motion data and the initial relative position data. Equipped with, method.

6. The method according to claim 3, The above (b) is, After determining the initial instruction position, the system includes acquiring the current posture and operation data of the controller and the current relative position data with respect to the display screen in real time. The above (c) is, This includes determining the current indicated position within the display coordinate system based on the initial posture and motion data, the initial relative position data, the current posture and motion data, and the current relative position data; The above (d) is, This involves generating an overlay image. The process includes determining the size of the overlay image based on the current relative position data of the controller, further enlarging a portion of the base image based on the current indicated position, and generating the overlay image including the enlarged portion. The above (e) is, This includes superimposing the generated overlay image onto the indicated position of the substrate image on the display screen in real time. method.

7. A display device that is wirelessly controlled by a controller, (A0) Display screen for displaying images; (B0) A communication unit that wirelessly receives controller attitude and operation data and controller relative position data with respect to the display screen from the controller; (C0) processor, (i) to determine the indicated position in the display coordinate system based on the received attitude and motion data and relative position data; and (ii) To generate an overlay image to be displayed at the indicated position; A processor configured to perform the following actions; and (D0) A display control unit configured to display the overlay image at the indicated position on the display screen; Equipped with, A display device configured such that, in response to detection of a reset operation to the controller, the processor calculates the difference distance from the origin of the display screen based on the posture and operation data at the time of the reset operation, in synchronization with the determination of the instruction position, sets the position where the longitudinal extension of the controller intersects the display screen at the time of the reset operation as a new initial position based on the difference distance, and stores the new initial position in the storage unit within the controller.

8. A display device according to claim 7, The aforementioned processor, The size of the overlay image is determined based on the relative position data, and the portion of the base image corresponding to the indicated position is enlarged to generate the overlay image; It is configured to perform, The aforementioned display control unit is The generated overlay image is configured to be superimposed and displayed at the indicated position on the base image. Display device.

9. A controller for controlling the image output to the display screen of a display device, (1) An input interface for detecting user operations; (2) A motion measurement unit that generates the attitude and motion data and relative position data of the controller relative to the display screen based on user operations detected by the input interface; (3) A communication unit that wirelessly transmits the generated posture and motion data and relative position data to the display device; and (4) A memory unit for storing data; Equipped with, The aforementioned controller, Based on the aforementioned posture and motion data and the aforementioned relative position data, the system is configured to identify the indicated position in the display coordinate system. A controller configured to respond to detection of a reset operation via the input interface, and in synchronization with the determination of the indicated position, calculate the difference distance from the origin of the display screen based on the posture and motion data at the time of the reset operation, set the position where the longitudinal extension of the controller intersects the display screen at the time of the reset operation as a new initial position based on the difference distance, and store the new initial position in the storage unit.

10. A controller according to claim 9, The aforementioned motion measurement unit is A posture and motion measurement module for measuring the posture and motion data of the controller; and A distance measuring module for measuring the relative position of the controller with respect to the display device; Equipped with, controller.

11. A controller according to claim 10, The motion measurement module includes a 9-axis sensor, controller.

12. A controller according to claim 10, The distance measurement module includes a laser distance sensor. controller.

13. A controller according to claim 9, The controller has a cylindrical outer shape with a longitudinal direction extending from one end to the other, and a cavity penetrating through the longitudinal direction, and is configured to be used with the longitudinal direction facing the display screen. The device further includes a function that automatically switches from pointing mode to a specific operation mode, which is different from pointing mode and uses the body movements to operate the controller, in response to changes in the controller's posture due to the user's body movements. controller.

14. A controller according to claim 13, The cylindrical outer shape is open at both ends, A controller that automatically switches a specific operating mode in response to the body's movements, including changes in posture and motion data, when viewing the display screen through the cavity.

15. A controller according to claim 14, The cylindrical controller is further provided with an operating ring on its outer circumference, configured to control the operation of the display device by rotation. The operating ring is further equipped with a touch sensor that detects the movement of the fingers of the hand gripping the controller, in addition to rotational operation, and is configured to automatically switch to a specific operating mode based on the detection of a change in a specific contact pattern with respect to the operating ring by the touch sensor, or the detection of a change in the posture and motion data. controller.

16. A controller according to claim 10, The input interface further includes a position reset button configured to, when operated, display a reset cursor on the display screen and reset the initial position of the controller. In response to the detection of the reset operation by operating the position reset button, and in synchronization with the determination of the indicated position, the difference distance from the origin of the display screen is calculated based on the posture and motion data at the time of the reset operation, and based on the difference distance, the position where the longitudinal extension of the controller intersects the display screen at the time of the reset operation is set as a new initial position, and the new initial position is stored in the storage unit. controller.

17. A method for wirelessly controlling images on a display screen using a controller, To detect the distance between the controller and a predetermined area corresponding to the indicated position on the display screen; Zooming in or zooming out of the image of the predetermined region according to the detected distance; and During the period in which the magnification is dynamically changed for zooming in or zooming out, the motion detection sensitivity of the XY axes for determining the indicated position is reduced compared to the normal state: A method for providing this.

18. A controller for controlling the image output to the display screen of a display device, (1) An input interface for detecting user operations; (2) An action measurement unit that detects the distance between the controller and a predetermined area corresponding to the indicated position on the display screen; (3) A communication unit that wirelessly transmits data to the display device to zoom in or zoom out of the image of the predetermined area according to the distance detected by the motion measurement unit; Equipped with, A controller configured to perform control that reduces the motion detection sensitivity of the X and Y axes compared to normal only during the period in which the magnification is dynamically changed according to the aforementioned distance.

19. A display device that is wirelessly controlled by a controller, (1) A display screen for displaying images; (2) A communication unit that wirelessly receives from the controller the distance to a predetermined area corresponding to the indicated position on the display screen, which has been detected by an action measurement unit provided in the controller; (3) A display control unit that zooms in or zooms out of the image of the predetermined area according to the distance received via the communication unit; Equipped with, A display device configured to send a signal to the controller that reduces the motion detection sensitivity of the X and Y axes to a lower level than normal, only during the period when the magnification is dynamically changed in order to perform zoom-in or zoom-out display in response to distance detection.

20. An image system comprising a display device and a controller for wirelessly controlling the image output to the display screen of the display device, (1) The controller An input interface that detects user actions; A motion measurement unit that detects the distance between the controller and a predetermined area corresponding to an indicated position on the display screen; A communication unit that transmits distance data detected by the motion measurement unit to the display device; Equipped with, (2) The display device is An interface for receiving the distance detected by the motion measurement unit from the communication unit, The system includes a calculation module that zooms in or zooms out of the image of a predetermined region according to the distance data received by the interface, The calculation module is configured to perform control that reduces the motion detection sensitivity of the controller's XY axes compared to normal only during the zoom execution period when the magnification is dynamically changing. Image system.

Images