Information display method, processing device thereof and information display system

Abstract

The present application provides an information display method, an information display system and a processing device. The system includes a plurality of light-transmissive displays and a plurality of processing devices. The processing devices are connected and communicated with each other via a gateway. A first processing device is selected from the processing devices according to position information of a user. The first processing device determines line-of-sight information of the user according to the position information and posture information of the user. A second processing device different from the first processing device calculates target coordinates of a target object. The first processing device selects a third processing device from the processing devices according to the line-of-sight information of the user. The third processing device determines display position information of a virtual object according to user coordinates and the target coordinates, and controls one of the displays to display the virtual object according to the display position information. Thus, the calculation efficiency can be improved to avoid delayed display of the virtual-real fusion display service, and the viewing experience of the user is improved.

Description

Technical Field

[0001] This disclosure relates to an information display technology. Background Technology

[0002] With the development of image processing and spatial positioning technologies, the application of transparent displays has gradually gained attention. This type of technology allows displays to be paired with physical objects, supplemented by virtual objects, and to generate interactive experiences according to user needs, enabling information to be presented in a more intuitive way.

[0003] Furthermore, virtual objects associated with physical objects can be displayed at specific locations on a transparent display, allowing users to simultaneously view the physical object and the virtual object superimposed on or to one side of it. For example, by setting up a transparent display on an observation deck, viewers can simultaneously view the landscape and the landscape information provided by the transparent display. However, in some large-scale application scenarios, it may be necessary to use a combination of multiple transparent displays to provide a virtual-real integrated information display service, and the number of physical objects and users is also greater. Therefore, if a single central computing device is used to handle all computing tasks, computational latency may occur due to excessive computational load or other factors, making it impossible to provide real-time virtual-real integrated display services to viewers. Summary of the Invention

[0004] This disclosure relates to an information display method, processing apparatus, and information display system, which can allocate computational tasks to a selected processing device based on the user's location and the user's line of sight.

[0005] In one exemplary embodiment disclosed herein, the information display system includes multiple light-transmitting displays, multiple sensing information acquisition devices, and multiple processing devices. The sensing information acquisition devices acquire user location and posture information, as well as target object location information. The processing devices are respectively associated with the displays and are interconnected and communicate with each other via multiple gateways. A first processing device is selected from the processing devices based on the user's location information, and the first processing device determines the user's gaze information based on the user's location and posture information provided by the sensing information acquisition devices. A second processing device, distinct from the first processing device, performs coordinate transformation based on the target object location information provided by the sensing information acquisition devices to calculate the target object's coordinates. The first processing device selects a third processing device from the processing devices based on the user's gaze information. The third processing device determines the display position information of the virtual object based on the user's coordinates and the target object's coordinates, and controls one of the displays to display the virtual object according to the virtual object's display position information.

[0006] In one exemplary embodiment disclosed herein, the above-described information display method is applicable to an information display system having multiple light-transmitting displays, multiple sensing information acquisition devices, and multiple processing devices, and includes the following steps: Acquiring user location and posture information and target object location information using the sensing information acquisition devices. Selecting a first processing device from the processing devices based on the user's location information. Determining the user's gaze information using the first processing device based on the user's location and posture information provided by the sensing information acquisition devices. Calculating target object coordinates by performing coordinate transformation using a second processing device (different from the first processing device) based on the target object's location information provided by the sensing information acquisition devices. Selecting a third processing device from the processing devices based on the user's gaze information. Determining the display position information of a virtual object using the third processing device based on the user coordinates and the target object coordinates, and controlling one of the displays to display the virtual object according to the virtual object's display position information.

[0007] This disclosure provides an exemplary embodiment of a processing device connected to a light-transmitting display and a sensing information acquisition device, and connected to multiple other processing devices via multiple gateways. The sensing information acquisition device acquires user position and posture information. The processing device includes a memory and a processor. The memory stores data, and the processor is configured to perform the following steps: The sensing information acquisition device determines that the distance between the processing device and the user is less than the distance between each of the other processing devices and the user. Based on the user's position and posture information provided by the sensing information acquisition device, the processing device determines the user's gaze information. Based on the user's gaze information, the processing device is selected, and the user's gaze information is transmitted to one of the processing devices via the gateways. One of the processing devices determines the display position information of a virtual object based on the gaze information, user coordinates, and target object coordinates, and controls the display device or another display connected to the other processing devices to display the virtual object based on the virtual object's display position information. Attached Figure Description

[0008] Figure 1A This is a block diagram of an information display system illustrated according to an exemplary embodiment of this disclosure;

[0009] Figure 1A This is a schematic diagram of an information display system illustrated according to an exemplary embodiment of this disclosure;

[0010] Figure 2 This is a flowchart illustrating an information display method according to an exemplary embodiment of this disclosure;

[0011] Figure 3A This is a schematic diagram illustrating an application scenario of an information display system according to an exemplary embodiment of this disclosure;

[0012] Figure 3B This is a flowchart illustrating an information display method according to an exemplary embodiment of this disclosure;

[0013] Figure 4A This is a schematic diagram illustrating an application scenario of an information display system according to an exemplary embodiment of this disclosure;

[0014] Figure 4B This is a flowchart illustrating an information display method according to an exemplary embodiment of this disclosure;

[0015] Figure 5A and Figure 5B This is a schematic diagram illustrating the estimated line-of-sight position according to the embodiments disclosed herein;

[0016] Figure 6A This is a schematic diagram illustrating an application scenario of an information display system according to an exemplary embodiment of this disclosure;

[0017] Figure 6B This is a flowchart illustrating an information display method according to an exemplary embodiment of this disclosure;

[0018] Figure 7 This is a schematic diagram illustrating an application scenario of an information display system according to an exemplary embodiment of this disclosure;

[0019] Figure 8A This is a schematic diagram illustrating an application scenario of an information display system according to an exemplary embodiment of this disclosure;

[0020] Figure 8B This is a flowchart illustrating an information display method according to an exemplary embodiment of this disclosure;

[0021] Figure 9 This is a block diagram of a processing apparatus illustrated according to an embodiment of the present disclosure.

[0022] Explanation of reference numerals in the attached figures

[0023] 10. Information display system;

[0024] 110_1~110_N: Monitor;

[0025] 120_1~120_N: Sensor information acquisition devices;

[0026] 130_1~130_N: Processing device;

[0027] G1~Gk: Gate devices;

[0028] N1: Network topology;

[0029] Vf1, Vf2: Virtual objects;

[0030] RF1: Reference display object frame;

[0031] U1, U2, U3, U4: Users;

[0032] Obj1, Obj2: Target objects;

[0033] E1, E2, E3, E4, E5, E6: Line of sight information;

[0034] P1~P4: Reference points;

[0035] 901: Memory;

[0036] 902: Processor;

[0037] 903: Transmission component;

[0038] S210~S260, S302~S324, S402~S426, S602~S638, S802~S830: Steps. Detailed Implementation

[0039] Reference will now be made in detail to exemplary embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same component reference numerals are used in the drawings and description to denote the same or similar parts.

[0040] Figure 1A This is a block diagram of an information display system illustrated according to an exemplary embodiment of this disclosure. First... Figure 1A First, we will introduce the various components of the system and their configuration relationships. Detailed functions will be revealed together with the flowcharts of the subsequent example embodiments.

[0041] Please refer to Figure 1A The information display system 10 in this exemplary embodiment may include multiple displays 110_1, 110_2, 110_3, ..., 110_N, multiple sensing information acquisition devices 120_1, 120_2, 120_3, ..., 120_N, and multiple processing devices 130_1, 130_2, 130_3, ..., 130_N. The processing devices 130_1 to 130_N may be wirelessly, wiredly, or electrically connected to the displays 110_1 to 110_N and the sensing information acquisition devices 120_1 to 120_N, respectively. It should be noted that... Figure 1A The example described uses a processing device connected to a display and a sensing information acquisition device, such as processing device 130_1 connected to display 110_1 and sensing information acquisition device 120_1, but this disclosure is not limited to this. In other examples, a processing device may be connected to multiple sensing information acquisition devices or multiple displays.

[0042] The displays 110_1 to 110_N can be used to display information and may include a combination of one or more display devices. The display devices may be, for example, transmissive displays such as liquid crystal displays (LCDs), field-sequential color liquid crystal displays, light-emitting diode (LED) displays, electrohumidification displays, or projection-type transmissive displays.

[0043] The sensing information acquisition devices 120_1 to 120_N can be used to acquire the user's location and posture information. The sensing information acquisition devices 120_1 to 120_N include sensing devices for acquiring user information. In some embodiments, the sensing information acquisition devices 120_1 to 120_N may include at least one image sensor or at least one image sensor combined with at least one depth sensor to acquire image data toward the user located in front of the displays 110_1 to 110_N, thereby performing image recognition and positioning of the user. The aforementioned image sensor may be a visible light sensor or a non-visible light sensor such as an infrared sensor. Furthermore, the sensing information acquisition devices 120_1 to 120_N may also include an optical locator for optical spatial positioning of the user. In some embodiments, the sensing information acquisition devices 120_1 to 120_N can also identify the posture of the user's limbs, torso, and head through various human posture recognition technologies. For example, sensory information acquisition devices 120_1 to 120_N can identify the human skeleton and human feature points based on image data, thereby recognizing the user's posture. Any device or combination thereof that can locate the user's location information and recognize the user's posture information falls within the scope of sensory information acquisition devices 120_1 to 120_N.

[0044] On the other hand, the sensing information acquisition devices 120_1 to 120_N can be used to acquire the position information of target objects in a physical scene. The sensing information acquisition devices 120_1 to 120_N include sensing devices for acquiring target object information. In some embodiments, the sensing information acquisition devices 120_1 to 120_N may include at least one image sensor or at least one image sensor combined with at least one depth sensor to acquire image data toward the target object located behind the display 110_1 to 110_N, thereby performing image recognition and positioning of the target object. The aforementioned image sensor may be a visible light sensor or a non-visible light sensor such as an infrared sensor. Any device or combination thereof capable of locating the position information of a target object falls within the scope of the sensing information acquisition devices 120_1 to 120_N.

[0045] In the embodiments disclosed herein, the image sensor described above can be used to acquire images and includes a camera lens with a lens and a photosensitive component. The depth sensor described above can be used to detect depth information, which can be implemented using active depth sensing technology and passive depth sensing technology. Active depth sensing technology can calculate depth information by actively emitting signals such as light sources, infrared light, ultrasound, and lasers, combined with time-of-flight ranging technology. Passive depth sensing technology can use two image sensors to acquire two images in front of them from different perspectives, and use the parallax of the two images to calculate depth information.

[0046] In some embodiments, the sensing information acquisition devices 120_1 to 120_N can transmit information to the processing devices 130_1 to 130_N via their respective communication interfaces in a wired or wireless manner. The processing devices 130_1 to 130_N are computing devices with computational capabilities. The processing devices 130_1 to 130_N can be deployed within the field of the information display system 10, and can be computing devices respectively built into or connected to the displays 110_1 to 110_N. The processing devices 130_1 to 130_N correspond to the displays 110_1 to 110_N and can be used to control the displays 110_1 to 110_N connected to them. For example, the processing device 130_1 can be used to control the display 110_1 to display content.

[0047] For example, Figure 1B This is a schematic diagram of an information display system according to an exemplary embodiment of this disclosure. For ease of explanation and clarity, Figure 1B The description uses three displays 110_1 to 110_3 and three sensing information acquisition devices 120_1 to 120_3 as examples, but this disclosure is not limited to these. Please refer to... Figure 1B User U1 and target object Obj1 are located at the front and rear of displays 110_1 to 110_3, respectively. In this example, user U1 can view the physical scene of virtual object Vf1 containing target object Obj1 through display 110_2. Virtual object Vf1 can be regarded as augmented reality content augmented based on target object Obj1.

[0048] It should be specifically noted that the processing devices 130_1 to 130_N are interconnected and communicate with each other via multiple gateways G1, G2, ..., Gk. Each gateway G1 to Gk supports a wireless transmission protocol or a wired transmission protocol and can establish a link with a nearby gateway or processing device 130_1 to 130_N. This disclosure does not limit the type of wireless or wired transmission protocol; it can be WiFi, ZigBee, mobile communication, or Ethernet standards, etc. In some embodiments, gateways G1 to Gk can form a network topology N1. However, this disclosure does not limit the number of gateways G1 to Gk or the pattern of the network topology. Each processing device 130_1 to 130_N can be connected to at least one of the gateways G1 to Gk. Through the links between gateways G1 to Gk, the processing devices 130_1 to 130_N can transmit information and communicate with each other via gateways G1 to Gk.

[0049] It should be noted that by linking multiple processing devices 130_1 to 130_N with gateways G1 to Gk, the computational tasks required to display the virtual object Vf1 based on the position and attitude information of user U1 and the position information of target object Obj1 can be distributed among a portion of the processing devices 130_1 to 130_N. This distributed processing architecture improves computational efficiency and avoids delayed display of virtual objects.

[0050] Figure 2 This is a flowchart illustrating an information display method according to an exemplary embodiment of this disclosure. Please also refer to... Figure 1A , Figure 1B as well as Figure 2 ,and Figure 2 The method and process can be derived from Figure 1A and Figure 1B The information display system 10 is used to achieve this.

[0051] In step S210, the location and attitude information of user U1 and the location information of target object Obj1 are acquired using the sensing information acquisition devices 120_1 to 120_N. As mentioned above, the sensing information acquisition devices 120_1 to 120_N are, for example, image sensors, depth sensors, or combinations thereof capable of locating the positions of user U1 and target object Obj1.

[0052] In step S220, a first processing device is selected from processing devices 130_1 to 130_N based on the location information of user U1. In some embodiments, the first processing device is the one among processing devices 130_1 to 130_N that is closest to the location information of user U1. That is, the distance between the first processing device and user U1 is smaller than the distance between each of the other processing devices and user U1. Specifically, at least one of the sensing information acquisition devices 120_1 to 120_N can locate the location information of user U1. Furthermore, since processing devices 130_1 to 130_N are fixedly installed in the field of the information display system 10, the location information of processing devices 130_1 to 130_N is known. Therefore, at least one of the processing devices 130_1 to 130_N can obtain the distance between each processing device 130_1 to 130_N and user U1 based on the location information of user U1 and the known location information of each processing device 130_1 to 130_N. In this way, the first processing device that is closest to the location information of user U1 among multiple processing devices can be selected. It is known that, in response to the dynamic movement of user U1, the first processing device closest to user U1 may change accordingly.

[0053] In step S230, the first processing device determines the user's gaze information E1 based on the position and posture information of user U1 provided by the perception information acquisition devices 120_1 to 120_N. Specifically, after selecting the first processing device, the first processing device can directly obtain the user's position and posture information from one of the connected perception information acquisition devices 120_1 to 120_N, or obtain the user's position and posture information from the gateways G1 to Gk. Thus, the first processing device can identify the gaze information E1 based on the user U1's position and posture information. The gaze information E1 includes a gaze vector.

[0054] In step S240, a second processing device, distinct from the first processing device, performs coordinate transformation on the position information of the target object Obj1 provided by the perception information acquisition devices 120_1 to 120_N to calculate the target object coordinates of Obj1. In other words, the second processing device performs coordinate transformation on the position information of the target object Obj1 (e.g., camera coordinates or image coordinates) provided by at least one of the perception information acquisition devices 120_1 to 120_N to obtain the target object coordinates in the three-dimensional display coordinate system.

[0055] In step S250, the first processing device selects a third processing device from processing devices 130_1 to 130_N based on the user U1's gaze information E1. Specifically, after the first processing device obtains the user U1's gaze information E1, it identifies one of the displays 110_1 to 110_N based on the user U1's gaze information E1, and selects the corresponding third processing device from processing devices 130_1 to 130_N based on one of the displays 110_1 to 110_N. In some embodiments, the first processing device can calculate the viewing angle range corresponding to one of the displays 110_1 to 110_N based on the user U1's position information. Reflecting that the user U1's gaze information falls within the viewing angle range, the first processing device identifies one of the displays 110_1 to 110_N from displays 110_1 to 110_3. Figure 1B For example, the first processing device can calculate the viewing angle range corresponding to the display 110_2 based on the position information of user U1. Since the viewing angle information of user U1 falls within the viewing angle range of display 110_2, it can be determined that the viewing position of user U1 falls on display 110_2.

[0056] In other words, the first processing device can identify the display that the user is looking at based on the user U1's gaze information E1. Since the displays 110_1 to 110_N are respectively controlled by their corresponding processing devices 130_1 to 130_N, the first processing device can select the processing device corresponding to the display that the user is looking at as the third processing device. It should be noted that the first processing device may be the same as or different from the third processing device. More specifically, in a scenario where the displays 110_1 to 110_N are arranged side by side and the processing devices 130_1 to 130_N are respectively located adjacent to their corresponding displays 110_1 to 110_N, when the user looks at the display directly in front of them, the first processing device closest to the user U1 will be the same as the third processing device that the user U1 is looking at; when the user looks at the displays on the left and right sides, the first processing device closest to the user U1 will be different from the third processing device that the user U1 is looking at.

[0057] In some embodiments, the third processing device performs coordinate transformation on the location information of user U1 provided by the sensing information acquisition devices 120_1 to 120_N to calculate the user's coordinates. In other words, the third processing device performs coordinate transformation on the location information of user U1 (e.g., camera coordinates or image coordinates) provided by at least one of the sensing information acquisition devices 120_1 to 120_N to obtain the user's coordinates in the three-dimensional display coordinate system.

[0058] In step S260, the third processing device determines the display position information of the virtual object Vf1 based on the user coordinates and the target object coordinates, and controls one of the displays 110_1 to 110_N to display the virtual object Vf1 according to the display position information of the virtual object Vf1. In some embodiments, the second processing device can transmit the target object coordinates of the target object Obj1 to the third processing device via gateways G1 to Gk. Similarly, if the first processing device is different from the third processing device, the first processing device can also transmit the user U1's gaze information E1 to the third processing device via gateways G1 to Gk. Based on this, the third processing device can determine the display position information of the virtual object Vf1 based on the user coordinates, the gaze information E1, and the target object coordinates. Specifically, the display position information can be regarded as the landing point or area of ​​the user's gaze projected onto the display plane when viewing the target object Obj1. Based on various needs or different applications, the third processing device can determine the actual display position of the virtual object Vf1 according to the display position information, so that the user U1 can see the virtual object Vf1 displayed near the target object Obj1 or see the virtual object Vf1 superimposed on the target object Obj1.

[0059] Therefore, by linking multiple processing devices 130_1 to 130_N through gateways G1 to Gk, this disclosure can distribute the computational workload required to display the virtual object Vf1 to multiple processing devices, thereby significantly improving computational efficiency and avoiding display delays of the virtual object.

[0060] The following examples, in conjunction with the information display system 10, illustrate the implementation of this disclosure, which determines the third processing device based on gaze information, and the single-user and multi-user implementations. For ease of explanation and clarity, subsequent embodiments will be described using the example of three processing devices 130_1 to 130_3 respectively connected to three displays 110_1 to 110_3 and three sensing information acquisition devices 120_1 to 120_3, but this disclosure is not limited thereto. The processing devices 130_1 to 130_3 may be respectively arranged adjacent to the corresponding displays 110_1 to 110_3.

[0061] Figure 3A This is a schematic diagram illustrating an application scenario of an information display system according to an exemplary embodiment of this disclosure. Figure 3B This is a flowchart illustrating an information display method according to an exemplary embodiment of this disclosure. Please also refer to... Figure 3A and Figure 3B In this embodiment, user U1 is located in front of display 110_2 and is looking at display 110_2 located directly in front of user U1.

[0062] The sensing information acquisition device 120_2 can acquire the position and attitude information of user U1 (step S302) and transmit the position and attitude information of user U1 to, for example, processing device 130_2. In response to receiving the position information of user U1, processing device 130_2 can calculate the distance between each processing device 130_1 to 130_3 and the position information of user U1. Furthermore, processing device 130_2 can select a first processing device based on the distance between each processing device 130_1 to 130_3 and the position information of user U1 (step S304). Here, the first processing device is the one among processing devices 130_1 to 130_3 that is closest to the position information of user U1. In this example, it is assumed that processing device 130_2 is the first processing device closest to user U1. That is, in one embodiment, the processing device 130_2 can determine through the sensing information acquisition device 120_2 that the distance between the processing device 130_2 and the user U1 is less than the distance between each of the other processing devices 130_1, 130_3 and the user U1.

[0063] Next, the processing device 130_2 can identify the user U1's gaze information E1 based on the user U1's position and posture information (step S306). More specifically, the processing device 130_2 is selected to calculate the user U1's gaze information E1 and determine which display the user U1's gaze information E1 falls on. In this example, the processing device 130_2 can determine that the user U1's gaze information E1 falls on the display 110_2 based on the user U1's gaze information E1, and select a third processing device based on the display 110_2 on which the gaze information E1 is projected (step S308). In this example, the processing device 130_2 used to control the display 110_2 is the third processing device. That is, the first processing device and the third processing device in this example are the same and are both processing device 130_2. Then, the processing device 130_2 performs coordinate transformation based on the user U1's position information provided by the perception information acquisition device 120_2 to calculate the user's user coordinates (step S310).

[0064] On the other hand, the sensing information acquisition devices 120_1 to 120_3 can acquire the location information of the target object Obj1 (step S312). Since the processing device 130_2 has been selected as the first processing device, the processing device 130_1 or the processing device 130_3 can be selected as the second processing device (step S314). The following description will continue with the example of the processing device 130_3 as the second processing device. The processing device 130_3 can receive the location information of the target object Obj1 and other related information to further process the target object identification related to the target object Obj1 (step S316). Next, the processing device 130_3 performs coordinate transformation based on the location information of the target object Obj1 provided by the sensing information acquisition devices 120_1 to 120_3 to calculate the target object coordinates of the target object Obj1 (step S318), so as to transform the location information of the user U1 and the location information of the target object Obj1 to the same coordinate system. Processing device 130_3 can transmit the target coordinates of target object Obj1 to processing device 130_2, which is a third processing device, via at least one of the gates G1 to Gk (step S320).

[0065] Finally, the processing device 130_2 determines the display position information of the virtual object Vf1 based on the user coordinates and the target object coordinates (step S322), and controls the display 110_2 to display the virtual object according to the display position information (step S324). In this way, the processing device 130_2 can display the virtual object Vf1 using the display position information as a reference.

[0066] Figure 4A This is a schematic diagram illustrating an application scenario of an information display system according to an exemplary embodiment of this disclosure. Figure 4B This is a flowchart illustrating an information display method according to an exemplary embodiment of this disclosure. Please also refer to... Figure 4A and Figure 4B In this embodiment, user U1 is located in front of display 110_3 and is looking at display 110_1 located to the left of user U1.

[0067] The sensing information acquisition device 120_3 can acquire the position and posture information of user U1 (step S402) and transmit the position and posture information of user U1 to, for example, processing device 130_3. Processing device 130_3 can select a first processing device based on the distance between each processing device 130_1 to 130_3 and the position information of user U1 (step S404). In this example, the first processing device is the processing device 130_3 among processing devices 130_1 to 130_3 that is closest to the position information of user U1. Next, processing device 130_3, as the first processing device, can identify the gaze information E2 of user U1 based on the position and posture information of user U1 (step S406). In this example, processing device 130_3 can determine that the gaze information E2 of user U1 falls on the display 110_1 based on the gaze information E2, and select a third processing device based on the display 110_1 projected by the gaze information E2 (step S408). In this example, the processing device 130_1 used to control the display 110_1 is a third processing device. That is, the first processing device and the third processing device in this example are different. The processing device 130_3 can transmit the user U1's gaze information E2 to the processing device 130_1, which is the third processing device, via at least one of the gateways G1 to Gk (step S410).

[0068] On the other hand, the sensing information acquisition devices 120_1 to 120_3 can acquire the location information of the target object Obj2 (step S412). Since the processing device 130_3 has been selected as the first processing device and the processing device 130_1 has been selected as the third processing device, the processing device 130_2 can be selected as the second processing device (step S414). The processing device 130_2 can receive the location information of the target object Obj2 and other related information to further process the target object identification related to the target object Obj2 (step S416). Next, the processing device 130_2 performs coordinate transformation based on the location information of the target object Obj2 provided by the sensing information acquisition devices 120_1 to 120_3 to calculate the target object coordinates of the target object Obj2 (step S418). The processing device 130_2 can transmit the target object coordinates of the target object Obj2 to the processing device 130_1, which is the third processing device, via at least one of the gates G1 to Gk (step S420).

[0069] Processing device 130_1 can receive the position information of user U1 through gateways G1 to Gk or directly from sensing information acquisition device 120_1. Then, processing device 130_1 performs coordinate transformation based on the position information of user U1 to calculate the user's coordinates (step S422). Processing device 130_1 determines the display position information of virtual object Vf2 based on user coordinates, target object coordinates, and line-of-sight information E2 (step S424), and controls display 110_1 to display virtual object Vf2 according to the display position information of virtual object Vf2 (step S426). In this example, processing device 130_3 (i.e., the first processing device) can analyze the line-of-sight information E2 of user U1. Processing device 130_2 (i.e., the second processing device) can process object identification and coordinate transformation of target object Obj2. Processing device 130_1 (i.e., the third processing device) determines the display position information of virtual object Vf2 based on user coordinates and target object coordinates.

[0070] In some embodiments, the first processing device can calculate the viewing angle range corresponding to a certain display based on the position information of user U1. If the viewing angle information of user U1 falls within this viewing angle range, the first processing device can identify the display that user U1 is looking at from displays 110_1 to 110_3.

[0071] In detail, Figure 5A and Figure 5B This is a schematic diagram illustrating the estimated line-of-sight position according to an embodiment of this disclosure. Please refer to... Figure 5A and Figure 5B The width of the display 110_2 is dw. By setting reference points P1 to P4 at known positions in front of the display 110_2, the processing device 130_2 can estimate the lateral offset distance X of the user U1 from the left edge of the display 110_2 based on the pixel positions of the reference points P1 to P4 on the image acquired by the perception information acquisition device 120_2. The reference points P1 to P4 can be any markers, and this disclosure is not limited thereto.

[0072] More specifically, the depth information for reference points P1-P2 is D1, and the depth information for reference points P3-P4 is D2. The ratio of the difference between the X-axis pixel coordinate D1R of reference point P1 and the X-axis pixel coordinate D1L of reference point P2, to the depth information D1, is equal to the ratio of the difference between the X-axis pixel coordinate DuR and the X-axis pixel coordinate DuL, to the depth information D. Similarly, the ratio of the difference between the X-axis pixel coordinate D2R of reference point P3 and the X-axis pixel coordinate D2L of reference point P4, to the depth information D2, is equal to the ratio of the difference between the X-axis pixel coordinate DuR and the X-axis pixel coordinate DuL, to the depth information D. Therefore, since the X-axis pixel coordinates DuR and DuL can be calculated, the lateral offset distance X of user U1 from the left edge of display 110_2 can be determined through, for example, interpolation and based on the user's depth information D and width dw.

[0073] In this way, the line-of-sight angle range θ can be calculated based on the lateral offset distance X, depth information D, and the tangent function. For example... Figure 5B As shown, if user U1's gaze information E3 does not fall within the gaze angle range θ, it means user U1 is looking at the display 110_1 on the left. Conversely, if user U1's gaze information falls within the gaze angle range θ, it means user U1 is looking at the display 110_2.

[0074] Figure 6A This is a schematic diagram illustrating an application scenario of an information display system according to an exemplary embodiment of this disclosure. Figure 6B This is a flowchart illustrating an information display method according to an exemplary embodiment of this disclosure. Please also refer to... Figure 6A and Figure 6B In this embodiment, user U1 is located in front of display 110_2, and the user changes from looking at display 110_2, which is directly in front of user U1, to looking at display 110_1, which is to the left of user U1.

[0075] The sensing information acquisition device 120_2 can acquire the position and posture information of user U1 (step S602) and transmit the position and posture information of user U1 to, for example, processing device 130_2. Processing device 130_2 can select a first processing device based on the distance between each processing device 130_1 to 130_3 and the position information of user U1 (step S604). In this example, the first processing device can be the processing device 130_2 among processing devices 130_1 to 130_3 that is closest to the position information of user U1. Next, as the first processing device, processing device 130_2 can identify the gaze information E1 of user U1 based on the position and posture information of user U1 (step S606). In this example, processing device 130_2 can determine that the gaze information E1 of user U1 falls on the display 110_2 based on the gaze information E1, and select a third processing device based on the display 110_1 projected by the gaze information E1 (step S608). In this example, before the user U1's gaze information changes, the processing device 130_2 used to control the display 110_2 is also a third processing device. Therefore, the processing device 130_2 calculates the user's coordinates (step S610). The processing device 130_2 determines the display position information of the virtual object Vf1 (step S612). The processing device 130_2 controls the display 110_2 to display the virtual object Vf1 (step S614). The detailed operational details of steps S602 to S614 have been described in detail in the foregoing embodiments and will not be repeated here.

[0076] It is particularly important to note that, in response to user U1's turn or head movement, processing device 130_2 detects changes in user U1's gaze information (step S616). In this example, the user's gaze information E1 changes to gaze information E3. After the change in user U1's gaze information, processing device 130_2 determines whether user U1's gaze information E3 still falls within the gaze angle range of one of the displays 110_1 to 110_3 (i.e., display 110_2) (step S618). If the user U1's gaze information E3 does not fall within the viewing angle range of the display 110_2 (step S618 determines no), the processing device 130_2 identifies one of the displays 110_1 to 110_3 (i.e., display 110_1) based on the user's gaze information E3, and selects another third processing device from the processing devices 130_1 to 130_3 based on the other display 110_1 to 110_3 (step S620). In this example, after the user U1's gaze information changes, the processing device 130_1 used to control the display 110_1 is identified as another third processing device. Then, the processing device 130_2 transmits the gaze information E3 to the subsequent processing device 130_1 responsible for display control (step S622).

[0077] On the other hand, the sensing information acquisition devices 120_1 to 120_3 can acquire the position information of the target objects Obj1 and Obj2 (step S624). Since the processing device 130_2 has been selected as the first processing device, the processing device 130_3 can be selected as the second processing device (step S626). The processing device 130_3 can receive the position information of the target objects Obj1 and Obj2 and other related information to further process the target object identification related to the target objects Obj1 and Obj2 (step S628). Next, the processing device 130_3 performs coordinate transformation based on the position information of the target objects Obj1 and Obj2 provided by the sensing information acquisition devices 120_1 to 120_3 to calculate the target object coordinates of the target objects Obj1 and Obj2 (step S630). Processing device 130_3 can transmit the target coordinates of target objects Obj1 and Obj2 to processing device 130_1 and processing device 130_2, which are third processing devices, via at least one of the gates G1 to Gk (step S632).

[0078] Similar to the aforementioned operating principle, processing device 130_1 performs coordinate transformation based on the user U1's position information to calculate the user's coordinates (step S634). Processing device 130_1 determines the display position information of virtual object Vf2 based on the user coordinates, target object coordinates, and line-of-sight information E3 (step S636), and controls display device 110_1 to display virtual object Vf2 according to its display position information (step S638). In this example, the third processing device responsible for display control switches from processing device 130_2 to 130_1 in response to changes in line-of-sight.

[0079] Figure 7 This is a schematic diagram illustrating an application scenario of an information display system according to an exemplary embodiment of this disclosure. Please refer to... Figure 7 When the number of users exceeds one, two of the processing devices 130_1 to 130_3 can serve as the first processing device for calculating line-of-sight information E2 and E4. Figure 7 In this example, since the perception information acquisition device 120_1 detects user U2, the processing device 130_1, which is closest to user U2, will be selected as the first processing device to calculate user U2's gaze information E4. On the other hand, since the perception information acquisition device 120_1 detects user U1, the processing device 130_3, which is closest to user U1, will be selected as the first processing device to calculate user U1's gaze information E2. Furthermore, based on the gaze positions of users U1 and U2 projected onto displays 110_1 and 110_2, processing devices 130_1 and 130_2 can be used to calculate the display position information of virtual objects Obj1 and Obj2, respectively.

[0080] Figure 8A This is a schematic diagram illustrating an application scenario of an information display system according to an exemplary embodiment of this disclosure. Figure 8B This is a flowchart illustrating an information display method according to an exemplary embodiment of this disclosure. Please also refer to... Figure 8A and Figure 8B In this embodiment, user U1 and several other users U3 and U4 are located in front of display 110_3, and user U1 and several other users U3 and U4 are all looking at display 110_1 located on the left side.

[0081] The sensing information acquisition device 120_3 can acquire the position and attitude information of user U1 and multiple other users U3 and U4 (step S802), and transmit the position and attitude information of user U1 and multiple other users U3 and U4 to, for example, the processing device 130_3. In this embodiment, the processing device 130_3 can select a first processing device based on the distance between each processing device 130_1 to 130_3 and the position information of user U1 and multiple other users U3 and U4 (step S804). Specifically, the processing device 130_3 can calculate the distance between processing devices 130_1 to 130_3 and user U1. Similarly, the processing device 130_3 can calculate the distance between processing devices 130_1 to 130_3 and multiple other users U3 and U4 respectively. The processing device 130_3 can find the minimum distance from all the above distances and select the processing device associated with the minimum distance as the first processing device. In this example, the user U1 is at a minimum distance from the processing device 130_3, so the processing device 130_3 is selected as the first processing device.

[0082] Next, the processing device 130_3, as the first processing device, can identify the line-of-sight information E3, E5, and E6 between user U1 and multiple other users U3 and U4 based on the position and posture information of user U1 and multiple other users U3 and U4 (step S806).

[0083] Processing device 130_3 determines whether one of the sensing information acquisition devices 120_1 to 120_3 (i.e., sensing information acquisition device 120_3) simultaneously detects user U1 and multiple other users U3 and U4 (step S808). In response to sensing information acquisition device 120_3 simultaneously detecting user U1 and other users U3 and U4 (step S808 determines yes), processing device 130_3 calculates a common viewing direction based on user U1's gaze information E3 and the gaze information E5 and E6 of other users U3 and U4 (step S810), and selects a third processing device and one of the displays 110_1 to 110_3 from processing devices 130_1 to 130_3 based on the common viewing direction (step S812). In some embodiments, processing device 130_3 may calculate the average value of the components of user U1's gaze information E3 and the gaze information E5 and E6 of other users U3 and U4 in each axial direction to obtain the common viewing direction.

[0084] It should be noted that, in some embodiments, before calculating a common line-of-sight direction, the processing device 130_3 may also determine whether the difference in line-of-sight direction between user U1's line-of-sight information E3 and other users U3 and U4's line-of-sight information E5 and E6 meets a preset condition. Specifically, the processing device 130_3 may determine whether the angular difference between user U1's line-of-sight vector and other users U3 and U4's line-of-sight vectors is less than a threshold value. If so, it can be determined that the difference in line-of-sight direction between user U1's line-of-sight information E3 and other users U3 and U4's line-of-sight information E5 and E6 meets a preset condition, indicating that user U1 and other users U3 and U4 are looking at similar positions.

[0085] Furthermore, in this example, since the common viewing direction falls on the display 110_1, the processing device 130_3 selects a third processing device based on the display 110_1 projected by the common viewing direction. In this example, the processing device 130_1 used to control the display 110_1 is the third processing device. The processing device 130_3 can transmit the common viewing direction to the processing device 130_1, which is the third processing device, via at least one of the gateways G1 to Gk (step S814).

[0086] On the other hand, the sensing information acquisition devices 120_1 to 120_3 can acquire the location information of the target object Obj2 (step S816). Since the processing device 130_3 has been selected as the first processing device and the processing device 130_1 has been selected as the third processing device, the processing device 130_2 can be selected as the second processing device (step S818). The processing device 130_2 can receive the location information of the target object Obj2 and other related information to further process the target object identification related to the target object Obj2 (step S820). Next, the processing device 130_2 performs coordinate transformation based on the location information of the target object Obj2 provided by the sensing information acquisition devices 120_1 to 120_3 to calculate the target object coordinates of the target object Obj2 (step S822). The processing device 130_2 can transmit the target object coordinates of the target object Obj2 to the processing device 130_1, which is the third processing device, via at least one of the gates G1 to Gk (step S824).

[0087] The processing device 130_1 can receive the position information of user U1 through the gateways G1 to Gk or directly from the sensing information acquisition device 120_1. Then, the processing device 130_1 performs coordinate transformation based on the position information of user U1 to calculate the user's coordinates (step S826). The processing device 130_1 determines the display position information of virtual object Vf2 based on the user coordinates, target object coordinates, and common line-of-sight direction (step S828), and controls the display 110_1 to display virtual object Vf2 according to the display position information of virtual object Vf2 (step S830).

[0088] Figure 9 This is a block diagram of a processing apparatus according to an embodiment of the present disclosure. The processing apparatus 900 may be the processing apparatus 130_1 to 130_N of the aforementioned embodiments. Please refer to... Figure 9 The processing device 900 may include a memory 901, a processor 902, and a transmission component 903. The memory 901 may be, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk, or other similar devices, integrated circuits, or combinations thereof. The processor 902 may be, for example, a central processing unit (CPU), an application processor (AP), or other programmable general-purpose or special-purpose microprocessor, digital signal processor (DSP), image signal processor (ISP), graphics processing unit (GPU), or other similar devices, integrated circuits, or combinations thereof. The transmission component 903 is a communication device supporting wired / wireless transmission protocols, such as a combination of a transceiver and an antenna. The processor 902 can execute instructions, program code, or software modules recorded in the memory 901 to implement the information display method disclosed herein.

[0089] The information display method, processing apparatus, and information display system disclosed in this exemplary embodiment can allocate computational resources among multiple processing devices based on the user's position and line of sight, thereby improving computational efficiency and avoiding delayed display in virtual-real fusion display services. As a result, virtual objects can be displayed in real-time and smoothly, significantly enhancing the user's viewing experience.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An information display system, characterized in that, include: Multiple transparent displays; Multiple sensing information acquisition devices are used to acquire the user's location and posture information, as well as the location information of the target object; Multiple processing devices, each corresponding to the display, are interconnected and communicate with each other via multiple gateways. The first processing device is selected from the processing devices based on the user's location information. The first processing device determines the user's gaze information based on the user's location and posture information provided by the perception information acquisition device. In this process, a second processing device, distinct from the first processing device, performs coordinate transformation based on the target object's position information provided by the sensing information acquisition device to calculate the target object's coordinates. In this process, the first processing device selects a third processing device based on the user's gaze information. The third processing device determines the display position information of the virtual object based on the user's coordinates and the target object's coordinates, and controls one of the displays to display the virtual object according to the virtual object's display position information. The first processing device is the one that is closest to the user's location information among the processing devices.

2. The information display system according to claim 1, characterized in that, The first processing device may be the same as or different from the third processing device, and the third processing device calculates the user's coordinates by performing coordinate transformation based on the user's location information provided by the sensing information acquisition device.

3. The information display system according to claim 1, characterized in that, The second processing device transmits the target object coordinates to the third processing device via the gate.

4. The information display system according to claim 1, characterized in that, The first processing device identifies one of the displays based on the user's gaze information, and selects the third processing device from the processing device based on the one of the displays.

5. The information display system according to claim 4, characterized in that, The first processing device calculates the viewing angle range corresponding to one of the displays based on the user's location information. If the user's viewing information falls within the viewing angle range, the first processing device identifies one of the displays.

6. The information display system according to claim 5, characterized in that, In response to changes in the user's gaze information, the first processing device determines whether the user's gaze information still falls within the gaze angle range of one of the displays; and in response to the user's gaze information not falling within the gaze angle range of one of the displays, the first processing device identifies another display based on the user's gaze information, so as to select another third processing device from the processing device based on the other display.

7. The information display system according to claim 1, characterized in that, The first processing device determines whether one of the sensing information acquisition devices simultaneously detects the user and multiple other users. In response to the sensing information acquisition device simultaneously detecting the user and the other users, the first processing device calculates a common gaze direction based on the user's gaze information and the gaze information of the other users, and selects the third processing device and the display from the processing devices based on the common gaze direction.

8. The information display system according to claim 7, characterized in that, The difference in gaze direction between the user's gaze information and the gaze information of other users meets a preset condition.

9. An information display method, characterized in that, The method is applicable to an information display system having multiple light-transmitting displays, multiple sensing information acquisition devices, and multiple processing devices, wherein the method includes: The sensor information acquisition device is used to acquire the user's location and posture information, as well as the location information of the target object; Select a first processing device from the processing devices based on the user's location information; The first processing device determines the user's gaze information based on the user's location and posture information provided by the perception information acquisition device. The target object coordinates are calculated by performing coordinate transformation based on the position information of the target object provided by the sensing information acquisition device through a second processing device different from the first processing device; Selecting a third processing device from the processing device based on the user's gaze information; and The third processing device determines the display position information of the virtual object based on the user's coordinates and the target object's coordinates, and controls one of the displays to display the virtual object according to the display position information. The step of selecting the first processing device from the processing device based on the user's location information includes: The first processing device is selected from the processing devices that is closest to the user's location information.

10. The information display method according to claim 9, characterized in that, The first processing device may be the same as or different from the third processing device, and the method further includes: The third processing device performs coordinate transformation based on the user's location information provided by the sensing information acquisition device to calculate the user's coordinates.

11. The information display method according to claim 9, characterized in that, The method further includes: The target coordinates of the target object are transmitted to the third processing device via the second processing device through a gateway.

12. The information display method according to claim 9, characterized in that, The step of selecting the third processing device from the processing device based on the user's gaze information includes: The first processing device identifies one of the displays based on the user's gaze information; and The third processing device is selected from the processing device according to one of the displays.

13. The information display method according to claim 12, characterized in that, The step of identifying one of the displays based on the user's gaze information through the first processing device includes: The first processing device calculates the viewing angle range corresponding to one of the displays based on the user's location information; and Responding to the user's gaze information falling within the gaze angle range, the first processing device identifies one of the displays from the display.

14. The information display method according to claim 13, characterized in that, The method further includes: In response to changes in the user's gaze information, the first processing device determines whether the user's gaze information still falls within the viewing angle range of one of the displays; and If the user's gaze information does not fall within the viewing angle range of one of the displays, the first processing device identifies another display based on the user's gaze information, and selects another third processing device from the processing device based on the other display.

15. The information display method according to claim 9, characterized in that, The method further includes: The first processing device determines whether one of the sensing information acquisition devices simultaneously detects the user and multiple other users. In response to the sensing information acquisition device, where one of them simultaneously detects the user and the other users, the first processing device calculates a common gaze direction based on the user's gaze information and the gaze information of the other users. The step of selecting the third processing device from the processing devices based on the user's gaze information includes: The third processing device and the display are selected from the processing device based on the common line of sight.

16. The information display method according to claim 15, characterized in that... The difference in gaze direction between the user's gaze information and the gaze information of other users meets a preset condition.

17. A processing apparatus, characterized in that, The device is connected to a light-transmitting display and a sensing information acquisition device, and is connected to multiple other processing devices via multiple gateways. The sensing information acquisition device is used to acquire a user's location and posture information. The processing devices include: Memory, used to store data; and Processor, connected to and configured with the memory: The sensory information acquisition device determines that the distance between the processing device and the user is less than the distance between each of the plurality of other processing devices and the user; The user's gaze information is determined based on the user's location and posture information provided by the perception information acquisition device, wherein, compared with the plurality of other processing devices, the processing device is closest to the user's location information; and Based on the user's gaze information, one of the plurality of processing devices is selected, and the user's gaze information is transmitted to one of the plurality of processing devices via the gateway. One of the plurality of processing devices determines the display position information of the virtual object based on the line-of-sight information, user coordinates, and target object coordinates, and controls the display or another display connected to the other processing devices to display the virtual object based on the display position information of the virtual object.

Images