Display device and webpage display method

Abstract

The embodiment of the invention discloses a display device and a webpage display method. The method comprises the steps that somatosensory coordinates are received; converting the somatosensory coordinates into webpage coordinates; performing track segmentation analysis on the plurality of webpage coordinates and generating a predicted track of the cursor; calculating a first correlation degree and a second correlation degree; according to the first association degree and / or the second association degree, calculating the moving distance, the moving direction, the magnification size and / or the display level of the webpage element; and moving the webpage elements according to the moving direction and distance, displaying the webpage elements in an enlarged size, and / or displaying the webpage elements in a display level. According to the embodiment of the invention, the cursor track is predicted according to the somatosensory coordinates, the correlation degree between the webpage element and the track where the cursor is located and the predicted track is calculated, the webpage element is moved towards the direction of the cursor and the predicted track according to the correlation degree, the size of the webpage element is enlarged, and the hierarchy of the webpage element is set, so that the somatosensory operation distance can be shortened; and the interaction convenience is improved.

Description

Technical Field

[0001] This application relates to the field of display device technology, and in particular to a display device and a web page display method. Background Technology

[0002] When browsing the web on a TV browser, traditional remote controls use directional keys to move the cursor item by item. However, this is very time-consuming for complex or information-heavy web pages, making it difficult to meet users' interactive needs. Motion-sensing interaction technology is gradually being introduced into smart TV systems. Through motion-sensing devices, user gestures or movements can be captured, enabling rapid cursor movement.

[0003] When a user interacts with a motion-sensing device, the TV browser can receive the user's two-dimensional coordinate data from the device, parse and verify this data, calculate the new cursor coordinates, and update the cursor position on the screen. When the user issues a click command through the motion-sensing device, the browser can trigger a click event on the corresponding webpage element based on the cursor's current position. Therefore, when browsing webpages through a motion-sensing device, the browser only supports simple cursor movement and static two-dimensional layouts of webpage elements, resulting in long user motion-sensing operation distances and low interaction efficiency. Summary of the Invention

[0004] Some embodiments of this application provide a display device and a webpage display method, which predicts the cursor trajectory based on motion coordinates, calculates the correlation between webpage elements and the cursor's trajectory and the predicted trajectory, moves the webpage elements in the direction of the cursor and the predicted trajectory based on the correlation, enlarges the size of the webpage elements, and sets the hierarchy of webpage elements, which can shorten the motion operation distance and improve the convenience of interaction.

[0005] In a first aspect, some embodiments of this application provide a display device, including: The monitor is configured to display a web page, which includes a cursor and multiple web page elements. A communication device is configured to establish a communication connection with a motion-sensing device; The controller is configured as follows: Receive multiple motion coordinates sent by the motion sensing device during user activity; Multiple motion coordinates are mapped onto the coordinate system of the webpage to obtain multiple webpage coordinates; Trajectory segmentation analysis is performed on multiple web page coordinates to obtain trajectory features of multiple segmented trajectories. The segmented trajectory is the movement trajectory of the cursor between multiple consecutively obtained web page coordinates, and the trajectory features are used to characterize the movement information of the cursor moving according to the segmented trajectory. Based on the trajectory characteristics of multiple segmented trajectories and the trajectory characteristics of the user's historical operations, a predicted trajectory of the cursor is generated. Calculate the first correlation degree between the web page element and the target segment trajectory, and calculate the second correlation degree between the web page element and the predicted trajectory. The target segment trajectory is the segment trajectory where the cursor is located. Based on the first relevance and / or the second relevance, calculate the movement distance and direction of the web page element, the magnified size and / or the display level, whereby the display level is used to characterize the display priority of the web page element; Move web page elements according to the direction and distance of movement to enlarge the size of the web page elements, and / or display web page elements to show hierarchy.

[0006] The above technical solution has the following advantages or beneficial effects: predicting the cursor trajectory based on the motion coordinates, calculating the correlation between the web page element and the cursor's trajectory and the predicted trajectory, moving the web page element in the direction of the cursor and the predicted trajectory based on the correlation, enlarging the size of the web page element, and setting the hierarchy of the web page element can shorten the motion operation distance and improve the convenience of interaction.

[0007] In some embodiments, the controller performs a calculation of the first correlation degree between the web page element and the target segmented trajectory, which is further configured to: Calculate the first shortest distance and the area of ​​the first intersection region, and obtain the matching value corresponding to the element type of the web page element. The first shortest distance is the shortest distance between the edge point of the web page element and the target segment trajectory. The area of ​​the first intersection region is the minimum area of ​​the closed region enclosed by the display area of ​​the web page element and the target segment trajectory. The matching value is generated based on the number of times or frequency of the user selects the element type within a preset period. The first correlation degree is obtained by weighting the first shortest distance, the area of ​​the first intersection region, and the matching value.

[0008] The above technical solution has the following advantages or beneficial effects: it simultaneously considers the first shortest distance, the area of ​​the first intersection region, and the user behavior matching value, avoiding misjudgments caused by a single indicator. By weightedly fusing the first shortest distance, the area of ​​the first intersection region, and the matching value, the first relevance more comprehensively reflects the user's focus.

[0009] In some embodiments, the controller performs a calculation of a second correlation between web page elements and the predicted trajectory, which is further configured to: Calculate the second shortest distance and the area of ​​the second intersection region. The second shortest distance is the shortest distance between the edge point of the web page element and the predicted segmented trajectory. The area of ​​the second intersection region is the minimum area of ​​the closed region enclosed by the display area of ​​the web page element and the predicted trajectory. The second correlation degree is obtained by weighting the second shortest distance, the area of ​​the second intersection region, and the matching value.

[0010] The above technical solution has the following advantages or beneficial effects: it simultaneously considers the second shortest distance, the area of ​​the second intersection region, and the user behavior matching value, avoiding misjudgments caused by a single indicator. By weightedly fusing the second shortest distance, the area of ​​the second intersection region, and the matching value, the second correlation degree more comprehensively reflects the user's focus.

[0011] In some embodiments, the controller performs the calculation of the movement distance and direction of the web page element based on a first relevance and a second relevance, which is further configured to: If the first relevance is greater than or equal to the second relevance, calculate the first endpoint position of the web page element based on the first relevance and the first shortest distance; Get the current position of a webpage element; Calculate the distance and direction of movement based on the first endpoint location and the current location.

[0012] The above technical solution has the following advantages or beneficial effects: when the first relevance is greater than or equal to the second relevance, the web page element can be moved in the direction of the cursor. Based on the first relevance and the first shortest distance, the endpoint position of the web page element can be accurately calculated, and combined with the current position, the movement distance and direction can be determined, thereby achieving the effect of the web page element snapping to the cursor.

[0013] In some embodiments, the controller performs the calculation of the movement distance and direction of the web page element based on a first relevance and a second relevance, which is further configured to: If the first relevance is greater than or equal to the second relevance, calculate the second endpoint position of the web page element based on the second relevance and the second shortest distance. Get the current position of a webpage element; Calculate the distance and direction of movement based on the second endpoint location and the current location.

[0014] The above technical solution has the following advantages or beneficial effects: when the first correlation degree is less than the second correlation degree, the web page element can be moved in the direction of the predicted trajectory. Based on the second correlation degree and the second shortest distance, the endpoint position of the web page element can be accurately calculated, and combined with the current position, the moving distance and moving direction can be determined, thereby achieving the effect of the web page element adhering to the predicted trajectory.

[0015] In some embodiments, the controller performs the calculation of the enlarged size of the web page element based on a first relevance and a second relevance, which is further configured to: If the first degree of relevance is greater than or equal to the second degree of relevance, the enlarged size of the web page element is calculated based on the magnification ratio corresponding to the first degree of relevance and the original size of the web page element. If the first degree of relevance is less than the second degree of relevance, the enlarged size of the web page element is calculated based on the magnification ratio corresponding to the second degree of relevance and the original size of the web page element.

[0016] The above technical solution has the following advantages or beneficial effects: The scaling ratio of web page elements is determined by the highest current relevance, which can prevent visual clutter caused by simultaneous scaling of web page elements. Calculating the magnification size based on the scaling ratio corresponding to the relevance ensures that the higher the relevance, the larger the scaling ratio, and the more prominent the web page element becomes.

[0017] In some embodiments, the controller performs the calculation of the hierarchy of web page elements based on a first relevance and a second relevance, and is further configured to: Z-axis depth information is obtained from motion coordinates, with the Z-axis established based on the orientation of the display device; User actions are identified based on Z-axis depth information; If the first degree of relevance is greater than or equal to the second degree of relevance, and the user's operation behavior is to move closer to the display device, the sum of the original level of the web page element and the level increment corresponding to the first degree of relevance is determined as the display level. If the user's operation is away from the display device, the difference between the original level of the web page element and the level increment corresponding to the first degree of relevance is determined as the display level; If the first relevance is less than the second relevance, and the user's action is to move closer to the display device, the sum of the original level of the web page element and the level increment corresponding to the second relevance is determined as the display level. If the user's action is to move away from the display device, the difference between the original level of the web page element and the level increment corresponding to the second degree of relevance is determined as the display level.

[0018] The above technical solution has the following advantages or beneficial effects: Based on the user's approach / distance movement in three-dimensional space, combined with the layer increment corresponding to the current optimal relevance, the display layer of web page elements is dynamically adjusted, which can highlight the most relevant web page elements, suppress interference from other web page elements, and achieve the effect of displaying elements near the cursor or predicted trajectory in front.

[0019] In some embodiments, the trajectory features include feature points, user actions, and motion parameters. Motion parameters include the cursor's direction vector, velocity, and acceleration. Feature points include a start point, an end point, velocity change points, and a resting point. User actions include moving closer to or further away from the display device. The controller performs segmented analysis on multiple webpage coordinates to obtain trajectory features for multiple segmented trajectories, which are further configured as follows: Multiple webpage coordinates are segmented to obtain multiple segmented trajectories, and the distance the cursor moves in each segmented trajectory is the target length. Based on the webpage coordinates corresponding to the segmented trajectory, feature points and user operation behavior are obtained, and the direction vector, movement speed and acceleration of the cursor in the segmented trajectory are calculated.

[0020] The above technical solution has the following advantages or beneficial effects: obtaining the feature points of the segmented trajectory and user operation behavior, as well as the direction vector, movement speed and acceleration of the cursor in the segmented trajectory, helps to accurately predict the future trajectory of the cursor.

[0021] In some embodiments, the controller generates a predicted trajectory for the cursor based on the trajectory characteristics of multiple segmented trajectories and the trajectory characteristics of the user's historical operations, which is further configured to: The predictive trajectory of the cursor is generated by using a rule engine based on the trajectory characteristics of multiple segmented trajectories and the trajectory characteristics of the user's historical operations.

[0022] The above technical solution has the following advantages or beneficial effects: by matching and reasoning the trajectory features of the current segmented trajectory with the user's past operating habits through the rule engine, the possible future movement path of the cursor can be accurately predicted in advance.

[0023] Secondly, some embodiments of this application provide a webpage display method, including: Receive multiple motion coordinates sent by the motion sensing device during user activity; Multiple motion coordinates are mapped onto the coordinate system of the webpage to obtain multiple webpage coordinates; Trajectory segmentation analysis is performed on multiple web page coordinates to obtain trajectory features of multiple segmented trajectories. The segmented trajectory is the movement trajectory of the cursor between multiple consecutively obtained web page coordinates, and the trajectory features are used to characterize the movement information of the cursor moving according to the segmented trajectory. Based on the trajectory characteristics of multiple segmented trajectories and the trajectory characteristics of the user's historical operations, a predicted trajectory of the cursor is generated. Calculate the first correlation degree between the web page element and the target segment trajectory, and calculate the second correlation degree between the web page element and the predicted trajectory. The target segment trajectory is the segment trajectory where the cursor is located. Based on the first relevance and / or the second relevance, calculate the movement distance and direction of the web page element, the magnified size and / or the display level, whereby the display level is used to characterize the display priority of the web page element; Move web page elements according to the direction and distance of movement to enlarge the size of the web page elements, and / or display web page elements to show hierarchy.

[0024] The above technical solution has the following advantages or beneficial effects: predicting the cursor trajectory based on the motion coordinates, calculating the correlation between the web page element and the cursor's trajectory and the predicted trajectory, moving the web page element in the direction of the cursor and the predicted trajectory based on the correlation, enlarging the size of the web page element, and setting the hierarchy of the web page element can shorten the motion operation distance and improve the convenience of interaction.

[0025] This application embodiment can receive multiple motion coordinates sent by a motion-sensing device during user activity, perform coordinate transformation and trajectory segmentation analysis to generate trajectory features of multiple segmented trajectories, and then generate a predicted cursor trajectory based on the trajectory features of the multiple segmented trajectories and the trajectory features of the user's historical operations. The correlation degree between web page elements and the segmented trajectory where the cursor is located and the predicted trajectory is calculated. Based on the correlation degree, the movement distance and direction of movement, the enlarged size and / or display level of the web page elements are calculated, and the web page elements are arranged using this data. This application embodiment can predict the cursor trajectory based on motion coordinates, calculate the correlation degree between web page elements and the trajectory where the cursor is located and the predicted trajectory, move the web page elements in the direction of the cursor and the predicted trajectory based on the correlation degree, enlarge the size of the web page elements, and set the display level of the web page elements, which can shorten the motion-sensing operation distance and improve the convenience of interaction. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application; Figure 2 This is a schematic diagram of the hardware configuration of a display device provided in some embodiments of this application; Figure 3 This is a schematic diagram of the software configuration of a display device provided in some embodiments of this application; Figure 4 A flowchart illustrating a webpage display method provided in some embodiments of this application; Figure 5 A flowchart illustrating another webpage display method provided in some embodiments of this application; Figure 6 A schematic diagram of a webpage provided for some embodiments of this application; Figure 7 A schematic diagram comparing the first type of webpage display provided for some embodiments of this application; Figure 8 A schematic diagram comparing a second type of webpage display provided for some embodiments of this application; Figure 9 A schematic diagram comparing a third type of webpage display provided in some embodiments of this application; Figure 10 A schematic diagram comparing a fourth type of webpage display provided in some embodiments of this application; Figure 11 A schematic diagram comparing a fifth type of webpage display provided in some embodiments of this application; Figure 12 A schematic diagram comparing a sixth type of webpage display provided in some embodiments of this application; Figure 13 A schematic diagram comparing a seventh type of webpage display provided in some embodiments of this application; Figure 14 A flowchart illustrating yet another webpage display method provided in some embodiments of this application; Figure 15 A system architecture diagram provided for some embodiments of this application; Figure 16 This is a timing diagram illustrating a webpage display method provided in some embodiments of this application. Detailed Implementation

[0028] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0029] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0030] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0031] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0032] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0033] In this embodiment, display device 200 generally refers to a device with screen display and data processing capabilities. For example, display device 200 includes, but is not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, and augmented reality devices. Display device 200 may also include projection devices.

[0034] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application. For example... Figure 1 As shown, users can operate the display device 200 via touch operation, mobile terminal 300, and control device 100. For example, control device 100 can be a remote control, stylus, gamepad, etc.

[0035] The mobile terminal 300 can function as a control device for human-computer interaction between the user and the display device 200. It can also function as a communication device for establishing a communication connection with the display device 200 and exchanging data. In some embodiments, the mobile terminal 300 can have software applications installed on it and communicate with the display device 200 via network communication protocols to achieve one-to-one control and data communication. Furthermore, it can transmit audio and video content displayed on the mobile terminal 300 to the display device 200 for synchronized display.

[0036] like Figure 1 The diagram also shows that the display device 200 communicates with the server 400 via various communication methods. This allows the display device 200 to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0037] Display device 200 can provide broadcast television reception function, and can also be equipped with intelligent network television function that provides computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0038] Figure 2 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of display device 200.

[0039] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface 280.

[0040] In some embodiments, detector 230 is used to acquire signals from the external environment or to interact with the outside world. For example, detector 230 includes a light receiver, a sensor for acquiring ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to acquire external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0041] In some embodiments, the display 260 includes display function components for presenting images and driving components for driving image display. The display 260 is used to receive and display image signals output from the controller 250. For example, the display 260 can be used to display video content, image content, menu control interface components, and user control UI interfaces, etc.

[0042] In some embodiments, the communication device 220 is a component used to communicate with external devices or the server 400 according to various communication protocol types. The display device 200 may have multiple communication devices 220 depending on the supported communication methods. For example, when the display device 200 supports wireless network communication, the communication device 220 may include a WiFi module. When the display device 200 supports Bluetooth connection communication, the communication device 220 may include a Bluetooth module.

[0043] The communication device 220 enables the display device 200 to communicate with external devices or the server 400 via wireless or wired connections. Wired connections utilize data cables, interfaces, or other components to connect the display device 200 to external devices. Wireless connections utilize wireless signals or wireless networks. The display device 200 can directly establish a connection with external devices or indirectly through gateways, routers, or other connection devices.

[0044] In this embodiment, the display device 200 can establish a communication connection with the motion-sensing device via a communication device. The communication connection can be Bluetooth, WiFi, USB, or a dedicated wireless protocol connection. A motion-sensing device refers to a hardware device capable of sensing a user's natural behaviors such as body movements, postures, or voice, and converting them into electronic signals or commands for interaction with the display device 200. Motion-sensing devices may include motion-sensing remote controls, motion-sensing cameras, gamepads, mobile terminals, and wearable devices. Wearable devices include smartwatches, etc.

[0045] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200.

[0046] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0047] In some embodiments, a user can input user commands through a graphical user interface (GUI) displayed on a display 260, and the user input interface receives user input commands through the graphical user interface (GUI).

[0048] In some embodiments, the audio output device 270 can be a built-in speaker of the display device 200 or an external audio output device connected to the display device 200. For the external audio output device connected to the display device 200, the display device 200 may also be provided with an external audio output terminal, through which the audio output device can be connected to the display device 200 to output sound from the display device 200.

[0049] In some embodiments, the user input interface 280 can be used to receive instructions from user input. For example, the user input interface 280 can receive text information entered by the user in the user interface. The user input interface 280 can also receive confirmation instructions from the user regarding controls in the user interface. The user input interface 280 can also receive voice instructions entered by the user.

[0050] In some embodiments, to enable user interaction, the display device 200 may run an operating system. An operating system is a computer program that manages and controls the hardware and software resources of the display device 200. The operating system can control the display device to provide a user interface; for example, the operating system can directly control the display device to provide a user interface, or it can provide a user interface by running applications. The operating system also allows users to interact with the display device 200.

[0051] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system that is deeply customized based on a specific operating platform, or an independent operating system specifically developed for display devices.

[0052] like Figure 3As shown, the display device system is divided into three layers, from top to bottom: the application layer, the middleware layer, and the hardware layer.

[0053] The application layer mainly includes commonly used applications on TV, as well as the application framework. The commonly used applications are mainly browser-based applications, such as HTML5 apps, and native apps.

[0054] An application framework is a complete program model that has all the basic functions required by standard application software, such as file access, data exchange, etc., as well as the user interface for these functions (toolbar, status bar, menu, dialog box).

[0055] Native apps can support online or offline access, push notifications, or access to local resources.

[0056] The middleware layer includes various television protocols, multimedia protocols, and system components. Middleware can use the basic services (functions) provided by system software to connect different parts of application systems or different applications on the network, achieving resource sharing and function sharing.

[0057] The hardware layer mainly includes the HAL interface, hardware, and drivers. The HAL interface is a unified interface for all TV chips, with the specific logic implemented by each chip. The drivers mainly include: audio drivers, display drivers, Bluetooth drivers, camera drivers, Wi-Fi drivers, USB drivers, HDMI drivers, sensor drivers (such as fingerprint sensors, temperature sensors, pressure sensors, etc.), and power drivers.

[0058] It should be noted that the above examples are merely a simple division of operating system functions and do not limit the specific form of the operating system of the display device 200 in this application embodiment. Depending on the function of the display device, the type of operating system, and other factors, the number of levels and the specific level type of the operating system may be expressed in other forms.

[0059] When browsing the web on a TV browser, traditional remote controls use directional keys to move the cursor item by item. However, this is very time-consuming for complex or information-heavy web pages, making it difficult to meet users' interactive needs. Motion-sensing interaction technology is gradually being introduced into smart TV systems. Through motion-sensing devices, user gestures or movements can be captured, enabling rapid cursor movement.

[0060] In some embodiments, such as Figure 4As shown, after receiving a user's instruction to open a browser application on display device 200, a webpage is displayed. After the user interacts with the motion-sensing device, the device sends its two-dimensional coordinate data to the browser application. The browser application processes the two-dimensional coordinate data, i.e., parses and verifies the data, calculates the new cursor position coordinates, and then updates the cursor position on the screen. When the user issues a click command through the motion-sensing device, the browser can trigger a click event on the corresponding webpage element based on the current cursor position. Therefore, when browsing webpages through a motion-sensing device, the browser application only supports simple cursor movement and static two-dimensional layout of webpage elements, resulting in long user motion-sensing operation distances and low interaction efficiency.

[0061] To shorten the user's haptic distance and improve interaction efficiency, this application provides a display device 200. The structure and functions of each part of the display device 200 can be found in the above embodiments. Furthermore, based on the display device 200 shown in the above embodiments, this embodiment further improves some functions of the display device 200. For example... Figure 5 As shown, controller 250 is configured to perform the following steps: Step S501: Receive multiple motion coordinates sent by the motion sensing device when the user is active.

[0062] Upon receiving a user's instruction to open a browser application, the system controls the monitor 260 to display the browser application's page, i.e., a webpage. The webpage includes a cursor and multiple webpage elements. A browser application is a software program that runs within a web browser and is accessed via the internet or a local server. Users can open the browser and enter a URL to display the corresponding webpage.

[0063] Web page elements are the basic units that constitute the content and structure of a web page. Web page elements include text, images, buttons, input boxes, and videos, etc. Web page elements can be displayed independently, or multiple elements can be displayed overlappingly. Each web page element has a hierarchy, with higher-level elements displayed first. The cursor is a visual marker in the user interface of the display device 200 used to indicate the current position or focus of operation. In this embodiment, the cursor position can be updated by motion coordinates sent by a motion-sensing device.

[0064] In some embodiments, taking a motion-sensing device as a motion-sensing remote control as an example, a user can hold the motion-sensing remote control and change its spatial position. The motion-sensing device can continuously send its motion data to the browser application of the display device 200. Alternatively, after detecting a change in the spatial position of the motion-sensing device, the device sends its motion data to the browser application of the display device 200. The motion data includes motion coordinates, as well as auxiliary information such as motion direction, speed, and acceleration.

[0065] In some embodiments, taking a motion-sensing camera as an example, the motion-sensing camera can be built into the display device 200 or externally connected to it. The motion-sensing camera can emit specific light signals, such as infrared light or structured light, into a designated space where the user can move. The camera can receive the light signals reflected back from the user's body and calculate the distance of each pixel using an algorithm to generate a depth map. By combining the RGB image and the depth map, the human skeletal joints can be identified. For example, the cursor position can move according to the position of the user's right hand, sending the right-hand motion data to the browser application on the display device 200.

[0066] The motion-sensing coordinates include three-dimensional coordinate data. In some embodiments, a coordinate system is established with the center of the motion-sensing device or display device 200 itself as the origin, the horizontal direction of the motion-sensing device or display device 200 as the X-axis, the vertical direction of the motion-sensing device or display device 200 as the Y-axis, and the front-back direction of the motion-sensing device or display device 200, i.e., the orientation, as the Z-axis. This coordinate system can be fixed or can change as the motion-sensing device rotates.

[0067] In some embodiments, the motion-sensing device may have built-in sensors such as accelerometers, gyroscopes, and magnetometers, which can be used to detect direction, angular velocity, and acceleration. The motion-sensing device can also send auxiliary information such as motion direction, speed, and acceleration to the browser application of the display device 200.

[0068] Step S502: Map multiple motion coordinates to the coordinate system of the web page to obtain multiple web page coordinates.

[0069] In some embodiments, after receiving the motion coordinates sent by the motion sensing device, the browser application performs noise reduction processing on the motion coordinates. This noise reduction processing can utilize Kalman filtering and moving average methods to eliminate data jitter and outliers, thereby improving the stability of the trajectory data.

[0070] One implementation of mapping multiple motion coordinates or denoised motion coordinates to the coordinate system of a webpage to obtain multiple webpage coordinates may include: obtaining the resolution W×H of the webpage; normalizing the X and Y coordinates in the motion coordinates according to a preset effective user operation range to obtain the corresponding normalized values ​​u and v. Since the Y-axis of the webpage coordinate system points downwards, while the Y-axis of the motion coordinate system usually points upwards, v needs to be inverted: v1 = 1 - v. The normalized values ​​are then scaled to the webpage coordinate system to obtain the webpage coordinates. Specifically, this can be done by using the current webpage resolution to convert (u, v1) into webpage coordinates: pagex = u×W, pagey = v1×H.

[0071] It should be noted that the Z-axis depth information is retained in the internal data structure for subsequent spatial hierarchy operations.

[0072] Step S503: Perform trajectory segmentation analysis on multiple webpage coordinates to obtain trajectory features of multiple segmented trajectories.

[0073] The segmented trajectory refers to the movement trajectory of the cursor among multiple continuously obtained webpage coordinates. The trajectory features are used to characterize the motion information of the cursor moving according to the segmented trajectory. The trajectory features may include feature points, user operation behavior, and the cursor's direction vector, movement speed, and acceleration. Feature points include the start point, end point, speed change point, and dwell point. User operation behavior includes sliding, dwelling, and moving closer to or further away from the display device 200.

[0074] In some embodiments, a method for segmenting and analyzing multiple webpage coordinates to obtain trajectory features of multiple segmented trajectories may include: segmenting multiple webpage coordinates to obtain multiple segmented trajectories; then, based on the webpage coordinates corresponding to the segmented trajectories, acquiring feature points and user operation behaviors, and calculating the cursor's direction vector, movement velocity, and acceleration within the segmented trajectories. Essentially, the segmented trajectories are lines connecting corresponding position points of multiple webpage coordinates.

[0075] In some embodiments, one implementation of segmenting multiple webpage coordinates to obtain multiple segmented trajectories may include: segmenting the multiple webpage coordinates according to a target length, such that the cursor movement distance in each segmented trajectory is the target length. The cursor movement distance is the sum of the distances between corresponding positions of two adjacent webpage coordinates in the segmented trajectory.

[0076] It should be noted that the target length can be fixed or variable. If the number of feature points detected in the segmented trajectory exceeds the first preset number, it indicates that the segmented trajectory requires further detailed analysis, and the target length can be reduced. If the number of feature points detected in the segmented trajectory is less than the second preset number, it indicates that the segmented trajectory does not require detailed analysis, and the target length can be increased. The subsequent webpage coordinates are segmented based on the reduced or increased target length. The first preset number is greater than the second preset number. The distance the cursor moves within the segmented trajectory, i.e., the target segmented trajectory, does not necessarily have to be equal to the target length.

[0077] In some embodiments, one implementation of segmenting multiple webpage coordinates to obtain multiple segmented trajectories may include: segmenting multiple webpage coordinates according to a target duration or a target number, such that the difference in reception time between the start and end points of each segmented trajectory is the target duration, or the number of webpage coordinates in each segmented trajectory is the target number.

[0078] It should be noted that the target duration and target quantity can be fixed or variable. If the number of feature points detected in the segmented trajectory exceeds the first preset number, the target duration or target quantity can be reduced. If the number of feature points detected in the segmented trajectory is lower than the second preset number, the target duration or target quantity can be increased. The subsequent webpage coordinates are segmented based on the reduced or increased target duration or target quantity. The difference in reception time between the start and end points in the segmented trajectory where the cursor is located does not have to be the target duration, and the number of webpage coordinates in the segmented trajectory where the cursor is located, i.e., the target segmented trajectory, does not have to be the target quantity.

[0079] In some embodiments, after determining the segmented trajectory, trajectory features of each segmented trajectory can be extracted. For example, the motion direction, velocity, and acceleration corresponding to each webpage coordinate in the segmented trajectory can be obtained. The mean, median, or total number of all motion directions, velocities, and accelerations can be calculated, and the mean, median, or total number can be used as the direction vector, motion velocity, and acceleration of the segmented trajectory.

[0080] For example, the first point of each segmented trajectory is set as the start point, and the last point is set as the end point. If a specified number of consecutive webpage coordinates are identical in a segmented trajectory, the point corresponding to those webpage coordinates is set as a stop point. If the speed difference between two adjacent webpage coordinates exceeds a threshold, the point corresponding to the preceding webpage coordinate is set as the speed change point.

[0081] For example, if the difference in Z-axis depth information between the start and end points of the segmented trajectory is positive, the user's action is to move closer to the display device 200. If the difference in Z-axis depth information between the start and end points of the segmented trajectory is negative, the user's action is to move away from the display device 200. If there is a stopping point in the segmented trajectory, the user's action is to stay still; if there is no stopping point in the segmented trajectory, the user's action is to slide.

[0082] The trajectory features of multiple segmented trajectories can be summarized into a list of trajectory segments, a set of feature points, motion parameters, and user actions. The list of trajectory segments includes the feature points and motion parameters for each segment. The set of feature points includes all feature points. The user actions include all user actions.

[0083] Step S504: Generate the predicted trajectory of the cursor based on the trajectory characteristics of multiple segmented trajectories and the trajectory characteristics of the user's historical operations.

[0084] In some embodiments, one implementation of generating a predicted cursor trajectory based on the trajectory features of multiple segmented trajectories and the trajectory features of the user's historical operations may include: using a rule engine to generate the predicted cursor trajectory based on the trajectory features of multiple segmented trajectories and the trajectory features of the user's historical operations. The trajectory features of the user's historical operations are the trajectory features corresponding to user operations that were successfully triggered in the past, such as the trajectory features corresponding to a user's hand gesture.

[0085] Multiple rules are constructed using a rule engine based on the trajectory features of the user's historical operations. The rule engine matches the trajectory features of the currently extracted multiple segmented trajectories with all rule conditions. For successfully matched rules, a score is assigned based on confidence (such as historical hit rate and feature fit). The rule with the highest score is selected as the prediction result. The predicted trajectory of the cursor is then generated based on this prediction result.

[0086] For example, a webpage such as Figure 6 As shown, point A is the cursor's starting position, point B is the cursor's current position, and line AB represents the cursor's current trajectory. Line AB is actually divided into multiple segmented trajectories. Using the rule engine, a predicted trajectory, i.e., line BC, can be generated from the trajectory features of these multiple segmented trajectories and the trajectory features of the user's historical operations. Point C is the predicted endpoint of the cursor.

[0087] Step S505: Calculate the first correlation degree between the web page element and the target segmented trajectory, and calculate the second correlation degree between the web page element and the predicted trajectory. The target segmented trajectory is the segmented trajectory where the cursor is located.

[0088] In some embodiments, one implementation of calculating the first correlation degree between a webpage element and a target segmented trajectory includes: calculating a first shortest distance between the webpage element and the target segmented trajectory, and calculating a first correlation degree based on the first shortest distance. The smaller the shortest distance, the greater the correlation degree. The first shortest distance between the webpage element and the target segmented trajectory is the shortest perpendicular distance between an edge point of the webpage element and the target segmented trajectory. It is important to emphasize that the edge point of the webpage element is the edge point of the original position of the webpage element, not the edge point of the webpage element after it has been moved. If the webpage element intersects with the target segmented trajectory or the target segmented trajectory is within the webpage element, the first shortest distance is 0.

[0089] For example, the first shortest distance d between the web page element and the target segment trajectory, and the first correlation R = 1 / (1+d).

[0090] In some embodiments, one implementation of calculating a second correlation between a web page element and a predicted trajectory includes: calculating a second shortest distance between the web page element and the predicted trajectory, and calculating a second correlation based on the second shortest distance.

[0091] In some embodiments, one implementation of calculating the first correlation degree between a web page element and a target segmented trajectory includes: calculating the first shortest distance and the area of ​​the first intersection region between the web page element and the target segmented trajectory; obtaining a matching value corresponding to the type of the web page element from a user habit list; and performing a weighted calculation on the first shortest distance, the area of ​​the first intersection region, and the matching value to obtain the first correlation degree.

[0092] The area of ​​the first intersection region between the webpage element and the target segment trajectory is the minimum area of ​​the closed region enclosed by the webpage element and the target segment trajectory. If the webpage element and the target segment trajectory do not intersect or do not form a closed region, the area of ​​the intersection region can be set to 0.

[0093] The user habit list includes different matching values ​​for different element types. The matching value is generated based on the number or frequency of times the user has historically selected each element type within a preset period. The more times or the higher the frequency, the larger the matching value. For example, if a user selects text, image, input box, and video 5 times, 9 times, 12 times, and 39 times respectively within a week, the matching values ​​for text, image, input box, and video would be 1, 2, 3, and 4 respectively. The matching value can also be calculated by normalizing the number of historical selections.

[0094] For example, the first shortest distance d corresponds to the weight w1, the area of ​​the first intersection region s corresponds to the weight w2, the matching value p corresponding to the type of web page element corresponds to the weight w3, and the first relevance R = 1 / (1+d)×w1+s×w2+p×w3.

[0095] In some embodiments, when calculating the first relevance, the angle α between the target line and the target segmented trajectory can also be introduced as a reference value. The target line is the line connecting the center point of the web page element and the current position of the cursor. For example, the smaller the angle between the target line and the target segmented trajectory, the greater the probability that the web page element will be on the future trajectory of the cursor, and the greater the relevance. An inverse proportional function 1 / (1+a) can be set, and a weight w4 corresponding to the angle can be set and added to the calculation of the first relevance, that is, the first relevance R = 1 / (1+d)×w1 + s×w2 + p×w3 + 1 / (1+a)×w4.

[0096] In some embodiments, when calculating the first correlation degree, the predicted trajectory endpoint can also be introduced as a reference value. It is determined whether the predicted trajectory endpoint is within the display area of ​​the webpage element. If the predicted trajectory endpoint is within the display area of ​​the webpage element, a value b and a weight w5 corresponding to the predicted trajectory endpoint are set and added to the calculation of the first correlation degree, i.e., the first correlation degree R = 1 / (1+d)×w1 + s×w2 + p×w3 + 1 / (1+a)×w4 + b×w5.

[0097] In some embodiments, one implementation of calculating the second correlation degree between a webpage element and a predicted trajectory may include: calculating a second shortest distance and a second intersection area between the webpage element and the predicted trajectory, and then weighting the second shortest distance, the second intersection area, and the matching value to obtain the second correlation degree. The second shortest distance is the shortest distance between the edge point of the webpage element and the predicted segmented trajectory, and the second intersection area is the minimum area of ​​the closed region enclosed by the display area of ​​the webpage element and the predicted trajectory.

[0098] It should be noted that the calculation method for the second degree of correlation is similar to that for the first degree of correlation, and will not be repeated here.

[0099] In some embodiments, a trajectory space model is established. Basic information about the webpage elements, along with the functional expression or trajectory features of the target segmented trajectory or predicted trajectory, are input into the trajectory space model. The trajectory space model then outputs the first and second correlation degrees corresponding to each webpage element. The basic information includes the original position coordinates, original size, and original hierarchy of the webpage elements.

[0100] Step S506: Calculate the movement distance and direction, magnification size, and / or hierarchy of the webpage element based on the first and / or second relevance. The hierarchy is used to characterize the display priority of the webpage element.

[0101] Step S507: Move the web page elements according to the direction and distance of movement to enlarge the size of the web page elements, and / or display the web page elements in a hierarchical manner.

[0102] In some embodiments, an implementation of calculating the movement distance and movement direction of a web page element based on a first relevance may include: calculating a first endpoint position of the web page element based on the first relevance and a first shortest distance, then obtaining the current position of the web page element, and then calculating the movement distance and movement direction based on the first endpoint position and the current position.

[0103] One implementation of calculating the first endpoint position of a webpage element based on a first correlation degree and a first shortest distance may include: obtaining the movement ratio n corresponding to the first correlation degree from a movement ratio data table, and then calculating the predicted distance d1 = d × n based on the movement ratio and the first shortest distance d. The direction from the original position of the webpage element towards the target segmented trajectory is determined as the prediction direction. The first endpoint position of the webpage element is calculated based on the original position, predicted distance, and predicted direction. It should be noted that the predicted distance and predicted direction are the distance and direction assumed for the webpage element to move from its original position. Since the webpage element has already moved according to the first correlation degree at a previous moment, it can be moved from its current position to the first endpoint position. The movement ratio data table includes movement ratios corresponding to different correlation degree ranges.

[0104] The above method allows you to calculate the movement direction and distance for each webpage element, and then move each element in the corresponding direction by the corresponding distance. Alternatively, you can calculate the movement direction and distance only when the initial relevance exceeds a preset relevance. If the initial relevance does not exceed the preset relevance, there is no need to calculate the movement distance, and the webpage element can be displayed directly in its original position.

[0105] In some embodiments, a smooth interpolation algorithm is used to calculate the movement speed of web page elements. The smooth interpolation algorithm incorporates continuity, differentiability, and even acceleration control during the interpolation process, ensuring that the change process is not only continuous but also free from abrupt changes, stutters, or jitter. The smooth interpolation algorithm can use Bézier curves, i.e., curves with initial acceleration and final deceleration, which conform to the human eye's expectation of natural motion and can improve the visual smoothness of web page element movement. If multiple web page elements are close to the cursor, their priority is sorted according to their relevance to prevent layout chaos. Spatial distribution optimization algorithms (such as force-directed layout) can also be used to automatically adjust the spacing between web page elements to avoid occlusion and visual clutter.

[0106] For example, Figure 7 In (a) is Figure 4 The display effect of the technical solution, and Figure 7 (b) shows the effect of a webpage element moving towards the segment trajectory where the cursor is located. The solid line next to the cursor represents the target segment trajectory. The higher the first degree of relevance, the greater the movement ratio.

[0107] In this embodiment, when the cursor approaches a webpage element, the browser can calculate the relevance and the moving distance based on the relevance. The webpage element can then move closer to the cursor, achieving a natural sense of attraction.

[0108] In some embodiments, one implementation of calculating the enlarged size of a webpage element based on a first relevance degree may include: obtaining the enlargement ratio m corresponding to the first relevance degree from an enlargement ratio data table, and then calculating the enlarged size w1 = w0 × m and h1 = h0 × m for each webpage element based on the enlargement ratio and the original size w0 and h0 of the webpage element. The enlarged size of each webpage element can be calculated using this method, and then the webpage element can be displayed at the enlarged size. Alternatively, the enlarged size of the webpage element can be calculated only when the first relevance degree exceeds a preset relevance degree; if the first relevance degree does not exceed the preset relevance degree, there is no need to calculate the enlarged size, i.e., there is no need to enlarge the webpage element. The enlargement ratio data table includes enlargement ratios corresponding to different relevance degree ranges.

[0109] In some embodiments, linear easing technology is used to achieve a smooth zoom-in animation effect for web page elements. When multiple web page elements deform simultaneously, a spatial distribution optimization algorithm automatically adjusts the spacing between the web page elements to avoid occlusion and visual clutter.

[0110] In some embodiments, the transparency of web page elements can also be calculated based on a first correlation degree. Specifically, this involves obtaining the transparency gain ratio p corresponding to the first correlation degree from a transparency gain ratio data table, and then calculating the transparency P1 = P0 × p for each web page element based on the transparency gain ratio and the original transparency P0 of the web page element. The transparency of each web page element can be calculated using this method, and then the web page element can be displayed with that transparency. Alternatively, the transparency of the web page element can be calculated only when the first correlation degree exceeds a preset correlation degree; if the first correlation degree does not exceed the preset correlation degree, no transparency calculation is needed, and the web page element is displayed with its original transparency. The transparency gain ratio data table includes gain ratios corresponding to different correlation degree ranges.

[0111] In some embodiments, the highlight level corresponding to the first degree of relevance is obtained from the highlight level data table, and then the highlight display method corresponding to the highlight level is obtained, and the web page element is displayed in the highlight display method. The highlight display method includes border highlighting, background color change, and shadow glow effect, etc. The highlight level data table includes highlight display methods corresponding to different degree of relevance.

[0112] For example, Figure 8 In (a) is Figure 4 The display effect of the technical solution, and Figure 8 (b) shows the effects of changes in the size, transparency, and highlight of web page elements. The higher the correlation with the segment trajectory where the cursor is located, the greater the magnification ratio, the greater the transparency gain coefficient, and the more obvious the highlight effect.

[0113] According to the embodiment of this application, the size, transparency, highlight and other visual attributes of the web page elements are dynamically adjusted based on the correlation between the cursor trajectory and the web page elements. When the cursor approaches, the web page elements are enlarged and highlighted, and when the cursor moves away, the web page elements return to their original state.

[0114] In some embodiments, one implementation of calculating the hierarchy of a webpage element based on a first degree of relevance may include: obtaining Z-axis depth information from motion coordinates, and then identifying user operation behavior based on the Z-axis depth information. The hierarchy increment corresponding to the first degree of relevance is obtained from a hierarchy increment data table. If the first degree of relevance is greater than or equal to a second degree of relevance, and the user operation behavior is moving closer to the display device 200, the sum of the original hierarchy of the webpage element and the hierarchy increment corresponding to the first degree of relevance is determined as the display hierarchy. If the user operation behavior is moving away from the display device 200, the difference between the original hierarchy of the webpage element and the hierarchy increment corresponding to the first degree of relevance is determined as the display hierarchy. The display hierarchy of the webpage element must not be less than the original hierarchy. The hierarchy increment data table includes hierarchy increments corresponding to different relevance ranges.

[0115] For example, Figure 9 In (a) is Figure 4 The display effect of the technical solution, and Figure 9 (b) illustrates the effect of layering. The original layer of a text element on a webpage is higher than that of an image element, meaning the text element is displayed on top of the image element. However, when the user's action is to move closer to the display device by 200 degrees, the image element's display layer becomes higher than the text element, thus displaying the image element on top of the text element.

[0116] This embodiment of the application can identify the user's approach or departure based on the Z-axis information of the motion sensing data. When the user approaches the display device 200, the web page elements near the cursor are automatically brought to the front for accurate selection. When the user moves away from the display device 200, the web page elements near the cursor are automatically moved back to the original layer. Layer switching uses a layered rendering algorithm to ensure smooth animation, and a parallax scrolling algorithm is used during layer switching to increase the depth of field effect and enhance the sense of space.

[0117] It should be noted that, depending on the actual needs, one or more of the following can be calculated based on the first degree of relevance: movement distance and direction, scaling size, transparency, highlighting, and display level, in order to perform one or more transformation operations on the web page element.

[0118] In some embodiments, an implementation of calculating the movement distance and direction of a web page element based on a second correlation may include: calculating a second endpoint position of the web page element based on a second correlation and a second shortest distance, then obtaining the current position of the web page element, and then calculating the movement distance and direction based on the second endpoint position and the current position.

[0119] One implementation of calculating the second endpoint position of a webpage element based on a second correlation degree and a second shortest distance may include: obtaining the movement ratio corresponding to the second correlation degree, and then calculating the predicted distance based on the movement ratio and a first shortest distance. The direction from the original position of the webpage element towards the target segmented trajectory is determined as the predicted direction. The second endpoint position of the webpage element is calculated based on the original position of the webpage element, the predicted distance, and the predicted direction.

[0120] For example, Figure 10 In (a) is Figure 4 The display effect of the technical solution, and Figure 10 In (b), the effect of the web page element moving towards the predicted trajectory is shown, where the dashed line represents the predicted trajectory. Secondly, the higher the correlation, the greater the movement ratio.

[0121] In some embodiments, one implementation of calculating the enlarged size of a web page element based on a second correlation degree may include: obtaining the enlargement ratio corresponding to the second correlation degree from an enlargement ratio data table, and then calculating the enlarged size of each web page element based on the enlargement ratio and the original size of the web page element.

[0122] For example, Figure 11 In (a) is Figure 4 The display effect of the technical solution, and Figure 11 (b) shows the effects of changes in webpage element size, transparency, and highlight, where the dashed line represents the predicted trajectory. The higher the correlation with the predicted trajectory, the larger the magnification ratio, the larger the transparency gain coefficient, and the better the highlight effect.

[0123] In some embodiments, the transparency of web page elements can also be calculated based on the second correlation degree. Specifically, this involves obtaining the transparency gain ratio corresponding to the second correlation degree from a transparency gain ratio data table, and then calculating the transparency of each web page element based on the transparency gain ratio and the original transparency of the web page element.

[0124] In some embodiments, the highlight level corresponding to the second correlation degree is obtained from the highlight level data table, and then the highlight display method corresponding to the highlight level is obtained, and the web page element is displayed in the highlight display method.

[0125] In some embodiments, one implementation of calculating the hierarchy of a webpage element based on a second correlation degree may include: obtaining Z-axis depth information from motion coordinates, and then identifying user operation behavior based on the Z-axis depth information. The hierarchy increment corresponding to the second correlation degree is obtained from a hierarchy increment data table. If the user operation behavior is moving closer to the display device 200, the sum of the original hierarchy of the webpage element and the hierarchy increment corresponding to the second correlation degree is determined as the display hierarchy. If the user operation behavior is moving away from the display device 200, the difference between the original hierarchy of the webpage element and the hierarchy increment corresponding to the second correlation degree is determined as the display hierarchy. The display hierarchy of the webpage element must not be less than the original hierarchy.

[0126] For example, Figure 12 In (a) is Figure 4 The display effect of the technical solution, and Figure 12 (b) illustrates the effect of layer changes, where the dashed line represents the predicted trajectory. In a webpage, the original layer of a text element is higher than that of an image element, meaning the text element is displayed on top of the image element. However, when the user's action is to move closer to the display device by 200 degrees, the image element's display layer becomes higher than the text element, and thus the image element is displayed on top of the text element.

[0127] In this embodiment, the method of calculating the moving distance and direction, magnification, transparency, highlighting and display level of web page elements based on the second correlation is similar to the method of calculating the moving distance and direction, magnification, transparency, highlighting and display level of web page elements based on the first correlation, and will not be described again here.

[0128] It should be noted that the predicted trajectory can refer to a single trajectory from the current cursor position to the predicted endpoint. Alternatively, the predicted trajectory can refer to multiple trajectories from the current cursor position to the predicted endpoint, with the movement distance calculated based on the webpage element and the trajectory with the highest correlation.

[0129] In this embodiment, the browser application can calculate the second relevance of web page elements near the predicted trajectory, i.e., sort them according to functional importance and spatial distance. Based on the predicted trajectory containing the predicted endpoint and direction vector, the motion trajectory extension area is determined in the web page coordinate system. Web page elements with high second relevance are moved to the trajectory extension area. The transformation property of Cascading Style Sheets 3 (CSS3) is used to achieve smooth movement of web page elements and pre-layout in the extension direction of the motion trajectory, reducing the path length of continuous operations and improving interaction efficiency.

[0130] In some embodiments, one implementation of calculating the movement distance and direction of a webpage element based on a first relevance degree and a second relevance degree includes: if the first relevance degree is greater than or equal to the second relevance degree, calculating a first endpoint position of the webpage element based on the first relevance degree and a first shortest distance; then obtaining the current position of the webpage element; and finally calculating the movement distance and direction based on the first endpoint position and the current position. If the first relevance degree is greater than or equal to the second relevance degree, calculating a second endpoint position of the webpage element based on the second relevance degree and the second shortest distance; then obtaining the current position of the webpage element; and finally calculating the movement distance and direction based on the second endpoint position and the current position.

[0131] In some embodiments, an implementation of calculating the enlarged size of a webpage element based on a first relevance and a second relevance may include: if the first relevance is greater than or equal to the second relevance, calculating the enlarged size of the webpage element based on the magnification ratio corresponding to the first relevance and the original size of the webpage element; if the first relevance is less than the second relevance, calculating the enlarged size of the webpage element based on the magnification ratio corresponding to the second relevance and the original size of the webpage element.

[0132] In some embodiments, one implementation of calculating the hierarchy of a webpage element based on a first correlation degree and a second correlation degree may include: obtaining Z-axis depth information from motion coordinates, and identifying user operation behavior based on the Z-axis depth information. If the first correlation degree is greater than or equal to the second correlation degree, and the user operation behavior is moving closer to the display device, the sum of the original hierarchy of the webpage element and the corresponding hierarchy increment of the first correlation degree is determined as the display hierarchy. If the user operation behavior is moving away from the display device, the difference between the original hierarchy of the webpage element and the corresponding hierarchy increment of the first correlation degree is determined as the display hierarchy.

[0133] If the first relevance is less than the second relevance, and the user's action is to move closer to the display device, the sum of the original level of the webpage element and the corresponding level increment of the second relevance is determined as the display level. If the user's action is to move away from the display device, the difference between the original level of the webpage element and the corresponding level increment of the second relevance is determined as the display level.

[0134] For example, Figure 13 In (a) is Figure 4 The display effect of the technical solution, and Figure 13 (b) shows the effect of changes in the layout of web page elements. AB is the target segment trajectory, and BC is the predicted trajectory. Web page elements near the target segment trajectory AB can move and scale towards the target segment trajectory AB. Web page elements near the predicted trajectory BC can move and scale towards the predicted trajectory BC.

[0135] In some embodiments, the flowchart of the webpage display method may be as follows: Figure 14 As shown. After receiving motion-sensing data from the motion-sensing device, the data is denoised. The motion-sensing data includes 3D coordinate data and motion assistance information. The denoising methods used are Kalman filtering and moving average. The denoised 3D coordinate data is projected and transformed into 2D webpage coordinates while retaining Z-axis information. The 2D webpage coordinates are segmented, and their feature points are extracted to calculate motion parameters and determine user operation behavior. The above data is organized into a segmented trajectory list, a feature point set, and user operation behavior. The predicted trajectory of the cursor is generated using the above data and a rule engine. The first correlation degree between the webpage element and the segmented trajectory where the cursor is located is calculated, and the second correlation degree between the webpage element and the predicted trajectory is calculated. The webpage elements are adaptively laid out based on the first and second correlation degrees. Adaptive layout includes element snapping, element deformation, layer switching, and pre-layout.

[0136] In some embodiments, the system architecture diagram can be as follows: Figure 15 As shown, the system architecture's operating environment now supports motion-sensing devices to receive and collect raw user motion data. The data layer of the system architecture adds a 3D data acquisition module for interface with and processing user 3D motion data, a data denoising module to remove unreasonable values ​​from the motion data and improve data accuracy, an auxiliary information processing module to process non-core motion data such as speed, direction, and acceleration, supplementing motion data content for more accurate calculations, and a coordinate transformation module to map 3D motion data to the 2D space of the webpage for adaptive webpage layout.

[0137] The system architecture's business layer has added a trajectory feature extraction module to label trajectory features, providing analytical data for spatial modeling and haptic intent prediction. A new spatial modeling module or correlation calculation module has been added to associate user haptic actions with webpage elements in real time, i.e., calculate the correlation degree, providing a spatial relationship foundation for subsequent interaction algorithms. A new trajectory prediction module has been added to determine the webpage location corresponding to the user's next haptic action based on the current trajectory, for use in adaptive webpage layout. A new adaptive webpage layout module has been added to dynamically adjust the layout and visual effects of webpage elements based on their correlation degree.

[0138] The presentation layer of the system architecture receives the adaptive layout results output by the business layer, and displays the changes of web page elements on the screen in real time through the browser's rendering engine, presenting them to the user.

[0139] In some embodiments, the timing diagram of the web page display method can be as follows: Figure 16 As shown. The display device 200 includes a coordinate transformation module, a trajectory analysis module, a trajectory prediction module, a correlation calculation module, and a page layout module. The coordinate transformation module receives multiple motion-sensing data points from the motion-sensing device, including motion coordinates and auxiliary information. The coordinate transformation module maps these multiple motion coordinates to the coordinate system of the webpage, obtaining multiple webpage coordinates and their corresponding Z-axis information, and then sends these coordinates and Z-axis information to the trajectory analysis module. The trajectory analysis module performs trajectory segmentation analysis on the multiple webpage coordinates, obtaining trajectory features of multiple segmented trajectories. These trajectory features include feature points, user operation behavior, and motion parameters. The trajectory features of the multiple segmented trajectories are sent to the trajectory prediction module, and the trajectory of the segment where the cursor is located (i.e., the target segmented trajectory), or the trajectory features of the target segmented trajectory, are sent to the correlation calculation module. The trajectory prediction module generates a predicted trajectory for the cursor based on the trajectory features of the multiple segmented trajectories and the trajectory features of the user's historical operations, and sends the predicted trajectory to the correlation calculation module. The correlation calculation module calculates the first correlation degree between each webpage element and the target segment trajectory, and the second correlation degree between each webpage element and the predicted trajectory, and sends the first and second correlation degrees of each webpage element to the webpage layout module. If the first correlation degree is greater than or equal to the second correlation degree, the webpage layout module calculates the movement distance and direction, zoom size, and display level of the webpage element based on the first correlation degree. If the first correlation degree is less than the second correlation degree, the module calculates the movement distance and direction, zoom size, and display level of the webpage element based on the second correlation degree. The webpage element is then moved the corresponding movement distance in the corresponding movement direction to increase its size and display level.

[0140] This application fully utilizes the three-dimensional spatial information provided by motion-sensing devices, such as spatial position, direction of movement, and acceleration, to diversify the user's motion-sensing operation effects in a browser scenario and provide rich interaction methods.

[0141] This application's embodiments differ from existing browsers in that their processing of motion-sensing trajectories is limited to basic path tracking. They can extract and analyze key feature points of the trajectory, such as speed change points and pause points, and have dynamic responses to motion-sensing operation types (such as approaching, moving away, swiping, etc.). They can realize spatial association between motion-sensing trajectories and web page elements, and optimize the interaction process according to the user's motion-sensing operation intentions.

[0142] This application's embodiment differs from existing browsers that simply update the cursor position based on motion sensing data. It can combine trajectory features and historical trajectory features for intelligent prediction and trend analysis of the motion sensing path, thereby enabling the browser to proactively identify the user's motion sensing operation intent and optimize the interactive experience in advance.

[0143] This application's embodiments can dynamically adjust the layout of web page elements in real time based on user motion data. For example, based on motion trajectory analysis, target elements on the web page can be automatically snapped to the vicinity of the cursor, reducing the distance the user's motion travels. Elements near the cursor can also be highlighted using methods such as zooming and highlighting. When the user's motion moves closer to or further away from the display, the browser automatically brings web page elements near the cursor to the front, enhancing the user's spatial experience through layer switching, facilitating accurate selection or global browsing. When the user's motion movements are relatively fixed, the browser predicts the user's possible operation path and pre-lays web page elements along the extended path of the motion trajectory, improving the convenience of user interaction.

[0144] Some embodiments of this application also provide a computer-readable storage medium that can store a program. When the computer storage medium is configured in a display device or server, the program, when executed, can include the program steps involved in the webpage display method in the above embodiments. The computer storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0145] This application provides an electronic device, which includes a processor and a memory for storing processor-executable instructions. The processor is configured to read executable instructions from the memory and execute the instructions to implement the webpage display method described in the above embodiments.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A display device, characterized in that, include: A display is configured to show a web page, the web page including a cursor and multiple web page elements; A communication device is configured to establish a communication connection with a motion-sensing device; The controller is configured as follows: Receive multiple motion coordinates sent by the motion sensing device during user activity; The multiple motion coordinates are mapped onto the coordinate system of the webpage to obtain multiple webpage coordinates; The trajectory segmentation analysis is performed on the multiple web page coordinates to obtain the trajectory features of multiple segmented trajectories. The segmented trajectory is the movement trajectory of the cursor between the multiple continuously obtained web page coordinates. The trajectory features are used to characterize the motion information of the cursor moving according to the segmented trajectory. Based on the trajectory characteristics of the multiple segmented trajectories and the trajectory characteristics of the user's historical operations, the predicted trajectory of the cursor is generated. Calculate the first correlation degree between the web page element and the target segmented trajectory, and calculate the second correlation degree between the web page element and the predicted trajectory, wherein the target segmented trajectory is the segmented trajectory where the cursor is located; Based on the first correlation degree and / or the second correlation degree, the moving distance and moving direction, the magnified size and / or the display level of the web page element are calculated, wherein the display level is used to characterize the display priority of the web page element; The web page element is moved according to the moving direction and the moving distance, and the web page element is displayed at the enlarged size, and / or the web page element is displayed at the display level.

2. The display device according to claim 1, characterized in that, The controller performs a calculation of the first correlation degree between the web page element and the target segmented trajectory, and is further configured to: Calculate the first shortest distance and the area of ​​the first intersection region, and obtain the matching value corresponding to the element type of the web page element. The first shortest distance is the shortest distance between the edge point of the web page element and the target segment trajectory. The area of ​​the first intersection region is the minimum area of ​​the closed region enclosed by the display area of ​​the web page element and the target segment trajectory. The matching value is generated based on the number of times or frequency of the user selecting the element type within a preset period. The first correlation degree is obtained by weighting the first shortest distance, the area of ​​the first intersection region, and the matching value.

3. The display device according to claim 2, characterized in that, The controller performs a calculation of the second correlation degree between the web page element and the predicted trajectory, and is further configured to: Calculate the second shortest distance and the area of ​​the second intersection region. The second shortest distance is the shortest distance between the edge point of the web page element and the predicted segmented trajectory. The area of ​​the second intersection region is the minimum area of ​​the closed region enclosed by the display area of ​​the web page element and the predicted trajectory. The second correlation degree is obtained by weighting the second shortest distance, the area of ​​the second intersection region, and the matching value.

4. The display device according to claim 2, characterized in that, The controller, by calculating the movement distance and direction of the webpage element based on the first and second correlation degrees, is further configured to: If the first relevance is greater than or equal to the second relevance, the first endpoint position of the web page element is calculated based on the first relevance and the first shortest distance; Get the current position of the webpage element; The moving distance and the moving direction are calculated based on the first destination position and the current position.

5. The display device according to claim 3, characterized in that, The controller, which calculates the movement distance and direction of the webpage element based on the first correlation and the second correlation, is further configured to: If the first relevance is greater than or equal to the second relevance, the second endpoint position of the web page element is calculated based on the second relevance and the second shortest distance; Get the current position of the webpage element; The moving distance and the moving direction are calculated based on the second endpoint position and the current position.

6. The display device according to claim 1, characterized in that, The controller, which calculates the enlarged size of the webpage element based on the first relevance and the second relevance, is further configured to: If the first correlation degree is greater than or equal to the second correlation degree, the magnified size of the web page element is calculated based on the magnification ratio corresponding to the first correlation degree and the original size of the web page element; If the first correlation degree is less than the second correlation degree, the magnified size of the web page element is calculated based on the magnification ratio corresponding to the second correlation degree and the original size of the web page element.

7. The display device according to claim 1, characterized in that, The controller, which calculates the hierarchy of the webpage elements based on the first and second relevance, is further configured to: Z-axis depth information is obtained from the somatosensory coordinates, wherein the Z-axis is established with respect to the orientation of the display device; User operation behavior is identified based on the Z-axis depth information; If the user's operation behavior is to move closer to the display device, and the first correlation degree is greater than or equal to the second correlation degree, the sum of the original level of the web page element and the level increment corresponding to the first correlation degree is determined as the display level. If the user's operation is to move away from the display device, the difference between the original level of the web page element and the level increment corresponding to the first correlation degree is determined as the display level; If the user's operation is to move closer to the display device, and the first correlation degree is less than the second correlation degree, the sum of the original level of the web page element and the level increment corresponding to the second correlation degree is determined as the display level. If the user's operation is to move away from the display device, the difference between the original level of the web page element and the level increment corresponding to the second correlation degree is determined as the display level.

8. The display device according to claim 1, characterized in that, The trajectory features include feature points, user operation behavior, and motion parameters. The motion parameters include the cursor's direction vector, movement speed, and acceleration. The feature points include a start point, an end point, a speed change point, and a rest point. The user operation behavior includes moving closer to or further away from the display device. The controller performs segmented analysis on the multiple webpage coordinates to obtain trajectory features of multiple segmented trajectories, which are further configured as follows: The coordinates of the multiple web pages are segmented to obtain multiple segmented trajectories, and the movement distance of the cursor in the segmented trajectory is the target length; Based on the webpage coordinates corresponding to the segmented trajectory, feature points and user operation behaviors are obtained, and the direction vector, movement speed, and acceleration of the cursor in the segmented trajectory are calculated.

9. The display device according to claim 8, characterized in that, The controller generates a predicted trajectory for the cursor based on the trajectory features of the multiple segmented trajectories and the trajectory features of the user's historical operations, and is further configured to: The predicted trajectory of the cursor is generated using a rule engine based on the trajectory features of the multiple segmented trajectories and the trajectory features of the user's historical operations.

10. A method for displaying a webpage, characterized in that, include: Receive multiple motion coordinates sent by the motion sensing device during user activity; The multiple motion coordinates are mapped onto the coordinate system of the webpage to obtain multiple webpage coordinates; The trajectory segmentation analysis is performed on the multiple web page coordinates to obtain the trajectory features of multiple segmented trajectories. The segmented trajectory is the movement trajectory of the cursor between the multiple continuously obtained web page coordinates. The trajectory features are used to characterize the motion information of the cursor moving according to the segmented trajectory. Based on the trajectory characteristics of the multiple segmented trajectories and the trajectory characteristics of the user's historical operations, the predicted trajectory of the cursor is generated. Calculate the first correlation degree between the web page element and the target segmented trajectory, and calculate the second correlation degree between the web page element and the predicted trajectory, wherein the target segmented trajectory is the segmented trajectory where the cursor is located; Based on the first correlation degree and / or the second correlation degree, the moving distance and moving direction, the magnified size and / or the display level of the web page element are calculated, wherein the display level is used to characterize the display priority of the web page element; The web page element is moved according to the moving direction and the moving distance, and the web page element is displayed at the enlarged size, and / or the web page element is displayed at the display level.

Images