Display control method and device for augmented reality equipment and related equipment

Abstract

The embodiment of the invention provides a display control method and device for augmented reality equipment and related equipment, and relates to the technical field of augmented reality interaction.The method comprises the steps that touch operation used for display control is detected, a first parameter is determined according to the touch operation, the first parameter is a parameter corresponding to a target object, and the target object is displayed according to the first parameter. The first parameter comprises at least one of a first horizontal coordinate, a first vertical coordinate and a first display area size; the first horizontal coordinate and the first vertical coordinate are coordinate values in a first coordinate system; converting the first parameter into a second parameter corresponding to the augmented reality device; and sending the second parameter to the augmented reality device, wherein the augmented reality device is used for updating the display effect of the target object according to the second parameter. The display mapping control of what you see is what you get is realized, and the operation precision and the interaction experience are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of augmented reality interaction, and in particular to a display control method and device for an augmented reality device and related equipment. BACKGROUND

[0002] With the development of augmented reality technology, augmented reality (AR) devices (such as augmented reality glasses, head-mounted displays, etc.) are increasingly widely used in navigation assistance, real-time translation, visual enhancement, remote collaboration and other scenarios.

[0003] In the prior art, AR devices usually control the position and size of the display content through gestures, voice or head gestures, and some AR devices also make simple function settings or content synchronization through the matching application on the mobile phone.

[0004] However, the inventors have found that when interacting in the above manner, the following technical problems exist:

[0005] 1. Display adjustment is not intuitive: the user needs to directly operate the AR device, lacks visual reference, and it is difficult to accurately control the information position; 2. Control method is single: there is only control instruction communication between the mobile phone and the AR device, and there is a lack of intuitive mapping and presentation of the display content in position and area; 3. Poor operation consistency: the content set on the mobile phone often does not match the actual effect displayed on the AR device, resulting in a poor interactive experience; 4. Insufficient real-time preview capability: the user cannot observe the final effect displayed on the AR device on the mobile phone in real time, and can only rely on repeated trial and error adjustment.

[0006] Therefore, there is an urgent need for a new augmented reality display control method. SUMMARY

[0007] The present application provides a display control method and device for an augmented reality device and related equipment to at least solve the above technical problems in the prior art.

[0008] The technical solution of the present application embodiment is as follows: In a first aspect, the present application provides a display control method for an augmented reality device, applied to a terminal, the method comprising: detecting a touch operation for display control, determining a first parameter according to the touch operation, the first parameter being a parameter corresponding to a target object, the first parameter including at least one of the following: a first horizontal coordinate, a first vertical coordinate, and a first display area size; the first horizontal coordinate and the first vertical coordinate being coordinate values in a first coordinate system; convert the first parameter into a second parameter corresponding to the augmented reality device; send the second parameter to the augmented reality device, the augmented reality device being configured to update a display effect of the target object according to the second parameter.

[0009] In a second aspect, an embodiment of the present application provides a display control method for an augmented reality device, applied to the augmented reality device, and the method comprises: receiving a second parameter from a terminal, the second parameter being a parameter corresponding to a target object; updating a display effect of the target object according to the second parameter; wherein the second parameter is converted from a first parameter of the terminal, the first parameter being a parameter corresponding to the target object, and the first parameter comprises at least one of a first horizontal coordinate, a first vertical coordinate, and a first display area size; the first horizontal coordinate and the first vertical coordinate are coordinate values in a first coordinate system.

[0010] In a third aspect, an embodiment of the present application provides a display control apparatus for an augmented reality device, applied to a terminal, and the apparatus comprises: a first processing module configured to detect a touch operation for display control, determine a first parameter according to the touch operation, the first parameter being a parameter corresponding to a target object, and the first parameter comprising at least one of a first horizontal coordinate, a first vertical coordinate, and a first display area size; the first horizontal coordinate and the first vertical coordinate being coordinate values in a first coordinate system; a second processing module configured to convert the first parameter into a second parameter corresponding to the augmented reality device; a first communication module configured to send the second parameter to the augmented reality device, the augmented reality device being configured to update a display effect of the target object according to the second parameter.

[0011] In a fourth aspect, an embodiment of the present application provides a display control apparatus for an augmented reality device, applied to the augmented reality device, and the apparatus comprises: a second communication module configured to receive a second parameter from a terminal, the second parameter being a parameter corresponding to a target object; a third processing module configured to update a display effect of the target object according to the second parameter; wherein the second parameter is converted from a first parameter of the terminal, the first parameter being a parameter corresponding to the target object, and the first parameter comprising at least one of a first horizontal coordinate, a first vertical coordinate, and a first display area size; the first horizontal coordinate and the first vertical coordinate being coordinate values in a first coordinate system.

[0012] In a fifth aspect, an electronic device is provided, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any of the terminal-side display control methods for an augmented reality device; or to enable the at least one processor to perform any of the augmented reality device-side display control methods for an augmented reality device.

[0013] In a sixth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, and the computer instructions are used to cause a computer to perform any of the terminal-side display control methods for an augmented reality device; or the computer instructions are used to cause a computer to perform any of the augmented reality device-side display control methods for an augmented reality device.

[0014] The embodiments of the present application have the following beneficial effects: The display control method, apparatus and related device for an augmented reality device provided by the embodiments of the present application are applied, and the method includes: a terminal detects a touch operation for display control, determines a first parameter according to the touch operation, the first parameter is a parameter corresponding to a target object, and the first parameter includes at least one of the following: a first horizontal coordinate, a first vertical coordinate and a first display area size; the first horizontal coordinate and the first vertical coordinate are coordinate values in a first coordinate system; the first parameter is converted into a second parameter corresponding to the augmented reality device; and the second parameter is sent to the augmented reality device, and the augmented reality device is used to update a display effect of the target object according to the second parameter. Correspondingly, the augmented reality device receives the second parameter from the terminal, the second parameter is a parameter corresponding to a target object; and a display effect of the target object is updated according to the second parameter. In this way, a user can intuitively adjust the position and area of the display content of the augmented reality device on the touchable and displayable interface of the familiar terminal through the touch operation, realize the display mapping control of “what you see is what you get”, and thus improve the operation precision and the interactive experience.

[0015] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 A flowchart of a display control method for an augmented reality device provided by the embodiments of the present application is shown; Figure 2A schematic diagram illustrating the coordinate mapping relationship between a terminal and an augmented reality device, provided for an application embodiment of this application; Figure 3 A flowchart illustrating another display control method for augmented reality devices provided in this application embodiment; Figure 4 A schematic diagram illustrating the interaction between a mobile phone and augmented reality glasses, provided as an application embodiment of this application; Figure 5 This application provides a schematic diagram of the structure of a display control device for an augmented reality device. Figure 6 This is a schematic diagram of another display control device for augmented reality devices provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0017] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0019] Figure 1 This application provides a flowchart illustrating a display control method for an augmented reality device. Figure 1 As shown, the method is applied to a terminal, and the method includes: Step 101: Detect the touch operation used for display control, and determine the first parameter based on the touch operation. The first parameter is a parameter corresponding to the target object. The first parameter includes at least one of the following: a first horizontal coordinate, a first vertical coordinate, and a first display area size; the first horizontal coordinate and the first vertical coordinate are coordinate values ​​in a first coordinate system. Step 102: Convert the first parameter into the second parameter corresponding to the augmented reality device; Step 103: Send the second parameter to the augmented reality device, which updates the display effect of the target object according to the second parameter.

[0020] In some embodiments, the terminal may be such as a mobile phone, smartphone, laptop computer, digital broadcast receiver, personal digital assistant (PDA), tablet computer (PAD), portable multimedia player (PMP), wearable device (such as smart bracelet, smartwatch, etc.).

[0021] The terminal has wireless and / or wired communication capabilities, and can communicate with augmented reality devices in real time using any of these methods.

[0022] The touch operation can be an operation for controlling the display of the augmented reality device, such as clicking, swiping, or dragging with a finger.

[0023] The first coordinate system is the terminal's own coordinate system. The first horizontal coordinate can also be denoted as the X coordinate, the first vertical coordinate can also be denoted as the Y coordinate, and the first display area size can refer to the size of the display area presented in the terminal.

[0024] Augmented reality devices refer to devices that use augmented reality technology to integrate virtual information or digital content (such as images, text, and sound) with images or environments in the real world, thereby enhancing or improving a user's perception and interaction with the real world. Examples include augmented reality glasses (AR glasses), head-mounted displays (AR helmets), smart glasses, AR projection devices, and mixed reality devices.

[0025] Augmented Reality (AR) technology uses optoelectronic display technology, interactive technology, multiple sensor technology, and computer graphics and multimedia technology to integrate a computer-generated virtual environment with the user's surrounding real environment, so that the user can be convinced from the sensory effect that the virtual environment is a part of the real environment around them.

[0026] Here, by detecting touch operations used for display control on the terminal, users can interact with the device more intuitively. For example, they can adjust the displayed content through direct actions such as touching, swiping, and zooming, reducing the limitations of traditional control methods (such as buttons and voice commands) and allowing users to operate the augmented reality device more smoothly and efficiently. Thus, touch operations enable flexible adjustments, allowing users to control various parameters of the displayed content according to their needs, and customize the display effect of the augmented reality device according to their own intentions.

[0027] The second parameter corresponding to the augmented reality device is a parameter for the target object. The second parameter includes at least one of the following: a second horizontal coordinate, a second vertical coordinate, and a second display area size; the second horizontal coordinate and the second vertical coordinate are coordinate values ​​in a second coordinate system. The target object is the object specifically targeted by the touch operation (the specific operation).

[0028] By converting the effect of touch operations in the first coordinate system (i.e., the first parameter) into the effect in the second coordinate system of the augmented reality device (i.e., the second parameter), consistency of operation across different devices can be ensured, reducing inconvenience caused by device differences. This method allows the augmented reality device to dynamically adjust the size and position of the display area according to the needs of different users, with different display operations corresponding to different display effects, increasing the flexibility and adaptability of augmented reality applications.

[0029] The method of controlling the display of the augmented display device through terminal operation can update the display content in real time. When the user performs touch operation, he / she can immediately see the change feedback, improve the intuitiveness of operation, and reduce the confusion and delay that the user may encounter during operation.

[0030] In some embodiments, detecting a touch operation for display control includes: Based on the display requirements of the augmented reality device, a visual control interface corresponding to the display content of the augmented reality device is presented; the visual control interface is used to display at least one object that can be mapped to the display area of ​​the augmented reality device. Detect touch operations used for display control in response to the operation of the visual control interface.

[0031] Here, based on the display requirements of the augmented reality device, the terminal generates a visual control interface corresponding to the display content of the augmented reality device. This interface corresponds to the display content of the augmented reality device, specifically in terms of location, display area, and specific content presented.

[0032] For example, a virtual interface or control panel can be displayed, containing information or parameters of various objects that the user can control.

[0033] The visual control interface can display objects. Objects refer to the specific information of the content to be displayed on the augmented reality device, which can include displayed objects, images, text, etc., and this content is usually integrated with the real-world environment. Examples include text boxes, maps, navigation signs, and other virtual objects in augmented reality. These objects can be adjusted and updated based on the user's interaction with the visual control interface.

[0034] Here, when a user interacts with the visual control interface, the terminal detects the user's touch operations, which can include swiping, clicking, zooming, etc. These operations will affect the performance of the visual control interface, and thus affect the content displayed on the augmented reality device.

[0035] In this way, users interact with a visual interface through touch operations, and this interface directly affects the content displayed on the augmented reality device. That is, users can control the content displayed on the augmented reality device through simple and intuitive touch operations, thus improving the interactive experience.

[0036] In some embodiments, the at least one object includes at least one of the following: a text module, an image module, a navigation indicator module, an augmented reality prompt window, and a display preview window; The display preview window is used to display the display effect of the mapped augmented reality device in real time; Each of the objects has an operable display boundary in the visual control interface, and the operation includes at least one of the following: dragging, zooming in, and zooming out.

[0037] Here, the object can include, but is not limited to, at least one of the following: A text module refers to a module that contains text content; users can edit or adjust the text content through the control interface. An image module refers to a module that contains image or graphic content; users can adjust the display method or position of the image through operation. The navigation indicator module refers to the module used to provide navigation information, such as directions and locations; users can adjust these modules to change the display method or position of the indicator content. Augmented reality prompt windows are typically used to provide prompts or instructions in an augmented reality environment, such as displaying operation tips or information boxes in the AR environment.

[0038] The preview window is a dedicated module for displaying the real-time effect of the mapped augmented reality device. In other words, it allows users to see a preview on the control interface, showing how their adjustments will affect the final AR display. For example, when a user adjusts the position, size, or display mode of a virtual object on the control interface, the preview window updates in real time, showing how these changes will be presented on the augmented reality device.

[0039] Here, objects are mapped onto the display area of ​​the augmented reality device, representing what the user sees when operating the visual control interface, which ultimately affects the virtual content displayed in the augmented reality device.

[0040] There can be one or more objects, and each object (such as a text module, an image module, etc.) has an operable display boundary. Users can manipulate and adjust the boundaries of these object modules through touch or other interactive methods. For example, they can change the display of these modules by dragging or scaling, or change the position and shape of the objects in the augmented reality device.

[0041] It should be noted that the first parameter refers to the object's first horizontal coordinate, first vertical coordinate, and / or first display area size; if multiple objects are being modified, the first parameter will be in multiple groups, with each group corresponding to the first parameter of each object.

[0042] Here, touch operations include at least one of the following: clicking, dragging, zooming in, and zooming out; where clicking is used to select a control item or set a parameter; dragging (sliding) is used to adjust the position or size of the display area; zooming out and zooming in (i.e., scaling) are used to control the size or viewing angle of the displayed content.

[0043] In one example, the terminal is a mobile phone, and a visual control interface corresponding to the content displayed on the AR glasses is shown on the phone's screen. This interface includes selectable objects for the user. The control interface provides operation options including dragging, zooming, etc., allowing the user to adjust the module in real time via touch gestures, thereby changing the module's position and size in the phone's coordinate system.

[0044] Thus, when users interact with different object modules in the visual control interface, these actions not only affect the display on the control interface but are also reflected in the augmented reality device in real time. Through dragging, zooming in, zooming out, and other operations, users can precisely control the presentation and position of the displayed content, thereby achieving more personalized and flexible interaction.

[0045] In some embodiments, the method further includes: generating a mapping model; the generated mapping model includes: Determine the screen information and first display resolution of the terminal; Obtain the field of view (FOV) parameter and the second display resolution of the augmented reality device; Based on the screen information, the first display resolution, the second display resolution, and the field of view parameters, a mapping model between the first coordinate system and the second coordinate system is established, wherein the first coordinate system is the display coordinate system of the terminal, and the second coordinate system is the display coordinate system of the augmented reality device; Accordingly, converting the first parameter into a second parameter corresponding to the augmented reality device includes: Using the mapping model, the first parameter is converted into a second parameter corresponding to the augmented reality device.

[0046] Here, the first coordinate system refers to the terminal's display coordinate system, which represents the position of each point on the terminal screen.

[0047] The second coordinate system refers to the display coordinate system of the augmented reality device. Because augmented reality devices have different display methods (such as through a lens, projection, etc.), this coordinate system is different from the terminal's coordinate system.

[0048] To enable accurate conversion and alignment of display content between terminals and augmented reality devices based on their respective display characteristics (such as screen resolution and field of view), and to ensure content synchronization and matching between the two, a method for generating mapping models is proposed.

[0049] In some embodiments, establishing a mapping model between a first coordinate system and a second coordinate system based on the screen information, the first display resolution, the second display resolution, and the field of view parameter includes: A scaling factor is determined based on the first display resolution and the second display resolution; The vertical scaling factor is determined based on the scaling factor, the vertical pixel height in the screen information, and the vertical display range in the field of view parameter. The horizontal scaling factor is determined based on the scaling factor, the horizontal pixel height in the screen information, and the horizontal display range in the field of view parameter. A mapping model between the first coordinate system and the second coordinate system is established based on the scaling factor, the vertical scaling factor, and the horizontal scaling factor.

[0050] Here, the field of view parameters can include: vertical display range and horizontal display range. The vertical display range refers to the actual area of ​​content displayed in the vertical direction, typically determined by the vertical field of view and the physical dimensions of the device (such as the height of an AR device screen). The horizontal display range refers to the actual area of ​​content displayed in the horizontal direction, determined by the horizontal field of view and the physical dimensions of the device (such as the width of an AR device screen).

[0051] Specifically, let's assume the first display resolution of the terminal screen is denoted as R. m Let Rg be the second display resolution of the augmented reality device, then we can establish the first scaling factor K = Rg / R m The first scaling factor obtained is used to achieve consistency in display range between the terminal and the augmented reality device; Assume the vertical pixel height of the terminal screen is denoted as H. m The vertical display range corresponding to the field of view of an augmented reality device is denoted as H. FOV Then the second proportionality coefficient H is obtained. m / H FOV The vertical scaling factor can be determined based on K and the second scaling factor. Assume the horizontal pixel width of the terminal screen is denoted as W. m The horizontal display range corresponding to the field of view of the augmented reality device is denoted as W. FOV Then we obtain the third proportionality coefficient W. m / WFOV The horizontal scaling factor can be determined based on K and the third scaling factor.

[0052] The resulting mapping model can be represented as a two-dimensional mapping matrix: M = f (K, H) m / H FOV W m / W FOV ); Among them, the mapping model (i.e. M) represents the two-dimensional transformation relationship between the first coordinate system of the terminal interface and the second coordinate system displayed by the augmented reality device. It is used to transform the coordinates of the terminal interface into the real coordinates of the augmented reality device, so as to realize cross-device regional location mapping and consistent presentation. f This represents the mapping function, that is, M can be based on the above K and H. m / H FOV W m / W FOV The resulting arbitrary mapping matrix; The first scaling factor (K) reflects the overall scaling ratio between the two resolutions; The second proportionality coefficient (i.e., H) m / H FOV This describes the normalized scale of the terminal interface relative to the visible area of ​​the augmented reality device in the vertical dimension. The third proportionality coefficient (i.e., W) m / W FOV This describes the normalized scale of the terminal interface relative to the visible area of ​​the augmented reality device in the horizontal dimension.

[0053] By using a two-dimensional mapping matrix M, the corresponding position of terminal interface elements in the display plane of augmented reality devices can be determined without relying on depth information, thereby achieving stable and consistent content mapping between devices of different sizes and different field of view.

[0054] In some embodiments, the method further includes: Determine the first pixel density of the terminal's screen and the second pixel density of the augmented reality device; Determine whether the difference between the first pixel density and the second pixel density exceeds a difference threshold; If the difference exceeds the difference threshold, the scaling factor is corrected based on the first pixel density and the second pixel density to obtain the corrected scaling factor; Accordingly, the vertical scaling factor is determined based on the scaling factor, the vertical pixel height in the screen information, and the vertical display range in the field of view parameter, including: determining the vertical scaling factor based on the corrected scaling factor, the vertical pixel height in the screen information, and the vertical display range in the field of view parameter; Determining a horizontal scaling factor based on the scaling factor, the horizontal pixel height in the screen information, and the horizontal display range in the field of view parameter includes: determining a horizontal scaling factor based on the corrected scaling factor, the horizontal pixel height in the screen information, and the horizontal display range in the field of view parameter. Establishing a mapping model between the first coordinate system and the second coordinate system based on the scaling factor, the vertical scaling factor, and the horizontal scaling factor includes: establishing a mapping model between the first coordinate system and the second coordinate system based on the modified scaling factor, the vertical scaling factor, and the horizontal scaling factor.

[0055] Here, pixel density (PPI, Pixels Per Inch) refers to the number of pixels per inch on a screen, determining the clarity and detail of the display. The pixel density of the terminal device and the augmented reality device may differ; generally, a higher pixel density results in a more detailed screen display. Considering that a large difference in pixel density between the two devices could lead to inaccurate matching of displayed content between the two devices, affecting the display effect, a difference threshold-based judgment method is proposed. The difference threshold is a set standard used to determine whether the difference in pixel density is large enough to warrant adjustment of the scaling factor.

[0056] If the difference exceeds a threshold, the scaling factor (K) is adjusted based on the first pixel density and the second pixel density. This is because if the pixel density of the two devices differs significantly, this difference will affect the level of detail in the displayed content. For example, a device with a high pixel density will display more detailed content, and the scaling factor needs to be adjusted to ensure accurate mapping.

[0057] The corrected scaling factor can be an adjustment factor based on the difference in pixel density between the two devices. This is used to make the display ratio of the terminal device and the augmented reality device more consistent, and to avoid display distortion caused by differences in pixel density.

[0058] In this way, the display mapping ratio between the two devices is adjusted according to the difference in pixel density, so as to ensure that the displayed content can be displayed clearly and accurately on both devices.

[0059] The following provides a specific correction method. First, it is necessary to determine whether modification is required: Scenario 1: When the pixel density of the terminal screen is detected to be close to that of the augmented reality device, or when the user selects the full-screen stretch mode, no additional physical size correction is required.

[0060] Scenario 2: When the pixel density difference between the terminal screen and the augmented reality device exceeds a preset threshold (e.g., difference > 10%), correction is required to ensure that objects appear at a consistent physical visual size on both devices (i.e., to avoid icons on a high-resolution screen appearing too large on low-resolution glasses). Calculation formula: K' = K (PPI) g / PPI m ) Where K' is the corrected scaling factor, and K is the scaling factor determined based on the display resolution; PPI g To enhance the pixel density of displays on real-world devices; PPI m This refers to the pixel density of the terminal display screen.

[0061] like Figure 2 The diagram illustrates the coordinate mapping relationship between a terminal and an augmented reality device. The terminal can detect and collect real-time parameters of the user's touch operations, forming a set of two-dimensional spatial datasets, including: horizontal coordinates (denoted as X). m ), vertical coordinate (denoted as Y) m Display size (denoted as S) m The terminal can ensure smooth mapping response through a high-precision touch sampling rate (e.g., 120Hz or higher). The above coordinate information is defined in the terminal screen coordinate system, where the origin O can be located at the upper left corner of the screen, the X-axis is to the right, and the Y-axis is downward. Then, the terminal can establish a mapping model M between the terminal screen coordinate system (i.e., the first coordinate system) and the AR device coordinate system (i.e., the second coordinate system).

[0062] In the diagram, the first parameter in the display control interface of the terminal (such as a mobile phone) can be represented as: X m Y m S m ; The second parameter in the display coordinate system of an augmented reality device (such as AR glasses) can be represented as: Xg, Yg, Sg; The mapping relationship can be represented by the following matrix transformation:

[0063] For the functional relation M = f(K, H) m / H FOV W m / W FOVThe diagonal transformation matrix (or affine matrix) generated by this specific transformation can be represented by the following formula:

[0064] The horizontal scaling factor, i.e., the horizontal mapping factor, can be determined based on K and the third scaling factor. FOV / W m This horizontal mapping factor ensures that the horizontal movement of the terminal screen is accurately mapped to the horizontal field of view of the augmented reality device. Here, K is the scaling factor from the terminal display scale to the augmented reality glasses display scale, K = Rg / R m .

[0065] The vertical scaling factor, i.e., the vertical mapping factor, can be determined based on K and the second scaling factor. FOV / H m This vertical scaling factor is used to adapt to the inconsistency between the aspect ratio of the terminal and the aspect ratio of the augmented reality field of view, preventing the image from being stretched or compressed.

[0066] The scaling factor, K or its modified form K', can be determined based on the actual resolution. K or K' reflects the overall scaling ratio between the two resolutions, and can also be denoted as the size scaling factor Sscale, used to control the display size of real-world objects in augmented reality devices. Sscale can be either the scaling factor K or the modified form K'. The modified scaling factor K' is then used in the construction and calculation of the mapping matrix M.

[0067] Through the aforementioned matrix M, the terminal can accurately map the two-dimensional operations of the terminal interface to the three-dimensional field of view of the augmented reality device without relying on depth information, thus realizing relational mapping in mathematical terms.

[0068] In practical applications, after receiving mapping data from a terminal (such as a mobile phone), augmented reality devices (such as AR glasses) draw the target object in their display coordinate system according to the corresponding Xg, Yg, and Sg parameters. The rendering engine projects the module to the corresponding position in the user's field of view based on the field-of-view geometry. This allows users to intuitively adjust the position and area of ​​the content displayed on the augmented reality device through touch operations on a familiar touch-sensitive and displayable interface of the terminal, achieving "what you see is what you get" display mapping control, thereby improving operational accuracy and interactive experience.

[0069] In some embodiments, obtaining the field of view parameters and the second display resolution of the augmented reality device includes: After the terminal establishes a connection with the augmented reality device, it sends an acquisition request to the augmented reality device; Receive the field of view parameters and the second display resolution fed back by the augmented reality device in response to the acquisition request; The terminal is connected to the augmented reality device via wireless or wired communication; each terminal is connected to one or more augmented reality devices.

[0070] Here, the wireless communication method may include, but is not limited to: Wi-Fi, Bluetooth (BLE), UWB or other low-latency transmission methods, to transmit the calculated mapping parameters to the AR glasses in real time.

[0071] Among various communication methods, the communication protocol can adopt a data packet frame structure, which includes fields such as coordinates, area size, font size, and synchronization timestamp, to ensure that augmented reality devices can respond and update within 10~30ms.

[0072] Each terminal is connected to one or more augmented reality devices, meaning that a terminal can control one or more augmented reality devices.

[0073] In some embodiments, the method may further include: By combining projection geometry correction coefficients, the tilt angle of the augmented reality device, and / or perspective offset, the mapping model between the first and second coordinate systems is corrected.

[0074] In augmented reality applications, projection geometry correction coefficients are used to correct distortions or deviations caused by different projection methods of devices (such as the field of view and focal length of AR devices).

[0075] Considering that the projection of augmented reality devices is usually not a perfect rectangle, but may have perspective distortion (such as asymmetrical field of view or lens distortion), projection geometry correction coefficients can be used to adjust the display coordinate system of the augmented reality device, correct these geometric errors, and make the displayed content more visually consistent with the content displayed on the terminal device.

[0076] Here, augmented reality devices are typically handheld or worn through head-mounted displays, which may be tilted. The tilt angle of the device affects the direction and angle of the display, and thus affects the mapping of the coordinate system.

[0077] Therefore, the terminal can obtain the tilt angle of the augmented reality device and then adjust the coordinate system mapping to correct visual deviations caused by device tilt. For example, if the augmented reality device is tilted, the position of the displayed content can be corrected through rotation or other geometric transformations.

[0078] Here, perspective shift is typically caused by differences in viewing angle, field of view, or projection method of the device. To accurately map content between the first and second coordinate systems, the mapping model can be dynamically corrected based on perspective shift. For example, through dynamic perspective correction, the depth and position of the displayed content can be adjusted so that the content displayed on the augmented reality device can be correctly aligned with the content on the terminal device from different viewing angles.

[0079] In some embodiments, the method further includes: In response to the confirmation command of the detection, the first parameter is saved to the configuration file; In response to the detection call instruction, the first parameter in the configuration file is invoked, and a mapping layout is generated based on the first parameter.

[0080] Here, when the terminal detects that the user has finished performing an operation, it stores the current mapping parameters, coordinates, and scale configuration information as a user mapping template. This template can be directly loaded in different subsequent scenarios (such as navigation, prompting, and auxiliary display) to achieve rapid layout reuse.

[0081] Each terminal connects to one or more augmented reality devices. When saving, each mapping template can be marked with an augmented reality device identifier, so that the corresponding mapping template can be saved and / or loaded directly later for each augmented reality device.

[0082] Figure 3 This is a flowchart illustrating a display control method for an augmented reality device provided in an embodiment of this disclosure; as shown below. Figure 3 As shown, the method is applied to an augmented reality device, and the method includes: Step 301: Receive a second parameter from the terminal, where the second parameter is a parameter corresponding to the target object; Step 302: Update the display effect of the target object according to the second parameter; The second parameter is obtained by converting the first parameter of the terminal. The first parameter is a parameter corresponding to the target object. The first parameter includes at least one of the following: a first horizontal coordinate, a first vertical coordinate, and a first display area size. The first horizontal coordinate and the first vertical coordinate are coordinate values ​​in a first coordinate system.

[0083] Here, the second parameter may include: a second horizontal coordinate (Xg), a second vertical coordinate (Yg), and a second display area size (Sg), which are parameters describing the position and size of a target object (such as a virtual object or image) in the display environment. The target object can be a virtual item in an AR application (such as a 3D model, icon, label, etc.).

[0084] Augmented reality devices can adjust the position, size, or display mode of the target object in real time based on the second parameters (i.e., Xg, Yg, Sg) transmitted by the terminal, to ensure that the target object is displayed correctly in the AR environment.

[0085] Specifically, the augmented reality device uses the received Xg and Yg coordinates as the position of the target object in the AR view, and uses Sg as the scaling factor of the target object, so as to correctly display the target object in the AR view.

[0086] In one example, an AR application might need to dynamically adjust the position and size of virtual objects based on the user's movement or changes in the environment to maintain alignment with the real world. Users interact with the augmented reality device in real time through touch controls on a familiar, touch-enabled interface, allowing for precise adjustments to the target object's position and size within the augmented reality environment (drag, zoom, zoom, etc.) that are consistent with the user's perspective and actions.

[0087] In another example, combining the above... Figure 2 For example, after receiving the mapping data transmitted from the mobile phone, the AR glasses draw the target object in its display coordinate system according to the corresponding Xg, Yg, and Sg parameters. The rendering engine projects the target object onto the corresponding position in the user's field of view based on the field-of-view geometry, achieving a synchronized display effect where "what the phone sees is what the glasses show".

[0088] Here, the method for obtaining the second parameter has already been explained. Figure 1 The method is explained in detail in the diagram, and will not be repeated here.

[0089] In some embodiments, the method further includes: After the terminal establishes a connection with the augmented reality device, it accepts the acquisition request sent by the augmented reality device; The request requests feedback on the field of view parameters and the second display resolution; The terminal is connected to the augmented reality device via wireless or wired communication; each terminal is connected to one or more augmented reality devices.

[0090] Figure 4 This diagram illustrates an interaction between a mobile phone and augmented reality glasses, provided as an application embodiment of this application. The mobile phone represents an example of a terminal, and the augmented reality glasses represent an example of an augmented reality device. The mobile phone includes a touch operation module, a mapping calculation module, and a communication module for interacting with the augmented reality glasses. The augmented reality glasses also correspondingly include a communication module for interacting with the mobile phone, enabling communication between the two. Furthermore, the augmented reality glasses also include a display touch module and a rendering engine module.

[0091] The specific details for mobile devices are as follows: The touch operation module can be used to present an interactive interface and collect touch information such as dragging and zooming on the screen by the user. The mapping calculation module can be used to calculate the mapping relationship between the mobile phone screen coordinates and the glasses display coordinates, generating a mapping matrix M; and, using the mapping matrix M, to perform coordinate mapping on the objects corresponding to the operations collected by the touch operation module, as described above. Calculation; After obtaining Xg, Yg, and Sg, they are transmitted to the augmented reality glasses via the communication modules at both ends.

[0092] The specific details regarding augmented reality glasses are as follows: The display touch module can be used to control the position and size of the target object image display based on the obtained mapping parameters (such as Xg, Yg, Sg mentioned above); The rendering engine module can be used to render augmented reality content corresponding to a target object on an optical display component.

[0093] Figure 5 This application provides a schematic diagram of the structure of a display control device for an augmented reality device, as shown in the embodiments of the present application. Figure 5 As shown, the device is applied to a terminal, and the device includes: A first processing module is configured to detect touch operations for display control and determine a first parameter based on the touch operation. The first parameter is a parameter corresponding to a target object and includes at least one of the following: a first horizontal coordinate, a first vertical coordinate, and a first display area size. The first horizontal coordinate and the first vertical coordinate are coordinate values ​​in a first coordinate system. The second processing module is used to convert the first parameter into a second parameter corresponding to the augmented reality device; A first communication module is used to send the second parameter to the augmented reality device, and the augmented reality device is used to update the display effect of the target object according to the second parameter.

[0094] In some embodiments, the second processing module is further configured to generate a mapping model; the generation of the mapping model includes: Determine the screen information and first display resolution of the terminal; Obtain the field of view parameters and second display resolution of the augmented reality device; Based on the screen information, the first display resolution, the second display resolution, and the field of view parameters, a mapping model between the first coordinate system and the second coordinate system is established, wherein the first coordinate system is the display coordinate system of the terminal, and the second coordinate system is the display coordinate system of the augmented reality device; Accordingly, converting the first parameter into a second parameter corresponding to the augmented reality device includes: Using the mapping model, the first parameter is converted into a second parameter corresponding to the augmented reality device.

[0095] In some embodiments, the second processing module is configured to determine a scaling factor based on the first display resolution and the second display resolution; The vertical scaling factor is determined based on the scaling factor, the vertical pixel height in the screen information, and the vertical display range in the field of view parameter. The horizontal scaling factor is determined based on the scaling factor, the horizontal pixel height in the screen information, and the horizontal display range in the field of view parameter. A mapping model between the first coordinate system and the second coordinate system is established based on the scaling factor, the vertical scaling factor, and the horizontal scaling factor.

[0096] In some embodiments, the second processing module is further configured to determine a first pixel density of the terminal's screen and a second pixel density of the augmented reality device; Determine whether the difference between the first pixel density and the second pixel density exceeds a difference threshold; If the difference exceeds the difference threshold, the scaling factor is corrected based on the first pixel density and the second pixel density to obtain the corrected scaling factor; Accordingly, the vertical scaling factor is determined based on the scaling factor, the vertical pixel height in the screen information, and the vertical display range in the field of view parameter, including: determining the vertical scaling factor based on the corrected scaling factor, the vertical pixel height in the screen information, and the vertical display range in the field of view parameter; Determining a horizontal scaling factor based on the scaling factor, the horizontal pixel height in the screen information, and the horizontal display range in the field of view parameter includes: determining a horizontal scaling factor based on the corrected scaling factor, the horizontal pixel height in the screen information, and the horizontal display range in the field of view parameter. Establishing a mapping model between the first coordinate system and the second coordinate system based on the scaling factor, the vertical scaling factor, and the horizontal scaling factor includes: establishing a mapping model between the first coordinate system and the second coordinate system based on the modified scaling factor, the vertical scaling factor, and the horizontal scaling factor.

[0097] In some embodiments, the first communication module is further configured to send an acquisition request to the augmented reality device after the terminal establishes a connection with the augmented reality device; Receive the field of view parameters and the second display resolution fed back by the augmented reality device in response to the acquisition request; The terminal is connected to the augmented reality device via wireless or wired communication; each terminal is connected to one or more augmented reality devices.

[0098] In some embodiments, the first processing module is configured to present a visual control interface corresponding to the display content of the augmented reality device according to the display requirements of the augmented reality device; the visual control interface is configured to display at least one object that can be mapped to the display area of ​​the augmented reality device; Detect touch operations used for display control in response to the operation of the visual control interface; The at least one object includes at least one of the following: a text module, an image module, a navigation indicator module, an augmented reality prompt window, and a display preview window; The display preview window is used to display the display effect of the mapped augmented reality device in real time; Each of the objects has an operable display boundary in the visual control interface, and the operation includes at least one of the following: dragging, zooming in, and zooming out.

[0099] In some embodiments, the second processing module is further configured to save the first parameter to a configuration file in response to a confirmation command for detection; In response to the detection call instruction, the first parameter in the configuration file is invoked, and a mapping layout is generated based on the first parameter.

[0100] It is understood that the display control device for augmented reality devices provided in the above embodiments, when implementing the corresponding display control method for augmented reality devices, can allocate the above processing to different program modules as needed to complete all or part of the processing described above. Furthermore, the device and the corresponding method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0101] Figure 6 This application provides a schematic diagram of the structure of a display control device for an augmented reality device, as shown in the embodiments of the present application. Figure 6 As shown, the device is applied to an augmented reality device, and the device includes: The second communication module is used to receive a second parameter from the terminal, wherein the second parameter is a parameter corresponding to the target object; The third processing module is used to update the display effect of the target object according to the second parameter; The second parameter is obtained by converting the first parameter of the terminal. The first parameter is a parameter corresponding to the target object. The first parameter includes at least one of the following: a first horizontal coordinate, a first vertical coordinate, and a first display area size. The first horizontal coordinate and the first vertical coordinate are coordinate values ​​in a first coordinate system.

[0102] It is understood that the display control device for augmented reality devices provided in the above embodiments, when implementing the corresponding display control method for augmented reality devices, can allocate the above processing to different program modules as needed to complete all or part of the processing described above. Furthermore, the device and the corresponding method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0103] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a display control method for an augmented reality device.

[0104] This application provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are executed by a processor, causing the processor to execute the display control method for augmented reality devices provided in this application.

[0105] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0106] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0107] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0108] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0109] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; as shown below. Figure 7 As shown, the electronic device 70 includes a processor 701 and a memory 702 communicatively connected to the processor 701; the memory 702 stores instructions executable by the processor 701. The instructions are executed by the processor 701 to enable the processor 701 to perform the aforementioned display control method for the augmented reality device on the terminal side; or, to enable the processor 701 to perform the aforementioned display control method for the augmented reality device on the augmented reality device side.

[0110] The electronic devices provided in the above embodiments and the corresponding display control methods for augmented reality devices belong to the same concept. For details of their implementation, please refer to the method embodiments, which will not be repeated here.

[0111] In practical applications, the electronic device 70 may further include at least one network interface 703. The various components of the electronic device 70 are coupled together via a bus system 704. It is understood that the bus system 704 is used to implement communication between these components. In addition to a data bus, the bus system 704 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 7 All buses are labeled as bus system 704. The number of processors 701 and the number of memories 702 can be at least one. The network interface 703 is used for wired or wireless communication between the electronic device 70 and other devices.

[0112] The memory 702 in this embodiment is used to store various types of data to support the operation of the electronic device 70.

[0113] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 701. Processor 701 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 701 or by instructions in software form. The processor 701 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 701 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 702. Processor 701 reads the information in memory 702 and, in conjunction with its hardware, completes the steps of the aforementioned display control method for augmented reality devices.

[0114] In some embodiments, the electronic device 70 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned methods.

[0115] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0116] In the above description, the term "some embodiments" refers to a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0117] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this application is for the purpose of describing embodiments of this application only and is not intended to be limiting of this application.

[0118] It should be understood that in the various embodiments of this application, the sequence number of each implementation process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0119] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0120] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A display control method for an augmented reality device, characterized by, Applied to a terminal, the method comprises: detecting a touch operation for display control, determining a first parameter according to the touch operation, the first parameter being a parameter of a corresponding target object, the first parameter comprising at least one of a first horizontal coordinate, a first vertical coordinate and a first display area size; the first horizontal coordinate and the first vertical coordinate being coordinate values in a first coordinate system; converting the first parameter into a second parameter corresponding to the augmented reality device; sending the second parameter to the augmented reality device, the augmented reality device being configured to update a display effect of the target object according to the second parameter.

2. The method of claim 1, wherein, The method further comprises generating a mapping model, and the generating of the mapping model comprises: determining screen information and a first display resolution of the terminal; obtaining a field of view angle parameter and a second display resolution of the augmented reality device; establishing a mapping model between a first coordinate system and a second coordinate system according to the screen information, the first display resolution, the second display resolution and the field of view angle parameter, the first coordinate system being a display coordinate system of the terminal, and the second coordinate system being a display coordinate system of the augmented reality device; correspondingly, the converting of the first parameter into the second parameter corresponding to the augmented reality device comprises: converting the first parameter into the second parameter corresponding to the augmented reality device by using the mapping model.

3. The method of claim 2, wherein, The establishing of the mapping model between the first coordinate system and the second coordinate system according to the screen information, the first display resolution, the second display resolution and the field of view angle parameter comprises: determining a proportionality coefficient according to the first display resolution and the second display resolution; determining a vertical scaling factor according to the proportionality coefficient, a vertical pixel height in the screen information and a vertical display range in the field of view angle parameter; determining a horizontal scaling factor according to the proportionality coefficient, a horizontal pixel height in the screen information and a horizontal display range in the field of view angle parameter; establishing the mapping model between the first coordinate system and the second coordinate system according to the proportionality coefficient, the vertical scaling factor and the horizontal scaling factor.

4. The method of claim 3, wherein, The method further comprises: determining a first pixel density of a screen of the terminal and a second pixel density of the augmented reality device; judging whether a difference degree of the first pixel density and the second pixel density exceeds a difference degree threshold; if the difference degree exceeds the difference degree threshold, correcting the proportionality coefficient according to the first pixel density and the second pixel density to obtain a corrected proportionality coefficient; correspondingly, the determining of the vertical scaling factor according to the proportionality coefficient, the vertical pixel height in the screen information and the vertical display range in the field of view angle parameter comprises: determining the vertical scaling factor according to the corrected proportionality coefficient, the vertical pixel height in the screen information and the vertical display range in the field of view angle parameter. According to the proportion coefficient, and the horizontal pixel height in the screen information and the horizontal display range in the field of view angle parameter, a horizontal scaling factor is determined, including: according to the modified proportion coefficient, and the horizontal pixel height in the screen information and the horizontal display range in the field of view angle parameter, a horizontal scaling factor is determined; According to the proportion coefficient, the vertical scaling factor and the horizontal scaling factor, a mapping model between the first coordinate system and the second coordinate system is established, including: according to the modified proportion coefficient, the vertical scaling factor and the horizontal scaling factor, a mapping model between the first coordinate system and the second coordinate system is established.

5. The method of claim 2, wherein, The field of view angle parameter and the second display resolution of the augmented reality device are obtained, including: After the terminal and the augmented reality device establish a connection, a request for obtaining is sent to the augmented reality device; The field of view angle parameter and the second display resolution fed back by the augmented reality device in response to the request for obtaining are received; The terminal and the augmented reality device are connected in a manner of wireless communication or wired communication; each terminal is connected with one or more augmented reality devices.

6. The method of claim 1, wherein, A touch operation for display control is detected, including: According to the display requirement of the augmented reality device, a visual control interface corresponding to the display content of the augmented reality device is presented; the visual control interface is used to display at least one object that can be mapped to the display area of the augmented reality device; A touch operation for display control is detected for the visual control interface operation; The at least one object includes at least one of the following: a text module, an image module, a navigation instruction module, an augmented reality prompt window and a display preview window; The display preview window is used to display the display effect of the augmented reality device after mapping in real time; Each object has an operable display boundary in the visual control interface, and the operation includes at least one of the following: dragging, zooming in and zooming out.

7. The method of claim 1, wherein, The method further includes: In response to the detected confirmation instruction, the first parameter is saved to a configuration file; In response to the detected calling instruction, the first parameter in the configuration file is called, and a mapping layout is generated based on the first parameter.

8. A display control method for an augmented reality device, characterized by, Applied to an augmented reality device, the method includes: A second parameter corresponding to a target object is received from a terminal; The display effect of the target object is updated according to the second parameter; The second parameter is converted from a first parameter of the terminal, and the first parameter corresponds to the target object, and the first parameter includes at least one of the following: a first horizontal coordinate, a first vertical coordinate and a first display area size; the first horizontal coordinate and the first vertical coordinate are coordinate values in a first coordinate system.

9. A display control device for an augmented reality device, characterized by, The device includes: The first processing module is configured to detect a touch operation for display control, determine a first parameter according to the touch operation, the first parameter being a parameter of a target object, and the first parameter including at least one of a first horizontal coordinate, a first vertical coordinate, and a first display area size; the first horizontal coordinate and the first vertical coordinate being coordinate values in a first coordinate system. The second processing module is configured to convert the first parameter into a second parameter corresponding to the augmented reality device. The first communication module is configured to send the second parameter to the augmented reality device, and the augmented reality device is configured to update a display effect of the target object according to the second parameter.

10. A display control device for an augmented reality device, characterized by, The device is applied to an augmented reality device, and includes: The second communication module is configured to receive a second parameter from a terminal, the second parameter being a parameter of a target object. The third processing module is configured to update a display effect of the target object according to the second parameter. The second parameter is converted from a first parameter of the terminal, the first parameter being a parameter of a target object, and the first parameter including at least one of a first horizontal coordinate, a first vertical coordinate, and a first display area size; the first horizontal coordinate and the first vertical coordinate being coordinate values in a first coordinate system.

11. An electronic device, comprising: The device includes: At least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1 to 7; or to enable the at least one processor to execute the method of claim 8.

12. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to execute the method of any one of claims 1 to 7; or the computer instructions are used to enable the computer to execute the method of claim 8.

Images