AR / VR multi-state interaction method and head-mounted display

By dynamically adjusting the color and shape of the AR/VR cursor, the interaction difficulties caused by the similarity between the cursor and the background color is solved, high-precision and natural interaction control are achieved, and user experience is improved.

CN119024992APending Publication Date: 2024-11-26HISENSE VISUAL TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310605849.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-25
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

During the AR/VR interaction process, the color and shape of the cursor are the same or similar to the application interface background, making it difficult for users to position the cursor quickly and accurately, resulting in interaction errors and degraded user experience.

Method used

By intercepting the target image of the application interface, segmenting the background area, extracting the background color, and selecting candidate colors with color differences from the background color from the preset color table, dynamically adjusting the color and shape of the cursor to eliminate background interference and ensure that the cursor is in Visibility in the interface.

Benefits of technology

It improves the accuracy and speed of interaction, reduces erroneous interactions, and improves the quality of user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119024992A_ABST
    Figure CN119024992A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of AR / VR, and provides an AR / VR multi-state interaction method and head-mounted display equipment, which are used for improving the accuracy of interaction control. According to the method, the intercepted target image of the application interface is segmented to obtain at least one background reference area, and the background color of each background reference area is extracted, so that the interference of each background color on the cursor operation is avoided; according to the method, at least one candidate color which is visually different from each background color is screened from the preset color table to set the target color of the cursor controlled by the head-mounted display or the handle, so that a user can quickly and accurately find the cursor in an application interface to realize high-precision and natural interaction control; in addition, the color and the shape of the cursor can be set in a personalized mode, different user requirements are met, and the user experience can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of AR / VR technology and provides an AR / VR polymorphic interaction method and a head-mounted display. Background Art

[0002] With the development of technologies such as virtual reality (VR) and augmented reality (AR), AR / VR helmets, AR / VR glasses and other devices have been widely used in various industries, such as education and training, fire drills, virtual driving, real estate, etc., bringing users an immersive visual feast. At the same time, natural and smooth interaction is also the key to improving user satisfaction experience.

[0003] At present, when VR / AR helmets, AR / VR glasses and other devices are interactively operated through AR / VR technology, head control and handle control are indispensable interaction methods. In some actual scenarios of AR / VR applications, during the interaction process, the cursor controlled by the head mounted display (HMD) or the handle is constantly moving or switching in the application interface, while the shape and color of the cursor are fixed during the current interaction process. In this way, when the color and shape of the cursor are the same or similar to the background in the application interface, the user cannot quickly and clearly obtain the cursor position, and thus cannot interact or generates erroneous interactions. In addition, the same or similar colors and shapes are prone to cause penetration misunderstandings, which seriously affects the user experience. Summary of the invention

[0004] The embodiments of the present application provide an AR / VR polymorphic interaction method and a head-mounted display, which are used to improve the accuracy and reliability of interactive control.

[0005] On the one hand, an embodiment of the present application provides an AR / VR polymorphic interaction method, including:

[0006] Intercepting an application interface played by a head-mounted display to obtain a target image; wherein the head-mounted display includes an AR device or a VR device;

[0007] Segmenting the target image to obtain at least one background reference area in the target image, and extracting the background color of each background reference area;

[0008] According to each background color, at least one candidate color available for the cursor controlled by the handle or the head mounted display is selected from a preset color table, each candidate color having a visual color difference from each background color;

[0009] A target color of the cursor is determined from the at least one candidate color, and the cursor is displayed in the application interface with the target color.

[0010] On the other hand, an embodiment of the present application provides a head-mounted display, the head-mounted display comprising a VR device and an AR device, the head-mounted display comprising a processor, a memory and a display, the memory, the display and the processor being connected via a bus;

[0011] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0012] Capture the playing application interface to obtain the target image;

[0013] Segmenting the target image to obtain at least one background reference area in the target image, and extracting the background color of each background reference area;

[0014] According to each background color, at least one candidate color available for the cursor controlled by the handle or the head mounted display is selected from a preset color table, each candidate color having a visual color difference from each background color;

[0015] A target color of the cursor is determined from the at least one candidate color, and the cursor is displayed in the application interface in the target color through the display.

[0016] Optionally, the processor determines the target color of the cursor from the at least one candidate color by:

[0017] Automatically select a candidate color from the at least one candidate color as the target color of the cursor edge area, keeping the original color of the cursor fill area unchanged; or

[0018] Automatically select a candidate color from the at least one candidate color as the target color of the cursor fill area, keeping the original color of the cursor edge area unchanged; or

[0019] From the at least one candidate color, a candidate color is automatically selected as a target color for the cursor fill area and the cursor edge area.

[0020] Optionally, the processor further executes:

[0021] Sets the color of the rays connected by the cursor to the target color.

[0022] Optionally, the processor further executes:

[0023] According to the preset shape list, a target shape is determined as the shape of the cursor, and the cursor is displayed in the application interface in the target shape through the display.

[0024] Optionally, after determining the target color of the cursor and before displaying the cursor, the processor further executes:

[0025] Circularly switching the at least one selected candidate color to provide the user with a personalized color selection;

[0026] Replace the target color with the candidate color selected by the user.

[0027] Optionally, each color in the preset color table corresponds to a shape, and the shape of the cursor automatically switches as the color switches.

[0028] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute the steps of an AR / VR polymorphic interaction method provided in an embodiment of the present application.

[0029] In an AR / VR polymorphic interaction method and head-mounted display provided by an embodiment of the present application, at least one background reference area is obtained by segmenting a captured target image of an application interface, and the background color of each background reference area is extracted. Since each background color will interfere with the color of the cursor, at least one candidate color that has a visual color difference with each background color is screened out from a preset color table, and then the target color of the cursor controlled by the head-mounted display or the handle is set from the at least one candidate color to eliminate the interference of the same or similar background colors, so that the user can quickly and accurately find the cursor in the application interface, thereby achieving high-precision and natural interactive control.

[0030] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0032] Figure 1A schematic diagram of an AR / VR controller provided in an embodiment of the present application;

[0033] Figure 2 A schematic diagram of an AR / VR interaction scenario provided in an embodiment of the present application;

[0034] Figure 3A This is the interface picture when the cursor color is the same as the background color of the application interface;

[0035] Figure 3B This is the interface image when the cursor color is similar to the background color of the application interface;

[0036] Figure 4A An architectural diagram of the AR / VR polymorphic interaction method provided in an embodiment of the present application;

[0037] Figure 4B A structural diagram of the AR / VR handle provided in an embodiment of the present application;

[0038] Figure 5 A flowchart of the AR / VR polymorphic interaction method provided in an embodiment of the present application;

[0039] Figure 6 A comparison diagram of the interference of different application interfaces provided in the embodiment of the present application on the cursor of the same form;

[0040] Figure 7 The preset colors provided in the embodiments of the present application represent the intended purpose;

[0041] Fig. 8A A schematic diagram of a process for determining a cursor color based on a background color is provided for an embodiment of the present application;

[0042] Figure 8B Another schematic diagram of a process for determining a cursor color based on a background color is provided for an embodiment of the present application;

[0043] Fig. 9 Provide a display effect diagram after the cursor color is changed for the embodiment of the present application;

[0044] Fig.10 Provide a display effect diagram after the ray color is changed for the embodiment of the present application;

[0045] Fig.11 This is a comparison chart of cursors of different colors and shapes in an application interface with the same background color;

[0046] Fig.12 Provide a color cycle switching schematic diagram for the embodiment of the present application;

[0047] Fig.13 A schematic diagram of cyclic switching of colors and shapes is provided for an embodiment of the present application;

[0048] Fig.14 The structure of a head-mounted display provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0050] Some of the terms used in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0051] Cursor: An icon displayed in different forms.

[0052] Focus: The location where the cursor is activated. The control at the focus can be operated, including but not limited to clicking, dragging, scrolling, etc.

[0053] Polymorphism: refers to the cursor having multiple presentation states, including changes in color, size, and shape.

[0054] The following is an overview of the design concepts of the embodiments of the present application.

[0055] For electronic devices such as AR / VR, head control and handle control are indispensable interaction methods, just as mouse or touch control methods are important for personal computers (PCs). During the interaction process, the rays emitted by the handle or HMD form a cursor in the application interface. The user moves the cursor to the focus and clicks, drags, scrolls, and other operations on the control at the focus to control the application interface.

[0056] Taking handle control as an example, AR / VR and other head-mounted display devices have positioning and tracking functions for handles, which are divided into 3DOF (only posture) positioning and 6DOF (including posture and position) positioning. Since 6DOF has higher positioning accuracy, it is the current mainstream interactive control method. After the handle is located, the user uses the buttons on the handle to operate the controls at the focus of the cursor to realize the interactive function.

[0057] like Figure 1 As shown in the figure, it is a schematic diagram of the handle, including the trigger button, side grip button, home button, return button, joystick, AB button, etc. During the interaction, the ray emitted by the handle points to the application interface and forms a cursor on the application interface, such as Figure 2When the cursor is currently focused on the "OK" control, the user can press the confirmation key or pull the trigger key to control the application interface, thereby realizing interaction with head-mounted display devices such as AR / VR.

[0058] In the current interaction mode, the color and shape of the cursor controlled by the handle or HMD in the application interface are usually fixed, such as a round white cursor. In actual AR / VR interaction scenarios, the cursor in the application interface of the ray emitted by the handle or HMD will be constantly moved and switched, which will cause the following problems: 1) When the ray and the cursor are the same or similar to the background color in the application interface, the background color will interfere with the cursor, and the user may not be able to quickly and accurately find the cursor, so that the interactive operation cannot be performed or the interactive operation may be wrong, affecting the user experience; 2) When the ray and the cursor are the same color as the objects in the background (such as text, vases, etc.), ray penetration may occur, affecting the realism of the picture.

[0059] For example, Figure 3A The following is an interface diagram when the cursor color is the same as the background color of the application interface. When the user wants to operate the target object in the application interface (in Figure 3A When you need to move the circular cursor (in the Figure 3A However, since the color of the circular cursor is the same as the background color, the user cannot quickly find the current position of the cursor, which reduces the interaction speed.

[0060] For example, Figure 3B The following is an interface diagram when the cursor color is similar to the background color of the application interface. When the user wants to operate the target object in the application interface (in Figure 3A When you need to move the circular cursor (in the Figure 3A However, since the color of the circular cursor is similar to the background color, the user cannot quickly find the current location of the cursor, which reduces the interaction speed.

[0061] In view of this, the embodiment of the present application provides an AR / VR polymorphic interaction method, which can adaptively adjust the color of the cursor according to the background color of the application interface, thereby reducing the interference of the same or similar background color on the cursor, facilitating the user to quickly and accurately find the cursor, thereby improving the speed and accuracy of the interaction and ensuring the quality of user experience. At the same time, the method can adjust the shape of the cursor.

[0062] It should be noted that the AR / VR polymorphic interaction method provided in the embodiment of the present application can be applied to scenarios in which a handle is used for interaction, and can also be applied to scenarios in which interaction is performed by wearing a head mounted display (HMD).

[0063] Take the example of using a handle to interact, see Figure 4A , is an architecture diagram of the AR / VR polymorphic interaction method provided in an embodiment of the present application, where the HMD is connected to the handle via a wireless module, such as Figure 4B The figure shows the structure of the handle, which includes a microcontroller unit (MCU), a light ring (infrared or LED), physical function keys, a wireless module, a reset circuit, and a power supply. Based on the handle image including the light ring captured by the camera on the HMD, the 6DoF position of the handle is determined through the positioning and tracking system. Based on the 6DoF position of the handle, the position of the cursor on the application interface of the ray emitted by the handle is determined, and the control at the focus of the cursor is operated based on the physical function keys, so as to realize the control of the HMD application interface by the handle.

[0064] During the interaction, the cursor constantly moves or switches in the application interface. To avoid the background color in the application interface interfering with the cursor, the currently playing application interface can be captured. For the captured image, a color-based image segmentation algorithm is used to segment it and divide it into larger areas. Usually, the large area in the application interface is generally the background. Therefore, through color segmentation, the background reference area where the background color that interferes with the cursor is located can be obtained, so that the background color is compared with the color in the preset color table, and the color with a large color difference from the background color is selected as the color available to the cursor, thereby eliminating the interference of the background color and allowing users to quickly and accurately find the cursor. At the same time, the shape of the cursor can also be adaptively changed.

[0065] The implementation process of an AR / VR polymorphic interaction method provided in the embodiment of the present application is as follows: Figure 5 As shown, a head mounted display is applied, wherein the head mounted display includes an AR device or a VR device. The process mainly includes the following steps:

[0066] S501: intercepting the application interface played by the head mounted display to obtain a target image.

[0067] During the interaction process of controlling the HMD by head movement or by a handle, the rays emitted by the HMD or the handle form a cursor on the application interface, and the cursor will run through the entire interaction process, constantly switching and moving in different scene interfaces or interface areas. In this way, the background color of the application interface may be the same or similar to the color of the ray and the cursor, making it impossible for the user to quickly and accurately find the cursor during the interaction process.

[0068] For example, Figure 6 As shown, the left picture shows the case where the background color of the application interface is blue. When the ray emitted by the handle or HMD is a white circle in the application interface, the background color in the application interface will not interfere with the cursor due to the large visual color difference between blue and white. In the right picture, the background color of the application interface is white and light gray. When the ray emitted by the handle or HMD is a white circle in the application interface, since white and light gray are the same and similar to the color of the cursor, the background color in the application interface will interfere with the cursor, making it difficult to locate the cursor for interactive operations.

[0069] Generally, in an interactive scenario, the background color of the application interface cannot be changed or adjusted. Therefore, in order to solve the interference of the background color on the cursor, the AR / VR polymorphic interaction method provided in the embodiment of the present application can dynamically adjust the color and shape of the cursor.

[0070] In one example, after starting an AR / VR application, the first frame of the application interface being played is captured to obtain a target image, and then background color analysis is performed based on the captured target image to eliminate the interference of the background color on the cursor color.

[0071] Considering that the application interface played by the HMD is constantly changing, the background colors of different application interfaces may be different. Therefore, in one example, multiple target images of the application interface can be periodically captured at a preset frequency, so that when the background color of the application interface changes, the color and shape of the cursor can be flexibly adjusted.

[0072] In one example, the background color in the application interface can also be identified in real time. When the identified background color is the same as or similar to the color of the current cursor, the process of adjusting the color and shape of the cursor is triggered.

[0073] It should be noted that the application interface in the embodiment of the present application can be a virtual screen displayed by a VR helmet, or it can be a virtual and real combined screen displayed by AR glasses.

[0074] S502: Segment the captured target image to obtain at least one background reference area in the target image, and extract the background color of each background reference area.

[0075] Considering that the background is generally a larger area in the application interface, the intercepted target image can be segmented according to the area, and the area with an area ratio greater than a preset ratio threshold is used as a background reference area. There can be one or more background reference areas in the application interface, that is, there can be one or more background colors.

[0076] For example, Figure 6As shown, in the left figure, the background reference area occupies the entire application interface. A pixel is selected from the background reference area, and the values ​​of the three channels R, G, and B of the pixel are read. It can be obtained that the color of the background reference area is blue, that is, the color that interferes with the cursor in the application interface is blue; the right figure contains two background reference areas, and the extracted colors are white and light gray, that is, the colors that interfere with the cursor in the application interface are white and light gray.

[0077] S503: Select at least one candidate color available for a cursor controlled by a handle or a head mounted display from a preset color table according to each background color.

[0078] Since the cursor controlled by the handle or HMD will constantly move or switch in the application page, when adjusting the color of the cursor, the interference of the background colors of different background reference areas on the cursor must be taken into account.

[0079] The embodiment of the present application pre-sets a preset color table containing a variety of colors with high saturation and easy to distinguish, and numbers each color for the cursor to adjust the color.

[0080] For example, Figure 7 As shown, the preset color table includes 6 colors, namely, No. 1 red, No. 2 yellow, No. 3 green, No. 4 cyan, No. 5 blue and No. 6 purple.

[0081] Since the cursor will move and switch in the entire application interface according to the user's interaction needs, all background colors need to be excluded when adjusting the cursor color. In specific implementation, after determining the background color of each background reference area, a color contrast algorithm can be used to select at least one candidate color available for the cursor controlled by the handle or HMD from a preset color table to exclude colors that are the same as or similar to each background color. Among them, each candidate color has a significant visual color difference from each background color. In this way, any candidate color in at least one of the selected candidate colors can be used as a color available to the cursor, thereby maximizing the exclusion of background color interference and reducing erroneous interaction operations and penetration misunderstandings.

[0082] For example, Fig. 8A As shown, the background color of the background reference area S1 in the application interface is blue No. 5, and the background color of the background reference area S2 is yellow No. 2, that is, there are two interfering colors. At this time, the remaining colors in the preset color table (namely red No. 1, green No. 3, cyan No. 4 and purple No. 6) can be used as candidate colors available for the cursor.

[0083] In one example, after screening out at least one candidate color available for the cursor, a different candidate color may be assigned to the cursor in each background reference area according to the number of background reference areas included in the application interface.

[0084] For example, Figure 8B As shown, the available candidate colors for the cursor in the application interface are red 1, green 3, cyan 4 and purple 6, and the application interface contains two background reference areas. At this time, the four candidate colors can be equally divided into S1 and S2, that is, when the cursor is located at S1, the available candidate colors for the cursor are red 1 and green 3, and when the cursor is located at S1, the available candidate colors for the cursor are cyan 4 and purple 6.

[0085] S504: Determine a target color of the cursor from at least one candidate color, and display the cursor in the application interface with the target color.

[0086] After excluding colors that are the same as or similar to each background color from the preset color table, the color of the cursor can be intelligently changed from at least one remaining candidate color to eliminate interference from the background color.

[0087] In one example, there are three main ways to change the cursor color: the first is to automatically select a candidate color from at least one candidate color as the target color of the cursor edge area, keeping the original color of the cursor fill area unchanged; the second is to automatically select a candidate color from at least one candidate color as the target color of the cursor fill area, keeping the original color of the cursor edge area unchanged; the third is to automatically select a candidate color from at least one candidate color as the target color of the cursor fill area and the cursor edge area.

[0088] After determining the target color of the cursor, the cursor is displayed in the application interface with the target color, so that the user can quickly and accurately move the cursor to the corresponding control for interactive operations.

[0089] For example, if the background color of the background reference area in the application interface is light gray, when the first method is used to change the cursor color, the cursor display effect is as follows: Fig. 9 When the second method is used to change the cursor color, the cursor display effect is as follows: Fig. 9 As shown in (b) in the figure; when the third method is used to change the cursor color, the cursor display effect is as follows Fig. 9 As shown in (c) in .

[0090] In the AR / VR polymorphic interaction method provided in the embodiment of the present application, in order to address the problem that the cursor in the AR / VR application is easily interfered by the background color during the interaction process, that is, the color of the cursor controlled by the handle or HMD during the interaction process is the same as or similar to the background color, thereby interfering with the user's operation, the color of the cursor itself can be automatically adjusted according to the actual scene, thereby eliminating the interference of the same or similar background color, allowing the user to quickly and accurately move the cursor to the corresponding control, thereby achieving high-precision natural interactive control.

[0091] In some embodiments, when the rays emitted by the handle or HMD are visible, in order to improve the coordination of the interactive page, after determining the target color of the cursor, the color of the ray connected by the cursor can also be set to the target color, such as Fig.10 shown.

[0092] In some embodiments, the shape of the cursor may be the same or similar to the target object in the application interface, for example, the round cursor in the billiard scene is the same as the colorful billiard ball. When there are multiple target objects in the application interface, the shape of the cursor can be adjusted to make the user find the cursor faster.

[0093] In specific implementation, multiple shapes can be pre-set to generate a preset shape list. During the interaction process, a target shape is determined from the preset shape list as the cursor shape, and the cursor is displayed in the application interface with the target shape. The target shape can be selected by the user or automatically selected by the HMD according to the actual application scenario.

[0094] like Fig.11 The following figure shows the contrast effect of cursors of different colors and shapes in the application interface with the same background color. By dynamically adjusting the color and shape of the cursor itself, the cursor and the background color and shape of the target object in the application interface are clearly distinguished, eliminating operational interference to the greatest extent, thereby improving the accuracy and reliability of interactive control and ensuring a natural and smooth interactive experience for users.

[0095] In some embodiments, after the cursor color is changed, at least one selected candidate color may be cyclically switched to allow the user to select a personalized color, such as Fig.12 As shown, the target color is replaced with the candidate color selected by the user to achieve personalized color replacement.

[0096] In some embodiments, each color in the preset color table corresponds to a shape, so that when the at least one candidate color selected is cyclically switched for the user to select a personalized color, the shape of the cursor automatically switches with the switching of the color, such as Fig.13 As shown, personalized color change is achieved while personalized cursor shape change is achieved.

[0097] In an optional implementation, the HMD and the handle can individually change the color and shape of the cursor according to needs based on specific actions.

[0098] For example, when the cursor is controlled by the HMD, the user can double-click the HMD to cycle through the cursor color and shape, and select a personalized color and shape by pressing the trigger key; when the cursor is controlled by the handle, the user can double-click the function key to cycle through the cursor color and shape, and select a personalized color and shape by pressing the trigger key.

[0099] It should be noted that the control method of the HMD and the handle can be actually operated according to the actual hardware functions, and the embodiments of the present application do not make any restrictive requirements.

[0100] In the AR / VR polymorphic interaction method provided in the embodiment of the present application, by cyclically switching the color and shape of the cursor, while eliminating the interference of the background color and the shape of the target object in the application interface on the cursor operation, the personalized change of the cursor color and shape can be realized, which improves the interactive response speed and accuracy, and helps to enhance the user's AR / VR experience.

[0101] In one example, when using a handle for interaction, after determining the target color and target shape of the cursor, the HMD may also send the target color and target shape to the handle through a wireless module, so that the handle changes the configuration of the cursor.

[0102] Based on the same technical concept, an embodiment of the present application provides a head-mounted display, which can implement the steps of the above-mentioned AR / VR polymorphic interaction method and achieve the same technical effect.

[0103] See also Fig.14 The head mounted display comprises a processor 1401, a memory 1402 and a display 1403, wherein the display 1403, the memory 1402 and the processor 1401 are connected via a bus 1404;

[0104] The memory 1402 stores a computer program, and the processor 1401 performs the following operations according to the computer program:

[0105] Capture the playing application interface to obtain the target image;

[0106] Segmenting the target image to obtain at least one background reference area in the target image, and extracting the background color of each background reference area;

[0107] According to each background color, at least one candidate color available for a cursor controlled by a handle or a head mounted display is selected from a preset color table, each candidate color having a visual color difference from each background color;

[0108] A target color of the cursor is determined from at least one candidate color, and the cursor is displayed in the application interface with the target color through the display 1403 .

[0109] Optionally, the processor 1401 determines a target color of the cursor from at least one candidate color by:

[0110] Automatically select a candidate color from at least one candidate color as the target color of the cursor edge area, and keep the original color of the cursor fill area unchanged; or

[0111] Automatically select a candidate color from at least one candidate color as the target color of the cursor fill area, and keep the original color of the cursor edge area unchanged; or

[0112] A candidate color is automatically selected from at least one candidate color as a target color for a cursor fill area and a cursor edge area.

[0113] Optionally, the processor 1401 further executes:

[0114] Sets the color of the ray emitted by the handle or HMD that the cursor is connected to to the target color.

[0115] Optionally, the processor 1401 further executes:

[0116] According to the preset shape list, a target shape is determined as the shape of the cursor, and the cursor is displayed in the application interface in the target shape through the display.

[0117] Optionally, after determining the target color of the cursor and before displaying the cursor, the processor 1401 further executes:

[0118] Circularly switching at least one of the selected candidate colors for the user to select a personalized color;

[0119] Replaces the target color with a candidate color selected by the user.

[0120] Optionally, each color in the preset color table corresponds to a shape, and the shape of the cursor automatically switches as the color switches.

[0121] In the head-mounted display provided in the embodiment of the present application, by segmenting the captured target image of the application interface, at least one background reference area is obtained, and the background color of each background reference area is extracted. Since each background color will interfere with the color of the cursor, at least one candidate color that has a visual color difference with each background color is selected from a preset color table, and then the target color of the cursor controlled by the head-mounted display or the handle is set from the at least one candidate color to eliminate the interference of the same or similar background colors, so that the user can quickly and accurately find the cursor in the application interface, thereby achieving high-precision and natural interactive control.

[0122] It should be noted that Fig.14This is just an example, and provides the necessary hardware for the head mounted display to execute the steps of an AR / VR polymorphic interaction method provided in an embodiment of the present application. If not shown, the head mounted display may also include the hardware of conventional AR / VR devices such as speakers, microphones, communication interfaces, and power supplies.

[0123] Embodiments of the present application Fig.14 The processor involved may be a central processing unit (CPU), a general-purpose processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.

[0124] An embodiment of the present application also provides a computer-readable storage medium for storing some instructions, which, when executed, can complete an AR / VR polymorphic interaction method in the aforementioned embodiment.

[0125] An embodiment of the present application also provides a computer program product for storing a computer program, wherein the computer program is used to execute an AR / VR polymorphic interaction method in the aforementioned embodiment.

[0126] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0127] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0128] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0129] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0130] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An AR / VR polymorphic interaction method, characterized in that: The method comprises: Intercepting an application interface played by a head-mounted display to obtain a target image; wherein the head-mounted display includes an AR device or a VR device; Segmenting the target image to obtain at least one background reference area in the target image, and extracting the background color of each background reference area; According to each background color, at least one candidate color available for the cursor controlled by the handle or the head mounted display is selected from a preset color table, each candidate color having a visual color difference from each background color; A target color of the cursor is determined from the at least one candidate color, and the cursor is displayed in the application interface with the target color.

2. The method according to claim 1, characterized in that The step of determining a target color of the cursor from the at least one candidate color comprises: Automatically select a candidate color from the at least one candidate color as the target color of the cursor edge area, keeping the original color of the cursor fill area unchanged; or Automatically select a candidate color from the at least one candidate color as the target color of the cursor fill area, keeping the original color of the cursor edge area unchanged; or From the at least one candidate color, a candidate color is automatically selected as a target color for the cursor fill area and the cursor edge area.

3. The method according to claim 2, characterized in that The color of the ray emitted by the handle or the head mounted display to which the cursor is connected is set to the target color.

4. The method according to claim 1, characterized in that The method further comprises: According to the preset shape list, a target shape is determined as the shape of the cursor, and the cursor is displayed in the application interface with the target shape.

5. The method according to claim 1, characterized in that After determining the target color of the cursor and before displaying the cursor, the method further includes: Circularly switching the at least one selected candidate color to provide the user with a personalized color selection; Replace the target color with the candidate color selected by the user.

6. The method according to claim 5, characterized in that Each color in the preset color table corresponds to a shape, and the shape of the cursor automatically switches as the color switches.

7. A head mounted display, characterized in that: The head mounted display includes a VR device and an AR device, the head mounted display includes a processor, a memory and a display, and the memory, the display and the processor are connected via a bus; The memory stores a computer program, and the processor performs the following operations according to the computer program: Capture the playing application interface to obtain the target image; Segmenting the target image to obtain at least one background reference area in the target image, and extracting the background color of each background reference area; According to each background color, at least one candidate color available for the cursor controlled by the handle or the head mounted display is selected from a preset color table, each candidate color having a visual color difference from each background color; A target color of the cursor is determined from the at least one candidate color, and the cursor is displayed in the application interface in the target color through the display.

8. The head mounted display according to claim 7, wherein: The processor also executes: The color of the ray emitted by the handle or the head mounted display to which the cursor is connected is set to the target color.

9. The head mounted display according to claim 7, wherein: The processor also executes: According to the preset shape list, a target shape is determined as the shape of the cursor, and the cursor is displayed in the application interface in the target shape through the display.

10. The head mounted display according to claim 7, wherein: After determining the target color of the cursor and before displaying the cursor, the processor further executes: Circularly switching the at least one selected candidate color to provide the user with a personalized color selection; Replace the target color with the candidate color selected by the user.