Method, apparatus, and related device for searching operation control of extended reality device

Abstract

The application provides a method for searching an operation control of an extended reality device and a related device. The method comprises the following steps: detecting that a user wears the extended reality device, and acquiring a communication state between the extended reality device and the operation control; determining an information acquisition mode corresponding to the communication state, and acquiring an operation control position according to the information acquisition mode; and prompting or displaying the operation control position on a display screen. The application selects a corresponding information acquisition mode according to the communication state with the operation control. Different communication states are matched with different information acquisition modes. Then, the operation control position is acquired according to the determined information acquisition mode, and the operation control position is prompted or displayed on the display screen of the extended reality device. Therefore, the user can easily find the operation control according to the prompted or displayed position, thereby saving the time for searching the operation control and facilitating the use of the user.

Description

Technical Field

[0001] This application relates to the field of data detection and analysis technology, and in particular to a method, apparatus, and related equipment for extending the search operation controls of a reality device. Background Technology

[0002] In the current technology, there are more and more extended reality devices that can display virtual space, especially in some gaming fields.

[0003] Currently, when users wear augmented reality devices, they may not know the location of the control controls. As a result, users have to fumble around to find the control controls, which causes a lot of inconvenience. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a method, apparatus, and related equipment for extending the search operation controls of real-world devices to solve or partially solve the above-mentioned technical problems.

[0005] To achieve the above objectives, the first aspect of this application provides a method for extending the search operation controls of a real-world device, comprising:

[0006] The system detects that the user is wearing an augmented reality device and obtains the communication status between the augmented reality device and the operation controls.

[0007] Determine the information acquisition method corresponding to the communication state, and obtain the position of the operation control according to the information acquisition method;

[0008] The location of the operation controls is displayed or prompted on the screen.

[0009] Based on the same inventive concept, a second aspect of this application proposes an apparatus for extending the search operation controls of a real-world device, comprising:

[0010] The communication status determination module is configured to detect when a user puts on an extended reality device and obtain the communication status between the extended reality device and the operation controls;

[0011] The location acquisition module is configured to determine the information acquisition method corresponding to the communication state, and acquire the position of the operation control according to the information acquisition method.

[0012] The prompt or display module is configured to display the position of the operation control on the display screen.

[0013] Based on the same inventive concept, a third aspect of this application proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0014] Based on the same inventive concept, a fourth aspect of this application proposes a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method described in the first aspect.

[0015] Based on the same inventive concept, the fifth aspect of this application proposes a computer program product, including computer program instructions that, when run on a computer, cause the computer to perform the method described in the first aspect.

[0016] As can be seen from the above, the method, apparatus, and related devices for searching operation controls in extended reality devices provided in this application allow the user to select an appropriate information acquisition method based on the communication state with the operation controls when the user wears the extended reality device. Different communication states will match different information acquisition methods. Then, the location of the operation controls will be obtained according to the determined information acquisition method, and the location of the operation controls will be prompted or displayed on the display screen of the extended reality device. In this way, the user can easily find the operation controls according to the prompted or displayed location, thereby saving time in searching for the operation controls and making it more convenient for the user. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this application;

[0019] Figure 2 This is a flowchart illustrating a method for searching operation controls on an extended reality device according to an embodiment of this application.

[0020] Figure 3 This is a structural block diagram of an apparatus for searching operation controls for extended reality devices according to an embodiment of this application;

[0021] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0022] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0023] The principles and spirit of this application will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement this application, and are not intended to limit the scope of this application in any way. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0024] It is understood that before using the technical solutions of the various embodiments in this disclosure, users will be informed of the type, scope of use, and usage scenarios of the personal information involved in an appropriate manner, and user authorization will be obtained.

[0025] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose, based on the prompt message, whether to provide personal information to the software or hardware such as electronic devices, applications, servers, or storage media performing the operations of this disclosed technical solution.

[0026] As an optional but not limited implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0027] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0028] In this article, it is important to understand that any number of elements in the accompanying figures is for illustrative purposes and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0029] Based on the above background description, the following situations also exist in the related technologies:

[0030] When a user wears a head-mounted display (a type of extended reality device), the screen shows a virtual space, in which the controllers (a type of control) are not visible. Methods to display controllers on a head-mounted display include:

[0031] The controller requires external force to be detected, such as shaking or clicking a button, to be woken up. The controller will usually enter sleep mode within a few seconds after being left idle.

[0032] Turn on the see-through mode to see the real situation and find the controller. You can see the controller in this way, but there is no prompt if the desktop is cluttered or there are multiple controllers on the desktop, so clear prompts and feedback are needed.

[0033] Based on the above description, the principles and spirit of this application will be explained in detail below with reference to several representative embodiments.

[0034] refer to Figure 1 This is a schematic diagram illustrating an application scenario of the information processing method for objects of interest provided in this application embodiment. The application scenario includes an extended reality device 101, a server 102, and a data storage system 103. The extended reality device 101, server 102, and data storage system 103 can all be connected via wired or wireless communication networks. The extended reality XR (Extended Reality) device 101 includes, but is not limited to, head-mounted AR (Augmented Reality) devices, VR (Virtual Reality) devices, and MR (Mixed Reality) devices. The server 102 and data storage system 103 can both be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0035] The extended reality device 101 can be a head-mounted device or glasses, etc. When the user wears the extended reality device, the system matches the corresponding information acquisition method based on the communication status between the extended reality device and the operation controls. The location of the operation controls is then obtained according to this information acquisition method and displayed on the screen, allowing the user to locate them. The user can then operate the device using the control, and the extended reality device interacts with the server 102, displaying the interactive content on the extended reality device's screen. The data storage system 103 provides data storage support for the operation of the server 102, for example, storing the data content interacting with the extended reality device.

[0036] The following is combined with Figure 1 The above application scenarios are used to describe a method for searching operation controls on an extended reality device according to exemplary embodiments of this application. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way in this respect. Rather, the embodiments of this application can be applied to any applicable scenario.

[0037] This application provides a method for searching operation controls on an extended reality device, wherein the extended reality device includes: virtual reality glasses or virtual reality headset.

[0038] like Figure 2 As shown, the method includes:

[0039] Step 201: Detect that the user is wearing an augmented reality device and obtain the communication status between the augmented reality device and the operation controls.

[0040] In practice, the detection and wearing process of this extended reality device includes:

[0041] The augmented reality device is equipped with a proximity sensor. When a user wears the device, if the distance detected by the proximity sensor is less than a threshold, it indicates that the user's head is obstructing the sensor, and the augmented reality device can then determine that the user is wearing it. The device then powers on and acquires the communication status between itself and the control panel. The communication status is divided into two states: a stable connection allowing normal data communication, and a disconnected state when the control panel is powered off or in standby mode.

[0042] The operation controls include at least one of the following: a remote control, a joystick, and a handle; in this embodiment, a handle is preferred.

[0043] Step 202: Determine the information acquisition method corresponding to the communication state, and acquire the position of the operation control according to the information acquisition method.

[0044] In practical implementation, considering both communication connection and disconnection states, the database needs to store at least the information retrieval methods corresponding to these two states. Once the communication state is determined, the appropriate information retrieval method can be matched based on that state. This information retrieval method is essentially the method for determining the position of the control.

[0045] This ensures that the position of the control can be correctly obtained regardless of whether the control can communicate normally.

[0046] Step 203: Display the location of the operation control on the screen.

[0047] In practice, the user's position relative to the control is determined based on the control's location, and a prompt or display is shown on the screen accordingly. For example, if the control is located to the user's left front based on its position, a prompt or display will be shown to the user on the left front of the screen, prompting the user to find the control to the left front.

[0048] With the above solution, when a user wears the augmented reality device, the system selects the appropriate information acquisition method based on the communication status with the control. Different communication statuses will match different information acquisition methods. Then, the system retrieves the location of the control based on the determined information acquisition method and displays the location of the control on the augmented reality device's screen. This allows the user to easily find the control based on the displayed location, saving time in searching for the control and making it more convenient for the user.

[0049] In some embodiments, step 201 includes:

[0050] Step 2011: The system detects that the user is wearing the augmented reality device and collects the user's hand movements.

[0051] Step 2012: The hand gesture is identified as a search operation control gesture, and the communication status between the extended reality device and the operation control is obtained.

[0052] In practice, it is necessary to collect a large amount of video data (e.g., more than or equal to 100) of hand groping as samples in advance. A first initial neural network model is then pre-built using CNN (Convolutional Neural Networks).

[0053] Then, the initial model of the first neural network is trained using video data samples of hand search (such as groping up and down or groping left and right), so that the initial model of the first neural network has increasingly higher accuracy in recognizing hand search actions. After the training is completed, a hand search action recognition model is obtained.

[0054] The acquired image information corresponding to the hand gesture is then input into the hand search action recognition model for identification. The model determines that the hand gesture is a search operation control action. After confirming that it is a search operation control action, it is determined that the user needs to find the operation control, and then the communication status between the extended reality device and the operation control is retrieved.

[0055] The above scheme can accurately identify the user's hand movements, and after determining that the user's hand movement is a search operation control movement, it can obtain the communication status between the extended reality device and the operation control, and then perform subsequent operations based on the obtained communication status.

[0056] In some embodiments, step 2011 includes:

[0057] Step 20111: The system detects that the user is wearing an augmented reality device and uses an image acquisition device to detect whether there is a platform in front of the user.

[0058] Step 20112: In response to the presence of the shelf, collect the user's hand movements.

[0059] In practice, a large amount of image data with or without a platform can be collected in advance as samples, as needed. A second initial neural network model is then pre-built using a CNN (Convolutional Neural Network).

[0060] Then, the initial model of the second neural network is trained using collected image data samples with or without a shelf, so that the initial model of the second neural network can recognize the shelf. As the number of training times increases, the accuracy of recognizing the shelf becomes higher and higher. After training is completed, a shelf recognition model is obtained.

[0061] The image acquired by the image acquisition device is then input into the tabletop recognition model for recognition. The tabletop recognition model is used to determine whether there is a tabletop (e.g., a desktop) in the image. After the tabletop recognition model outputs that there is a tabletop, the image acquisition device is used to acquire the user's hand movements, and the hand movements are recognized according to the process described in the above embodiment.

[0062] With the above solution, since the operation controls are usually placed on a shelf, it is necessary to first determine whether there is a shelf in front of the user before recognizing the user's hand movements. If there is no shelf in front of the user, there is no need to collect the user's hand movements.

[0063] In some embodiments, step 202 includes:

[0064] Step 2021: In response to the communication state being a communication connection state, determine that the information acquisition method is a direct acquisition method, and directly acquire the position of the operation control of the operation control.

[0065] In practice, when the extended reality device and the operating control are in a communication connection state, it proves that the device can normally interact with the operating control, and the operating control can directly send its location to the extended reality device via wired or wireless means.

[0066] Alternatively, in step 2022, in response to the communication state being a communication disconnected state, the information acquisition method is determined to be an indirect acquisition method. An image of the user's surrounding environment is acquired through an image acquisition device, an operation control image is found in the environment image, and the position of the operation control is determined based on the position of the operation control image.

[0067] In practice, when communication between the extended reality device and the control is disconnected, it indicates that the control is powered off or in standby mode and cannot interact normally with the extended reality device. In this case, to determine the position of the control, an image acquisition device (e.g., a camera) installed on the extended reality device needs to be activated to capture images of the user's surroundings. If the image of the control is present in the environmental image, its position relative to the user is determined based on the positional relationship of the control image within the environmental image and the positional relationship of the image acquisition device relative to the user.

[0068] With the above solution, the location of the operating control can be determined normally regardless of whether the extended reality device and the operating control can communicate or not. This facilitates the normal display of the operating control location on the screen, prompting the user to find the operating control correctly based on its location.

[0069] In some embodiments, step 203 includes:

[0070] Step 203A1: Determine the virtual model diagram of the operation control, and determine the display position of the operation control in the virtual interface of the display screen according to the position of the operation control.

[0071] Step 203A2: Display the virtual model diagram at the display location.

[0072] In practice, the corresponding virtual model diagram can be a model diagram constructed based on the operation controls, which may be the same as or different from the operation control graphics. The style of the virtual model diagram can be set or selected by the user. The style can be a graphic drawn by the user, or a target style selected by the user from multiple style templates.

[0073] In this way, after determining the location of the control, the virtual model can be displayed on the screen at the corresponding display position of the control, so that the user can search for the control based on the location of the virtual model.

[0074] In addition, the solutions in steps 203A1 and 203A2 above are preferably applicable to the prompting or display scheme of the position of the operation control corresponding to the communication connection status.

[0075] In some embodiments, step 203 includes:

[0076] Step 203B1: Determine the display position of the operation control in the virtual interface of the display screen based on the position of the operation control.

[0077] Step 203B2: Generate a prompt message and display the prompt message at the display location.

[0078] In practice, a prompt message can be displayed on the screen at the location corresponding to the operation control. The prompt message can be at least one of the following: a prompt icon, prompt text, or prompt animation.

[0079] With the above solution, after determining the location of the operation control, a prompt message can be displayed at the corresponding display position on the screen, thereby prompting the user that the operation control is located at the position of the prompt message, making it easier for the user to search for the operation control based on the location of the prompt message.

[0080] In addition, the solutions in steps 203B1 and 203B2 above are preferably applicable to the prompting or display scheme of the position of the operation control obtained in the communication disconnection state.

[0081] In some embodiments, step 203 includes:

[0082] Step 203C1: Determine the display position on the screen corresponding to the position of the operation control, and enable the see-through function in the predetermined display area containing the display position so that the user can see the real environment image displayed through the screen in the predetermined display area.

[0083] Step 203C2: In response to determining that the user has found the operation control, the perspective function is turned off.

[0084] In practice, perspective can be enabled within a predetermined display area on the screen corresponding to the location of the control (e.g., a circle with a radius of 10cm centered on the control). This allows the user to see an image of the real environment containing the control within this predetermined display area. The control can then be seen within this image of the real environment, and its location on the screen can be determined based on its corresponding position. Once the control is located, the perspective function becomes useless and interferes with the further display of the virtual image; therefore, it needs to be disabled.

[0085] In some embodiments, after step 203C1 and before step 203C2, the method further includes:

[0086] Step 203C1': The predetermined display area displays a real operation control image, and at the same time, a virtual visual marker is superimposed on the real operation control image.

[0087] In practice, the graphic of the virtual visual marker in the predetermined display area can be set according to actual needs. In this embodiment, the virtual graphic of the operation control is preferred.

[0088] The above method overlays virtual visual markers onto the operation control image, allowing the user's eyes to see the operation control more quickly and thus find it more easily.

[0089] In some embodiments, after step 203C1 and before step 203C2, the method further includes:

[0090] Step C11: Determine whether a user's hand image exists within the predetermined display area.

[0091] Step C12: In response to the presence of a user's hand image within the predetermined display area, the display position of the user's hand image is determined in real time, and the virtual hand image is followed in real time at the display position of the user's hand image.

[0092] Step C13: In response to the absence of a user hand image within the predetermined display area, the virtual hand image is displayed at the display position corresponding to the operation control position within the predetermined display area.

[0093] In practice, if the user's hand moves into the predetermined display area (i.e., the area corresponding to the location of the operation control), an image of the user's hand will appear within that area. To better guide the user, the virtual hand image will move along with the user's hand image. If no user hand image is present in the predetermined display area, it indicates that the user's hand has not yet moved into the area. In this case, the corresponding virtual hand image does not need to move; it simply appears at the display position corresponding to the operation control location within the predetermined display area.

[0094] Then, if the user finds and grasps the control, the perspective function will be turned off.

[0095] The above solutions provide at least three visual options for prompting or displaying information. Users can choose from these three options based on their specific needs and circumstances, making it easier for them to quickly find the control functions based on the prompts or displays.

[0096] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0097] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0098] Based on the same inventive concept, corresponding to the method of searching operation controls for extended reality devices in any of the above embodiments, this application also provides an apparatus for searching operation controls for extended reality devices.

[0099] refer to Figure 3 The device includes:

[0100] The communication status determination module is configured to detect when a user puts on an extended reality device and obtain the communication status between the extended reality device and the operation controls;

[0101] The location acquisition module is configured to determine the information acquisition method corresponding to the communication state, and acquire the position of the operation control according to the information acquisition method.

[0102] The prompt or display module is configured to display the position of the operation control on the display screen.

[0103] In some embodiments, the communication status determination module includes:

[0104] The motion capture unit is configured to detect when a user is wearing an augmented reality device and capture the user's hand movements;

[0105] The motion recognition unit is configured to recognize the hand motion as a search operation control action and to obtain the communication status between the extended reality device and the operation control.

[0106] In some embodiments, the motion acquisition unit is further configured to:

[0107] The system detects that the user is wearing an augmented reality device and uses an image acquisition device to detect whether there is a platform in front of the user; in response to the presence of a platform, it captures the user's hand movements.

[0108] In some embodiments, the location acquisition module includes:

[0109] The direct acquisition unit is configured to, in response to the communication state being a communication connection state, determine that the information acquisition method is a direct acquisition method and directly acquire the position of the operation control of the operation control; or...

[0110] The indirect acquisition unit is configured to, in response to the communication state being a communication disconnected state, determine that the information acquisition method is an indirect acquisition method, acquire an image of the user's surrounding environment through an image acquisition device, locate the operation control image from the environment image, and determine the position of the operation control based on the position of the operation control image.

[0111] In some embodiments, the prompting or display module includes:

[0112] The virtual model prompt or display unit is configured to determine the virtual model diagram of the operation control, determine the display position of the operation control in the virtual interface of the display screen according to the position of the operation control, and display the virtual model diagram at the display position.

[0113] In some embodiments, the prompting or display module includes:

[0114] The prompt information display unit is configured to determine the display position of the operation control in the virtual interface of the display screen based on the position of the operation control; generate prompt information; and display the prompt information at the display position.

[0115] In some embodiments, the prompting or display module includes:

[0116] The perspective display unit is configured to determine the display position on the display screen corresponding to the position of the operation control, enable the perspective function in a predetermined display area including the display position, so that the user can see the perspective real environment image displayed in the predetermined display area of ​​the display screen; and disable the perspective function in response to determining that the user has found the operation control.

[0117] In some embodiments, after enabling the perspective function in a predetermined display area including the display location, the response is to disable the perspective function before determining that the user has found the operation control;

[0118] The perspective display unit is also configured as follows:

[0119] The predetermined display area displays an image of the real operation control, while a virtual visual marker is overlaid on the image of the real operation control.

[0120] In some embodiments, after enabling the perspective function in a predetermined display area including the display location, the response is to disable the perspective function before determining that the user has found the operation control;

[0121] The perspective display unit is also configured as follows:

[0122] Determine whether a user's hand image exists within the predetermined display area; in response to the presence of a user's hand image within the predetermined display area, determine the display position of the user's hand image in real time and follow the virtual hand image at the display position of the user's hand image in real time; in response to the absence of a user's hand image within the predetermined display area, display the virtual hand image at the display position corresponding to the operation control position within the predetermined display area.

[0123] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0124] The apparatus of the above embodiments is used to implement the corresponding method in any of the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0125] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any of the above embodiments.

[0126] Figure 4 This illustration shows a more specific hardware structure diagram of an electronic device provided in this embodiment. The device may include: a processor 410, a memory 420, an input / output interface 430, a communication interface 440, and a bus 450. The processor 410, memory 420, input / output interface 430, and communication interface 440 are interconnected internally via the bus 450.

[0127] The processor 410 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0128] The memory 420 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 420 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 420 and is called and executed by the processor 410.

[0129] Input / output interface 430 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0130] The communication interface 440 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (e.g., USB, Ethernet cable) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth).

[0131] Bus 450 includes a pathway for transmitting information between various components of the device (e.g., processor 410, memory 420, input / output interface 430, and communication interface 440).

[0132] It should be noted that although the above-described device only shows the processor 410, memory 420, input / output interface 430, communication interface 440, and bus 450, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0133] The electronic devices described above are used to implement the corresponding extended reality device search operation controls in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0134] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to perform the methods described in any of the above embodiments.

[0135] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0136] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to perform the methods described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0137] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides a computer program product, including computer program instructions. When the computer program instructions are run on a computer, the computer causes the computer to perform the method described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0138] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0139] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0140] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0141] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A method for extending the search operation controls of a real-world device, characterized in that, include: The system detects that the user is wearing an augmented reality device and collects the user's hand movements. The hand gesture is identified as a search operation control action, and the communication status between the extended reality device and the operation control is obtained; Determine the information acquisition method corresponding to the communication state, and obtain the position of the operation control according to the information acquisition method; The location of the operation controls will be indicated or displayed on the screen; The step of prompting or displaying the position of the operation control on the display screen includes: Determine the display position on the screen corresponding to the position of the operation control, and enable the perspective function in the predetermined display area containing the display position so that the user can see the real environment image displayed through the perspective in the predetermined display area of ​​the screen; In response to determining that the user has found the operation control, the perspective function is turned off.

2. The method according to claim 1, characterized in that, The detection of the user wearing the augmented reality device and the collection of the user's hand movements include: The system detects that the user is wearing an augmented reality device and uses an image acquisition device to detect whether there is a platform in front of the user. In response to the presence of a shelf, the system collects the user's hand gestures.

3. The method according to claim 1 or 2, characterized in that, The step of determining the information acquisition method corresponding to the communication state and acquiring the position of the operation control according to the information acquisition method includes: In response to the communication state being a communication connection state, the information acquisition method is determined to be a direct acquisition method, directly acquiring the position of the operation control of the operation control; or... In response to the communication state being disconnected, the information acquisition method is determined to be an indirect acquisition method. An image of the user's surrounding environment is acquired through an image acquisition device, an image of the operation control is found in the image of the environment, and the position of the operation control is determined based on the position of the operation control image.

4. The method according to claim 1, characterized in that, The step of prompting or displaying the position of the operation control on the display screen includes: Determine the virtual model diagram of the operation control, and determine the display position of the operation control in the virtual interface of the display screen based on the position of the operation control; The virtual model diagram is displayed at the specified display location.

5. The method according to claim 1, characterized in that, The step of prompting or displaying the position of the operation control on the display screen includes: The display position of the operation control in the virtual interface of the display screen is determined based on the position of the operation control; Generate a prompt message and display the prompt message at the specified display location.

6. The method according to claim 1, characterized in that, After enabling the perspective function in the predetermined display area including the display position, and before disabling the perspective function in response to determining that the user has found the operation control, the method further includes: The predetermined display area displays an image of the real operation control, while a virtual visual marker is overlaid on the image of the real operation control.

7. The method according to claim 1, characterized in that, After enabling the perspective function in the predetermined display area including the display position, and before disabling the perspective function in response to determining that the user has found the operation control, the method further includes: Determine whether a user's hand image exists within the predetermined display area; In response to the presence of a user's hand image within the predetermined display area, the display position of the user's hand image is determined in real time, and the virtual hand image is moved to follow the display position of the user's hand image in real time; or, In response to the absence of a user's hand image within the predetermined display area, the virtual hand image is displayed at the display position corresponding to the operation control position within the predetermined display area.

8. A device for extending the search operation controls of a reality device, characterized in that, include: The communication status determination module is configured to detect when a user is wearing an extended reality device and collect the user's hand movements; The hand gesture is identified as a search operation control action, and the communication status between the extended reality device and the operation control is obtained; The location acquisition module is configured to determine the information acquisition method corresponding to the communication state, and acquire the position of the operation control according to the information acquisition method. The prompting or display module is configured to prompt or display the position of the operation control on the display screen; The prompting or display module is specifically configured as follows: Determine the display position on the screen corresponding to the position of the operation control, and enable the perspective function in the predetermined display area containing the display position so that the user can see the real environment image displayed through the perspective in the predetermined display area of ​​the screen; In response to determining that the user has found the operation control, the perspective function is turned off.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method as described in any one of claims 1 to 7.

11. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed on a computer, the computer causes the computer to perform the method as described in any one of claims 1 to 7.

Images