Demand function providing method and apparatus, device, and medium

Abstract

The present invention relates to the technical field of head mounting. Disclosed are a demand function providing method and apparatus, a device, and a medium. The method is applied to a control device for a head-mounted device, and comprises: receiving a plurality of first sub-programs capable of executing first demand functions of the control device, wherein the first demand functions at least comprise a multi-device fusion function and a control function for a streaming device of the head-mounted device; on the basis of the first sub-programs, generating first sub-function modules corresponding to the first demand functions; receiving a first input for selecting a target first sub-function module from among the first sub-function modules; and in response to the first input, providing an interface of the target first sub-function module and interface description information. By means of the method, a control device can provide a user with a service meeting a demand. That is, the control device can provide a user with a personalized service.

Description

Method, device, equipment and medium for providing required functions

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 16, 2023, with application number 202311034703.1 and invention name “Method, device, equipment and medium for providing demand functions”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] With the development of science and technology and economy, head-mounted display devices (such as AR devices and VR devices, etc.) have been increasingly widely used.

[0003] Head-mounted display devices are categorized into two types: split-body and all-in-one. However, split-body devices offer superior power consumption, battery life, weight, and heat dissipation compared to all-in-one devices. Therefore, split-body devices are becoming a market and technological trend. Split-body devices are typically equipped with a PUCK device as a processing unit.

[0004] Currently, the functions that can be realized by the PUCK devices provided to users are fixed. Therefore, the PUCK devices cannot provide personalized services to users.

[0005] Summary of the Invention

[0006] An object of the present invention is to provide a new technical solution for providing the required functionality.

[0007] According to a first aspect of the present invention, a method for providing a required function is provided, which is applied to a control device of a head-mounted device, comprising:

[0008] receiving a plurality of first subprograms capable of executing first required functions of the control device, wherein the first required functions include at least a multi-device integration function and a function of controlling a streaming device of the head-mounted device;

[0009] generating a first sub-function module corresponding to a first required function according to the first sub-program;

[0010] receiving a first input for selecting a target first sub-functional module among the first sub-functional modules;

[0011] In response to the first input, an interface and interface description information of the target first sub-function module are provided.

[0012] Optionally, the method further includes:

[0013] receiving a second input based on interface description information corresponding to the target interface;

[0014] In response to the second input, a first sub-function module corresponding to the target interface is executed.

[0015] Optionally, the target interface is an interface of a target first sub-functional module corresponding to a multi-device fusion function, and the step of running the target first sub-functional module corresponding to the target interface in response to the second input includes:

[0016] Acquire an image of at least one streaming device and a local image of the control device according to the interface description information corresponding to the target interface;

[0017] fusing an image of at least one streaming device and a local image of the control device to obtain a fused image;

[0018] The fused image is sent to the head mounted device so that the head mounted device displays the fused image.

[0019] Optionally, the method further includes:

[0020] In an eye movement interaction mode, obtaining a gaze state of a wearer of the head mounted device and performing an operation matching the gaze state;

[0021] In the virtual marker and eye movement dual interaction mode, the gaze position of the head mounted device and the projection position of the virtual marker in the fused image are obtained, and an operation matching the gaze position and the projection position is performed.

[0022] Optionally, receiving at least one second subroutine capable of executing a second required function of the control device, wherein the second required function is a newly added required function;

[0023] generating a second sub-function module of the second required function according to the second sub-program;

[0024] In case of receiving a third input selecting the second sub-functional module, in response to the third input, providing an interface and interface description information of the second sub-functional module.

[0025] Optionally, receiving a plurality of third subroutines capable of executing a third required function of the head mounted device;

[0026] generating a third sub-function module corresponding to the third required function according to the third sub-program;

[0027] When the control device is in communication with the head mounted device, the interface and interface description information of the third sub-function module are sent to the head mounted device.

[0028] Optionally, receiving a plurality of fourth subroutines capable of executing a fourth required function of the streaming device;

[0029] generating a fourth sub-function module corresponding to a fourth required function according to the fourth sub-program;

[0030] When the control device is in communication with the streaming device, the interface and interface description information of the fourth sub-function module are sent to the streaming device.

[0031] Optionally, providing the interface and interface description information of the target first sub-function module in response to the first input includes:

[0032] In response to the first input, the interface and interface description information of the target first sub-function module are provided in the form of a software installation package.

[0033] According to a second aspect of the present invention, there is provided a device for providing a required function, which is applied to a control device of a head-mounted device, comprising:

[0034] A first receiving module is configured to receive a plurality of first subprograms capable of executing first required functions of the control device, wherein the first required functions include at least a multi-device integration function and a function of controlling a streaming device of the head-mounted device;

[0035] a generating module, configured to generate a first sub-function module corresponding to a first required function according to the first sub-program;

[0036] a second receiving module, configured to receive a first input for selecting a target first sub-functional module among the first sub-functional modules;

[0037] A providing module is used to provide the interface and interface description information of the target first sub-function module in response to the first input.

[0038] According to a third aspect of the present invention, a control device is provided, the control device comprising the device for providing the required function as described in the second aspect; or

[0039] The control device includes a memory and a processor, the memory is used to store computer instructions, and the processor is used to call the computer instructions from the memory to execute the method for providing the required function as described in any one of the first aspects.

[0040] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer program implements the method for providing the required function according to any one of the first aspects.

[0041] In an embodiment of the present invention, a method for providing required functions is provided. This method is applied to a control device for a head-mounted device. The method modularizes the first required functions of the control device by receiving multiple first subroutines capable of executing the first required functions of the control device, wherein the first required functions include at least a multi-device fusion function and a function for controlling the head-mounted device's streaming device, and then generates, based on the first subroutines, first sub-function modules corresponding to the first required functions. This method then obtains a first sub-function module corresponding to each first required function. Furthermore, by receiving a first input for selecting a target first sub-function module within the first sub-function modules and, in response to the first input, providing an interface and interface description information for the target first sub-function module, the method specifies a target first required function that meets the requirements and customizes the first required function in the control device. Based on this, the control device can provide users with services that meet the requirements. That is, the control device can provide users with personalized services.

[0042] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0044] 1 is a block diagram 1 of the hardware configuration of a control device for implementing a method for providing a demand function provided by an embodiment of the present invention;

[0045] FIG2 is a flow chart of a method for providing a demand function according to an embodiment of the present invention;

[0046] FIG3 is a schematic diagram of a functional architecture of a control device, a head-mounted device, and a streaming device provided in an embodiment of the present invention;

[0047] 4 is a schematic structural diagram of a device for providing a demand function according to an embodiment of the present invention;

[0048] FIG5 is a second block diagram of the hardware configuration of a control device for implementing a method for providing a demand function provided by an embodiment of the present invention.

[0049] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0051] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), such directional indications are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0052] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the meaning of "and / or" appearing throughout the text includes three parallel schemes. Taking "A and / or B" as an example, it includes scheme A, or scheme B, or a scheme in which A and B are satisfied at the same time. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0053] FIG1 is a block diagram 1 of the hardware configuration of a control device for implementing a method for providing a demand function provided by an embodiment of the present invention.

[0054] The control device is specifically a data processing unit of the head-mounted device. The control device is a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in this embodiment of the present invention. Furthermore, the head-mounted device is a split-type head-mounted device. The split-type head-mounted device may be a split-type head-mounted device such as an AR, MR, VR, or XR head-mounted device.

[0055] The control device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and the like. The processor 1100 may be a central processing unit (CPU), a microprocessor (MCU), or the like. The memory 1200 may include, for example, a ROM (read-only memory), a RAM (random access memory), or a non-volatile memory such as a hard disk. The interface device 1300 may include, for example, a USB interface or a headphone jack. The communication device 1400 may be capable of wired or wireless communication. The display device 1500 may be, for example, an LCD display or a touchscreen display. The input device 1600 may include, for example, a touchscreen or a keyboard. A user may input and output voice information via the speaker 1700 and the microphone 1800.

[0056] Although FIG1 shows a plurality of devices for the control device 1000 , the present invention may only relate to some of the devices. For example, the control device 1000 may only relate to the memory 1200 and the processor 1100 .

[0057] In the embodiment of the present invention, the memory 1200 of the control device 1000 is used to store instructions, and the instructions are used to control the processor 1100 to execute the method for providing the required function provided by the embodiment of the present invention.

[0058] In the above description, a person skilled in the art can design instructions according to the solution disclosed in the present invention. How instructions control the operation of a processor is well known in the art and will not be described in detail here.

[0059] An embodiment of the present invention provides a method for providing a required function, which is applied to a control device of a head-mounted device. As shown in FIG2 , the method includes the following steps S2100 to S2400:

[0060] S2100: Receive a plurality of first subroutines capable of executing a first required function of a control device.

[0061] Among them, the first required function includes at least device integration and control of the streaming device of the head-mounted device.

[0062] It should be noted that the method for providing required functions provided in the embodiments of the present invention is applied to a streaming scenario. This scenario includes a control device, a headset, and a streaming device, with the headset and streaming device each being communicatively connected to the control device. The communication connection can be via Wi-Fi or USB. Exemplary streaming devices include mobile phones, PADs, or PCs.

[0063] In the embodiment of the present invention, the first demand function is a default demand function of the control device in the initial state. One first demand function corresponds to one first subroutine, that is, there is a one-to-one correspondence between the first demand function and the first subroutine.

[0064] In one embodiment of the present invention, the first required functions may illustratively include: streaming / communication functions, control functions for the streaming device of the head-mounted device, local control functions for the control device, multi-application opening functions, third-party algorithm functions, plug-in functions, OpenXR Loader functions, and multi-device integration functions. The multi-device integration function may be implemented through the XRRuntime function.

[0065] Regarding the above content, the streaming / communication function includes the streaming function and the communication function, where the streaming function specifically refers to: the function of receiving the display image of the streaming device, and the function of displaying the display image of the streaming device; and the communication function specifically refers to: the function of transmitting control commands between the streaming device and the control device, and the function of transmitting 3Dof / 6Dof, Audio, Camera and other data between the control device and the head-mounted device;

[0066] The multi-device fusion function specifically refers to the following functions: completing the fusion of the local display image of the control device and the display image of the streaming device, and transmitting the fused image to the head-mounted device;

[0067] The multi-app feature specifically means launching multiple apps (local apps and / or apps on the streaming device) simultaneously, with each app running on a different screen of the headset. These apps are all in the foreground, and subsequent apps and windows launched by an app must run on the same screen. This feature requires customization at the Framework layer.

[0068] The control function of the streaming device of the headset specifically refers to: supporting multiple interactive methods such as ray, gesture, and voice to control the application on the streaming device, as well as supporting mouse and keyboard interaction on each screen of the headset after multi-opening, and supporting screen-based volume control. This function requires customization of the Framework layer;

[0069] The local control function of the control device specifically refers to: supporting multiple interactive methods such as ray, gesture, and voice to control local applications, as well as supporting mouse and keyboard interaction on each screen of the headset after multi-opening, and supporting screen-based volume control. This function requires customization at the Framework layer;

[0070] The third-party algorithm function specifically refers to: using the 3Dof / 6Dof and Camera data of the head-mounted device to achieve spatial anchoring, gesture recognition, plane detection and other functions.

[0071] For any of the aforementioned first required functions, a developer inputs a program into the control device that can execute (i.e., implement) the first required function. In this embodiment of the present invention, the aforementioned program is referred to as a first subroutine. Based on this, the control device receives multiple first subroutines that can execute the first required function of the control device.

[0072] S2200: Generate a first sub-function module corresponding to a first required function according to a first sub-program.

[0073] In an embodiment of the present invention, for any first subroutine, the control device encapsulates and protects the first subroutine to obtain a corresponding first sub-functional module. Furthermore, the first sub-functional module includes an interface for calling the first subroutine and interface description information for the interface. The interface description information is used to describe how to use the interface.

[0074] Based on the above S2100 and S2200, the modularization of the first required function of the control device can be achieved, which can provide a basis for subsequent users to customize the first required function to meet their own needs.

[0075] S2300: Receive a first input for selecting a target first sub-functional module among the first sub-functional modules.

[0076] In the embodiment of the present invention, the first sub-function module that meets the user's requirements among the first sub-function modules is recorded as the target first sub-function module. The first input is an input for selecting the target first sub-function module from the first sub-function modules.

[0077] There can be at least one target first sub-function. It should be noted that in the embodiment of the present invention, the number of target first sub-functions is not limited.

[0078] The above S2300 can complete the designation of the target first requirement function that meets the requirements.

[0079] S2400 : In response to a first input, provide an interface and interface description information of a target first sub-function module.

[0080] In an embodiment of the present invention, the first sub-function module selected based on the first input is used as the target first sub-function module. After the target first sub-function module is determined, the interface and interface description information of the target first sub-function module are provided. Thus, based on the interface description information, it is possible to determine how to call the corresponding interface. Furthermore, after calling the interface based on the interface description information, the control device can execute the first required function corresponding to the target first sub-function module corresponding to the interface.

[0081] In one example, when the target first sub-functional module is the first sub-functional module of the multi-device fusion function and the control function of the streaming device of the head-mounted device, based on the above S2400, the control device can only implement the multi-device fusion function and the control function of the streaming device of the head-mounted device.

[0082] Through the above S2400, the customization of the first required function in the control device can be completed. In this way, the control device can provide the user with a service that meets the user's needs. In other words, the control device can provide the user with personalized services.

[0083] In one embodiment of the present invention, the above S2400 may be implemented illustratively according to the following S2410:

[0084] S2410 . In response to the first input, provide the interface and interface information of the target first sub-function module in the form of a software installation package.

[0085] In an embodiment of the present invention, the target first sub-function interface and interface information are presented to the user via a software development kit (SDK). Since SDK is a commonly used method for users, and the interface information provided via the software installation kit also includes information such as interface operation examples, the above-mentioned S2410 can facilitate the user's use of the control device.

[0086] In an embodiment of the present invention, a method for providing required functions is provided. This method is applied to a control device for a head-mounted device. The method modularizes the first required functions of the control device by receiving multiple first subroutines capable of executing the first required functions of the control device, wherein the first required functions include at least a multi-device fusion function and a function for controlling the head-mounted device's streaming device, and then generates, based on the first subroutines, first sub-function modules corresponding to the first required functions. This method then obtains a first sub-function module corresponding to each first required function. Furthermore, by receiving a first input for selecting a target first sub-function module within the first sub-function modules and, in response to the first input, providing an interface and interface description information for the target first sub-function module, the method specifies a target first required function that meets the requirements and customizes the first required function in the control device. Based on this, the control device can provide users with services that meet the requirements. That is, the control device can provide users with personalized services.

[0087] In one embodiment of the present invention, the method for providing the required function provided by the embodiment of the present invention further includes the following S2510 and S2511:

[0088] S2510: Receive a second input based on the interface description information corresponding to the target interface.

[0089] In the embodiment of the present invention, the target interface is an interface that meets the user's call requirements. The user performs input of the target interface call operation according to the interface description information of the target interface. Based on this, the control device receives the second input.

[0090] S2511 . In response to the second input, run the target first sub-function module corresponding to the target interface.

[0091] In an embodiment of the present invention, upon receiving the second input, the control device executes the first subroutine corresponding to the target first sub-function module called by the second input. At this time, the control device can complete the required function corresponding to the target first sub-function module called by the second input.

[0092] Based on the embodiments shown in S2510 and S2511 above, when the target interface is the interface of the target first sub-function module corresponding to the multi-device fusion function, S2511 above is specifically implemented through the following S2511-1 to S2511-3:

[0093] S2511-1. Acquire an image of at least one streaming device and a local image of the control device according to the interface description information corresponding to the target interface.

[0094] SS2511-2. Fusing the image of at least one streaming device and the local image of the control device to obtain a fused image.

[0095] S2511-3. Send the fused image to the head-mounted device so that the head-mounted device displays the fused image.

[0096] In this embodiment of the present invention, images from at least one streaming device and local images from the control device are acquired in accordance with the acquisition method specified in the interface description information corresponding to the target interface. Furthermore, the images from the at least one streaming device and local images from the control device are spliced ​​together to produce a fused image. Image fusion is thus completed. Furthermore, the control device transmits the fused image to the head-mounted device for display.

[0097] In an embodiment of the present invention, when a user calls the interface of the target first sub-function module corresponding to the multi-device fusion function, the control device can complete the multi-device fusion function through the above S2511-1 to S2511-3.

[0098] Based on the embodiments shown in S2511-1 to S2511-3 above, the method for providing the required function provided by the embodiment of the present invention further includes the following S2511-4 and S2511-5:

[0099] S2511-4. In the eye movement interaction mode, obtain the gaze state of the wearer of the head mounted device, and perform operations that match the gaze state.

[0100] S2511-5. In the dual interaction mode of virtual identification and eye movement, obtain the gaze position of the wearer of the head-mounted device and the projection position of the virtual identification in the fused image, and perform operations that match the gaze position and the projection position.

[0101] In the embodiment of the present invention, there are two interaction modes for the interaction of the fused image: an eye movement interaction mode and a virtual marker and eye movement dual interaction mode.

[0102] In the embodiment of the present invention, the eye movement interaction mode refers to the way in which the eyes of the wearer of the head mounted device interact with the fused image.

[0103] The dual interaction mode of virtual markers and eye movements refers to a method in which a virtual marker emitted by a control device and the wearer's eyes interact with the fused image. In one embodiment of the present invention, the virtual marker is used to describe the path between the control device and the interaction location when the wearer holds the control device and points to the interaction location on the fused image. In one example, the virtual marker can be a straight line, ray, or curve, which is not limited in this embodiment of the present invention.

[0104] In one embodiment of the present invention, the two interaction modes described above can be selected by the user according to their needs, or automatically selected according to the type of application currently running on the control device. This embodiment of the present invention does not limit this.

[0105] In one embodiment of the present invention, in eye tracking mode, an eye tracking algorithm can be used to determine the wearer's gaze position on the fused image and the corresponding gaze time period. In this embodiment of the present invention, the gaze position and gaze time period are recorded as the gaze state.

[0106] Furthermore, in eye tracking mode, the control device performs different operations for different gaze states. For example, if the gaze state indicates that the gaze position remains unchanged and the gaze period corresponding to the gaze position reaches a preset duration, the wearer is determined to be in a state of gazing at the gaze position, and an operation matching the gaze position state is performed. The operation matching the gaze position state can be pre-specified. In one exemplary embodiment, the operation matching the gaze position state can be a confirmation operation.

[0107] For another example, if the gaze state indicates that the gaze position remains unchanged, but the gaze time period corresponding to the gaze position is two discontinuous, shorter time periods, the wearer is determined to be in a double-blink state, and an operation matching the double-blink state at the gaze position is performed. Similarly, the operation matching the double-blink state at the gaze position can be pre-specified. In one exemplary embodiment, the operation matching the double-blink state at the gaze position can be a copy operation.

[0108] In one embodiment of the present invention, in a dual interaction mode of virtual marker and eye movement, the wearer's gaze position on the fused image can be determined using an eye tracking algorithm. Furthermore, the projection position of the virtual marker on the fused image can be obtained by conventionally determining the projection position of the virtual marker on the fused image. Based on the above, the operations in S2511-5 that match the gaze position and the projection position are specifically as follows:

[0109] If the difference between the position of the virtual marker on the fused image and the wearer's gaze position is less than a preset threshold, the position of the virtual marker on the fused image is determined to coincide with the wearer's gaze position, and the operation corresponding to the gaze position and / or the projected position of the virtual marker is performed. For example, if the button corresponding to the gaze position and / or the projected position of the virtual marker is a "Confirm" button, the confirmation operation corresponding to that button is performed.

[0110] Correspondingly, when the difference between the position of the virtual marker on the fused image and the wearer's gaze position is less than a preset threshold, it is determined that the position of the virtual marker on the fused image does not coincide with the wearer's gaze position, and no operation is performed at this time.

[0111] For the dual interaction mode of virtual signage and eye movement, the eye tracking accuracy requirement is low, which can reduce the power consumption of the control device. Correspondingly, in the eye movement interaction mode, the eye tracking accuracy requirement is high.

[0112] In one embodiment of the present invention, the method for providing the required function provided by the embodiment of the present invention further includes the following steps S2610-S2612:

[0113] S2610: Receive at least one second subroutine as a second required function of an executable control device.

[0114] Among them, the second demand function is a new demand function.

[0115] In an embodiment of the present invention, when a developer adds a new required function, namely a second required function, to the default required function, namely the first required function, the developer inputs a program capable of executing the second required function into the control device. In this embodiment of the present invention, the aforementioned program is referred to as a second subroutine. Based on this, the control device receives at least one second subroutine capable of executing the second required function of the control device.

[0116] S2611. Generate a second sub-function module of the second required function according to the second sub-program.

[0117] In the embodiment of the present invention, the specific implementation of the above S2611 is the same as the specific implementation of the above S2200, and will not be repeated here.

[0118] S2612: When a third input selecting the second sub-functional module is received, provide an interface and interface description information of the second sub-functional module in response to the third input.

[0119] In one embodiment of the present invention, the specific implementation of the above S2612 can be: when receiving the third input selecting the second sub-function module, respond to the third input in the form of a software installation package and provide the interface and interface description information of the second sub-function module.

[0120] In this embodiment of the present invention, the third input is an input by the user selecting the second sub-function module. Receiving the third input indicates that the user desires the control device to implement the second required function corresponding to the second sub-function module. In this case, the control device provides the interface and interface description information of the second sub-function module for the user to invoke the second sub-function module.

[0121] In the embodiment of the present invention, the above steps S2610 to S2612 can realize the addition of required functions to the control device and allow the user to call the newly added required functions.

[0122] In one embodiment of the present invention, the method for providing the required function provided by the embodiment of the present invention further includes the following steps S2710-S2712:

[0123] S2710: Receive multiple third subroutines capable of executing a third required function of the head mounted device.

[0124] S2711. Generate a third sub-function module corresponding to the third required function according to the third sub-program.

[0125] S2712: When the control module is in communication with the head mounted device, send the interface and interface description information of the third sub-function module to the head mounted device.

[0126] It should be noted that the specific implementation of the above S2710 and S2711 is the same as the specific implementation of the above S2100 and S2200, and will not be repeated here.

[0127] In one embodiment of the present invention, the specific implementation of the above S2712 can be: when the control module is communicatively connected to the head-mounted device, the interface and interface description information of the third sub-function module provided in the form of a software installation package are sent to the head-mounted device.

[0128] In this embodiment of the present invention, the head-mounted device and the control device are compatible and mutually reinforcing. Therefore, the head-mounted device must implement the required functions of the control device. In this embodiment of the present invention, the function of the head-mounted device to adapt to the requirements of the control device is denoted as the third required function.

[0129] Furthermore, the control device generates a third sub-function module for the third required function, and transmits the interface and interface description information of the third sub-function module to the head-mounted device based on the communication connection between the control device and the head-mounted device. In this way, when the head-mounted device is adapted to the control device, no modification is required on the head-mounted device side.

[0130] In one embodiment of the present invention, the third required function may exemplarily include: the function of obtaining Camera data on the head-mounted device side, the function of obtaining Audio data on the head-mounted device side, the function of providing 3Dof / 6Dof data on the head-mounted device side to the control device, the encoding and decoding function, the TimeWarp function, the rendering on the screen function, the streaming / communication function, etc.

[0131] Corresponding to the previous embodiment, in one embodiment of the present invention, the method for providing the required function provided by the embodiment of the present invention further includes the following steps S2810-S2812:

[0132] S2810: executing a fourth subroutine capable of executing a fourth required function of the streaming device according to a plurality of fourth subroutines.

[0133] S2811. Generate a fourth sub-function module corresponding to a fourth required function according to the fourth sub-program.

[0134] S2812: When the control device is in communication with the streaming device, send the interface and interface description information of the fourth sub-function module to the streaming device.

[0135] It should be noted that the specific implementation of the above S2810 and S2811 is the same as the specific implementation of the above S2100 and S2200, and will not be repeated here.

[0136] In one embodiment of the present invention, the specific implementation of the above S2812 may be: when the control module is in communication with the streaming device, sending the interface and interface description information of the fourth sub-function module provided in the form of a software installation package to the streaming device.

[0137] In this embodiment of the present invention, the streaming device and the control device are compatible and mutually reinforcing. Therefore, the streaming device must implement the required functions of the control device. In this embodiment of the present invention, the function of the streaming device to adapt to the requirements of the control device is recorded as the fourth required function.

[0138] Furthermore, the control device generates a fourth sub-function module for the fourth required function, and transmits the interface and interface description information of the fourth sub-function module to the streaming device based on the communication connection between the control device and the streaming device. In this way, when the streaming device is adapted to the control device, no modification is required on the streaming device side.

[0139] In one embodiment of the present invention, the fourth required function may illustratively include: a streaming / communication function, a function of receiving control instructions from the control device side, and a function of converting the control instructions into coordinate time and reporting it to the application of the streaming device.

[0140] In combination with the above content, the functional architecture of the control device, head-mounted device and streaming device in an embodiment of the present invention is shown in FIG3 .

[0141] Among them, Figure 3 shows an example in which the control device supports the OpenXR standard and has an Android operating system. Based on this, the software installation package SDK of the control device is divided into: Plugin layer, Graphic Plugin layer, Platform Plugin layer, and Runtime process layer. And the SDK serves as the service layer of the APP layer. In addition, the control function of the streaming device of the head-mounted device, the control function of the local control device, and the multi-application opening are essentially implemented based on the framework layer. In addition, when implementing the control function of the streaming device of the head-mounted device, the control function of the local control device, and the multi-application opening based on the framework layer, the framework layer includes: application management, permission management, screen management, Audio management, display management, window management, Input management, and Camera management.

[0142] The embodiment of the present invention further provides a device 400 for providing a required function, which is applied to a control device of a head-mounted device, as shown in FIG4 , and includes:

[0143] A first receiving module 410 is configured to receive a plurality of first subprograms capable of executing first required functions of the control device, wherein the first required functions include at least a multi-device integration function and a function of controlling a streaming device of the head-mounted device;

[0144] A generating module 420, configured to generate a first sub-function module corresponding to a first required function according to the first sub-program;

[0145] A second receiving module 430 is configured to receive a first input for selecting a target first sub-functional module among the first sub-functional modules;

[0146] The providing module 440 is configured to provide the interface and interface description information of the target first sub-function module in response to the first input.

[0147] Based on the device for providing the required function provided by the embodiment of the present invention, the control device can provide the user with services that meet the requirements, that is, the control device can provide the user with personalized services.

[0148] In one embodiment of the present invention, the device 400 for providing the required function provided by the embodiment of the present invention further includes:

[0149] A third receiving module receives a second input based on the interface description information corresponding to the target interface;

[0150] The running module runs the first sub-function module corresponding to the target interface in response to the second input.

[0151] In one embodiment of the present invention, the target interface is an interface of a target first sub-function module corresponding to the multi-device fusion function, and the operation module is specifically configured to:

[0152] Acquire an image of at least one streaming device and a local image of the control device according to the interface description information corresponding to the target interface;

[0153] fusing an image of at least one streaming device and a local image of the control device to obtain a fused image;

[0154] The fused image is sent to the head mounted device so that the head mounted device displays the fused image.

[0155] In one embodiment of the present invention, the operation module is further configured to:

[0156] In an eye movement interaction mode, obtaining a gaze state of a wearer of the head mounted device and performing an operation matching the gaze state;

[0157] In the virtual marker and eye movement dual interaction mode, the gaze position of the head mounted device and the projection position of the virtual marker in the fused image are obtained, and an operation matching the gaze position and the projection position is performed.

[0158] In one embodiment of the present invention, the first receiving module 410 is further configured to receive at least one second subprogram capable of executing a second required function of the control device, where the second required function is a newly added required function;

[0159] The generating module 420 is further configured to generate a second sub-function module of the second required function according to the second sub-program;

[0160] The providing module 440 is further configured to provide the interface and interface description information of the second sub-functional module in response to a third input for selecting the second sub-functional module when the third input is received.

[0161] In one embodiment of the present invention, the first receiving module 410 is further configured to receive a plurality of third subroutines capable of executing a third required function of the head mounted device;

[0162] The generating module 420 is further configured to generate a third sub-function module corresponding to the third required function according to the third sub-program;

[0163] In this embodiment, the demand function providing device 400 further includes:

[0164] A sending module is used to send the interface and interface description information of the third sub-function module to the head-mounted device when the control device is communicatively connected to the head-mounted device.

[0165] In one embodiment of the present invention, the first receiving module 410 is further configured to receive a plurality of fourth subprograms capable of executing a fourth required function of the streaming device;

[0166] The generating module 420 is further configured to generate a fourth sub-function module corresponding to a fourth required function according to the fourth sub-program;

[0167] The sending module is further configured to send the interface and interface description information of the fourth sub-function module to the streaming device when the control device is communicatively connected to the streaming device.

[0168] In one embodiment of the present invention, the providing module 440 is specifically configured to provide the interface and interface description information of the target first sub-function module in the form of a software installation package in response to the first input.

[0169] An embodiment of the present invention further provides a control device 500 , which includes the demand function providing device 400 provided by any of the above device embodiments.

[0170] Alternatively, as shown in FIG5 , the control device 500 includes a memory 510 and a processor 520 , wherein the memory 510 is used to store computer instructions, and the processor 520 is used to call the computer instructions from the memory 510 to execute the method for providing the required function as described in any one of the above method embodiments.

[0171] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the method for providing the required function according to any one of the above method embodiments.

[0172] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0173] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0174] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0175] The computer program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), is personalized by utilizing the state information of the computer readable program instructions, and the electronic circuit can execute the computer readable program instructions, thereby realizing various aspects of the present invention.

[0176] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0177] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0178] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0179] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent. It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0180] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0181] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as mentioned above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, TV, or network device, etc.) to execute the methods of various embodiments of the present invention.

[0182] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made by using the contents of the present invention description and drawings under the inventive concept of the present invention, or direct / indirect application in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A method for providing a demand function, characterized in that: Control devices for head mounted devices include: Receiving a plurality of first subprograms capable of executing first required functions of the control device, wherein the first required functions at least include a multi-device fusion function and a control function of a streaming device of the head mounted device; According to the first subprogram, generating a first sub-function module corresponding to the first required function; receiving a first input for selecting a target first sub-functional module among the first sub-functional modules; In response to the first input, an interface and interface description information of the target first sub-function module are provided.

2. The method according to claim 1, characterized in that: The method further comprises: receiving a second input based on the interface description information corresponding to the target interface; In response to the second input, a first sub-function module corresponding to the target interface is executed.

3. The method according to claim 2, characterized in that The target interface is an interface of a target first sub-function module corresponding to the multi-device fusion function, and in response to the second input, running the target first sub-function module corresponding to the target interface includes: Acquire an image of at least one streaming device and a local image of the control device according to the interface description information corresponding to the target interface; fusing an image of at least one streaming device and a local image of the control device to obtain a fused image; The fused image is sent to the head mounted device so that the head mounted device displays the fused image.

4. The method according to claim 3, characterized in that The method further comprises: In an eye movement interaction mode, obtaining a gaze state of a wearer of the head mounted device, and performing an operation matching the gaze state; In the virtual identification and eye movement dual interaction mode, the gaze position of the head mounted device is obtained and The virtual marker is located at a projection position in the fused image and performs an operation that matches the gaze position and the projection position.

5. The method according to claim 1, characterized in that: The method further comprises: receiving at least one second subprogram capable of executing a second required function of the control device, wherein the second required function is a newly added required function; generating a second sub-function module of the second required function according to the second sub-program; In case of receiving a third input selecting the second sub-functional module, in response to the third input, providing an interface and interface description information of the second sub-functional module.

6. The method according to claim 1, characterized in that The method further comprises: receiving a plurality of third subroutines capable of executing a third required function of the head mounted device; According to the third subprogram, generating a third sub-function module corresponding to the third required function; When the control device is in communication connection with the head mounted device, the interface and interface description information of the third sub-functional module are sent to the head mounted device.

7. The method according to claim 1, characterized in that The method further comprises: receiving a plurality of fourth subprograms capable of executing a fourth required function of the streaming device; According to the fourth subprogram, generating a fourth sub-function module corresponding to the fourth required function; When the control device is in communication connection with the streaming device, the interface and interface description information of the fourth sub-function module are sent to the streaming device.

8. The method according to any one of claims 1 to 7, characterized in that: The step of providing, in response to the first input, an interface and interface description information of the target first sub-function module, comprises: In response to the first input, the interface and interface description information of the target first sub-function module are provided in the form of a software installation package.

9. A device for providing a demand function, characterized in that: Control devices for head mounted devices include: The first receiving module is used to receive a plurality of first required functions that can execute the control device. a subprogram, wherein the first required function at least includes a multi-device integration function and a control function of a streaming device of the head mounted device; A generating module, used for generating a first sub-function module corresponding to a first required function according to the first sub-program; A second receiving module, configured to receive a first input for selecting a target first sub-functional module in the first sub-functional modules; A providing module is used to provide the interface and interface description information of the target first sub-function module in response to the first input.

10. A control device, characterized in that: The control device comprises the demand function providing device as claimed in claim 9; or, The control device comprises a memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to call the computer instructions from the memory to execute the method for providing the required function as claimed in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method for providing a required function according to any one of claims 1 to 8 is implemented.

Images