Method and apparatus for providing requirement function, and device and medium

Abstract

The present invention relates to the technical field of head mounting. Disclosed are a method and apparatus for providing a requirement function, and a device and a medium. The method comprises: receiving display requirement information; when the display requirement information indicates a preset display requirement, receiving a plurality of first sub-programs which can execute a first requirement function of a head-mounted device, wherein the first requirement function at least comprises a multi-device fusion function, and a control function for a streaming device of the head-mounted device; on the basis of the first sub-programs, generating first sub-function modules corresponding to the first requirement function; receiving a first input for selecting a target first sub-function module from among the first sub-function modules; and in response to the first input, providing an interface of the target first sub-function module and interface description information. By means of the method, a head-mounted device can provide for a user a service that meets requirements. That is, the head-mounted device can provide a personalized service for the user.

Description

Method, device, equipment and medium for providing required functions

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 16, 2023, with application number 202311033709.7 and invention name “Method, device, equipment and medium for providing demand functions”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present invention relates to the field of head-mounted technology, and more particularly to a method for providing a required function, an apparatus for providing a required function, a head-mounted device, and a computer-readable storage medium. Background Art

[0003] With the development of science and technology and economy, head-mounted display devices (such as AR devices and VR devices, etc.) have been increasingly widely used.

[0004] Head-mounted display devices come in two forms: split-body and all-in-one. However, split-body devices offer superior power consumption, battery life, weight, and heat dissipation compared to all-in-one devices. Therefore, split-body devices are becoming a market and technological trend. Split-body devices are typically equipped with a PUCK device as the processing unit.

[0005] Currently, the functions that can be realized by head-mounted devices provided to users are fixed. Therefore, head-mounted devices cannot provide differentiated services to users.

[0006] Summary of the Invention

[0007] An object of the present invention is to provide a new technical solution for providing the required functionality.

[0008] According to a first aspect of the present invention, a method for providing a required function is provided, which is applied to a head-mounted device, comprising:

[0009] Receive display demand information;

[0010] When the display requirement information is a preset display requirement, receiving a plurality of first subroutines capable of executing first required functions of the head mounted device, wherein the first required functions include at least a multi-device fusion function and a function of controlling a streaming device of the head mounted device;

[0011] generating a first sub-function module corresponding to a first required function according to the first sub-program;

[0012] receiving a first input for selecting a target first sub-functional module among the first sub-functional modules;

[0013] In response to the first input, an interface and interface description information of the target first sub-function module are provided.

[0014] Optionally, the method further includes:

[0015] receiving a second input based on interface description information corresponding to the target interface;

[0016] In response to the second input, a first sub-function module corresponding to the target interface is executed.

[0017] Optionally, the target interface is an interface of a target first sub-functional module corresponding to a multi-device fusion function, and the step of running the target first sub-functional module corresponding to the target interface in response to the second input includes:

[0018] Acquire an image of at least one streaming device and an image of a control device of the head-mounted device according to the interface description information corresponding to the target interface;

[0019] fusing an image of at least one streaming device and an image of the control device to obtain a fused image;

[0020] The fused image is displayed.

[0021] Optionally, the method further includes:

[0022] In an eye movement interaction mode, obtaining a gaze state of a wearer of the head mounted device and performing an operation matching the gaze state;

[0023] In the virtual marker and eye movement dual interaction mode, the gaze position of the head mounted device and the projection position of the virtual marker in the fused image are obtained, and an operation matching the gaze position and the projection position is performed.

[0024] Optionally, the method further includes:

[0025] receiving at least one second subprogram capable of executing a second required function of the head mounted device, where the second required function is a newly added required function;

[0026] generating a second sub-function module of the second required function according to the second sub-program;

[0027] In case of receiving a third input selecting the second sub-functional module, in response to the third input, providing an interface and interface description information of the second sub-functional module.

[0028] Optionally, the method further includes:

[0029] receiving a plurality of third subroutines capable of executing a third required function of a control device of the head mounted device;

[0030] generating a third sub-function module corresponding to the third required function according to the third sub-program;

[0031] When the control device is in communication with the head mounted device, the interface and interface description information of the third sub-function module are sent to the control device.

[0032] Optionally, the method further includes:

[0033] receiving a plurality of fourth subroutines capable of executing a fourth required function of the streaming device;

[0034] generating a fourth sub-function module corresponding to a fourth required function according to the fourth sub-program;

[0035] When the head mounted device is communicatively connected to the streaming device, the interface and interface description information of the fourth sub-functional module are sent to the streaming device.

[0036] Optionally, providing the interface and interface description information of the target first sub-function module in response to the first input includes:

[0037] In response to the first input, the interface and interface description information of the target first sub-function module are provided in the form of a software installation package.

[0038] According to a second aspect of the present invention, there is provided a device for providing a required function, which is applied to a head-mounted device and includes:

[0039] A first receiving module, configured to receive display requirement information;

[0040] a second receiving module, configured to receive, when the display requirement information is a preset display requirement, a plurality of first subroutines capable of executing first required functions of the head-mounted device, wherein the first required functions include at least a multi-device fusion function and a function of controlling a streaming device of the head-mounted device;

[0041] a generating module, configured to generate a first sub-function module corresponding to a first required function according to the first sub-program;

[0042] a third receiving module, configured to receive a first input for selecting a target first sub-functional module among the first sub-functional modules;

[0043] A providing module is used to provide the interface and interface description information of the target first sub-function module in response to the first input.

[0044] According to a third aspect of the present invention, there is provided a head-mounted device, the head-mounted device comprising the device for providing the required function as described in the second aspect; or

[0045] The head mounted device includes a memory and a processor, the memory is used to store computer instructions, and the processor is used to call the computer instructions from the memory to execute the method for providing the required function as described in any one of the first aspects.

[0046] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer program implements the method for providing the required function according to any one of the first aspects.

[0047] In an embodiment of the present invention, a method for providing a desired function is provided. The method is applied to a head-mounted device. The method receives display demand information and, if the display demand information is a preset display demand, receives multiple first subroutines capable of executing a first desired function of the head-mounted device. The first desired function includes at least a multi-device integration function and a function for controlling a streaming device of the head-mounted device. Furthermore, the method generates, based on the first subroutines, first sub-function modules corresponding to the first desired functions. These three steps enable modularization of the first desired functions of the head-mounted device, resulting in a first sub-function module corresponding to each first desired function. Furthermore, the method receives a first input for selecting a target first sub-function module from among the first sub-function modules and, in response to the first input, provides an interface and interface description information for the target first sub-function module. These steps enable the designation of a target first desired function that meets the demand and the customization of the first desired function in the head-mounted device. Based on this, the head-mounted device can provide a user with a service that meets the demand. That is, the head-mounted device can provide a personalized service for the user.

[0048] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0050] FIG1 is a block diagram 1 of a hardware configuration of a head mounted device for implementing a method for providing a required function according to an embodiment of the present invention;

[0051] FIG2 is a flow chart of a method for providing a demand function according to an embodiment of the present invention;

[0052] FIG3 is a schematic diagram of a functional architecture of a control device, a head-mounted device, and a streaming device provided in an embodiment of the present invention;

[0053] 4 is a schematic structural diagram of a device for providing a demand function according to an embodiment of the present invention;

[0054] FIG5 is a second block diagram of the hardware configuration of a head mounted device for implementing a method for providing a required function according to an embodiment of the present invention.

[0055] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0057] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications will also change accordingly.

[0058] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the meaning of "and / or" appearing throughout the text includes three parallel schemes. Taking "A and / or B" as an example, it includes scheme A, or scheme B, or a scheme in which A and B are satisfied at the same time. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0059] FIG1 is a block diagram 1 of the hardware configuration of a head mounted device for implementing a method for providing a required function according to an embodiment of the present invention.

[0060] The head-mounted device is a split-type head-mounted device. The split-type head-mounted device can be an AR, MR, VR, or XR split-type head-mounted device. The head-mounted device is equipped with a corresponding control device, which is specifically a data processing unit of the head-mounted device. The control device is a device with an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in this embodiment of the present invention.

[0061] The head-mounted device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and the like. The processor 1100 may be a central processing unit (CPU), a microprocessor (MCU), or the like. The memory 1200 may include, for example, ROM (read-only memory), RAM (random access memory), or a non-volatile memory such as a hard disk. The interface device 1300 may include, for example, a USB interface or a headphone jack. The communication device 1400 may be capable of wired or wireless communication. The display device 1500 may be, for example, an LCD display or a touchscreen display. The input device 1600 may include, for example, a touchscreen or a keyboard. The user may input and output voice information through the speaker 1700 and microphone 1800.

[0062] Although FIG1 shows multiple devices for the head mounted device 1000 , the present invention may only relate to some of the devices. For example, the head mounted device 1000 may only relate to the memory 1200 and the processor 1100 .

[0063] In the embodiment of the present invention, the memory 1200 of the head mounted device 1000 is used to store instructions, which are used to control the processor 1100 to execute the method for providing the required function provided by the embodiment of the present invention.

[0064] In the above description, a person skilled in the art can design instructions according to the solution disclosed in the present invention. How instructions control the operation of a processor is well known in the art and will not be described in detail here.

[0065] An embodiment of the present invention provides a method for providing a required function, which is applied to a head-mounted device. As shown in FIG2 , the method includes the following steps S2100 to S2500 :

[0066] S2100: Receive display demand information.

[0067] In the embodiment of the present invention, the display requirement information is information indicating that the display requirement is a low-latency display requirement, or information indicating that the display requirement is a non-low-latency display requirement.

[0068] S2200: When the display requirement information is a preset display requirement, receive a plurality of first subroutines capable of executing a first required function of a control device.

[0069] Among them, the first required function includes at least device integration and control of the streaming device of the head-mounted device.

[0070] It should be noted that the method for providing required functions provided in the embodiments of the present invention is applied to a streaming scenario. This scenario includes a control device, a headset, and a streaming device, with the headset and streaming device each being communicatively connected to the control device. The communication connection can be via Wi-Fi or USB. Exemplary streaming devices include mobile phones, PADs, or PCs.

[0071] In an embodiment of the present invention, the preset display requirement may specifically be a low-latency display requirement.

[0072] The first required function is the default required function of the head mounted device in the initial state. One first required function corresponds to one first subroutine, that is, there is a one-to-one correspondence between the first required function and the first subroutine.

[0073] In one embodiment of the present invention, the first required functions may exemplarily include: streaming / communication function, multi-device fusion function, control function of the streaming device of the head-mounted device, function of obtaining Camera data on the head-mounted device side, function of obtaining Audio data on the head-mounted device side, function of obtaining 3Dof / 6Dof data on the head-mounted device side for image cropping and / or completing spatial anchoring, encoding and decoding function, TimeWarp function and rendering on screen function, etc.

[0074] Regarding the above content, the streaming / communication function includes the streaming function and the communication function. The streaming function specifically refers to the function of receiving the display image of the streaming device; and the function of displaying the display image of the streaming device. The communication function specifically refers to the function of transmitting control commands, audio, camera data, etc. between the streaming device and the head-mounted device;

[0075] The multi-device fusion function specifically refers to the function of completing the fusion of the display image of the control device and the display image of the streaming device.

[0076] For any of the aforementioned first required functions, the developer inputs a program into the head-mounted device that can execute (i.e., implement) the first required function. In this embodiment of the present invention, the aforementioned program is referred to as a first subroutine. Based on this, the head-mounted device receives multiple first subroutines that can execute the first required function of the head-mounted device.

[0077] If the display requirement information is a preset display requirement, i.e., a low-latency display requirement, the head-mounted device performs multi-device integration and controls the head-mounted device's streaming device, thereby meeting the low-latency display requirement. Based on this, if the display requirement information is a preset display requirement, multiple first subroutines capable of executing the first required function of the head-mounted device are received, including at least a multi-device integration function and a head-mounted device's streaming device control function.

[0078] Correspondingly, when the display requirement information is not the preset display requirement information, the operation of receiving multiple first subroutines that can execute the first required function of the head mounted device is not received or responded to.

[0079] S2300: Generate a first sub-function module corresponding to a first required function according to a first sub-program.

[0080] In an embodiment of the present invention, for any first subroutine, the head-mounted device encapsulates and protects the first subroutine to obtain a corresponding first sub-functional module. Furthermore, the first sub-functional module includes an interface for calling the first subroutine and interface description information for the interface. The interface description information describes how to use the interface.

[0081] Based on the above steps S2200 and S2300, the modularization of the first required function of the head mounted device can be achieved, which can provide a basis for subsequent users to customize the first required function to meet their own needs.

[0082] S2400: Receive a first input for selecting a target first sub-functional module among the first sub-functional modules.

[0083] In the embodiment of the present invention, the first sub-function module that meets the user's requirements among the first sub-function modules is recorded as the target first sub-function module. The first input is an input for selecting the target first sub-function module from the first sub-function modules.

[0084] There can be at least one target first sub-function. It should be noted that in the embodiment of the present invention, the number of target first sub-functions is not limited.

[0085] The above S2400 can complete the designation of the target first requirement function that meets the requirements.

[0086] S2500 : In response to a first input, provide an interface and interface description information of a target first sub-function module.

[0087] In an embodiment of the present invention, the first sub-function module selected based on the first input is used as the target first sub-function module. After the target first sub-function module is determined, the interface and interface description information of the target first sub-function module are provided. Thus, based on the interface description information, it is possible to determine how to call the corresponding interface. Furthermore, after calling the interface based on the interface description information, the head-mounted device can execute the first required function corresponding to the target first sub-function module corresponding to the interface.

[0088] In one example, when the target first sub-functional module is the first sub-functional module of the multi-device fusion function and the control function of the streaming device of the head-mounted device, based on the above S2500, the head-mounted device can realize the multi-device fusion function and the control function of the streaming device of the head-mounted device.

[0089] Through the above S2500, the customization of the first required function in the head-mounted device can be completed. In this way, the head-mounted device can provide the user with a service that meets the user's needs. In other words, the head-mounted device can provide the user with personalized services.

[0090] In one embodiment of the present invention, the above S2500 may be implemented illustratively according to the following S2510:

[0091] S2510 . In response to the first input, provide the interface and interface information of the target first sub-function module in the form of a software installation package.

[0092] In an embodiment of the present invention, the target first sub-function interface and interface information are presented to the user via a software development kit (SDK). Since SDKs are commonly used by users, and the interface information provided via software installation kits also includes information such as interface operation examples, the above-mentioned S2510 facilitates the user's use of the head-mounted device.

[0093] In an embodiment of the present invention, a method for providing a desired function is provided. The method is applied to a head-mounted device. The method receives display demand information and, if the display demand information is a preset display demand, receives multiple first subroutines capable of executing a first desired function of the head-mounted device. The first desired function includes at least a multi-device integration function and a function for controlling a streaming device of the head-mounted device. Furthermore, the method generates, based on the first subroutines, first sub-function modules corresponding to the first desired functions. These three steps enable modularization of the first desired functions of the head-mounted device, resulting in a first sub-function module corresponding to each first desired function. Furthermore, the method receives a first input for selecting a target first sub-function module from among the first sub-function modules and, in response to the first input, provides an interface and interface description information for the target first sub-function module. These steps enable the designation of a target first desired function that meets the demand and the customization of the first desired function in the head-mounted device. Based on this, the head-mounted device can provide a user with a service that meets the demand. That is, the head-mounted device can provide a personalized service for the user.

[0094] In one embodiment of the present invention, the method for providing the required function provided by the embodiment of the present invention further includes the following S2610 and S2611:

[0095] S2610: Receive a second input based on the interface description information corresponding to the target interface.

[0096] In the embodiment of the present invention, the target interface is an interface that meets the user's call requirements. The user performs input of the target interface call operation according to the interface description information of the target interface. Based on this, the control device receives the second input.

[0097] S2611 . In response to the second input, run the target first sub-function module corresponding to the target interface.

[0098] In an embodiment of the present invention, upon receiving the second input, the head-mounted device executes the first subroutine corresponding to the target first sub-function module called by the second input. At this point, the head-mounted device can complete the required function corresponding to the target first sub-function module called by the second input.

[0099] Based on the embodiments shown in S2610 and S2611 above, when the target interface is the interface of the target first sub-function module corresponding to the multi-device fusion function, S2611 above is specifically implemented through the following S2611-1 to S2611-3:

[0100] S2611-1. Acquire an image of at least one streaming device and an image of a control device of the head-mounted device according to the interface description information corresponding to the target interface.

[0101] SS2611-2. Fusing the image of at least one streaming device and the image of the control device to obtain a fused image.

[0102] S2611-3. Display the fused image.

[0103] In an embodiment of the present invention, an image from at least one streaming device and an image from a control device are acquired in accordance with the acquisition method specified in the interface description information corresponding to the target interface. Furthermore, the image from the at least one streaming device and the image from the control device are spliced ​​together to produce a fused image. Image fusion is thus completed. Furthermore, the head-mounted device displays the fused image.

[0104] In an embodiment of the present invention, when the user calls the interface of the target first sub-function module corresponding to the multi-device fusion function, the head-mounted device can complete the multi-device fusion function through the above S2611-1 to S2611-3.

[0105] Based on the embodiments shown in S2611-1 to S2611-3 above, the method for providing the required function provided by the embodiment of the present invention further includes the following S2611-4 and S2611-5:

[0106] S2611-4. In the eye movement interaction mode, obtain the gaze state of the wearer of the head mounted device and perform operations that match the gaze state.

[0107] S2611-5. In the dual interaction mode of virtual identification and eye movement, obtain the gaze position of the wearer of the head-mounted device and the projection position of the virtual identification in the fused image, and perform operations that match the gaze position and the projection position.

[0108] In the embodiment of the present invention, there are two interaction modes for the interaction of the fused image: an eye movement interaction mode and a virtual marker and eye movement dual interaction mode.

[0109] In the embodiment of the present invention, the eye movement interaction mode refers to a manner in which the eyes of the wearer of the head mounted device interact with the fused image.

[0110] The dual interaction mode of virtual markers and eye movements refers to a method in which a virtual marker emitted by a control device and the wearer's eyes interact with the fused image. In one embodiment of the present invention, the virtual marker is used to describe the path between the control device and the interaction location when the wearer holds the control device and points to the interaction location on the fused image. In one example, the virtual marker can be a straight line, ray, or curve, which is not limited in this embodiment of the present invention.

[0111] In one embodiment of the present invention, the two interaction modes described above can be selected by the user according to their needs, or automatically selected according to the type of application currently running on the control device. This embodiment of the present invention does not limit this.

[0112] In one embodiment of the present invention, in eye tracking mode, an eye tracking algorithm can be used to determine the wearer's gaze position on the fused image and the corresponding gaze time period. In this embodiment of the present invention, the gaze position and gaze time period are recorded as the gaze state.

[0113] Furthermore, in eye tracking mode, the control device performs different operations for different gaze states. For example, if the gaze state indicates that the gaze position remains unchanged and the gaze period corresponding to the gaze position reaches a preset duration, the wearer is determined to be in a state of gazing at the gaze position, and an operation matching the gaze position state is performed. The operation matching the gaze position state can be pre-specified. In one exemplary embodiment, the operation matching the gaze position state can be a confirmation operation.

[0114] For another example, if the gaze state indicates that the gaze position remains unchanged, but the gaze time period corresponding to the gaze position is two discontinuous, shorter time periods, the wearer is determined to be in a double-blink state, and an operation matching the double-blink state at the gaze position is performed. Similarly, the operation matching the double-blink state at the gaze position can be pre-specified. In one exemplary embodiment, the operation matching the double-blink state at the gaze position can be a copy operation.

[0115] In one embodiment of the present invention, in a dual interaction mode of virtual marker and eye movement, the wearer's gaze position on the fused image can be determined using an eye tracking algorithm. Furthermore, the projection position of the virtual marker on the fused image can be obtained by conventionally determining the projection position of the virtual marker on the fused image. Based on the above, the operations in S2611-5 that match the gaze position and the projection position are specifically as follows:

[0116] If the difference between the position of the virtual marker on the fused image and the wearer's gaze position is less than a preset threshold, the position of the virtual marker on the fused image is determined to coincide with the wearer's gaze position, and the operation corresponding to the gaze position and / or the projected position of the virtual marker is performed. For example, if the button corresponding to the gaze position and / or the projected position of the virtual marker is a "Confirm" button, the confirmation operation corresponding to that button is performed.

[0117] Correspondingly, when the difference between the position of the virtual marker on the fused image and the wearer's gaze position is less than a preset threshold, it is determined that the position of the virtual marker on the fused image does not coincide with the wearer's gaze position, and no operation is performed at this time.

[0118] For the dual interaction mode of virtual signage and eye movement, the eye tracking accuracy requirement is low, which can reduce the power consumption of the control device. Correspondingly, in the eye movement interaction mode, the eye tracking accuracy requirement is high.

[0119] In one embodiment of the present invention, the method for providing the required function provided by the embodiment of the present invention further includes the following steps S2710-S2712:

[0120] S2710: Receive at least one second subroutine as a second required function executable by the head mounted device.

[0121] Among them, the second demand function is a new demand function.

[0122] In an embodiment of the present invention, if a developer adds a new desired function, namely a second desired function, to the default desired function, namely the first desired function, the developer inputs a program capable of executing the second desired function into the head-mounted device. In this embodiment of the present invention, the aforementioned program is referred to as a second subroutine. Based on this, the control device receives at least one second subroutine capable of executing the second desired function of the head-mounted device.

[0123] S2711. Generate a second sub-function module of the second required function according to the second sub-program.

[0124] In the embodiment of the present invention, the specific implementation of the above S2711 is the same as the specific implementation of the above S2300, and will not be repeated here.

[0125] S2712: When a third input for selecting the second sub-functional module is received, provide an interface and interface description information of the second sub-functional module in response to the third input.

[0126] In one embodiment of the present invention, the specific implementation of the above S2712 can be: when receiving the third input selecting the second sub-function module, respond to the third input in the form of a software installation package and provide the interface and interface description information of the second sub-function module.

[0127] In this embodiment of the present invention, the third input is an input by the user selecting the second sub-function module. Receiving the third input indicates that the user desires the headset to implement the second desired function corresponding to the second sub-function module. In this case, the headset provides the interface and interface description information of the second sub-function module for the user to invoke the second sub-function module.

[0128] In the embodiment of the present invention, the above steps S2610-S2612 can be used to add a new function to the head mounted device and allow the user to call the newly added function.

[0129] In one embodiment of the present invention, the method for providing the required function provided by the embodiment of the present invention further includes the following steps S2810-S2812:

[0130] S2810: Receive multiple third subroutines capable of executing a third required function of the control device of the head mounted device.

[0131] S2811. Generate a third sub-function module corresponding to the third required function according to the third sub-program.

[0132] S2812: When the control module is in communication with the head mounted device, send the interface and interface description information of the third sub-function module to the control device.

[0133] It should be noted that the specific implementation of the above S2810 and S2811 is the same as the specific implementation of the above S2200 and S2300, and will not be repeated here.

[0134] In one embodiment of the present invention, the specific implementation of the above S2812 can be: when the control module is in communication connection with the head-mounted device, the interface and interface description information of the third sub-function module provided in the form of a software installation package are sent to the control device.

[0135] In this embodiment of the present invention, the head-mounted device and the control device are compatible and mutually reinforcing. Therefore, the control device must implement the required functions adapted to the head-mounted device. In this embodiment of the present invention, the function of the control device to adapt to the requirements of the head-mounted device is referred to as the third required function.

[0136] Furthermore, the head-mounted device generates a third sub-function module for the third required function, and transmits the interface and interface description information of the third sub-function module to the control device based on the communication connection between the control device and the head-mounted device. In this way, when the head-mounted device is adapted to the control device, no modification is required on the control device side.

[0137] In one embodiment of the present invention, the third required functions may exemplarily include: streaming / communication function, plug-in function, multi-application opening function, function of receiving control instructions of head-mounted devices, function of supporting control of local applications of control devices through various interactive methods such as rays, gestures, voice, etc., support for mouse and keyboard interaction of each screen after multi-application opening, support for screen-based volume control function, support for the use of three-party algorithm to use the camera data of the head-mounted device to realize gesture recognition, plane detection and other functions, etc.

[0138] Regarding the above content, the multi-application function specifically refers to: launching multiple applications (local applications and / or applications on streaming devices) at the same time, so that each application runs on a different screen of the head-mounted device, these applications are all in the foreground, and subsequent applications, windows, etc. launched by the application must run on the same screen as the application. This function requires customization of the Framework layer.

[0139] Corresponding to the previous embodiment, in one embodiment of the present invention, the method for providing the required function provided by the embodiment of the present invention further includes the following S2910-S2912:

[0140] S2910: executing a fourth subroutine capable of executing a fourth required function of the streaming device according to a plurality of fourth subroutines.

[0141] S2911. Generate a fourth sub-function module corresponding to a fourth required function according to the fourth sub-program.

[0142] S2912: When the head mounted device is in communication with the streaming device, send the interface and interface description information of the fourth sub-function module to the streaming device.

[0143] It should be noted that the specific implementation of the above S2910 and S2911 is the same as the specific implementation of the above S2200 and S2300, and will not be repeated here.

[0144] In one embodiment of the present invention, the specific implementation of the above S2912 can be: when the head-mounted module is communicatively connected to the streaming device, the interface and interface description information of the fourth sub-function module provided in the form of a software installation package are sent to the streaming device.

[0145] In this embodiment of the present invention, the streaming device and the head-mounted device are compatible and mutually reinforcing. Therefore, the streaming device must implement the required functions of the head-mounted device. In this embodiment of the present invention, the function of the streaming device to adapt to the requirements of the head-mounted device is recorded as the fourth required function.

[0146] Furthermore, the headset generates a fourth sub-function module for the fourth required function, and transmits the interface and interface description information of the fourth sub-function module to the streaming device based on the communication connection between the headset and the streaming device. In this way, if the streaming device is compatible with the headset, no modification is required on the streaming device side.

[0147] In one embodiment of the present invention, the fourth required function may illustratively include: a streaming / communication function, a function of receiving control instructions from the control device side, and a function of converting the control instructions into coordinate time and reporting it to the application of the streaming device.

[0148] In combination with the above content, the functional architecture of the control device, head-mounted device and streaming device in an embodiment of the present invention is shown in FIG3 .

[0149] Among them, Figure 3 shows an example in which the control device supports the OpenXR standard and has an Android operating system. Based on this, the software installation package SDK of the control device is divided into: Plugin layer, Graphic Plugin layer, Platform Plugin layer, and Runtime process layer. And the SDK serves as the service layer of the APP layer. In addition, the local control function of the control device and the multi-application opening are essentially implemented based on the framework layer. In addition, when implementing the local control function of the control device and the multi-application opening based on the framework layer, the framework layer includes: application management, permission management, screen management, Audio management, display management, window management, Input management and Camera management.

[0150] The embodiment of the present invention further provides a device 400 for providing a required function, which is applied to a control device of a head-mounted device, as shown in FIG4 , and includes:

[0151] A first receiving module 410 is configured to receive display requirement information;

[0152] A second receiving module 420 is configured to receive, when the display requirement information is a preset display requirement, a plurality of first subroutines capable of executing first required functions of the head mounted device, wherein the first required functions include at least a multi-device integration function and a function of controlling a streaming device of the head mounted device;

[0153] A generating module 430 is configured to generate a first sub-function module corresponding to a first required function according to the first sub-program;

[0154] A third receiving module 440 is configured to receive a first input for selecting a target first sub-functional module among the first sub-functional modules;

[0155] The providing module 450 is configured to provide the interface and interface description information of the target first sub-function module in response to the first input.

[0156] Based on the device for providing the required functions provided by the embodiment of the present invention, the head-mounted device can provide users with services that meet their needs. That is, the head-mounted device can provide users with personalized services.

[0157] In one embodiment of the present invention, the device 400 for providing the required function provided by the embodiment of the present invention further includes:

[0158] A fourth receiving module receives a second input based on the interface description information corresponding to the target interface;

[0159] The running module runs the first sub-function module corresponding to the target interface in response to the second input.

[0160] In one embodiment of the present invention, the target interface is an interface of a target first sub-function module corresponding to the multi-device fusion function, and the operation module is specifically configured to:

[0161] Acquire an image of at least one streaming device and an image of a control device of the head-mounted device according to the interface description information corresponding to the target interface;

[0162] fusing an image of at least one streaming device and an image of the control device to obtain a fused image;

[0163] The fused image is displayed.

[0164] In one embodiment of the present invention, the operation module is further configured to:

[0165] In an eye movement interaction mode, obtaining a gaze state of a wearer of the head mounted device and performing an operation matching the gaze state;

[0166] In the virtual marker and eye movement dual interaction mode, the gaze position of the head mounted device and the projection position of the virtual marker in the fused image are obtained, and an operation matching the gaze position and the projection position is performed.

[0167] In one embodiment of the present invention, the second receiving module 420 is further configured to receive at least one second subprogram capable of executing a second required function of the head mounted device, where the second required function is a newly added required function;

[0168] The generating module 430 is further configured to generate a second sub-function module of the second required function according to the second sub-program;

[0169] The providing module 450 is further configured to provide the interface and interface description information of the second sub-functional module in response to a third input for selecting the second sub-functional module when the third input is received.

[0170] In one embodiment of the present invention, the second receiving module 420 is further configured to receive a plurality of third subroutines capable of executing a third required function of the control device of the head mounted device;

[0171] The generating module 430 is further configured to generate a third sub-function module corresponding to the third required function according to the third sub-program;

[0172] In this embodiment, the demand function providing device 400 further includes:

[0173] A sending module is used to send the interface and interface description information of the third sub-function module to the control device when the control device is communicatively connected to the head-mounted device.

[0174] In one embodiment of the present invention, the second receiving module 420 is further configured to receive a plurality of fourth subprograms capable of executing a fourth required function of the streaming device;

[0175] The generating module 430 is further configured to generate a fourth sub-function module corresponding to a fourth required function according to the fourth sub-program;

[0176] The sending module is further configured to send the interface and interface description information of the fourth sub-function module to the streaming device when the head mounted device is communicatively connected to the streaming device.

[0177] In one embodiment of the present invention, a providing module 450 is specifically configured to provide, in response to the first input, an interface and interface description information of the target first sub-function module in the form of a software installation package.

[0178] An embodiment of the present invention further provides a head mounted device 500, which includes a device 400 for providing a required function as provided in any of the above device embodiments.

[0179] Alternatively, as shown in Figure 5, the head-mounted device 500 includes a memory 510 and a processor 520, wherein the memory 510 is used to store computer instructions, and the processor 520 is used to call the computer instructions from the memory 510 to execute the method for providing the required function as described in any one of the above method embodiments.

[0180] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the method for providing the required function according to any one of the above method embodiments.

[0181] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0182] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0183] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0184] The computer program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), is personalized by utilizing the state information of the computer readable program instructions, and the electronic circuit can execute the computer readable program instructions, thereby realizing various aspects of the present invention.

[0185] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0186] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0187] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0188] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent. It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0189] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0190] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as mentioned above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, TV, or network device, etc.) to execute the methods of various embodiments of the present invention.

[0191] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made by using the contents of the present invention description and drawings under the inventive concept of the present invention, or direct / indirect application in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A method for providing a demand function, characterized in that: Applied to head mounted devices, including: Receive display demand information; In a case where the display requirement information is a preset display requirement, receiving a plurality of first subprograms capable of executing first required functions of the head mounted device, wherein the first required functions at least include a multi-device fusion function and a control function of a streaming device of the head mounted device; According to the first subprogram, generating a first sub-function module corresponding to the first required function; receiving a first input for selecting a target first sub-functional module among the first sub-functional modules; In response to the first input, an interface and interface description information of the target first sub-function module are provided.

2. The method according to claim 1, characterized in that The method further comprises: receiving a second input based on the interface description information corresponding to the target interface; In response to the second input, a first sub-function module corresponding to the target interface is executed.

3. The method according to claim 2, characterized in that The target interface is an interface of a target first sub-function module corresponding to the multi-device fusion function, and in response to the second input, running the target first sub-function module corresponding to the target interface includes: Acquire an image of at least one streaming device and an image of a control device of the head mounted device according to the interface description information corresponding to the target interface; fusing an image of at least one streaming device and an image of the control device to obtain a fused image; The fused image is displayed.

4. The method according to claim 3, characterized in that: The method further comprises: In an eye movement interaction mode, obtaining a gaze state of a wearer of the head mounted device, and performing an operation matching the gaze state; In the virtual marker and eye movement dual interaction mode, the gaze position of the head mounted device and the projection position of the virtual marker in the fused image are obtained, and the gaze position and the projection position of the virtual marker are executed. Projection position matches the operation.

5. The method according to claim 1, characterized in that: The method further comprises: receiving at least one second subprogram capable of executing a second required function of the head mounted device, wherein the second required function is a newly added required function; generating a second sub-function module of the second required function according to the second sub-program; In case of receiving a third input selecting the second sub-functional module, in response to the third input, providing an interface and interface description information of the second sub-functional module.

6. The method according to claim 1, characterized in that The method further comprises: receiving a plurality of third subroutines capable of executing a third required function of a control device of the head mounted device; According to the third subprogram, generating a third sub-function module corresponding to the third required function; When the control device is communicatively connected to the head mounted device, the interface and interface description information of the third sub-function module are sent to the control device.

7. The method according to claim 1, characterized in that The method further comprises: receiving a plurality of fourth subprograms capable of executing a fourth required function of the streaming device; According to the fourth subprogram, generating a fourth sub-function module corresponding to the fourth required function; When the head mounted device is communicatively connected to the streaming device, the interface and interface description information of the fourth sub-functional module are sent to the streaming device.

8. The method according to any one of claims 1 to 7, characterized in that: The step of providing, in response to the first input, an interface and interface description information of the target first sub-function module, comprises: In response to the first input, the interface and interface description information of the target first sub-function module are provided in the form of a software installation package.

9. A device for providing a demand function, characterized in that: Applied to head mounted devices, including: A first receiving module, used for receiving display requirement information; A second receiving module is used to receive a plurality of first subprograms capable of executing first required functions of the head mounted device when the display requirement information is a preset display requirement, wherein the first required functions at least include a multi-device fusion function and a control function of a streaming device of the head mounted device; A generating module, used for generating a first sub-function module corresponding to a first required function according to the first sub-program; A third receiving module, configured to receive a first input for selecting a target first sub-functional module in the first sub-functional modules; A providing module is used to provide the interface and interface description information of the target first sub-function module in response to the first input.

10. A head mounted device, characterized in that: The head mounted device comprises the device for providing the required function as claimed in claim 9; or The head mounted device comprises a memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to call the computer instructions from the memory to execute the method for providing the required function as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method for providing a required function according to any one of claims 1 to 8 is implemented.

Images