Method, device, equipment and medium for providing demand function

Abstract

The present application discloses a method, device, equipment and medium for providing a demand function, and relates to the field of head-mounted technology. The method is applied to a control device of a head-mounted device, and includes: receiving a first subprogram of multiple executable control devices for a first demand function, wherein the first demand function includes at least a multi-device fusion function and a control function for a streaming device of the head-mounted device; generating a first sub-function module corresponding to the first demand function according to the first subprogram; receiving a first input for selecting a target first sub-function module in the first sub-function module; and providing an interface and interface description information of the target first sub-function module in response to the first input. Through this method, the control device can provide users with services that meet their needs. That is, the control device can provide personalized services for users.

Description

Technical Field

[0001] The present application relates to the field of head-mounted technology, and more specifically, to a method for providing a required function, an apparatus for providing a required function, a control device, and a computer-readable storage medium. Background Art

[0002] With the development of science and technology and economy, head-mounted display devices (such as AR devices and VR devices, etc.) have been used more and more widely.

[0003] Head-mounted display devices are divided into two types: split head-mounted devices and integrated head-mounted devices. However, compared with integrated head-mounted devices, split head-mounted devices have better power consumption, battery life, weight and heat dissipation. Therefore, split head-mounted devices have become a trend in market and technological development. For split head-mounted devices, they are usually equipped with a PUCK device as a processing unit.

[0004] Currently, the functions that can be realized by the PUCK devices provided to users are fixed. Therefore, the PUCK devices cannot provide personalized services for users. Summary of the invention

[0005] One object of the present application is to provide a new technical solution for providing required functions.

[0006] According to a first aspect of the present application, a method for providing a required function is provided, which is applied to a control device of a head mounted device, comprising:

[0007] Receiving a plurality of first subprograms capable of executing first required functions of the control device, wherein the first required functions at least include a multi-device fusion function and a control function of a streaming device of the head mounted device;

[0008] According to the first subprogram, generating a first sub-function module corresponding to the first required function;

[0009] receiving a first input for selecting a target first sub-functional module among the first sub-functional modules;

[0010] In response to the first input, an interface and interface description information of the target first sub-function module are provided.

[0011] Optionally, the method further comprises:

[0012] receiving a second input based on the interface description information corresponding to the target interface;

[0013] In response to the second input, a first sub-function module corresponding to the target interface is executed.

[0014] Optionally, the target interface is an interface of a target first sub-function module corresponding to the multi-device fusion function, and in response to the second input, running the target first sub-function module corresponding to the target interface includes:

[0015] Acquire an image of at least one streaming device and a local image of the control device according to the interface description information corresponding to the target interface;

[0016] fusing an image of at least one streaming device and a local image of the control device to obtain a fused image;

[0017] The fused image is sent to the head mounted device so that the head mounted device displays the fused image.

[0018] Optionally, the method further comprises:

[0019] In an eye movement interaction mode, obtaining a gaze state of a wearer of the head mounted device, and performing an operation matching the gaze state;

[0020] In the virtual marker and eye movement dual interaction mode, the gaze position of the head mounted device and the projection position of the virtual marker in the fused image are obtained, and an operation matching the gaze position and the projection position is performed.

[0021] Optionally, receiving at least one second subprogram capable of executing a second required function of the control device, wherein the second required function is a newly added required function;

[0022] generating a second sub-function module of the second required function according to the second sub-program;

[0023] In case of receiving a third input selecting the second sub-functional module, in response to the third input, providing an interface and interface description information of the second sub-functional module.

[0024] Optionally, receiving a plurality of third subroutines capable of executing a third required function of the head mounted device;

[0025] According to the third subprogram, generating a third sub-function module corresponding to the third required function;

[0026] When the control device is in communication connection with the head mounted device, the interface and interface description information of the third sub-functional module are sent to the head mounted device.

[0027] Optionally, receiving a plurality of fourth subprograms capable of executing a fourth required function of the streaming device;

[0028] According to the fourth subprogram, generating a fourth sub-function module corresponding to the fourth required function;

[0029] When the control device is in communication connection with the streaming device, the interface and interface description information of the fourth sub-function module are sent to the streaming device.

[0030] Optionally, in response to the first input, providing the interface and interface description information of the target first sub-function module includes:

[0031] In response to the first input, the interface and interface description information of the target first sub-function module are provided in the form of a software installation package.

[0032] According to a second aspect of the present application, a device for providing a required function is provided, which is applied to a control device of a head mounted device, comprising:

[0033] A first receiving module, configured to receive a plurality of first subprograms capable of executing first required functions of the control device, wherein the first required functions at least include a multi-device fusion function and a control function of a streaming device of the head mounted device;

[0034] A generating module, used for generating a first sub-function module corresponding to a first required function according to the first sub-program;

[0035] A second receiving module, configured to receive a first input for selecting a target first sub-functional module in the first sub-functional modules;

[0036] A providing module is used to provide the interface and interface description information of the target first sub-function module in response to the first input.

[0037] According to a third aspect of the present application, a control device is provided, the control device comprising the device for providing the demand function as described in the second aspect; or,

[0038] The control device includes a memory and a processor, the memory is used to store computer instructions, and the processor is used to call the computer instructions from the memory to execute the method for providing the required function as described in any one of the first aspects.

[0039] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for providing the required function according to any one of the first aspects is implemented.

[0040] In an embodiment of the present application, a method for providing a required function is provided, which is applied to a control device of a head-mounted device, and the method can complete the modularization of the first required function of the control device by receiving a plurality of first subprograms that can execute the first required function of the control device, wherein the first required function includes at least a multi-device fusion function and a control function of the streaming device of the head-mounted device, and generating a first sub-function module corresponding to the first required function according to the first subprogram, and obtain the first sub-function module corresponding to each first required function. Further, based on receiving a first input of selecting a target first sub-function module in the first sub-function module, and responding to the first input, providing an interface and interface description information of the target first sub-function module, the designation of the target first required function that meets the requirements and the customization of the first required function in the control device can be completed. On this basis, the control device can provide users with services that meet the requirements. That is, the control device can provide personalized services for users.

[0041] Other features and advantages of the present application will become apparent from the following detailed description of exemplary embodiments of the present application with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the application and, together with the description, serve to explain the principles of the application.

[0043] Figure 1 It is a frame of the hardware configuration of the control device for implementing a method for providing a required function provided in an embodiment of the present application. Figure 1 ;

[0044] Figure 2 It is a flowchart of a method for providing a required function provided in an embodiment of the present application;

[0045] Figure 3 It is a schematic diagram of a functional architecture of a control device, a head-mounted device, and a streaming device provided in an embodiment of the present application;

[0046] Figure 4 It is a structural schematic diagram of a device for providing a required function provided in an embodiment of the present application;

[0047] Figure 5 It is a frame of the hardware configuration of the control device for implementing a method for providing a required function provided in an embodiment of the present application. Figure 2 . DETAILED DESCRIPTION

[0048] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present application.

[0049] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present application, its application, or uses.

[0050] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0051] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0052] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0053] Figure 1 It is a frame of the hardware configuration of the control device for implementing a method for providing a required function provided in an embodiment of the present application. Figure 1 .

[0054] The control device is specifically a data processing unit of the head-mounted device. The control device is a device with an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application. Also, the head-mounted device is a split head-mounted device. The split head-mounted device may be a split head-mounted device such as AR, MR, VR or XR.

[0055] The control device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and the like. The processor 1100 may be a central processing unit CPU, a microprocessor MCU, and the like. The memory 1200 may include, for example, a ROM (read-only memory), a RAM (random access memory), a non-volatile memory such as a hard disk, and the like. The interface device 1300 may include, for example, a USB interface, a headphone interface, and the like. The communication device 1400 may, for example, be capable of wired or wireless communication. The display device 1500 may be, for example, a liquid crystal display screen, a touch display screen, and the like. The input device 1600 may include, for example, a touch screen, a keyboard, and the like. The user may input / output voice information through the speaker 1700 and the microphone 1800.

[0056] Despite Figure 1 In the figure, multiple devices are shown for the control device 1000, but the present application may only involve some of the devices, for example, the control device 1000 only involves the memory 1200 and the processor 1100.

[0057] In the embodiment of the present application, the memory 1200 of the control device 1000 is used to store instructions, and the instructions are used to control the processor 1100 to execute the method for providing the demand function provided in the embodiment of the present application.

[0058] In the above description, the technician can design instructions according to the scheme disclosed in this application. How instructions control the processor to operate is well known in the art, so it will not be described in detail here.

[0059] The embodiment of the present application provides a method for providing a required function, which is applied to a control device of a head mounted device, such as Figure 2 As shown, the method includes the following S2100-S2400:

[0060] S2100, receiving a plurality of first subroutines capable of executing a first required function of a control device.

[0061] Among them, the first required function includes at least device integration and control of the streaming device of the head-mounted device.

[0062] It should be noted that the method for providing the required function provided in the embodiment of the present application is applied to a streaming scenario. The scenario includes a control device, a head-mounted device, and a streaming device, and the head-mounted device and the streaming device are respectively connected to the control device in communication. The communication connection method can be Wifi or USB. The streaming device can be exemplarily a mobile phone, a PAD, or a PC.

[0063] In the embodiment of the present application, the first demand function is the default demand function of the control device in the initial state. One first demand function corresponds to one first subroutine, that is, the first demand function and the first subroutine are in a one-to-one correspondence.

[0064] In one embodiment of the present application, the first required functions may exemplarily include: streaming / communication function, control function of the streaming device of the head mounted device, control function of the local control device, multi-application opening function, third-party algorithm function, plug-in function, OpenXR Loader function, multi-device fusion function. Among them, the multi-device fusion function can be realized through the XRRuntime function.

[0065] For the above content, the streaming / communication function includes a streaming function and a communication function, wherein the streaming function specifically refers to: a function of receiving a display image of a streaming device, a function of displaying a display image of a streaming device, and the communication function specifically refers to: a control command transmission function between a streaming device and a control device, and a transmission function of 3Dof / 6Dof, Audio, Camera and other data between a control device and a head-mounted device;

[0066] The multi-device fusion function specifically refers to: completing the fusion of the local display image of the control device and the display image of the streaming device, and transmitting the fused image to the head-mounted device;

[0067] The multi-application function specifically refers to: launching multiple applications (local applications and / or applications on streaming devices) at the same time, so that each application runs on a different screen of the head-mounted device, these applications are in the foreground, and the subsequent applications and windows launched by the application must run on the same screen as the application. This function requires customization of the Framework layer;

[0068] The control function of the streaming device of the head-mounted device specifically refers to: supporting multiple interactive methods such as rays, gestures, and voice to control the application on the streaming device, supporting mouse and keyboard interaction on each screen of the head-mounted device after multiple openings, supporting screen-based volume control, etc. This function needs to be customized at the Framework layer;

[0069] The local control function of the control device specifically refers to: supporting multiple interactive methods such as ray, gesture, voice, etc. to control local applications, supporting mouse and keyboard interaction on each screen of the head-mounted device after multiple openings, supporting screen-based volume control, etc. This function needs to be customized at the Framework layer;

[0070] The third-party algorithm function specifically refers to: using the 3Dof / 6Dof and Camera data of the head-mounted device to achieve spatial anchoring, gesture recognition, plane detection and other functions.

[0071] For any of the above-mentioned first required functions, the developer inputs a program that can execute (i.e., realize) the first required function into the control device. In the embodiment of the present application, the above-mentioned program is recorded as a first subroutine. Based on this, the control device receives multiple first subroutines that can execute the first required function of the control device.

[0072] S2200: Generate a first sub-function module corresponding to a first required function according to a first sub-program.

[0073] In the embodiment of the present application, for any first subprogram, the control device encapsulates and protects the first subprogram to obtain a corresponding first sub-function module. And the first sub-function module has an interface for calling the first subprogram therein and interface description information of the interface. The interface description information is used to describe how to use the interface.

[0074] Based on the above S2100 and S2200, the modularization of the first required function of the control device can be realized. This can provide a customization basis for subsequent users to customize the first required function that meets their own needs. S2300, receiving a first input of selecting a target first sub-function module in the first sub-function module.

[0075] In the embodiment of the present application, the first sub-function module that meets the user's requirements in the first sub-function modules is recorded as the target first sub-function module. The first input is an input for selecting the target first sub-function module from the first sub-function modules.

[0076] There can be at least one target first sub-function. It should be noted that in the embodiment of the present application, the number of target first sub-functions is not limited.

[0077] The above S2300 can complete the designation of the target first requirement function that meets the requirements.

[0078] S2400 . In response to a first input, provide an interface and interface description information of a target first sub-function module.

[0079] In the embodiment of the present application, the first sub-function module selected based on the first input is used as the target first sub-function module. After the target first sub-function module is determined, the interface and interface description information of the target first sub-function module are provided. In this way, based on the description information of the interface, it can be determined how to call the corresponding interface. And, after calling the interface based on the description information of the interface, the control device can execute the first required function corresponding to the target first sub-function module corresponding to the interface.

[0080] In one example, when the target first sub-function module is the first sub-function module of the multi-device fusion function and the control function of the streaming device of the head-mounted device, based on the above S2400, the control device may only implement the multi-device fusion function and the control function of the streaming device of the head-mounted device.

[0081] Through the above S2400, the customization of the first required function in the control device can be completed. In this way, the control device can provide the user with a service that meets the user's needs. That is, the control device can provide personalized services for the user.

[0082] In one embodiment of the present application, the above S2400 may be implemented illustratively according to the following S2410:

[0083] S2410. In response to the first input, provide an interface and interface information of a target first sub-function module in the form of a software installation package.

[0084] In the embodiment of the present application, the target first sub-function interface and interface information are presented to the user in the form of a software installation package (Software Development Kit, SDK). Since SDK is a common method for users, and the interface information provided in the form of a software installation package also includes information such as interface operation examples, the above S2410 can facilitate the user to use the control device.

[0085] In an embodiment of the present application, a method for providing a required function is provided, which is applied to a control device of a head-mounted device, and the method can complete the modularization of the first required function of the control device by receiving a plurality of first subprograms that can execute the first required function of the control device, wherein the first required function includes at least a multi-device fusion function and a control function of the streaming device of the head-mounted device, and generating a first sub-function module corresponding to the first required function according to the first subprogram, and obtain the first sub-function module corresponding to each first required function. Further, based on receiving a first input of selecting a target first sub-function module in the first sub-function module, and responding to the first input, providing an interface and interface description information of the target first sub-function module, the designation of the target first required function that meets the requirements and the customization of the first required function in the control device can be completed. On this basis, the control device can provide users with services that meet the requirements. That is, the control device can provide personalized services for users.

[0086] In one embodiment of the present application, the method for providing the demand function provided in the embodiment of the present application further includes the following S2510 and S2511:

[0087] S2510. Receive a second input based on interface description information corresponding to the target interface.

[0088] In the embodiment of the present application, the target interface is an interface that meets the user's calling requirements. The user performs input of the target interface calling operation according to the interface description information of the target interface. Based on this, the control device receives the second input.

[0089] S2511. In response to the second input, run the target first sub-function module corresponding to the target interface.

[0090] In the embodiment of the present application, when receiving the second input, the control device executes the first subroutine corresponding to the target first sub-function module called by the second input. At this time, the control device can complete the required function corresponding to the target first sub-function module called by the second input.

[0091] On the basis of the embodiments shown in S2510 and S2511 above, when the target interface is the interface of the target first sub-function module corresponding to the multi-device fusion function, the above S2511 is specifically implemented by the following S2511-1 to S2511-3:

[0092] S2511-1. Acquire an image of at least one streaming device and a local image of the control device according to the interface description information corresponding to the target interface.

[0093] SS2511-2. Fuse the image of at least one streaming device and the local image of the control device to obtain a fused image.

[0094] S2511-3. Send the fused image to the head mounted device so that the head mounted device displays the fused image.

[0095] In the embodiment of the present application, the image of at least one streaming device and the local image of the control device are acquired according to the acquisition method specified by the interface description information corresponding to the target interface. And, the image of at least one streaming device and the local image of the control device are spliced ​​to obtain a fused image. Based on this, the image fusion is completed. Further, the control device sends the fused image to the head-mounted device for display by the head-mounted device.

[0096] In an embodiment of the present application, when a user calls the interface of the target first sub-function module corresponding to the multi-device fusion function, the control device can complete the multi-device fusion function through the above S2511-1 to S2511-3.

[0097] On the basis of the embodiments shown in S2511-1 to S2511-3 above, the method for providing the required function provided in the embodiment of the present application further includes the following S2511-4 and S2511-5:

[0098] S2511-4. In the eye movement interaction mode, obtain the gaze state of the wearer of the head mounted device, and perform operations matching the gaze state.

[0099] S2511-5. In the dual interaction mode of virtual identification and eye movement, obtain the gaze position of the wearer of the head mounted device and the projection position of the virtual identification in the fused image, and perform operations matching the gaze position and the projection position.

[0100] In the embodiment of the present application, there are two interaction modes for the interaction of the fused image, which are: eye movement interaction mode and virtual identification and eye movement dual interaction mode.

[0101] In the embodiment of the present application, the eye movement interaction mode refers to the way in which the eyes of the wearer of the head mounted device interact with the fused image.

[0102] The virtual marker and eye movement dual interaction mode refers to the way in which the virtual marker emitted by the control device and the eyes of the wearer of the head-mounted device interact with the fused image together. In one embodiment of the present application, the virtual marker is used to describe the path between the control device and the interaction location when the wearer holds the control device to point to the interaction location on the fused image. In one example, the virtual marker can be a straight line, a ray, or a curve, which is not limited in this embodiment of the present application.

[0103] In an embodiment of the present application, the above two interaction modes can be selected by the user according to his / her own needs, or can be automatically selected according to the application type currently running the application on the control device. This embodiment of the present application does not limit this.

[0104] In one embodiment of the present application, in the eye tracking mode, the gaze position of the wearer on the fused image and the gaze time period corresponding to the gaze position can be determined by an eye tracking algorithm. In the embodiment of the present application, the gaze position and the gaze time period are recorded as the gaze state.

[0105] Also, in the eye tracking mode, for different gaze states, the control device performs different operations. For example, when the gaze state indicates that the gaze position does not change, and the gaze time period corresponding to the gaze position reaches a preset duration, it is determined that the wearer is in a state of gazing at the gaze position, and an operation matching the state of the gaze position is performed. The operation matching the state of the gaze position can be specified in advance. In an exemplary embodiment, the operation matching the state of the gaze position can be a confirmation operation.

[0106] For another example, when the gaze state indicates that the gaze position remains unchanged, but the gaze time period corresponding to the gaze position is two discontinuous shorter time periods, it is determined that the wearer is in a state of blinking twice, and an operation matching the state of blinking twice at the gaze position is performed. Similarly, the operation matching the state of blinking at the gaze position can be specified in advance. In an exemplary embodiment, the operation matching the state of blinking at the gaze position can be a copy operation.

[0107] In one embodiment of the present application, in the dual interaction mode of virtual identification and eye movement, the gaze position of the wearer on the fused image can be determined by an eye tracking algorithm. And the projection position of the virtual identification on the fused image is obtained by the traditional method of determining the projection position of the virtual identification on the fused image. Based on the above content, the operation of performing the operation matching the gaze position and the projection position in the above S2511-5 is specifically:

[0108] When the difference between the position of the virtual marker on the fused image and the gaze position of the wearer is less than a preset threshold, it is determined that the position of the virtual marker on the fused image coincides with the gaze position of the wearer, and the operation corresponding to the gaze position and / or the virtual marker at the projected position is performed. For example, when the button corresponding to the gaze position and / or the virtual marker at the projected position is a "confirm" button, the confirmation operation corresponding to the button is performed.

[0109] Correspondingly, when the difference between the position of the virtual marker on the fused image and the wearer's gaze position is less than a preset threshold, it is determined that the position of the virtual marker on the fused image does not coincide with the wearer's gaze position, and no operation is performed at this time.

[0110] For the virtual identification and eye movement dual interaction mode, the eye movement tracking accuracy requirement is low, which can reduce the power consumption of the control device. Correspondingly, in the eye movement interaction mode, the eye movement tracking accuracy requirement is high.

[0111] In one embodiment of the present application, the method for providing the demand function provided in the embodiment of the present application further includes the following S2610-S2612:

[0112] S2610, receiving at least one second subroutine as a second required function of an executable control device.

[0113] Among them, the second demand function is a new demand function.

[0114] In the embodiment of the present application, when the developer adds a new requirement function, namely the second requirement function, on the basis of the default requirement function, namely the first requirement function, the developer inputs a program that can execute the second requirement function into the control device. In the embodiment of the present application, the aforementioned program is recorded as a second subroutine. Based on this, the control device receives at least one second subroutine that can execute the second requirement function of the control device.

[0115] S2611. Generate a second sub-function module of the second required function according to the second sub-program.

[0116] In the embodiment of the present application, the specific implementation of the above S2611 is the same as the specific implementation of the above S2200, which will not be repeated here.

[0117] S2612: When a third input for selecting the second sub-functional module is received, provide an interface and interface description information of the second sub-functional module in response to the third input.

[0118] In one embodiment of the present application, the specific implementation of the above S2612 may be: upon receiving a third input for selecting the second sub-function module, responding to the third input in the form of a software installation package, and providing an interface and interface description information of the second sub-function module.

[0119] In the embodiment of the present application, the third input is an input for the user to select the second sub-function module. When the third input is received, it means that the user hopes that the control device can realize the second required function corresponding to the second sub-function module. At this time, the control device provides the interface and interface description information of the second sub-function module for the user to call the second sub-function module.

[0120] In the embodiment of the present application, the above S2610-S2612 can realize the addition of the required functions of the control device, and the user can call the newly added required functions.

[0121] In one embodiment of the present application, the method for providing the demand function provided in the embodiment of the present application further includes the following S2710-S2712:

[0122] S2710, receiving a plurality of third subroutines capable of executing a third required function of the head mounted device.

[0123] S2711. Generate a third sub-function module corresponding to the third required function according to the third sub-program.

[0124] S2712: When the control module is in communication connection with the head mounted device, send the interface and interface description information of the third sub-function module to the head mounted device.

[0125] It should be noted that the specific implementation of the above S2710 and S2711 is the same as the specific implementation of the above S2100 and S2200, and will not be repeated here.

[0126] In one embodiment of the present application, the specific implementation of the above S2712 may be: when the control module is communicatively connected with the head mounted device, the interface and interface description information of the third sub-function module provided in the form of a software installation package are sent to the head mounted device.

[0127] In the embodiment of the present application, the head mounted device and the control device are adapted to each other, and the two complement each other. Therefore, the head mounted device needs to realize the required function adapted to the control device. In the embodiment of the present application, the function of the head mounted device to adapt to the requirements of the control device is recorded as the third required function.

[0128] Furthermore, the control device generates a third sub-function module of the third required function, and sends the interface and interface description information of the third sub-function module to the head-mounted device based on the communication connection between the control device and the head-mounted device. In this way, when the head-mounted device is adapted to the control device, no improvement is required on the head-mounted device side.

[0129] In one embodiment of the present application, the third required function may exemplarily include: a function of obtaining Camera data on the head-mounted device side, a function of obtaining Audio data on the head-mounted device side, a function of providing 3Dof / 6Dof data on the head-mounted device side to the control device, a coding function, a TimeWarp function, a rendering on-screen function, a streaming / communication function, etc.

[0130] Corresponding to the previous embodiment, in one embodiment of the present application, the method for providing the demand function provided in the embodiment of the present application further includes the following S2810-S2812:

[0131] S2810, executing a fourth subprogram of the streaming device according to a plurality of fourth required functions.

[0132] S2811. Generate a fourth sub-function module corresponding to a fourth required function according to the fourth sub-program.

[0133] S2812: When the control device is in communication connection with the streaming device, send the interface and interface description information of the fourth sub-function module to the streaming device.

[0134] It should be noted that the specific implementation of the above S2810 and S2811 is the same as the specific implementation of the above S2100 and S2200, and will not be repeated here.

[0135] In one embodiment of the present application, the specific implementation of the above S2812 may be: when the control module is in communication connection with the streaming device, the interface and interface description information of the fourth sub-function module provided in the form of a software installation package are sent to the streaming device.

[0136] In the embodiment of the present application, the streaming device is adapted to the control device, and the two complement each other. Therefore, the streaming device needs to realize the function of adapting to the requirements of the control device. In the embodiment of the present application, the function of adapting the streaming device to the requirements of the control device is recorded as the fourth required function.

[0137] Furthermore, the control device generates a fourth sub-function module of the fourth required function, and sends the interface and interface description information of the fourth sub-function module to the streaming device based on the communication connection between the control device and the streaming device. In this way, when the streaming device is adapted to the control device, no improvement is required on the streaming device side.

[0138] In one embodiment of the present application, the fourth required function may exemplarily include: a streaming / communication function, a function of receiving control instructions from the control device side, and a function of converting the control instructions into coordinate time and reporting it to the application of the streaming device.

[0139] In combination with the above content, the functional architecture of the control device, the head-mounted device and the streaming device in the embodiment of the present application is as follows: Figure 3 shown.

[0140] in, Figure 3 In the example, the control device supports the OpenXR standard and has an Android operating system. Based on this, the software installation package SDK of the control device is divided into: Plugin layer, Graphic Plugin layer, Platform Plugin layer, and Runtime process layer. And the SDK serves as the service layer of the APP layer. In addition, the control function of the streaming device of the head-mounted device, the control function of the local control device, and multiple application openings are essentially implemented based on the framework layer. In addition, when implementing the control function of the streaming device of the head-mounted device, the control function of the local control device, and multiple application openings based on the framework layer, the framework layer includes: application management, permission management, screen management, Audio management, display management, window management, Input management, and Camera management.

[0141] The embodiment of the present application also provides a device 400 for providing a required function, which is applied to a control device of a head mounted device, such as Figure 4 As shown, including:

[0142] A first receiving module 410 is used to receive a plurality of first subprograms capable of executing first required functions of the control device, wherein the first required functions at least include a multi-device fusion function and a control function of a streaming device of the head mounted device;

[0143] A generating module 420, configured to generate a first sub-function module corresponding to a first required function according to the first sub-program;

[0144] A second receiving module 430, configured to receive a first input for selecting a target first sub-functional module among the first sub-functional modules;

[0145] The providing module 440 is used to provide the interface and interface description information of the target first sub-function module in response to the first input.

[0146] Based on the device for providing the required function provided in the embodiment of the present application, the control device can provide the user with services that meet the requirements, that is, the control device can provide the user with personalized services.

[0147] In one embodiment of the present application, the device 400 for providing the demand function provided in the embodiment of the present application further includes:

[0148] A third receiving module receives a second input based on the interface description information corresponding to the target interface;

[0149] The running module runs the first sub-function module corresponding to the target interface in response to the second input.

[0150] In one embodiment of the present application, the target interface is an interface of a target first sub-function module corresponding to the multi-device fusion function, and the operation module is specifically used for:

[0151] Acquire an image of at least one streaming device and a local image of the control device according to the interface description information corresponding to the target interface;

[0152] fusing an image of at least one streaming device and a local image of the control device to obtain a fused image;

[0153] The fused image is sent to the head mounted device so that the head mounted device displays the fused image.

[0154] In one embodiment of the present application, the operation module is also used to:

[0155] In an eye movement interaction mode, obtaining a gaze state of a wearer of the head mounted device, and performing an operation matching the gaze state;

[0156] In the virtual marker and eye movement dual interaction mode, the gaze position of the head mounted device and the projection position of the virtual marker in the fused image are obtained, and an operation matching the gaze position and the projection position is performed.

[0157] In one embodiment of the present application, the first receiving module 410 is further configured to receive at least one second subprogram that can execute a second required function of the control device, where the second required function is a newly added required function;

[0158] The generating module 420 is further used to generate a second sub-function module of the second required function according to the second sub-program;

[0159] The providing module 440 is further configured to provide an interface and interface description information of the second sub-functional module in response to a third input for selecting the second sub-functional module when the third input is received.

[0160] In one embodiment of the present application, the first receiving module 410 is further configured to receive a plurality of third subprograms capable of executing a third required function of the head mounted device;

[0161] The generating module 420 is further used to generate a third sub-function module corresponding to the third required function according to the third sub-program;

[0162] In this embodiment, the demand function providing device 400 further includes:

[0163] A sending module is used to send the interface and interface description information of the third sub-function module to the head mounted device when the control device is in communication connection with the head mounted device.

[0164] In one embodiment of the present application, the first receiving module 410 is further configured to receive a plurality of fourth subprograms capable of executing a fourth required function of the streaming device;

[0165] The generating module 420 is further used to generate a fourth sub-function module corresponding to the fourth required function according to the fourth sub-program;

[0166] The sending module is further used to send the interface and interface description information of the fourth sub-function module to the streaming device when the control device is communicatively connected to the streaming device.

[0167] In one embodiment of the present application, a providing module 440 is specifically configured to provide, in response to the first input, an interface and interface description information of the target first sub-function module in the form of a software installation package.

[0168] The embodiment of the present application further provides a control device 500, which includes a demand function providing device 400 as provided in any of the above-mentioned device embodiments.

[0169] Or, if Figure 5 As shown, the control device 500 includes a memory 510 and a processor 520, wherein the memory 510 is used to store computer instructions, and the processor 520 is used to call the computer instructions from the memory 510 to execute the method for providing the required function as described in any one of the above method embodiments.

[0170] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements a method for providing a required function according to any one of the above method embodiments.

[0171] The present application may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present application.

[0172] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0173] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0174] The computer program instructions for performing the operation of the present application can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions can be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of a computer-readable program instruction to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA), the electronic circuit can execute a computer-readable program instruction, thereby realizing various aspects of the present application.

[0175] Various aspects of the present application are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0176] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0177] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0178] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0179] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present application is defined by the appended claims.

Claims

1. A method for providing a demand function, characterized in that: Control devices for head mounted devices include: Receiving a plurality of first subprograms capable of executing first required functions of the control device, wherein the first required functions at least include a multi-device fusion function and a function of controlling a streaming device of the head mounted device, wherein the multi-device fusion function specifically refers to: completing the fusion of a local display image of the control device and a display image of the streaming device, and transmitting the fused image to the head mounted device; According to the first subprogram, generating a first sub-function module corresponding to the first required function; receiving a first input for selecting a target first sub-functional module among the first sub-functional modules; In response to the first input, providing an interface and interface description information of the target first sub-function module; Receiving a second input based on interface description information corresponding to a target interface, wherein the target interface is an interface of a target first sub-function module corresponding to the multi-device fusion function; Acquire an image of at least one streaming device and a local image of the control device according to the interface description information corresponding to the target interface; fusing an image of at least one streaming device and a local image of the control device to obtain a fused image; Sending the fused image to the head mounted device so that the head mounted device displays the fused image; In an eye movement interaction mode, obtaining a gaze state of a wearer of the head mounted device, and performing an operation matching the gaze state; In the dual interaction mode of virtual identification and eye movement, the gaze position of the head-mounted device and the projection position of the virtual identification in the fused image are obtained, and operations matching the gaze position and the projection position are performed. The virtual identification is used to describe the path between the control device and the interaction position when the wearer holds the control device to point to the interaction position on the fused image.

2. The method according to claim 1, characterized in that: The method further comprises: receiving at least one second subprogram capable of executing a second required function of the control device, wherein the second required function is a newly added required function; generating a second sub-function module of the second required function according to the second sub-program; In case of receiving a third input selecting the second sub-functional module, in response to the third input, providing an interface and interface description information of the second sub-functional module.

3. The method according to claim 1, characterized in that The method further comprises: receiving a plurality of third subroutines capable of executing a third required function of the head mounted device; According to the third subprogram, generating a third sub-function module corresponding to the third required function; When the control device is in communication connection with the head mounted device, the interface and interface description information of the third sub-functional module are sent to the head mounted device.

4. The method according to claim 1, characterized in that: The method further comprises: receiving a plurality of fourth subprograms capable of executing a fourth required function of the streaming device; According to the fourth subprogram, generating a fourth sub-function module corresponding to the fourth required function; When the control device is in communication connection with the streaming device, the interface and interface description information of the fourth sub-function module are sent to the streaming device.

5. The method according to any one of claims 1 to 4, characterized in that: The step of providing, in response to the first input, an interface and interface description information of the target first sub-function module, comprises: In response to the first input, the interface and interface description information of the target first sub-function module are provided in the form of a software installation package.

6. A device for providing a demand function, characterized in that: Control devices for head mounted devices include: A first receiving module, configured to receive a plurality of first subprograms capable of executing a first required function of the control device, wherein the first required function at least includes a multi-device fusion function and a function of controlling a streaming device of the head mounted device, wherein the multi-device fusion function specifically refers to a function of completing fusion of a local display image of the control device and a display image of the streaming device, and transmitting the fused image to the head mounted device; A generating module, used for generating a first sub-function module corresponding to a first required function according to the first sub-program; A second receiving module, configured to receive a first input for selecting a target first sub-functional module in the first sub-functional modules; A providing module, configured to provide, in response to the first input, an interface and interface description information of the target first sub-function module; A third receiving module is used to receive a second input based on the interface description information corresponding to the target interface, wherein the target interface is an interface of the target first sub-function module corresponding to the multi-device fusion function; An operation module is used to obtain the image of at least one streaming device and the local image of the control device according to the interface description information corresponding to the target interface; fuse the image of at least one streaming device and the local image of the control device to obtain a fused image; send the fused image to the head-mounted device so that the head-mounted device displays the fused image; in an eye movement interaction mode, obtain the gaze state of the wearer of the head-mounted device, and perform operations matching the gaze state; in a virtual identification and eye movement dual interaction mode, obtain the gaze position of the head-mounted device and the projected position of the virtual identification in the fused image, and perform operations matching the gaze position and the projected position, wherein the virtual identification is used to describe the path between the control device and the interaction position when the wearer holds the control device to point to the interaction position on the fused image.

7. A control device, characterized in that: The control device comprises the demand function providing device as claimed in claim 6; or, The control device comprises a memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to call the computer instructions from the memory to execute the method for providing the required function as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method for providing a required function according to any one of claims 1 to 5 is implemented.

Images