Information display control method and apparatus, head-mounted device, and system

Abstract

The present disclosure relates to the technical field of virtual reality, and provides an information display control method and apparatus, a head-mounted device, and a system. The method comprises: acquiring device information of a target device in a reality scene; on the basis of the device information of the target device, constructing a virtual model corresponding to the target device; and fusing the virtual model corresponding to the target device with a current virtual scene, and displaying a fusion result. The present disclosure improves the convenience of acquiring reality scene condition by a user wearing the head-mounted device, thereby improving user experience.

Description

Information display control method and device, head-mounted device and system

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority from application number 202310899137.4, application date July 20, 2023, and invention title: Information Display Control Method and Device, Head-Mounted Device and System. The entire contents of this Chinese patent application are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of virtual reality technology, and in particular to an information display control method and device, a head-mounted device, and a system. Background Art

[0004] Head-mounted devices (such as VR devices) are used to provide users with an immersive experience, isolating them from the real world. However, in real-world scenarios, when wearing head-mounted devices, users often need to understand the real-world situation, such as the status of special groups of people or the operating status of household appliances. Currently, for this scenario, users need to frequently take off and put on their devices to obtain real-world information, which brings great inconvenience to users. Especially for users who already wear functional glasses, the process of taking off and putting on head-mounted devices is even more cumbersome, seriously affecting the user experience.

[0005] Summary of the Invention

[0006] According to the first aspect of the present disclosure, an information display control method is provided, which obtains device information of a target device in a real scene; constructs a virtual model corresponding to the target device based on the device information of the target device; merges the virtual model corresponding to the target device with the current virtual scene, and displays the fusion result.

[0007] Optionally, the fusion result includes a virtual device associated with the target device. After displaying the fusion result, the method further includes: in response to a target operation on the virtual device, controlling the target device to perform a corresponding operation.

[0008] Optionally, in response to the target operation on the virtual device, controlling the target device to perform the corresponding operation includes: in response to the target operation on the virtual device, calling a target application corresponding to the virtual device to control the target device to perform the corresponding operation through the target application.

[0009] Optionally, in response to a target operation for the virtual device, controlling the target device to perform a corresponding operation includes: generating a target control instruction in response to the target operation for the virtual device; sending the target control instruction to the target device associated with the virtual device so that the target device performs the corresponding operation based on the target control instruction.

[0010] Optionally, constructing a virtual model corresponding to the target device according to the device information of the target device includes: filtering out a corresponding virtual model in a virtual model library according to the device information of the target device, wherein the device information includes at least one of a device type, a device model, and a device identifier.

[0011] Optionally, fusing the virtual model corresponding to the target device with the current virtual scene includes: creating a target virtual subspace in the current virtual scene; and placing the virtual model corresponding to the target device in the target virtual subspace.

[0012] Optionally, the virtual model corresponding to the target device is integrated with the current virtual scene, including: obtaining attribute information of each virtual object in the current virtual scene; determining the target area of ​​the virtual model corresponding to the target device in the current virtual scene based on the attribute information of each virtual object and the virtual model corresponding to the target device; and placing the virtual model corresponding to the target device in the target area.

[0013] Optionally, determining the target area of ​​the virtual model corresponding to the target device in the current virtual scene includes: determining the spatial position relationship between each virtual object and the virtual model corresponding to the target device based on the attribute information of each virtual object and the virtual model corresponding to the target device; and determining the target area corresponding to the virtual model corresponding to the target device in the current virtual space based on the spatial position relationship.

[0014] Optionally, obtaining device information of a target device in a real scene includes: obtaining the device information of the target device through a local connection in response to a trigger operation on a target control.

[0015] Optionally, obtaining device information of a target device in a real scenario includes: establishing a communication connection with a control device associated with the target device; and performing a first interaction with the control device to obtain the device information of the target device through the control device.

[0016] The method further includes: performing a second interaction with the manipulation device to control the target device to perform a corresponding operation through the manipulation device.

[0017] Optionally, the target device includes a first type of device. After the target device performs a corresponding operation or when calling the target application fails, the method further includes: obtaining current first status information of the first type of device; and using the first status information to update the status of the virtual device corresponding to the first type of device.

[0018] Optionally, the target device includes a second type of device, and the method further includes: acquiring media data of the second type of device for the real scene; and displaying the media data in the current virtual scene.

[0019] Optionally, the media data includes a target object in a real scene, and the method further includes: obtaining a virtual digital object corresponding to the target object, updating the state of the virtual digital object based on second state information of the target object captured in real time; and updating the media data using the virtual digital object.

[0020] Optionally, the method further includes: sending a subscription request including subscription conditions to the target device; receiving a subscription notification sent by the target device when the subscription conditions are met; and generating and displaying the status of the virtual device associated with the target device based on the subscription notification.

[0021] Optionally, the method further includes: receiving an update notification sent by the target device, the update notification being triggered when a target event occurs on the target device and generated based on the current state information of the target device; and updating and displaying the state of a virtual device associated with the target device based on the update notification. Furthermore, when the target event occurs on the target device, receiving and displaying media data of the target device specific to a real-world scenario.

[0022] Optionally, the method further includes: obtaining facial information of the user; determining permission information of the user based on the facial information; and determining candidate operations of the current user for each virtual device based on the permission information.

[0023] According to a second aspect of the present disclosure, an information display control device is provided, comprising: an acquisition module, a model construction module and a fusion display module, wherein: the acquisition module is configured to acquire device information of a target device in a real scene; the model construction module is configured to construct a virtual model corresponding to the target device based on the device information of the target device; and the fusion display module is configured to fuse the virtual model associated with the target device with the current virtual scene and display the fusion result.

[0024] According to a third aspect of the present disclosure, a head mounted device is provided, comprising the information display control apparatus of the above embodiment.

[0025] According to a fourth aspect of the present disclosure, an Internet of Things system is provided, comprising the head-mounted device of the above embodiment.

[0026] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the method in the above-mentioned embodiment.

[0027] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0028] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0030] FIG1 schematically shows one of the flow charts of an information display control method according to an embodiment of the present disclosure.

[0031] FIG2 schematically shows a current virtual scene diagram according to an embodiment of the present disclosure.

[0032] FIG3 schematically illustrates a fusion result of a virtual scene and a virtual model corresponding to a target device, and a schematic diagram of a target device control scene based on a user interaction interface according to an embodiment of the present disclosure.

[0033] FIG4 schematically shows a schematic diagram of performing corresponding operations on a virtual device using a target application according to an embodiment of the present disclosure.

[0034] FIG5 schematically shows a schematic diagram of the display effect of media data of the second type of device in a virtual scene according to an embodiment of the present disclosure.

[0035] FIG6 schematically shows a schematic diagram of a status update display effect of a virtual device according to an embodiment of the present disclosure.

[0036] FIG7 schematically shows a second flow chart of the information display control method according to an embodiment of the present disclosure.

[0037] FIG8 schematically shows a block diagram of an information display control device according to an embodiment of the present disclosure.

[0038] FIG9 schematically shows a block diagram of a head mounted device according to an embodiment of the present disclosure.

[0039] FIG10 schematically shows a block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0041] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0042] In one embodiment of the present disclosure, an information display control method is proposed. FIG1 shows a flow chart of the information display control method. As shown in FIG1 , the information display control method can be applied to a head-mounted device and may include at least the following steps:

[0043] Step S110: Acquire device information of a target device in a real scene.

[0044] Step S120 : constructing a virtual model corresponding to the target device according to the current virtual scene and the device information of the target device.

[0045] Step S130: Fusing the virtual model associated with the target device with the current virtual scene and displaying the fusion result.

[0046] The information display control method of the disclosed embodiment can obtain the device information of the target device based on the interconnection between the head-mounted device and the target device in the real scene, and build a virtual model corresponding to the target device according to the device information of the target device; and fuse and display the virtual model associated with the target device with the current virtual scene. On the one hand, it can integrate the devices in the real scene into the virtual scene, so that the user can timely know the status of the devices in the real scene when wearing the head-mounted device, avoiding the problem of the user frequently taking off the device and ensuring the user experience. On the other hand, it can keep the user in the virtual scene all the time, avoiding the problem of poor user experience caused by the interruption of the immersive experience of the virtual scene, ensuring that the user always has a certain sense of immersion in the virtual scene, and reducing the impact of the real scene on the user experience.

[0047] In order to make the technical solution of the present disclosure clearer, each step of the information display control method is described below.

[0048] In step S110 , device information of a target device in a real scene is obtained.

[0049] In this exemplary embodiment, the target device refers to a smart device in a real scene. The real scene refers to a scene in the objective world outside the human brain. The objective world is a perceptible world.

[0050] For example, in a smart home scenario, the target device may be various smart home devices, such as the target device may include one or more of smart surveillance cameras, smart door locks, smart washing machines, smart refrigerators, smart ovens, smart TVs, smart air purifiers, etc., which is not limited in this example. The device information may include one or more of the device type, device identification, device model, device name, device capabilities (which may include device functions and function parameters, etc.), etc., which is not limited in this example. The head-mounted device and the target device may form an Internet of Things so that the devices can be connected through the network. For example, both the head-mounted device and the target device may be added to the local network for communication. The head-mounted device may be a variety of devices that need to be worn on the head and cover the user's eyes, such as various virtual reality (VR) devices or mixed reality (MR) devices, which is not limited in this example.

[0051] In some embodiments, in response to a trigger operation on a target control, device information of the target device may be obtained through a local connection.

[0052] In this example embodiment, a local connection can be established between the target device and the head-mounted device to achieve intercommunication between the devices based on the local connection. For example, the target device and the head-mounted device can exchange information through one of the Matter protocol, UPnP (Universal Plug and Play) protocol, OCF (Open Connectivity Foundation) protocol, etc., so that the head-mounted device can promptly discover the interconnected devices in the real scene. The target control refers to the control in the head-mounted device used to discover the device in the real scene, and the trigger operation can be implemented through an input device such as a handle of the head-mounted device, for example, clicking the target control with the handle. The trigger operation can also be other operation methods of the head-mounted device, which is not limited in this example.

[0053] In other embodiments, obtaining device information of a target device in a real scenario may include the steps of: establishing a communication connection with a control device associated with the target device; and performing a first interaction with the control device to obtain the device information of the target device through the control device.

[0054] In this example embodiment, the control device may be a terminal device that controls the target device. For example, the control device may be a mobile device that is communicatively connected to the target device, such as a mobile phone or a tablet computer. The head-mounted device may be added to a local network to achieve a communication connection with the mobile device. Exemplarily, the first interaction may be that the head-mounted device sends a discovery request for the target device to the mobile device, and the mobile device returns corresponding request information to the head-mounted device based on the request. The request information may include capability information of the target device associated with the mobile device. For example, the capability information may include at least one of the device type, device model, device identification, device function, and parameters corresponding to the function of the target device, and may also include other device-related information, which is not limited in this example. The head-mounted device may construct a corresponding virtual model based on the capability information of the target device.

[0055] In step S120 , a virtual model corresponding to the target device is constructed according to the device information of the target device.

[0056] In this example implementation, the head-mounted device can send the discovered device information to the server, and the server builds a corresponding virtual model based on the device information and returns it to the head-mounted device; the head-mounted device can also locally save the device virtual model library and select the corresponding virtual model based on the discovered device information; the head-mounted device can also scan the target device in real time and build a corresponding virtual model. This example does not specifically limit the method of building the virtual model.

[0057] For example, the corresponding virtual models can be filtered from the virtual model library based on the target device's device information, which includes at least one of the device type, device model, and device identifier. Device types can include refrigerators, door locks, televisions, and other different types. A primary screening of virtual models can be performed based on the device type, followed by a secondary screening based on the device model or device identifier. Alternatively, model identifiers can be pre-assigned to the virtual models in the virtual model library, a mapping relationship established between the model identifier and the device identifier, and the corresponding virtual models can be screened from the virtual model library based on this mapping relationship.

[0058] In some embodiments, different virtual models may be constructed for different virtual scenes so that the virtual model adapts to the current virtual scene, ensuring picture consistency and thus ensuring the effect presented to the user.

[0059] In step S130 , the virtual model corresponding to the target device is fused with the current virtual scene, and the fusion result is displayed.

[0060] In this example embodiment, the current virtual scene refers to the three-dimensional virtual image currently presented by the head-mounted device. The virtual scene may include a user interaction interface so that the user can select applications such as players, games, chat applications, etc. through an input device (such as a handle). For example, as shown in FIG2 , a typical virtual scene diagram is shown, which includes a virtual scene composed of virtual objects such as plateaus, snow-capped mountains, and villas, which can allow users to be placed in the virtual scene. The fusion result may include virtual devices associated with the virtual model. FIG3 shows the fusion result obtained in the smart home scene. As can be seen from FIG3 , the virtual scene incorporates the virtual models corresponding to the smart door lock and smart speaker in the real scene, that is, the target device in the real scene is added to the virtual scene, so that the user can obtain the target device status in the real scene without taking off the head-mounted device, which greatly facilitates the user and ensures the user experience.

[0061] Exemplarily, a target virtual subspace may be created in the current virtual scene; and then a virtual model corresponding to the target device is placed in the target virtual subspace to merge the virtual model corresponding to the target device with the current virtual scene.

[0062] In this example embodiment, the target virtual subspace can be created in the current virtual scene in advance, and then the virtual model corresponding to the target device can be placed in the target virtual subspace. The position of the virtual model can be fixed or dynamically changed. The target virtual subspace can also be created in real time, that is, the position of the target virtual subspace can change with the movement of the user's head posture and the change of the current virtual scene. For example, the specific fusion process is: by identifying the ground in the current virtual scene, a shelf is created on the ground, and the virtual models corresponding to the discovered target devices are placed on the shelf in order. The fusion process can be carried out dynamically.

[0063] Exemplarily, the attribute information of each virtual object in the current virtual scene can be obtained; based on the attribute information of each virtual object and the virtual model corresponding to the target device, the target area of ​​the virtual model corresponding to the target device in the current virtual scene is determined; the virtual model corresponding to the target device is placed in the target area to merge the virtual model corresponding to the target device with the current virtual scene.

[0064] In this example implementation, different identification information can be assigned to virtual objects with different attributes, and matching relationships can be established between the virtual object's identification information and different virtual models to match the virtual objects with the virtual models. For example, in Figure 3, the floor, table, and swimming pool can be assigned identifications A, B, and C, respectively. The virtual models corresponding to the door lock and speaker can then be matched with identification B. During the fusion process, the door lock and speaker can then be placed on the matching table. Alternatively, identification information can be assigned to the virtual model of each target device, directly establishing a matching mapping relationship between the virtual object and the virtual model corresponding to the target device, and matching and fusion of the virtual scene and the virtual model corresponding to the target device can be performed using the identification information.

[0065] Exemplarily, the target area of ​​the virtual model corresponding to the target device in the current virtual scene can be determined according to the following steps: first, based on the attribute information of each virtual object and the virtual model corresponding to the target device, the spatial position relationship between each virtual object and the virtual model corresponding to the target device is determined; then, based on the spatial position relationship, the target area corresponding to the virtual model corresponding to the target device in the current virtual space is determined.

[0066] In this example implementation, the spatial positional relationship between each virtual object and its virtual model can be determined based on its own attributes. Spatial positional correspondences between the attribute information of each virtual object and different virtual models can be pre-set, and the spatial positional relationship can be determined based on this correspondence, thereby determining the target area corresponding to the virtual model. For example, a spatial positional correspondence can be set between virtual objects such as tables and shelves and various small household appliances placed on the support plane of the virtual objects, allowing the target area to be quickly determined by identifying the virtual objects.

[0067] In some embodiments, the fusion result includes a virtual device associated with the target device. After displaying the fusion result, the method further includes: in response to a target operation on the virtual device, controlling the target device to perform a corresponding operation.

[0068] In this example embodiment, the virtual device can be a target device in the current virtual scene. After the virtual model corresponding to the target device is integrated with the current virtual scene, the virtual device can be displayed in the virtual scene. The user can control the target device in the real scene by operating the virtual device to perform corresponding operations, thereby achieving control of the target device in the real scene from the virtual scene.

[0069] For example, in response to a target operation on a virtual device, a target application corresponding to the virtual device may be called to control the target device to perform the corresponding operation through the target application.

[0070] In this example embodiment, a target application corresponding to a virtual device can be installed in the head-mounted device. The target application can be a control application for the target device. As shown in Figure 3, the target application can be selected and invoked through an input device 320 (such as a controller) in the interactive interface 310 of the head-mounted device. Various operations on the target application control the target device 330 to perform corresponding operations. For example, it can be used to control a smart door lock to open or adjust parameters such as the temperature or time of a smart oven.

[0071] Exemplarily, a target control instruction may be generated in response to a target operation on a virtual device; and the target control instruction may be sent to a target device associated with the virtual device, so that the target device performs a corresponding operation based on the target control instruction.

[0072] In this example embodiment, control of the target device can be achieved through the interaction of control commands between the head-mounted device and the target device. As shown in Figure 4, the user can select virtual device 410 (surveillance camera) in the virtual scene and perform different operations (such as target operations) on the virtual device (surveillance camera) using input device 420 (such as a handle) to control the target device to perform different operations, such as turning the camera on, rotating, or off. This approach does not increase the number of applications on the head-mounted device, saving computing resources.

[0073] In some embodiments, the head-mounted device obtains device information by interacting with the control device of the target device; at the same time, the head-mounted device can also perform a second interaction with the control device to control the target device to perform corresponding operations through the control device.

[0074] In this example embodiment, the second interaction can be the exchange of control instructions between the control device and the head-mounted device. For example, by clicking and selecting a virtual device, the head-mounted device sends a start control instruction to the control device, causing the control device to call the application corresponding to the target device. The head-mounted device can also trigger the control device to perform a corresponding operation on the target device by selecting a corresponding function of the virtual device (such as rotating the camera). This example saves the computing and storage resources of the head-mounted device by utilizing the application resources of the control device, thus solving the problem of limited resources of the head-mounted device.

[0075] In some embodiments, the target device includes a first type of device. After the target device performs a corresponding operation or when calling the target application fails, the method further includes: obtaining the current first status information of the first type of device; and using the first status information to update the status of the virtual device corresponding to the first type of device.

[0076] In this example embodiment, the first type of device refers to a device whose state changes after executing a corresponding control operation, such as the door lock, washing machine, stereo, television, oven, etc. in Figure 3, and may also include a surveillance camera. After executing the corresponding operation or when the head-mounted device does not have the target application installed or the program fails, the head-mounted device can obtain its current state information (first state information) from the server or target device, and use the first state information to update the state of the virtual device corresponding to the first type of device. For example, as shown in Figure 6, in response to the door opening operation for the virtual device (smart door lock), the state information of the smart door lock and the corresponding door after the door opening operation can be obtained, and the state of the smart door lock and the virtual device corresponding to the door can be updated based on the state information. For example, the state data after the operation can be obtained, and the corresponding virtual model can be updated based on the state data. The virtual model after the state update is generated and displayed, and the display effect is shown in Figure 6.

[0077] In some embodiments, the target device includes a second type of device, and the method further includes: acquiring media data of the second type of device for the real scene; and displaying the media data in the current virtual scene.

[0078] In this example embodiment, the second type of device refers to a device capable of reflecting a real-world scene, such as a surveillance camera, which captures images of the real-world scene to reflect the real-world scene. The head-mounted device can obtain media data (e.g., video data) captured by the surveillance camera from a server. Alternatively, the head-mounted device can obtain media data captured by the surveillance camera through a local connection, although this example is not limited to this. The head-mounted device displays the acquired media data in the current virtual scene. For example, the media data can be displayed at the center of the current virtual scene, or the display position can be determined based on the user's perspective, or at other designated locations, although this example is not limited to this. As shown in Figure 5, media data captured by the surveillance camera (e.g., images of the living room and bedroom) are displayed in the current virtual scene directly in front of the user's perspective for easy viewing. Different images can be displayed in a specified order (e.g., images of the living room and bedroom are arranged in sequence). The media data can be displayed as a pop-up, with a preconfigured display time (e.g., automatically combined after 5 seconds of display), or the display can be terminated based on a user-specified action, although this example is not limited to this.

[0079] Exemplarily, the media data includes a target object in a real scene, and the method further includes: obtaining a virtual digital object corresponding to the target object, updating the state of the virtual digital object based on the second state information of the target object captured in real time; and updating the media data using the virtual digital object.

[0080] In this example embodiment, the target object refers to an object that can autonomously change state, such as a person or various animals, such as the dog in the living room image in Figure 5. When the media data includes the target object, the target object (dog) can be first identified and then the virtual digital object corresponding to the target object (such as a virtual digital dog) can be called, and the virtual digital object (such as a virtual digital dog) can be used to update the target object (dog) in the media data. In this example, the state of the virtual digital object can be updated on the server side based on the state information (second state information) of the target object captured in real time on the device side, so as to achieve rapid and accurate update and display of key information in the real scene in the virtual scene, improving the immediacy of the display of key information.

[0081] In some embodiments, the method further includes: sending a subscription request including subscription conditions to the target device; receiving a subscription notification sent by the target device when the subscription conditions are met; and generating and displaying the status of the virtual device associated with the target device based on the subscription notification.

[0082] In this example embodiment, the head-mounted device can obtain the state change information of the target device by means of subscription information. The subscription condition can be that the state of the target device changes, that is, when the state of the target device changes, the subscription condition is satisfied. At this time, the target device sends a subscription notification to the subscriber (head-mounted device). The subscription notification may include the state information before and after the target device state changes, and may also include other information such as device information and subscription notification sending time. This example does not limit this. Based on the received subscription notification, the head-mounted device generates a virtual device associated with the target device corresponding to the changed state, as shown in Figure 6, which shows the new state of the virtual device (door open state). The subscription method enables users to promptly know the latest state of the target device in the real scene, avoid the problems of power consumption and poor user experience caused by the long-term target device monitoring process, and reduce the impact of the target device monitoring function on the battery life of the head-mounted device.

[0083] In this example embodiment, for a surveillance camera, when a user wears a head-mounted device, a subscription notification can be sent to the user and the monitored media data can be pushed based on the subscription request of the head-mounted device (the subscription condition can be a change in the state of the target object in the media data).

[0084] In some embodiments, when the user takes off the head mounted device, a cancellation request may be sent to the target device, and subscription notifications sent by the target device may no longer be received, thereby further reducing device power consumption.

[0085] In some embodiments, the method further includes: receiving an update notification sent by the target device, the update notification being triggered when a target event occurs on the target device and generated based on current status information of the target device; and updating and displaying the status of a virtual device associated with the target device based on the update notification.

[0086] In this example implementation, a target event capture capability may be configured on the target device or its server. The target event may be a change in the device's status information or a change in the status of a target object in media data. When a target event is captured, an update notification is sent to the head-mounted device. The update notification may include the current status information of the target device, and may also include information such as the notification time and notification level (such as emergency notification, important notification, and general notification). This example does not limit this. The head-mounted device may update the status of the associated virtual device based on the status information in the update notification. As shown in FIG6 , based on the door lock changing from a closed state to an open state, a corresponding update notification is sent to the head-mounted device. The head-mounted device updates the virtual model of the door associated with the door lock based on the update notification and displays the updated virtual model, i.e., the virtual model of the door in the open state.

[0087] In some embodiments, for a target device with a surveillance camera, when a target event (a change in the state of a target object in media data) is monitored, media data for a real scene pushed by the target device can be received and displayed.

[0088] In some embodiments, the method further includes: obtaining facial information of the user; determining permission information of the user based on the facial information; and determining candidate operations of the current user for each virtual device based on the permission information.

[0089] In this example embodiment, the user's facial information may include iris information, which is used to determine the user's identity and, in turn, the user's permissions. For example, adult users may have greater permissions than adolescent users, and registered users may have greater permissions than ordinary users, and so on. This example does not limit this. For users with greater permissions, more operations (candidate operations) for virtual devices may be enabled. Conversely, for users with less permissions, operational restrictions may be imposed to ensure operational security. For example, adolescent users may be restricted from operating high-temperature devices.

[0090] Referring to FIG7 , the information display control method disclosed herein can be executed in a system consisting of a head-mounted device, a target device, and a server, and implemented when a user wears the head-mounted device. Specifically, the information display control method may include the following steps:

[0091] Step S701: The head mounted device joins the local network.

[0092] Step S702: In response to a trigger operation on a target control, the head mounted device obtains device information of the target device through a local connection.

[0093] Step S703: The head mounted device sends the device information to the server.

[0094] Step S704: The server constructs a virtual model corresponding to the target device based on the received device information.

[0095] In step S705 , the server sends the data corresponding to the virtual model to the head mounted device.

[0096] In step S706 , the head mounted device fuses the received virtual model with the current virtual scene of the head mounted device and displays the virtual scene fused with the corresponding virtual device.

[0097] The process of the above steps S701-S706 is the process of integrating the device in the real scene with the virtual scene.

[0098] Step S707 : In response to the target operation on the virtual device, a target application corresponding to the virtual device is called to control the target device to perform the corresponding operation through the target application.

[0099] In step S708, the server monitors whether the target event occurs. If so, the process goes to S709; otherwise, the process continues monitoring.

[0100] In step S709 , the server sends an update notification to the head mounted device, or the server sends media data to the head mounted device.

[0101] In step S710 , the head mounted device updates and displays the status of the associated virtual device based on the received update notification, or displays the received media data.

[0102] The above steps S707-S710 are a process of using the head-mounted device to control the target device in the real scene to perform various operations and update the display.

[0103] The information display control method provided by the present disclosure can, on the one hand, integrate and display a target device in a real scene into the virtual scene of a head-mounted device, allowing the user to promptly understand the situation in the real scene without removing the head-mounted device, greatly facilitating user operation. While ensuring the immersive experience of the head-mounted device, it also takes into account the timely acquisition of necessary information in the real scene, thereby improving the user experience. On the other hand, the target device in the real scene can be controlled to perform corresponding operations through operations on the head-mounted device, allowing the user to control the corresponding operations of the target device through the head-mounted device, further facilitating the user's operation on the target device, ensuring the user experience while improving the control efficiency.

[0104] The following describes an embodiment of the device disclosed herein, which can be used to execute the information display control method disclosed herein. For details not disclosed in the embodiment of the device disclosed herein, please refer to the embodiment of the information display control method disclosed herein.

[0105] FIG8 schematically shows a block diagram of an information display control device according to an embodiment of the present disclosure.

[0106] As shown in Figure 8, the information display control device 800 according to an embodiment of the present disclosure includes: an acquisition module 810, a model construction module 820 and a fusion display module 830, wherein: the acquisition module 810 is configured to acquire device information of a target device in a real scene; the model construction module 820 is configured to construct a virtual model corresponding to the target device based on the device information of the target device; the fusion display module 830 is configured to fuse the virtual model associated with the target device with the current virtual scene and display the fusion result.

[0107] In some embodiments of the present disclosure, based on the aforementioned scheme, the fusion result includes a virtual device associated with the target device, and the apparatus 800 further includes: a control module, which is configured to: after displaying the fusion result, control the target device to perform a corresponding operation in response to a target operation on the virtual device.

[0108] In some embodiments of the present disclosure, based on the aforementioned solution, the control module is further configured to: in response to a target operation on the virtual device, call a target application corresponding to the virtual device to control the target device to perform a corresponding operation through the target application.

[0109] In some embodiments of the present disclosure, based on the aforementioned scheme, the control module is further configured to: generate a target control instruction in response to a target operation for the virtual device; and send the target control instruction to a target device associated with the virtual device so that the target device performs a corresponding operation based on the target control instruction.

[0110] In some embodiments of the present disclosure, based on the aforementioned solution model, the model building module 820 is further configured to: filter out the corresponding virtual model in the virtual model library according to the device information of the target device, where the device information includes at least one of the device type, device model and device identification.

[0111] In some embodiments of the present disclosure, based on the aforementioned solution, the fusion display module 830 is further configured to: create a target virtual subspace in the current virtual scene; and place the virtual model corresponding to the target device in the target virtual subspace.

[0112] In some embodiments of the present disclosure, based on the aforementioned scheme, the fusion display module 830 is further configured to: obtain attribute information of each virtual object in the current virtual scene; determine the target area of ​​the virtual model corresponding to the target device in the current virtual scene based on the attribute information of each virtual object and the virtual model corresponding to the target device; and place the virtual model corresponding to the target device in the target area.

[0113] In some embodiments of the present disclosure, based on the aforementioned scheme, the fusion display module 830 is further configured to: determine the spatial position relationship between each virtual object and the virtual model corresponding to the target device according to the attribute information of each virtual object and the virtual model corresponding to the target device; and determine the target area corresponding to the virtual model corresponding to the target device in the current virtual space according to the spatial position relationship.

[0114] In some embodiments of the present disclosure, based on the aforementioned solution, the acquisition module 810 is further configured to: in response to a trigger operation on a target control, acquire device information of the target device through a local connection.

[0115] In some embodiments of the present disclosure, based on the aforementioned solution, the acquisition module 810 is further configured to: establish a communication connection with a control device associated with the target device; perform a first interaction with the control device to obtain device information of the target device through the control device;

[0116] The control module is further configured to: perform a second interaction with the manipulation device to control the target device to perform a corresponding operation through the manipulation device.

[0117] In some embodiments of the present disclosure, based on the aforementioned scheme, the target device includes a first category device, and the apparatus 800 also includes an update module, which is configured to: obtain the current first status information of the first category device after the target device performs a corresponding operation or when calling the target application fails; and use the first status information to update the status of the virtual device corresponding to the first category device.

[0118] In some embodiments of the present disclosure, based on the aforementioned solution, the target device includes a second type of device, and the update module is further configured to: obtain media data of the second type of device for the real scene; and display the media data in the current virtual scene.

[0119] In some embodiments of the present disclosure, based on the aforementioned scheme, the media data includes a target object in a real scene, and the update module is further configured to: obtain a virtual digital object corresponding to the target object, and update the status of the virtual digital object based on the second status information of the target object captured in real time; and use the virtual digital object to update the media data.

[0120] In some embodiments of the present disclosure, based on the aforementioned scheme, the device 800 also includes a subscription module, which is configured to: send a subscription request including subscription conditions to the target device; receive a subscription notification sent by the target device when the subscription conditions are met; and the update module is also configured to generate and display the status of the virtual device associated with the target device based on the subscription notification.

[0121] In some embodiments of the present disclosure, based on the aforementioned scheme, the update module is further configured to: receive an update notification sent by the target device, the update notification being triggered when a target event occurs on the target device and generated based on the current status information of the target device; based on the update notification, update and display the status of the virtual device associated with the target device; and / or, in the event that a target event occurs on the target device, receive and display the media data of the target device for the real scene.

[0122] In some embodiments of the present disclosure, based on the aforementioned scheme, the device 800 also includes a determination module, which is configured to: obtain the user's facial information; determine the user's permission information based on the facial information; and determine the current user's candidate operations for each virtual device based on the permission information.

[0123] The detailed information of each module / unit in the information display control device of this embodiment has been described in detail in the information display control method of the aforementioned embodiment, and will not be repeated here.

[0124] 9 , a head mounted device 900 according to an embodiment of the present disclosure includes an information display control device 910 according to any of the above embodiments.

[0125] In this example embodiment, the head-mounted device refers to a device that a user needs to wear on the head and covers the user's eyes. For example, the head-mounted device can be a virtual reality device or a mixed reality device.

[0126] The present disclosure also provides an Internet of Things system. According to an embodiment of the present disclosure, the Internet of Things system may include the head-mounted device of any of the above embodiments.

[0127] It should be noted that although several modules or units of the device for execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0128] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods, or program products. Therefore, various aspects of the present invention may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0129] The electronic device 1000 according to this embodiment of the present invention is described below with reference to Figure 10. The electronic device 1000 shown in Figure 10 is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0130] As shown in FIG10 , electronic device 1000 is implemented as a general-purpose computing device. Components of electronic device 1000 may include, but are not limited to, the aforementioned at least one processing unit 1010, the aforementioned at least one storage unit 1020, a bus 1030 connecting various system components (including storage unit 1020 and processing unit 1010), and a display unit 1040.

[0131] The storage unit stores program code, which can be executed by the processing unit 1010, causing the processing unit 1010 to perform the steps of various exemplary embodiments of the present invention described in the information display control method described above in this specification. For example, the processing unit 1010 may perform steps S110 as shown in Figure 1: obtaining device information of a target device in a real scene; step S120: constructing a virtual model corresponding to the target device based on the device information of the target device; and step S130: fusing the virtual model corresponding to the target device with the current virtual scene and displaying the fusion result.

[0132] The storage unit 1020 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 10201 and / or a cache memory unit 10202 , and may further include a read-only memory unit (ROM) 10203 .

[0133] The storage unit 1020 may also include a program / utility 10204 having a set (at least one) of program modules 10205, such program modules 10205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0134] Bus 1030 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0135] Electronic device 1000 can also communicate with one or more external devices (e.g., a keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a viewer to interact with electronic device 1000, and / or any device that enables electronic device 1000 to communicate with one or more other computing devices (e.g., a router, modem, etc.). Such communication can occur via input / output (I / O) interface 1050. Furthermore, electronic device 1000 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via network adapter 1060. As shown, network adapter 1060 communicates with other modules of electronic device 1000 via bus 1030. It should be understood that, although not shown, other hardware and / or software modules can be used in conjunction with electronic device 1000, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0136] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0137] In this exemplary embodiment, a computer-readable storage medium is also provided, storing a program product capable of implementing the methods described above. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product comprising program code. When the program product is executed on a terminal device, the program code is configured to cause the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the "Exemplary Methods" section above.

[0138] According to an embodiment of the present invention, a program product for implementing the above-described method can be implemented in a portable compact disc read-only memory (CD-ROM) and include program code, and can be executed on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0139] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0140] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0141] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0142] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0143] Furthermore, the above-mentioned figures are merely illustrative of the processes included in the methods according to exemplary embodiments of the present disclosure and are not intended to be limiting. It is readily understood that the processes illustrated in the above-mentioned figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0144] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow from the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

[0145] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An information display control method, comprising: Get the device information of the target device in the real scene; Constructing a virtual model corresponding to the target device according to the device information of the target device; The virtual model corresponding to the target device is fused with the current virtual scene, and the fusion result is displayed.

2. The method according to claim 1, wherein: The fusion result includes a virtual device associated with the target device. After displaying the fusion result, the method further includes: In response to a target operation on the virtual device, the target device is controlled to perform a corresponding operation.

3. The method according to claim 2, wherein: In response to a target operation on the virtual device, controlling the target device to perform a corresponding operation includes: In response to the target operation for the virtual device, a target application corresponding to the virtual device is called to control the target device to perform a corresponding operation through the target application.

4. The method according to claim 2, wherein: In response to a target operation on the virtual device, controlling the target device to perform a corresponding operation includes: generating a target control instruction in response to a target operation on the virtual device; The target control instruction is sent to a target device associated with the virtual device, so that the target device performs a corresponding operation based on the target control instruction.

5. The method according to claim 1, wherein: Constructing a virtual model corresponding to the target device according to the device information of the target device, including: According to the device information of the target device, a corresponding virtual model is screened out in a virtual model library, wherein the device information includes at least one of a device type, a device model, and a device identifier.

6. The method according to claim 1, wherein: The virtual model corresponding to the target device is integrated with the current virtual scene, including: Creating a target virtual subspace in the current virtual scene; The virtual model corresponding to the target device is placed in the target virtual subspace.

7. The method according to claim 1, wherein: The virtual model corresponding to the target device is integrated with the current virtual scene, including: Acquire attribute information of each virtual object in the current virtual scene; Determining a target area of ​​the virtual model corresponding to the target device in the current virtual scene according to the attribute information of each virtual object and the virtual model corresponding to the target device; The virtual model corresponding to the target device is placed in the target area.

8. The method according to claim 7, wherein: Determining a target area of ​​the virtual model corresponding to the target device in the current virtual scene includes: According to the attribute information of each virtual object and the virtual model corresponding to the target device, the virtual object and the target device are determined. The spatial position relationship between the virtual models corresponding to the target devices; According to the spatial position relationship, a target area corresponding to the virtual model corresponding to the target device in the current virtual space is determined.

9. The method according to claim 1, wherein: Get device information of the target device in the real scene, including: In response to a trigger operation on a target control, device information of the target device is acquired through a local connection.

10. The method according to any one of claims 1 to 4, wherein: Get device information of the target device in the real scene, including: Establishing a communication connection with a control device associated with the target device; Performing a first interaction with the control device to obtain device information of the target device through the control device; The method further comprises: A second interaction is performed with the control device to control the target device to perform a corresponding operation through the control device.

11. The method according to claim 3 or 4, wherein: The target device includes a first type of device. After the target device performs a corresponding operation or when calling the target application fails, the method further includes: Obtaining current first status information of the first category device; The first status information is used to update the status of the virtual device corresponding to the first type of device.

12. The method according to claim 1, wherein: The target device includes a second type of device, and the method further includes: Acquire media data of the second type of device for real scenes; The media data is displayed in the current virtual scene.

13. The method according to claim 12, wherein: The media data includes a target object in a real scene, and the method further includes: Acquire a virtual digital object corresponding to the target object, wherein the virtual digital object updates its state based on the second state information of the target object captured in real time; The media data is updated using the virtual digital object.

14. The method according to claim 1, wherein: The method further comprises: Sending a subscription request including subscription conditions to the target device; receiving a subscription notification sent by the target device when the subscription condition is met; Based on the subscription notification, a status of a virtual device associated with the target device is generated and displayed.

15. The method according to claim 1, wherein: The method further comprises: receiving an update notification sent by the target device, wherein the update notification is triggered when a target event occurs on the target device and is generated based on current state information of the target device; Based on the update notification, updating and displaying the status of the virtual device associated with the target device; and / or, When a target event occurs on a target device, media data of the target device for a real scene is received and displayed.

16. The method according to any one of claims 2 to 4, wherein: The method further comprises: Get the user's facial information; Determining the user's authority information based on the facial information; According to the permission information, candidate operations for each virtual device by the current user are determined.

17. An information display control device, comprising: An acquisition module, configured to acquire device information of a target device in a real scene; A model building module, configured to build a virtual model corresponding to the target device according to the device information of the target device; The fusion display module is configured to fuse the virtual model associated with the target device with the current virtual scene and display the fusion result.

18. A head mounted device, comprising the information display control device according to claim 17.

19. An Internet of Things system, comprising the head-mounted device according to claim 18.

20. An electronic device, comprising: one or more processors; A storage device, used for storing one or more programs, when the one or more programs are executed by one or more processors, the one or more processors implement the method according to any one of claims 1 to 16.

21. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 16.

Images