Equipment interaction method and device, equipment and storage medium

Abstract

The invention provides an equipment interaction method and device, equipment and a storage medium, which are applied to virtual reality equipment in a remote virtual-real interconnection system, and remote communication connection exists between the virtual reality equipment and interactive equipment. The method comprises the steps that a first target virtual model is determined in a virtual reality scene of a virtual reality device, a target virtual model is displayed in the virtual reality scene, the target virtual model is a virtual model corresponding to an interactive device, and the first target virtual model is a target virtual model located on a user sight focus; sending an information reporting instruction to a first interactive device corresponding to the first target virtual model; receiving first equipment interaction information sent by the first interactive equipment; and displaying the first device interaction information in the virtual reality scene. According to the technical scheme, the synchronization of the device interaction information of the interactive device in the real space in the virtual reality device can be realized, and the experience of the user in the real space can be simulated in the virtual reality scene.

Description

Technical Field

[0001] The present application relates to the field of interaction, and in particular to a device interaction method, apparatus, device and storage medium. Background Art

[0002] With the development of science and technology, teaching methods have become diversified. In addition to traditional offline teaching using multimedia screens in the classroom, teachers and students can also conduct remote teaching online through the Internet and use virtual reality (VR) equipment for virtual reality teaching.

[0003] Normally, offline teaching, distance learning and virtual reality teaching are independent teaching methods and need to be conducted separately. In some scenarios, these teaching methods need to be integrated, that is, offline teaching, distance learning and virtual reality teaching are integrated together to complete offline teaching online. In this scenario, it is urgent to synchronize information between online and offline devices. Summary of the invention

[0004] The present application provides a device interaction method, apparatus, device and storage medium to solve the technical problem of information synchronization between online devices and offline devices.

[0005] In a first aspect, a device interaction method is provided, which is applied to a virtual reality device in a remote virtual-reality interconnection system, wherein the remote virtual-reality interconnection system includes the virtual reality device and an interactive device in a real space, and a remote communication connection is established between the virtual reality device and the interactive device; the method includes:

[0006] Determining a first target virtual model in a virtual reality scene of the virtual reality device, wherein a target virtual model is displayed in the virtual reality scene, the target virtual model is a virtual model corresponding to the interactive device, and the first target virtual model is a target virtual model located at the focus of the user's line of sight;

[0007] Sending an information reporting instruction to a first interactive device corresponding to the first target virtual model, wherein the information reporting instruction is used to instruct the first interactive device to report device interaction information of the first interactive device, wherein the device interaction information includes a current device state;

[0008] receiving first device interaction information sent by the first interactive device, where the first device interaction information is device interaction information of the first interactive device;

[0009] The first device interaction information is displayed in the virtual reality scene.

[0010] In this technical solution, a remote communication connection is established between the virtual reality device and the interactive device in the real space. The virtual reality device first determines a first target virtual model located at the focus of the user's line of sight in a virtual reality scene displaying a virtual model corresponding to the interactive device, and then sends an information reporting instruction to the first interactive device corresponding to the first target virtual model, then receives the first device interaction information sent by the first interactive device, and finally displays the first device interaction information in the virtual reality scene, thereby realizing synchronization of the device interaction information of the interactive device in the real space in the online virtual reality device, making it convenient for the user to remotely grasp the current device status of the interactive device in the offline environment and then control the interactive device in the offline environment; since the information reporting instruction is sent to the first interactive device corresponding to the first target virtual model located at the focus of the user's line of sight, it is equivalent to making the interactive device concerned by the user report the device interaction information, which can not only save the amount of data transmission, but also simulate the user's experience in the real space in the virtual reality scene.

[0011] In combination with the first aspect, in a possible implementation, after displaying the first device interaction information in the virtual reality scene, it also includes: receiving a first virtual interaction operation acting on the first target virtual model; in response to the first virtual interaction operation, sending a first interaction instruction corresponding to the first virtual interaction operation to the first interactive device, the first interaction instruction being used to instruct the first interactive device to update the current device state of the first interactive device according to the first interaction instruction. After receiving the virtual interaction operation acting on the target virtual model in the virtual reality scene, the virtual reality device sends an interaction instruction to the interactive device corresponding to the target virtual model, instructing the interactive device to update the current device state of the interactive device according to the interaction instruction, thereby realizing remote control of the interactive device in the offline environment through the VR device, thereby realizing the integration of offline teaching, remote teaching and virtual reality teaching.

[0012] In combination with the first aspect, in a possible implementation, after sending the first interaction instruction corresponding to the first virtual interaction operation to the first interactive device, it also includes: receiving state synchronization information sent by the first interactive device after updating the device state, the state synchronization information including the current device state of the first interactive device after updating the device state; and updating the first device interaction information in the virtual reality scene according to the state synchronization information. The device interaction information in the virtual reality scene is updated as the state of the interactive device in the offline environment is updated, so that the information in the virtual reality scene can be synchronized with the information of the interactive device in the offline environment.

[0013] In combination with the first aspect, in a possible implementation, the device interaction information also includes a device interactivity type; after receiving the first device interaction information sent by the first interactive device, it also includes: displaying a virtual interaction element corresponding to the first target virtual model in the virtual reality scene according to the first device interactivity type, the first device interactivity type being the device interactivity type in the first device interaction information, and the virtual interaction element being used to trigger an interaction instruction corresponding to the first device interactivity type. Displaying a virtual interaction element for controlling the triggering of an interaction instruction in a virtual reality scene according to the device type of the interactive device can facilitate users to control the interactive device in the virtual reality scene and improve operational convenience.

[0014] In combination with the first aspect, in a possible implementation manner, determining a first target virtual model in a virtual reality scene of the virtual reality device includes: determining a user's line of sight focus in the virtual reality scene; and determining a target virtual model corresponding to the user's line of sight focus as the first target virtual model.

[0015] In conjunction with the first aspect, in a possible implementation, displaying the first device interaction information in the virtual reality scene includes: obtaining a first device type, the first device type being the device type of the first interactive device; and displaying the first device interaction information in the virtual reality scene using a display method corresponding to the first device type. By displaying the first device interaction information in the virtual reality scene using a display method corresponding to the device type, the information display method can be adapted to the device type, giving the user a more realistic experience.

[0016] In combination with the first aspect, in a possible implementation, the position of the target virtual model in the virtual reality scene is the same as the position of the interactive device in the real space; before determining the first target virtual model in the virtual reality scene of the virtual reality device, it also includes: obtaining a first position, the first position being the position of the interactive device in the real space; and creating the target virtual model in the virtual reality scene according to the first position. Creating the target virtual model in the virtual reality scene according to the position of the interactive device in the real space can make the position of the target virtual model in the virtual reality scene the same as the position of the interactive device in the real space, thereby enhancing the realism of the target virtual model in the virtual reality scene.

[0017] In combination with the first aspect, in a possible implementation, the visual presentation effect of the target virtual model in the virtual reality scene is the same as the visual presentation effect of the interactive device in the real space; before determining the first target virtual model in the virtual reality scene of the virtual reality device, it also includes: obtaining the device type of the interactive device; and determining the virtual model corresponding to the device type of the interactive device as the target virtual model. Determining the virtual model corresponding to the device type of the interactive device as the target virtual model can make the visual presentation effect of the target virtual model in the virtual reality scene the same as the visual presentation effect of the interactive device in the real space, thereby enhancing the realism of the target virtual model in the virtual reality scene.

[0018] In a second aspect, a device interaction apparatus is provided, which is applied to a virtual reality device in a remote virtual-reality interconnection system, wherein the remote virtual-reality interconnection system includes the virtual reality device and an interactive device in a real space, and a remote communication connection is established between the virtual reality device and the interactive device; the apparatus includes:

[0019] A model determination module, configured to determine a first target virtual model in a virtual reality scene of the virtual reality device, wherein a target virtual model is displayed in the virtual reality scene, the target virtual model is a virtual model corresponding to the interactive device, and the first target virtual model is a target virtual model located at the focus of the user's line of sight;

[0020] a sending module, configured to send an information reporting instruction to a first interactive device corresponding to the first target virtual model, wherein the information reporting instruction is used to instruct the first interactive device to report device interaction information of the first interactive device, wherein the device interaction information includes a current device state;

[0021] A receiving module, configured to receive first device interaction information sent by the first interactive device, where the first device interaction information is device interaction information of the first interactive device;

[0022] A display module is used to display the first device interaction information in the virtual reality scene.

[0023] In a third aspect, a virtual reality device is provided, comprising a memory, a display panel, and one or more processors, wherein the memory and the display panel are connected to the one or more processors, and the one or more processors are used to execute one or more computer programs stored in the memory. When the one or more processors execute the one or more computer programs, the virtual reality device implements the device interaction method of the first aspect.

[0024] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the device interaction method of the first aspect.

[0025] The present application can achieve the following technical effects: it realizes the synchronization of device interaction information of interactive devices in real space in online virtual reality devices, making it convenient for users to remotely grasp the current device status of interactive devices in offline environments and then control the interactive devices in offline environments; since an information reporting instruction is sent to the first interactive device corresponding to the first target virtual model located at the focus of the user's line of sight, it is equivalent to making the interactive device that the user is paying attention to report the device interaction information, which can not only save data transmission volume, but also simulate the user's experience in real space in the virtual reality scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic diagram of a specific application scenario provided in an embodiment of the present application;

[0027] Figure 2 A schematic diagram of the system architecture of a remote virtual-real interconnection system provided in an embodiment of the present application;

[0028] Figure 3 A schematic diagram of the association between a classroom and a remote course provided in an embodiment of the present application;

[0029] Figure 4 A schematic diagram of a flow chart of a device interaction method provided in an embodiment of the present application;

[0030] Figure 5A-5E A schematic diagram of a virtual reality scene provided in an embodiment of the present application;

[0031] Figure 6 is a structural schematic diagram of a device interaction apparatus provided in an embodiment of the present application;

[0032] Figure 7 It is a structural schematic diagram of a virtual reality device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0034] The technical solution of this application is applicable to scenarios where online devices are combined with offline devices to complete remote tasks. A typical scenario where online devices are combined with offline devices to complete remote tasks is as follows: Figure 1 As shown, Figure 1 It shows the scenario of combining online devices with offline devices to complete remote teaching. Figure 1 In the virtual reality scene, teachers use online VR devices to give remote teaching to students in offline classrooms, and students watch the teacher's remote teaching content through the display screen in the real world.

[0035] To achieve Figure 1 In the scenario shown, a remote communication connection needs to be established between online devices and offline devices. Online VR devices and offline devices can form a remote virtual-reality interconnection system. Figure 2 , Figure 2 A schematic diagram of the system architecture of a remote virtual-real interconnection system provided in an embodiment of the present application. Figure 2 As shown, the remote virtual-reality interconnection system 10 includes a VR device 101 and an interactive device 102 in the real space. Among them, the VR device 101 is a terminal with virtual reality function, that is, the virtual reality terminal 101 has the ability to render and display virtual reality scenes. Specifically, the VR device 101 can be VR glasses, VR helmets, and the like. In this application, the VR device is an online device. The interactive device 102 is a device that can interact with the user in the real space, that is, a device that the user can control in the real space. When the real space is a classroom, the interactive device 102 includes but is not limited to audio equipment, display screens, experimental equipment, and the like in the classroom.

[0036] The interactive device 102 may be bound to the real space in advance, and the VR device 101 may find the interactive device 102 in the real space by searching the real space, and then establish a remote communication connection with the interactive device 102 .

[0037] For example, reference may be made to Figure 3 , for the above Figure 1 In the scenario shown, an association relationship between a classroom and a distance learning course can be established through a course schedule, an interactive device can be bound to a classroom, and a teacher can use a VR device in a distance learning course. The VR device can determine the classroom corresponding to the distance learning course according to the course schedule, and then obtain the communication information (such as communication address, communication protocol, service type, port number, etc.) of the interactive device bound to the classroom, complete the discovery of the interactive device, and then establish a remote communication connection between the interactive devices 102 in the classroom corresponding to the distance learning course.

[0038] The remote virtual-reality interconnection system 10 also includes a server 103 , which is connected to the VR device 101 and the interactive device 102 respectively. After the VR device 101 discovers the interactive device 102 , the VR device 101 and the interactive device 102 can establish a remote communication connection through the server 103 .

[0039] After the VR device 101 establishes a remote communication connection with the interactive device 102, the VR device 101 may send an information acquisition request to the interactive device 102 to request the device information of the interactive device 102; after receiving the information acquisition request sent by the VR device 101, the interactive device 102 sends the device information of the interactive device 101 to the VR device 101, and the device information of the interactive device 101 may include the name, device type, device model, etc. of the interactive device 101. In this way, the device identification of the interactive device 102 by the VR device 101 can be completed, and the VR device 101 can determine which interactive devices are included in the real space.

[0040] based on Figure 1 The remote virtual-reality interconnection system shown can implement the technical solution of the present application, and the technical solution of the present application is specifically applied to the VR device 101 in the remote virtual-reality interconnection system. The technical solution of the present application is specifically introduced below.

[0041] See also Figure 4 , Figure 4 A schematic diagram of a device interaction method provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the method comprises the following steps:

[0042] S201, the virtual reality device displays a virtual reality scene.

[0043] Here, the virtual reality scene is a three-dimensional scene displayed in a virtual reality device. The virtual reality scene includes various virtual models, and the virtual model refers to the three-dimensional model in the virtual reality scene. A target three-dimensional model is displayed in the virtual reality scene, and the target three-dimensional model is a virtual model corresponding to the interactive device, that is, a three-dimensional model of the interactive device in the virtual reality scene. After completing the device identification of the interactive device in the real space, the virtual reality device can display a virtual reality scene containing the target three-dimensional model. For example, the real space is a classroom, and the speakers and display screens in the classroom are interactive devices. After identifying the speakers and display screens in the classroom, the virtual reality scene displayed by the virtual reality device contains the virtual model corresponding to the speakers and the virtual model corresponding to the display screen. Exemplarily, the virtual reality scene can be as follows Figure 5A As shown, P1 is the virtual reality scene displayed in the virtual reality device, Y1 in P1 is the virtual model corresponding to the speaker, and Y2 in P1 is the virtual model corresponding to the display screen.

[0044] In some possible cases, the position of the target virtual model in the virtual reality scene is the same as the position of the interactive device in the real space. The position of the target virtual model in the virtual reality scene refers to the position of the target virtual model in the virtual world relative to the origin of the virtual world; the position of the interactive device in the real space refers to the position of the interactive device in the real world relative to the origin of the real world. The target three-dimensional model can be created through the following steps A1-A2:

[0045] A1. Get the first position.

[0046] Here, the first position is the position of the interactive device in the real space.

[0047] In one possible scenario, the first position may be pre-calibrated and stored in the interactive device. When the interactive device replies to the information acquisition request sent by the VR device, the interactive device may carry the first position in the device information of the interactive device; the VR device may obtain the first position from the device information replied by the interactive device.

[0048] In another possible situation, the first position can also be obtained by image recognition. A camera with a pre-calibrated posture can be set in the real space, and the camera can be used to shoot the real space containing the interactive device to obtain a real space image. The VR device can obtain the real space image shot by the camera, and then determine the image position of the interactive device in the real space image based on the target recognition algorithm, and finally determine the first position according to the posture of the camera and the image position of the interactive device in the real space image, wherein the VR device can determine the coordinate conversion relationship between the camera coordinate system and the real world coordinate system according to the posture of the camera, and transform the image position coordinates of the interactive device in the real space image from the image coordinate system to the camera coordinate system based on the camera intrinsic parameter transformation matrix to obtain the position coordinates of the interactive device in the camera coordinate system, and then transform the position coordinates of the interactive device in the camera coordinate system to the real world coordinate system according to the coordinate conversion relationship between the camera coordinate system and the real world coordinate system to obtain the position coordinates of the interactive device in the real world coordinate system, thereby obtaining the first position.

[0049] A2. Creating a target virtual model in the virtual reality scene according to the first position.

[0050] Here, a second position corresponding to the first position can be determined in the virtual world, and a target virtual model can be created at the second position in the virtual world. The position coordinates corresponding to the first position can be converted to the virtual world coordinate system according to a pre-calibrated coordinate conversion relationship between the virtual world coordinate system and the real world coordinate system to obtain the position coordinates corresponding to the second position.

[0051] In a virtual reality scene, a target virtual model is created according to the position of an interactive device in real space, so that the position of the target virtual model in the virtual reality scene is the same as the position of the interactive device in real space, thereby enhancing the realism of the target virtual model in the virtual reality scene.

[0052] In some possible cases, the visual presentation effect of the target virtual model in the virtual reality scene may also be the same as the visual presentation effect of the interactive device in the real space, that is, the size and appearance of the target virtual model in the virtual reality scene are the same or similar to the size and appearance of the interactive device in the real space; the target virtual model may be obtained through the following steps B1-B2:

[0053] B1. Obtain the device type of the interactive device.

[0054] Here, the VR device may obtain the device type of the interactive device from the device information of the interactive device.

[0055] B2. Determine the virtual model corresponding to the device type of the interactive device as the target virtual model.

[0056] In a feasible implementation, a virtual model library may be pre-set, and the virtual model library contains virtual models corresponding to various types of interactive devices. The virtual models in the virtual model library are obtained by three-dimensional modeling based on real interactive devices. The virtual model corresponding to the device type of the interactive device may be obtained from the virtual model library as the target virtual model. For example, if the device type of the interactive device is a display screen, the virtual model corresponding to the display screen is obtained from the virtual model library as the target virtual model.

[0057] In another feasible implementation, a virtual model corresponding to the device type of the interactive device can also be constructed based on a three-dimensional reconstruction method. After obtaining the device type of the interactive device, the corresponding interactive device can be photographed by a camera set in the real space to obtain a real space image containing the interactive device, and then the interactive device can be three-dimensionally reconstructed based on the real space image to obtain a target virtual model. For example, if the device type of the interactive device is a display screen, the display screen can be photographed by a camera to obtain a real space image containing the display screen, and then three-dimensional reconstruction can be performed based on the real space image containing the display screen to obtain a virtual model corresponding to the display screen as the target virtual model.

[0058] Determining the virtual model corresponding to the device type of the interactive device as the target virtual model can make the visual presentation effect of the target virtual model in the virtual reality scene the same as the visual presentation effect of the interactive device in the real space, thereby enhancing the realism of the target virtual model in the virtual reality scene.

[0059] S202: The virtual reality device determines a first target virtual model in a virtual reality scene of the virtual reality device.

[0060] Here, the first target virtual model is a target virtual model located at the focus of the user's sight.

[0061] The first target virtual model may be determined in the virtual reality scene of the virtual reality device through the following steps C1-C2:

[0062] C1. Determine the user's visual focus in the virtual reality scene.

[0063] Here, the user's line of sight focus may be determined by an eye tracking method such as an eye tracking method based on two-dimensional mapping or an eye tracking method based on appearance.

[0064] C2. Determine the target virtual model corresponding to the user's visual focus as the first target virtual model.

[0065] Here, the target virtual model corresponding to the user's line of sight focus is described as the target virtual model that the user is interested in. By determining the target virtual model corresponding to the user's line of sight focus as the first target virtual model, the scene that the user sees when looking at the interactive device in the real space can be simulated in the virtual reality scene, giving the user an experience in the real space.

[0066] S203: The virtual reality device sends an information reporting instruction to the first interactive device corresponding to the first target virtual model, and the first interactive device receives the information reporting instruction.

[0067] Here, the information reporting indication is used to indicate that the first interactive device reports the device interaction information of the first interactive device. The device interaction information refers to the relevant information for the interaction between the interactive device and the user. The device interaction information may include the current device state. The current device state is used to reflect the state of the interactive device. The device type of the interactive device is different, and the specific content of the current device state is different. The current device state may include the current volume of the device, the screen currently displayed in the device, or the current experimental progress of the device. Optionally, the device interaction information may also include the device interactive type, the interaction instruction corresponding to the device interactive type, and the like. Among them, the device interactive type refers to the type of operation that can be performed on the interactive device in the current device state of the interactive device. For example, the interactive device is a speaker. If the current device state of the speaker is off, the device interactive type may be an on operation; if the current device state of the speaker is on, the device interactive type may be an adjustment volume operation. The interaction instruction corresponding to the device interactive type refers to the operation instruction sent to the interactive device to execute the operation corresponding to the device interactive type in the current device state. For example, the interactive device is a speaker, and the device interactive type is a volume adjustment operation. Then, the interactive instructions corresponding to the device interactive type are volume down, volume up, and the like.

[0068] S204: The first interactive device sends first device interaction information to the virtual reality device, and the virtual reality device receives the first device interaction information.

[0069] Here, the first device interaction information is the device interaction information of the first interactive device, and the first device interaction information may include the current device state of the first interactive device, the interactive type of the first interactive device, and the interaction instructions corresponding to the interactive type of the first interactive device, and so on.

[0070] S205: The virtual reality device displays the first device interaction information in the virtual reality scene.

[0071] Among them, for interactive devices of different device types, the virtual reality device can display the device interaction information of the interactive devices of different device types according to the same display method, that is, the display method of the device interaction information of each interactive device is the same. For example, no matter which virtual model corresponding to the interactive device the user is looking at, the device interaction information of the interactive device the user is looking at is displayed in a floating window.

[0072] Optionally, for interactive devices of different device types, the virtual reality device may also use different display devices to display device interaction information for interactive devices of different device types, that is, the display methods of device interaction information of different interactive devices are different. The first device interaction information may be displayed through the following steps D1-D2:

[0073] D1. Obtain a first device type.

[0074] Here, the first device type is the device type of the first interactive device, and the VR device can obtain the first device type from the device information of the first interactive device.

[0075] D2. Display the first device interaction information in a virtual reality scene using a display method corresponding to the first device type.

[0076] For example, if the device type of the first interactive device is a display screen, which is used to display images and supports image switching, the display method corresponding to the first device type may be to display the current device state of the first interactive device in the first target virtual model corresponding to the first interactive device, that is, to display the image currently displayed in the first interactive device in the first target virtual model corresponding to the first interactive device. For another example, if the device type of the first interactive device is a speaker, which is used to play sounds and supports volume adjustment, the display method corresponding to the first device type may be to display the current device state of the first interactive device near the first target virtual model corresponding to the first interactive device.

[0077] For example, assuming that the virtual reality scene is Figure 5A As shown, when the first target virtual model is Figure 5A When Y1 in the example, the device type of the first interactive device is a speaker, a display method corresponding to the first device type is used in the virtual reality scene, and the first device interaction information can be displayed as follows: Figure 5B As shown in P2 in the figure. When the first target virtual model is Figure 5A When Y2 in , the device type of the first interactive device is a display screen. Assume that the screen displayed on the display screen is as follows Figure 5C As shown in p1 in FIG. 1 , in the virtual reality scene, a display method corresponding to the first device type is adopted to display the first device interaction information as follows: Figure 5C As shown in P3.

[0078] At this point, based on the above steps S201 to S205, the virtual reality device completes the information interaction with the first interactive device, which can facilitate the user to remotely grasp the current device status of the interactive device in the offline environment and then control the interactive device in the offline environment; because the information reporting instruction is sent to the first interactive device corresponding to the first target virtual model located at the focus of the user's line of sight, it is equivalent to making the interactive device that the user pays attention to report the device interaction information, which can not only save data transmission volume, but also simulate the user's experience in the real space in the virtual reality scene.

[0079] Optionally, after receiving the first device interaction information sent by the first interactive device, the virtual reality device can also display a virtual interaction element corresponding to the first target virtual model in the virtual reality scene according to the first device interactivity type, the first device interactivity type being the device interactivity type in the first device interaction information, and the virtual interaction element corresponding to the first target virtual model is used to trigger the interaction instruction corresponding to the first interactivity type.

[0080] For example, if the first interactive device is a display screen, and the interactive type of the first interactive device is screen switching, then the virtual interactive element corresponding to the first target virtual model is a virtual switching button, which is used to trigger the screen switching instruction. For another example, if the first interactive device is a speaker, and the interactive type of the first interactive device is volume adjustment, then the virtual interactive element corresponding to the first target virtual model is a volume adjustment button.

[0081] For example, assuming that the virtual reality scene is Figure 5A As shown, when the first target virtual model is Figure 5A When Y1 in , the first interactive device is a speaker, and the interactive type of the first interactive device is adjusting the volume, then the virtual interactive element corresponding to the first target virtual model displayed in the virtual reality scene can be as follows Figure 5D As shown in P4 in FIG. 1 , Q1 in P4 is the virtual interaction element corresponding to the first target virtual model; when the first target virtual model is Figure 5A When Y2 in , the first interactive device is a display screen, and the interactive type of the first interactive device is screen switching, then the virtual interactive element corresponding to the first target virtual model can be displayed in the virtual reality scene as follows: Figure 5E As shown in P5 in FIG. 1 , Q2 and Q3 in P5 are virtual interaction elements corresponding to the first target virtual model.

[0082] Displaying virtual interactive elements for controlling and triggering interactive instructions in a virtual reality scene according to the device type of the interactive device can facilitate users to control the interactive device in the virtual reality scene and improve operational convenience.

[0083] After receiving the first device interaction information, the user may also control the first interactive device in the virtual reality scene according to the first device interaction information displayed in the virtual reality scene. The device interaction method provided in the present application may also include the following steps:

[0084] S206: The virtual reality device receives a first virtual interaction operation acting on a first target virtual model.

[0085] Here, the first virtual interaction operation is a virtual operation performed by a user on a first target virtual model in a virtual reality scene. The VR device can receive the first virtual interaction operation by acquiring a position in the virtual reality scene corresponding to the user's line of sight focus.

[0086] For example, a virtual reality scene such as Figure 5E As shown in P5 in FIG. 1 , when it is detected that the user's sight focus is on the virtual interactive element Q2 , it can be determined that the first virtual interactive operation is a virtual interactive operation of turning the page down.

[0087] S207: The virtual reality device responds to the first virtual interaction operation and sends a first interaction instruction corresponding to the first virtual interaction operation to the first interactive device, and the first interactive device receives the first interaction instruction.

[0088] Here, the first interaction instruction is used to trigger the first interactive device to perform an interaction operation corresponding to the first virtual interaction operation. For example, if the first interactive device is a display screen and the first virtual interaction operation is a virtual interaction operation of turning a page down, the first interaction instruction is used to trigger the first interactive device to turn a page down.

[0089] S208: The first interactive device updates the current device state of the first interactive device according to the first interaction instruction.

[0090] For example, the first interactive device is a display screen, and the first interactive instruction is an interactive instruction for triggering the first interactive device to turn the page down, then the first interactive device can turn the page down according to the first interactive instruction.

[0091] For example, assuming that the display screen is displayed as Figure 5E As shown in p1 in , after turning the page down, the display of the first interactive device can be as follows Figure 5E As shown in p2.

[0092] S209: The first interactive device sends state synchronization information to the virtual reality device, and the virtual reality device receives the state synchronization information.

[0093] The state synchronization information includes the current device state of the first interactive device after the device state is updated. Optionally, the state synchronization information may also include the device interactive type, the interactive instruction corresponding to the device interactive type, and the like.

[0094] For example, the first interactive device is a display screen. Assume that the current device state after the first interactive device updates the device state is as follows: Figure 5E As shown in p2 in the figure, the first interactive device sends the picture in p2 to the virtual reality device, and the picture in p2 sent by the first interactive device to the virtual reality device is a part of the state synchronization information.

[0095] S210: The virtual reality device updates the first device interaction information in the virtual reality scene according to the state synchronization information.

[0096] Exemplarily, the virtual reality device may update the first device interaction information according to the state synchronization information as follows: Figure 5E As shown in P6.

[0097] Based on the above steps S206 to S210, after receiving the virtual interaction operation acting on the target virtual model in the virtual reality scene, the virtual reality device sends an interaction instruction to the interactive device corresponding to the target virtual model, so as to instruct the interactive device to update the current device state of the interactive device according to the interaction instruction, thereby realizing remote control of the interactive device in the offline environment through the VR device, thereby realizing the integration of offline teaching, remote teaching and virtual reality teaching; the device interaction information in the virtual reality scene is updated as the state of the interactive device in the offline environment is updated, and synchronization between the information in the virtual reality scene and the information of the interactive device in the offline environment can be achieved.

[0098] The method of the present application is introduced above, and the device of the present application is introduced below.

[0099] See also Figure 6 , Figure 6 Schematic diagram of a device interaction apparatus provided in an embodiment of the present application. A virtual reality device applied to a remote virtual-reality interconnection system, the remote virtual-reality interconnection system can be as follows Figure 2 As shown. Figure 6 As shown, the device interaction device 30 includes:

[0100] A model determination module 301 is used to determine a first target virtual model in a virtual reality scene of the virtual reality device, wherein a target virtual model is displayed in the virtual reality scene, the target virtual model is a virtual model corresponding to the interactive device, and the first target virtual model is a target virtual model located at the user's sight focus;

[0101] A sending module 302 is used to send an information reporting instruction to a first interactive device corresponding to the first target virtual model, wherein the information reporting instruction is used to instruct the first interactive device to report device interaction information of the first interactive device, wherein the device interaction information includes a current device state;

[0102] A receiving module 303 is configured to receive first device interaction information sent by the first interactive device, where the first device interaction information is device interaction information of the first interactive device;

[0103] The display module 304 is used to display the first device interaction information in the virtual reality scene.

[0104] In a possible design, the sending module 302 is also used to: receive a first virtual interaction operation acting on the first target virtual model; in response to the first virtual interaction operation, send a first interaction instruction corresponding to the first virtual interaction operation to the first interactive device, and the first interaction instruction is used to instruct the first interactive device to update the current device state of the first interactive device according to the first interaction instruction.

[0105] In a possible design, the receiving module 303 is further used to: receive status synchronization information sent by the first interactive device after updating the device status, wherein the status synchronization information includes the current device status of the first interactive device after updating the device status; the display module 304 is further used to: update the first device interaction information in the virtual reality scene according to the status synchronization information.

[0106] In a possible design, the device interaction information also includes a device interactivity type; the above-mentioned display module 304 is also used to: display a virtual interaction element corresponding to the first target virtual model in the virtual reality scene according to the first device interactivity type, the first device interactivity type is the device interactivity type in the first device interaction information, and the virtual interaction element is used to trigger an interaction instruction corresponding to the first device interactivity type.

[0107] In a possible design, the model determination module 301 is specifically used to: determine the user's line of sight focus in the virtual reality scene; and determine the target virtual model corresponding to the user's line of sight focus as the first target virtual model.

[0108] In a possible design, the display module 304 is specifically used to: obtain a first device type, where the first device type is the device type of the first interactive device; and display the first device interaction information in the virtual reality scene using a display method corresponding to the first device type.

[0109] In a possible design, the position of the target virtual model in the virtual reality scene is the same as the position of the interactive device in the real space; the above-mentioned model determination module 301 is also used to: obtain a first position, which is the position of the interactive device in the real space; and create the target virtual model in the virtual reality scene according to the first position.

[0110] In a possible design, the visual presentation effect of the target virtual model in the virtual reality scene is the same as the visual presentation effect of the interactive device in the real space; the above-mentioned model determination module 301 is also used to: obtain the device type of the interactive device; and determine the virtual model corresponding to the device type of the interactive device as the target virtual model.

[0111] It should be noted that Figure 6 For the contents not mentioned in the corresponding embodiments, please refer to the description of the aforementioned method embodiments, which will not be repeated here.

[0112] The above-mentioned device first determines the first target virtual model located at the focus of the user's line of sight in the virtual reality scene where the virtual model corresponding to the interactive device is displayed, then sends an information reporting instruction to the first interactive device corresponding to the first target virtual model, then receives the first device interaction information sent by the first interactive device, and finally displays the first device interaction information in the virtual reality scene, thereby realizing the synchronization of the device interaction information of the interactive device in the real space in the online virtual reality device, making it convenient for the user to remotely grasp the current device status of the interactive device in the offline environment and then control the interactive device in the offline environment; since the information reporting instruction is sent to the first interactive device corresponding to the first target virtual model located at the focus of the user's line of sight, it is equivalent to making the interactive device that the user focuses on report the device interaction information, which can not only save the data transmission amount, but also simulate the user's experience in the real space in the virtual reality scene.

[0113] See also Figure 7 , Figure 7 4 is a structural diagram of a virtual reality device provided in an embodiment of the present application, wherein the virtual reality device 40 includes a processor 401, a memory 402, and a display panel 403. The memory 402 and the display panel 403 are connected to the processor 401, for example, through a bus.

[0114] The processor 401 is configured to support the virtual reality device 40 to perform the corresponding functions of the method in the above method embodiment. The processor 401 can be a central processing unit (CPU), a network processor (NP), a hardware chip or any combination thereof. The above hardware chip can be an application specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0115] The memory 402 is used to store program codes, etc. The memory 402 may include a volatile memory (VM), such as a random access memory (RAM); the memory 402 may also include a non-volatile memory (NVM), such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the memory 402 may also include a combination of the above-mentioned types of memories.

[0116] The display panel 403 is used to display the virtual reality scene.

[0117] The processor 401 may call the program code to perform the following operations:

[0118] Determining a first target virtual model in a virtual reality scene of the virtual reality device, wherein a target virtual model is displayed in the virtual reality scene, the target virtual model is a virtual model corresponding to the interactive device, and the first target virtual model is a target virtual model located at the focus of the user's line of sight;

[0119] Sending an information reporting instruction to a first interactive device corresponding to the first target virtual model, wherein the information reporting instruction is used to instruct the first interactive device to report device interaction information of the first interactive device, wherein the device interaction information includes a current device state;

[0120] receiving first device interaction information sent by the first interactive device, where the first device interaction information is device interaction information of the first interactive device;

[0121] The first device interaction information is displayed in the virtual reality scene.

[0122] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the method described in the above embodiment.

[0123] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0124] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A device interaction method, characterized in that: A virtual reality device used in a remote virtual-reality interconnection system, wherein the remote virtual-reality interconnection system includes the virtual reality device and an interactive device in a real space, and a remote communication connection is established between the virtual reality device and the interactive device; the method includes: Determining a first target virtual model in a virtual reality scene of the virtual reality device, wherein a target virtual model is displayed in the virtual reality scene, the target virtual model is a virtual model corresponding to the interactive device, and the first target virtual model is a target virtual model located at the focus of the user's line of sight; Sending an information reporting instruction to a first interactive device corresponding to the first target virtual model, wherein the information reporting instruction is used to instruct the first interactive device to report device interaction information of the first interactive device, wherein the device interaction information includes a current device state; receiving first device interaction information sent by the first interactive device, where the first device interaction information is device interaction information of the first interactive device; The first device interaction information is displayed in the virtual reality scene.

2. The method according to claim 1, characterized in that After displaying the first device interaction information in the virtual reality scene, the method further includes: receiving a first virtual interaction operation acting on the first target virtual model; In response to the first virtual interaction operation, a first interaction instruction corresponding to the first virtual interaction operation is sent to the first interactive device, where the first interaction instruction is used to instruct the first interactive device to update a current device state of the first interactive device according to the first interaction instruction.

3. The method according to claim 2, characterized in that After sending the first interaction instruction corresponding to the first virtual interaction operation to the first interactive device, the method further includes: Receiving state synchronization information sent by the first interactive device after updating the device state, the state synchronization information including the current device state of the first interactive device after updating the device state; The first device interaction information is updated in the virtual reality scene according to the state synchronization information.

4. The method according to claim 1, characterized in that: The device interaction information also includes the device interactive type; After receiving the first device interaction information sent by the first interactive device, the method further includes: According to the first device interactivity type, a virtual interactive element corresponding to the first target virtual model is displayed in the virtual reality scene, the first device interactivity type is the device interactivity type in the first device interaction information, and the virtual interactive element is used to trigger an interaction instruction corresponding to the first device interactivity type.

5. The method according to any one of claims 1 to 4, characterized in that: The step of determining a first target virtual model in the virtual reality scene of the virtual reality device includes: Determining a user's visual focus in the virtual reality scene; The target virtual model corresponding to the user's visual focus is determined as the first target virtual model.

6. The method according to any one of claims 1 to 4, characterized in that: The displaying the first device interaction information in the virtual reality scene includes: Obtaining a first device type, where the first device type is a device type of the first interactive device; In the virtual reality scene, the first device interaction information is displayed using a display method corresponding to the first device type.

7. The method according to any one of claims 1 to 4, characterized in that: The position of the target virtual model in the virtual reality scene is the same as the position of the interactive device in the real space; Before determining the first target virtual model in the virtual reality scene of the virtual reality device, the method further includes: Acquire a first position, where the first position is a position of the interactive device in a real space; The target virtual model is created in the virtual reality scene according to the first position.

8. The method according to any one of claims 1 to 4, characterized in that: The visual presentation effect of the target virtual model in the virtual reality scene is the same as the visual presentation effect of the interactive device in the real space; Before determining the first target virtual model in the virtual reality scene of the virtual reality device, the method further includes: Obtaining the device type of the interactive device; The virtual model corresponding to the device type of the interactive device is determined as the target virtual model.

9. A device interaction apparatus, characterized in that: A virtual reality device used in a remote virtual-reality interconnection system, wherein the remote virtual-reality interconnection system includes the virtual reality device and an interactive device in a real space, and a remote communication connection is established between the virtual reality device and the interactive device; the device includes: A model determination module, configured to determine a first target virtual model in a virtual reality scene of the virtual reality device, wherein a target virtual model is displayed in the virtual reality scene, the target virtual model is a virtual model corresponding to the interactive device, and the first target virtual model is a target virtual model located at the focus of the user's line of sight; a sending module, configured to send an information reporting instruction to a first interactive device corresponding to the first target virtual model, wherein the information reporting instruction is used to instruct the first interactive device to report device interaction information of the first interactive device, wherein the device interaction information includes a current device state; A receiving module, configured to receive first device interaction information sent by the first interactive device, where the first device interaction information is device interaction information of the first interactive device; A display module is used to display the first device interaction information in the virtual reality scene.

10. A virtual reality device, characterized in that: The device comprises a memory, a processor and a display panel, wherein the memory and the display panel are connected to the processor, the processor is used to execute one or more computer programs stored in the memory, and the display panel is used to display a virtual reality scene. When the processor executes the one or more computer programs, the virtual reality device implements the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 8.

Images