Method, apparatus, electronic device and storage medium for virtual content interaction

Abstract

This application belongs to the field of image processing technology and discloses a method, apparatus, electronic device, and storage medium for virtual content interaction. The method includes: displaying a first original image using a display device; capturing an image of the surrounding environment of the display device to obtain an environmental image; identifying the environmental image to obtain a first image indication position of a target object in the surrounding environment relative to the first original image; and sending a first image rendering instruction containing the first image indication position to at least one virtual content display device, causing the virtual content display device to render and display the first image indication position of the first original image based on the first image rendering instruction. In this way, the virtual content position pointed to by the user can be rendered and presented to other users to guide other users to focus on the rendered position in the virtual content, thereby realizing user interaction with the virtual content.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to methods, apparatuses, electronic devices, and storage media for virtual content interaction. Background Technology

[0002] Virtual Reality (VR) head-mounted displays (such as smart glasses) and Augmented Reality (AR) head-mounted displays use VR or AR technology to block out a person's vision and hearing from the outside world, guiding the user to feel as if they are in a virtual environment. The display principle is that the left and right screens display images for the left and right eyes respectively; the human eye receives this differentiated information and then generates a sense of depth in the brain.

[0003] In some application scenarios (such as teaching, meetings, surgeries, and demonstrations), listeners (such as students) typically wear VR headsets to listen to the speaker (such as a teacher) explain the images displayed on the VR headset.

[0004] However, during the presentation, the speaker can only guide the listener to a specific location in the image through voice. The listener usually finds it difficult to locate the corresponding location in the image according to the speaker's location instructions. In other words, it is difficult to present the speaker's interaction with the virtual content (i.e., the image mentioned above) to the listener (e.g., pointing to a location in the image with a finger). Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, electronic device, and storage medium for virtual content interaction, so as to solve the problem that users have difficulty interacting with virtual content in application scenarios.

[0006] On the one hand, a method for virtual content interaction is provided, including:

[0007] The first target device uses a display device to display the first original image;

[0008] The surrounding environment of the display device is photographed to obtain environmental images;

[0009] The environmental image is identified to obtain the first image indication position of the target indication object in the surrounding environment relative to the first original image. The first image indication position is the position of the target element in the first original image pointed to by the target indication object in the first original image.

[0010] Send a first image rendering instruction containing a first image indication position to at least one virtual content display device, such that the virtual content display device renders and displays the first image indication position of the first original image based on the first image rendering instruction.

[0011] In the above implementation process, the virtual content location (i.e., the first original image displayed by the display device) pointed to by the user through the target indicator object (e.g., a finger) can be rendered and presented to other users to guide other users of VR head-mounted display devices to pay attention to the rendered location in the virtual content, thereby realizing the interaction between the user and the virtual content.

[0012] In one embodiment, the target indication object includes at least one of the following: a user's limb and an indication device;

[0013] The first image rendering instruction also includes at least one of the following rendering methods: resolution adjustment, highlighting, framed display, pointing icon display, and overlaying virtual images;

[0014] The first original image in the virtual content display device is obtained from the image distribution device or the first target device.

[0015] In one embodiment, before displaying the first original image using a display device, the method further includes:

[0016] Receive the first raw image sent by the image distribution device;

[0017] Retrieve the first raw image stored locally;

[0018] Alternatively, an image can be acquired using a first image acquisition device to obtain a first original image.

[0019] In the above implementation process, the first original image can be obtained in a variety of ways.

[0020] In one embodiment, capturing images of the surrounding environment of the display device to obtain an environmental image includes:

[0021] When the image acquisition triggering conditions are met, a second image acquisition device is used to capture images of the surrounding environment and obtain environmental images.

[0022] During the above implementation process, environmental images can be acquired according to application requirements.

[0023] In one embodiment, the method further includes:

[0024] According to the rendering method in the first image rendering instruction, the rendering operation is performed at the first image instruction position of the first original image;

[0025] Displays the first original image after rendering.

[0026] During the above implementation process, local rendering and display can also be performed.

[0027] In one embodiment, the method further includes:

[0028] Receive a second image rendering instruction for the second original image sent by the second target device;

[0029] Obtain the position of the second image indicator contained in the second image rendering indicator;

[0030] According to the rendering method in the second image rendering instruction, the second original image is rendered at the second image instruction position in the second original image, and the second original image is obtained from the second target device, the image distribution device, or locally;

[0031] Displays the rendered second original image.

[0032] In the above implementation process, image rendering and display can also be performed according to the rendering instructions of other devices.

[0033] On the one hand, a method for virtual content interaction is provided, applied to a virtual content display device, including:

[0034] Receive a first image rendering instruction sent by a first target device, which includes a first image indication position, wherein the first image indication position is the position of the target element in the first original image presented in the display device, which is pointed to by the target indication object in the surrounding environment of the display device;

[0035] Based on the first image rendering instruction, render the first image at the first image instruction position of the first original image;

[0036] Displays the first original image after rendering.

[0037] On the one hand, the target of the indication includes at least one of the following: the user's limbs and the indicating device;

[0038] The first image rendering instruction also includes at least one of the following rendering methods: resolution adjustment, highlighting, framed display, pointing icon display, and overlaying virtual images;

[0039] The first original image in the virtual content display device is obtained from the image distribution device or the first target device.

[0040] On the one hand, a device for virtual content interaction is provided, comprising:

[0041] A display unit is configured to display a first original image via a display device using a first target device;

[0042] The imaging unit is used to capture images of the surrounding environment of the display device to obtain environmental images;

[0043] The recognition unit is used to recognize the environmental image and obtain the first image indication position of the target indication object in the surrounding environment relative to the first original image. The first image indication position is the position of the target element in the first original image pointed to by the target indication object in the first original image.

[0044] A rendering unit is configured to send a first image rendering instruction containing a first image indication position to at least one virtual content display device, such that the virtual content display device renders and displays the first image indication position of the first original image based on the first image rendering instruction.

[0045] In one embodiment, the target indication object includes at least one of the following: a user's limb and an indication device;

[0046] The first image rendering instruction also includes at least one of the following rendering methods: resolution adjustment, highlighting, framed display, pointing icon display, and overlaying virtual images;

[0047] The first original image in the virtual content display device is obtained from the image distribution device or the first target device.

[0048] In one embodiment, the display unit further includes:

[0049] Receive the first raw image sent by the image distribution device;

[0050] Retrieve the first raw image stored locally;

[0051] Alternatively, an image can be acquired using a first image acquisition device to obtain a first original image.

[0052] In one embodiment, the imaging unit is used for:

[0053] When the image acquisition triggering conditions are met, a second image acquisition device is used to capture images of the surrounding environment and obtain environmental images.

[0054] In one implementation, the rendering unit is further configured to:

[0055] According to the rendering method in the first image rendering instruction, the rendering operation is performed at the first image instruction position of the first original image;

[0056] Displays the first original image after rendering.

[0057] In one implementation, the rendering unit is further configured to:

[0058] Receive a second image rendering instruction for the second original image sent by the second target device;

[0059] Obtain the position of the second image indicator contained in the second image rendering indicator;

[0060] According to the rendering method in the second image rendering instruction, the second original image is rendered at the second image instruction position in the second original image, and the second original image is obtained from the second target device, the image distribution device, or locally;

[0061] Displays the rendered second original image.

[0062] On the one hand, a device for virtual content interaction is provided, comprising:

[0063] The receiving unit is configured to receive a first image rendering instruction sent by the first target device, which includes a first image indication position. The first image indication position is the position of the target element in the first original image presented in the display device, which is pointed to by the target indication object in the surrounding environment of the display device.

[0064] A rendering unit is used to render a first image at a first image indication position based on a first image rendering indication;

[0065] The display unit is used to display the first original image after rendering.

[0066] In one embodiment, the target indication object includes at least one of the following: a user's limb and an indication device;

[0067] The first image rendering instruction also includes at least one of the following rendering methods: resolution adjustment, highlighting, framed display, pointing icon display, and overlaying virtual images;

[0068] The first original image is obtained from either the image distribution device or the first target device.

[0069] On one hand, an electronic device is provided, including a processor and a memory, the memory storing computer-readable instructions that, when executed by the processor, perform the steps of the method provided in any of the above-described alternative implementations of virtual content interaction.

[0070] On the one hand, a computer-readable storage medium is provided on which a computer program is stored, which, when executed by a processor, performs the steps of the methods provided in various alternative implementations of any of the aforementioned virtual content interaction methods.

[0071] On the one hand, a computer program product is provided that, when run on a computer, causes the computer to perform the steps of the methods provided in various alternative implementations of any of the aforementioned virtual content interaction methods.

[0072] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0073] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1 This application provides an architectural diagram of a virtual content interaction system.

[0075] Figure 2 A flowchart illustrating a method for virtual content interaction provided in an embodiment of this application;

[0076] Figure 3 A flowchart of a method for virtual content interaction in a first application scenario provided in this application embodiment;

[0077] Figure 4 A flowchart of a method for virtual content interaction in a second application scenario provided in this application embodiment;

[0078] Figure 5 A flowchart of a method for virtual content interaction in a third application scenario provided in this application embodiment;

[0079] Figure 6 A structural block diagram of a virtual content interaction device provided in an embodiment of this application;

[0080] Figure 7 A structural block diagram of another virtual content interaction device provided in an embodiment of this application;

[0081] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0082] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0083] First, some of the terms used in the embodiments of this application will be explained to facilitate understanding by those skilled in the art.

[0084] Terminal devices can be mobile terminals, fixed terminals, or portable terminals, such as mobile phones, sites, units, devices, multimedia computers, multimedia tablets, internet nodes, communicators, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, personal communication system devices, personal navigation devices, personal digital assistants, audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. It is also foreseeable that terminal devices can support any type of user-facing interface (e.g., wearable devices).

[0085] Servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0086] To address the problem of users having difficulty interacting with virtual content in VR or AR application scenarios, embodiments of this application provide methods, devices, electronic devices, and storage media for virtual content interaction.

[0087] See Figure 1 The diagram shown is an architectural schematic of a virtual content interaction system provided in an embodiment of this application. The virtual content interaction system includes a first target device (e.g., a VR / AR head-mounted display device for a speaker) and a virtual content display device (e.g., a VR / AR head-mounted display device for a listener), and may also include a second target device (e.g., a VR / AR head-mounted display device for another speaker) and an image distribution device (e.g., the cloud).

[0088] The virtual content display device is either a VR head-mounted display device or an AR head-mounted display device. The first target device and the second target device are electronic devices. These electronic devices can be servers or terminal devices. Optionally, the first target device and the second target device can also be VR head-mounted display devices or AR head-mounted display devices.

[0089] In one embodiment, the first target device is used to identify the specific image position (i.e., the first image indication position) in the first original image displayed on the display device pointed to by the user through a target indication object (e.g., a finger, a pointer, or an indicator rod), and to send a first image rendering instruction containing the first image indication position and a rendering method (the rendering method can be the default or the user-indicated) to the virtual content display device.

[0090] Furthermore, the first target device can also render and display the first original image locally according to the position indicated by the first image and the rendering method described above.

[0091] Furthermore, the first target device can also render and display a local or received second original image based on image rendering instructions sent by other devices (such as the second target device). The functions of the second target device and the first target device can be the same, and will not be elaborated here.

[0092] Virtual content display device: used to render and display an image at a first image indicator position in a first original image according to the received rendering method.

[0093] Optionally, the first original image and the second original image can be the same or different. If the first original image and the second original image are the same image, the same position or different positions of the same original image can be rendered separately according to the image rendering instructions sent by multiple devices (e.g., the first image rendering instruction and the second image rendering instruction).

[0094] It should be noted that the second target device and the virtual content display device can be the same device or different devices; this is not limited here. The first target device, the second target device, and the virtual content display device can all send image rendering instructions containing image location and rendering mode to other devices, or they can perform image rendering and display at the image location indicated in the original image according to the received rendering mode. See also... Figure 2 The diagram shown is a flowchart of a virtual content interaction method provided in an embodiment of this application, applied to a first target device, combined with... Figure 2 right Figure 1 The method for virtual content interaction in the virtual content interaction system is described below, and the specific implementation process of the method is as follows:

[0095] Step 200: Display the first original image using a display device.

[0096] In one embodiment, the implementation process of step 200 further includes, but is not limited to, at least one of the following methods for acquiring the first original image:

[0097] Method 1: Receive the first raw image sent by an image distribution device (e.g., the cloud).

[0098] Method 2: Obtain the first raw image stored locally on the first target device.

[0099] Method 3: Acquire the first original image by using a first image acquisition device (e.g., a camera device).

[0100] The first target device may also include a first image acquisition device. The first image acquisition device may also be a different device from the first target device. The display device may be a display screen, a projection device, etc.

[0101] It should be noted that the image (e.g., the first original image) can be a single picture or a document that can be displayed as an image, or a video frame in a video, and can be displayed on the first target device and multiple virtual display devices.

[0102] Step 201: Take pictures of the surrounding environment of the display device to obtain environmental images.

[0103] In one embodiment, when the image acquisition trigger condition is determined to be met, a second image acquisition device is used to capture images of the surrounding environment to obtain an environmental image. Optionally, the environmental image may include a first original image displayed on the display device, or it may not include the first original image.

[0104] As an example, a second image acquisition device is used to capture images of the surrounding environment in real time or periodically (e.g., at 1-second intervals) to obtain environmental images.

[0105] Optionally, the first target device may also include a second image acquisition device. The second image acquisition device may be a different device from the first target device. The first image acquisition device and the second image acquisition device may be the same device or different devices; this is not limited here.

[0106] Step 202: Recognize the environmental image to obtain the first image indication position of the target indication object in the surrounding environment relative to the first original image.

[0107] In one embodiment, the first image indication location is the position of the target element in the first original image pointed to by the target indication object (e.g., the coordinates of an element in the first original image). Furthermore, the first image indication location may include spatial information in addition to positional information.

[0108] Optionally, the target indication object is an object used to indicate any location in the first original image, and may include, but is not limited to, at least one of the following: a user's limb (e.g., a user's finger) and an indication device. The indication device may be a pointer or an indicator rod.

[0109] In practical applications, the target indication object can be set according to the actual application scenario, and there are no restrictions here.

[0110] As an example, a second image acquisition device (e.g., a second camera) is positioned 2 meters directly in front of the display screen (or projected image). The speaker's pointer is positioned between the display screen and the second image acquisition device. The speaker points to the center of the image displayed on the screen using the pointer. The second image acquisition device captures an environmental image of the display screen, including the speaker's pointer and the first original image displayed on the screen. The environmental image is then used to identify the pointer, determining the position where the pointer contacts the first original image (i.e., the first image pointing position).

[0111] In this way, the specific location in the first original image can be obtained by the user pointing with their finger or using a pointer, thus enabling interaction between the real user and the virtual content.

[0112] Step 203: Send a first image rendering instruction containing a first image indication position to at least one virtual content display device, so that the virtual content display device renders and displays the first image indication position of the first original image based on the first image rendering instruction.

[0113] In one embodiment, a first image rendering instruction, comprising a first image indication position and a rendering mode, is sent to at least one virtual content display device. The virtual content display device obtains the rendering mode in the first image rendering instruction, and performs a rendering operation at the first image indication position of the first original image according to the rendering mode, and displays the rendered first original image.

[0114] The first image rendering instruction also includes at least one of the following rendering methods: resolution adjustment, highlighting, framed display, pointing icon display, and overlaying virtual images (e.g., virtual animated characters); the first original image in the virtual content display device is obtained from the image distribution device or the first target device.

[0115] As an example, the virtual content display device receives a first original image sent by an image distribution device (e.g., the cloud), or receives a first original image sent by a first target device, or obtains a first original image that exists locally, and performs a rendering operation at the first image indication position of the first original image according to the rendering method in the first image rendering instruction, and displays the rendered first original image.

[0116] As another example, the virtual content display device acquires a first original image contained in the first image rendering instruction, and performs a rendering operation at the first image instruction position of the first original image according to the rendering method in the first image rendering instruction, and displays the rendered first original image.

[0117] Furthermore, the first target device can also render and display its own first original image.

[0118] In one embodiment, the first target device performs a rendering operation at the first image indication position of the first original image according to the rendering method in the first image rendering instruction, and displays the rendered first original image.

[0119] Furthermore, the first target device can also render and display images according to rendering instructions from other devices.

[0120] In one embodiment, a second image rendering instruction for a second original image is received from a second target device; a second image instruction position contained in the second image rendering instruction is obtained; rendering is performed at the second image instruction position in the second original image according to the rendering method in the second image rendering instruction; and the rendered second original image is displayed. The second original image is obtained from the second target device, an image distribution device, or locally.

[0121] The specific steps for the second target device to send a second image rendering instruction for the second original image are described in steps 200-203 above, and will not be repeated here.

[0122] The following section will discuss several application scenarios. Figure 2 The method of virtual content interaction is illustrated with an example.

[0123] In the first application scenario, the virtual content interaction system includes a first target device, a virtual content display device, and a cloud (i.e., an image distribution device). The first target device includes a second image acquisition device. (See also...) Figure 3 The diagram shows a flowchart of a method for virtual content interaction in the first application scenario. The specific implementation process of this method is as follows:

[0124] Step 300: The first target device receives the first raw image from the cloud and displays the first raw image on the screen.

[0125] Step 301: The first target device periodically captures images of the surrounding environment of the display screen through the second image acquisition device to obtain environmental images.

[0126] Optionally, the surrounding environment of the user on the display screen or the first target device can be captured. The second image acquisition device can be fixed or automatically moving; for example, visual tracking technology can be used to track and capture the target object.

[0127] It should be noted that when the first target device is a VR / AR head-mounted display device, the second image acquisition device may not be able to acquire the first original image on the display screen of the VR / AR head-mounted display device. If the first original image is displayed as a virtual image, the second image acquisition device will also be unable to acquire the virtual image. Therefore, in this embodiment, the surrounding environment is acquired so that in subsequent steps, the relative position and direction between the target pointing object and the first original image can be identified to determine where the target pointing object points to the first original image.

[0128] Step 302: The first target device identifies the environmental image and obtains the position of the first original image on the display screen that the user points to with the pointer, thus obtaining the position indicated by the first image.

[0129] Optionally, the first image indication position may also include the orientation of the first original image on the display screen to which the indicator is pointing.

[0130] Furthermore, if the recognition result determines that the environmental image does not contain the aforementioned indicator or that the aforementioned indicator does not point to the first original image on the display screen, then the environmental image is discarded and step 300 is executed.

[0131] Step 303: The first target device sends a first image rendering instruction containing the first image indication position and rendering method to the virtual content display device.

[0132] As an example, the virtual content display device is a VR headset, consisting of two eye screens (i.e., displays). The rendering method can be user-instructed or default.

[0133] Step 304: The virtual content display device receives the first raw image from the cloud and the first image rendering instruction from the first target device.

[0134] Step 305: The virtual content display device renders and displays the first image indicated position in the first original image according to the rendering method in the first image rendering instruction.

[0135] Specifically, the virtual content display device displays the first original image and, according to the rendering method in the first image rendering instruction (e.g., highlighting), highlights the unit image area (e.g., a 10x10 pixel area) where the first image instruction position of the first original image is located.

[0136] This approach transmits the first image rendering instruction and the first original image independently, rather than rendering the image first and then transmitting the rendered image. Firstly, since the first image rendering instruction typically involves less data (e.g., position signals, rendering method), and transmitting the rendered image after rendering it would increase the data volume, this method reduces the amount of data transmitted, especially for wireless transmission, and also improves transmission reliability. Secondly, because the first image rendering instruction involves less data, the latency is very low. Furthermore, since the first original image can be transmitted before the first image rendering instruction, the virtual content display device can display the first original image simultaneously with the first target device, reducing the latency of the virtual content display device displaying the first original image. Moreover, the virtual content display device can personalize its selection of various rendering methods and whether to use any rendering method according to the user's actual needs, improving the user experience.

[0137] The virtual content display device and the first target device can receive the first original image sent from the cloud via wired or wireless means. The wireless means can be cellular communication, WiFi wireless network communication technology, or Ultra Wide Band (UWB).

[0138] Furthermore, the first target device can also acquire and display the first original image stored locally, and send the first original image to the virtual content display device.

[0139] In the second application scenario, the virtual content interaction system includes a first target device, a virtual content display device, and a cloud (i.e., an image distribution device). The first target device includes a first image acquisition device and a second image acquisition device. (See also...) Figure 4 The diagram shows a flowchart of a virtual content interaction method for the second application scenario. The specific implementation process of this method is as follows:

[0140] Step 400: The first target device acquires an image through the first image acquisition device to obtain the first original image.

[0141] Optionally, the first target device may be equipped with a first image acquisition device and a second image acquisition device.

[0142] The first image acquisition device is used to acquire a first original image. The second image acquisition device is used to acquire an environmental image including the first original image displayed on the screen, so as to identify target objects around the screen in subsequent steps and indicate the position of the first image in the first original image displayed on the screen. The first image acquisition device and the second image acquisition device can be the same device or different devices.

[0143] Step 401: The first target device displays the first original image on the display screen.

[0144] Step 402: The first target device periodically captures images of the surrounding environment of the display screen using the second image acquisition device to obtain an environmental image containing the first original image displayed on the display screen.

[0145] Step 403: The first target device recognizes the environmental image and obtains the position of the first original image on the display screen indicated by the user through the pointer, thus obtaining the position indicated by the first image.

[0146] Step 404: The first target device sends a first image rendering instruction to the virtual content display device, which includes the first image indication position, rendering method, and the first original image.

[0147] Step 405: The virtual content display device renders and displays the first image indicated position in the received first original image according to the rendering method in the first image rendering instruction.

[0148] In the third application scenario, the virtual content interaction system includes a first target device and multiple virtual content display devices. The first target device includes a first image acquisition device, a second image acquisition device, and a display screen. The first target device is a VR head-mounted display device or an AR head-mounted display device.

[0149] See Figure 5 The diagram shows a flowchart of a method for virtual content interaction in the third application scenario. The specific implementation process of this method is as follows:

[0150] Step 500: The first target device acquires an image through the first image acquisition device, obtains a first original image, and displays the first original image on the display screen.

[0151] Furthermore, the first target device can also receive the first raw image from the cloud.

[0152] Step 501: The first target device uses the second image acquisition device to capture the surrounding environment of the user of the first target device in real time, and obtains an environmental image including the first original image displayed on the screen.

[0153] Step 502: The first target device recognizes the environmental image, obtains the position of the first original image on the display screen pointed to by the user's finger, and obtains the position indicated by the first image.

[0154] Step 503: The first target device renders and displays the first image indicated position in the first original image according to the rendering method instructed by the user.

[0155] Step 504: The first target device sends the rendered first original image to each virtual content display device.

[0156] Step 505: Each virtual content display device displays the received rendered first original image.

[0157] As an example, in a teaching scenario, both the teacher and students wear VR / AR head-mounted displays (e.g., smart glasses) and acquire video (i.e., the first raw image) from the cloud. When the teacher points to a specific area of ​​the video playing on the screen, the teacher's VR / AR head-mounted display captures an image of the environment. A recognition module then identifies the user's finger in the environmental image to determine the location of the video frame the teacher is pointing to (i.e., the first image indication location). The teacher's VR / AR head-mounted display shares this first image rendering indication, containing the location and rendering method of the video frame, with each student's VR / AR head-mounted display. Each student's VR / AR head-mounted display then renders the video frame based on the first image rendering indication, adding frames and other rendering techniques to the indicated location, and displays the rendered video frame, allowing students to focus more intently on the rendered area of ​​the video frame.

[0158] As another example, in a conference scenario (which could also include teaching, surgery, and demonstration scenarios), all participants wear VR / AR head-mounted displays. The conference speaker (who could also be a lecturer, a surgeon, or an instrument repairman) uses the first camera in their VR / AR head-mounted display to capture images (the camera could capture images of the lecturer's workbench, the surgeon's procedure, the repairman's work, or the product presenter's product, etc.) to obtain a first raw image, which is then shared with the VR / AR head-mounted displays of other participants. All participants' VR / AR head-mounted displays display the first raw image. When the conference speaker points to an area of ​​the first raw image using a pointer, their VR / AR head-mounted display uses a second camera to capture the scene where the pointer is pointing to that area, obtaining an environmental image. The speaker then identifies the pointer within the environmental image, determines the location indicated by the first image, and sends a first image rendering instruction containing that location to the VR / AR head-mounted displays of other participants. The VR / AR head-mounted display devices of other participants render the first original image by adding a frame to the first image indicator position based on the first image rendering instruction, and display the rendered first original image to guide other participants to focus on the rendered area.

[0159] In this embodiment of the application, in a VR / AR application scenario, users can watch the same image or video together through a VR / AR head-mounted display device. The device can identify and render any position in the image pointed to by a user, allowing each user to view the rendered image through their VR / AR head-mounted display device. This guides users to focus on a specific position in the image, thus realizing the interaction between users and virtual content in a VR / AR application scenario.

[0160] Based on the same inventive concept, this application also provides a device for virtual content interaction. Since the principle of the above-mentioned device and equipment in solving the problem is similar to that of a method for virtual content interaction, the implementation of the above-mentioned device can refer to the implementation of the method, and the repeated parts will not be described again.

[0161] like Figure 6 The diagram shown is a structural schematic of a virtual content interaction device provided in an embodiment of this application, comprising:

[0162] Display unit 601 is used to display a first original image via a display device through a first target device;

[0163] The imaging unit 602 is used to capture images of the surrounding environment of the display device to obtain environmental images;

[0164] The recognition unit 603 is used to recognize the environmental image and obtain the first image indication position of the target indication object in the surrounding environment relative to the first original image. The first image indication position is the position of the target element in the first original image pointed to by the target indication object in the first original image.

[0165] The rendering unit 604 is configured to send a first image rendering instruction containing a first image indication position to at least one virtual content display device, so that the virtual content display device renders and displays the first image indication position of the first original image based on the first image rendering instruction.

[0166] In one embodiment, the target indication object includes at least one of the following: a user's limb and an indication device;

[0167] The first image rendering instruction also includes at least one of the following rendering methods: resolution adjustment, highlighting, framed display, pointing icon display, and overlaying virtual images;

[0168] The first original image in the virtual content display device is obtained from the image distribution device or the first target device.

[0169] In one embodiment, the display unit 601 further includes:

[0170] Receive the first raw image sent by the image distribution device;

[0171] Retrieve the first raw image stored locally;

[0172] Alternatively, an image can be acquired using a first image acquisition device to obtain a first original image.

[0173] In one embodiment, the imaging unit 602 is used for:

[0174] When the image acquisition triggering conditions are met, a second image acquisition device is used to capture images of the surrounding environment and obtain environmental images.

[0175] In one embodiment, the rendering unit 604 is further configured to:

[0176] According to the rendering method in the first image rendering instruction, the rendering operation is performed at the first image instruction position of the first original image;

[0177] Displays the first original image after rendering.

[0178] In one embodiment, the rendering unit 604 is further configured to:

[0179] Receive a second image rendering instruction for the second original image sent by the second target device;

[0180] Obtain the position of the second image indicator contained in the second image rendering indicator;

[0181] According to the rendering method in the second image rendering instruction, the second original image is rendered at the second image instruction position in the second original image, and the second original image is obtained from the second target device, the image distribution device, or locally;

[0182] Displays the rendered second original image.

[0183] like Figure 7 The diagram shown is a structural schematic of another virtual content interaction device provided in this application embodiment, comprising:

[0184] The receiving unit 701 is used to receive a first image rendering instruction sent by the first target device, which includes a first image indication position. The first image indication position is the position of the target element in the first original image presented in the display device, which is pointed to by the target indication object in the surrounding environment of the display device.

[0185] Rendering unit 702 is used to render a first image at a first image indication position based on a first image rendering indication;

[0186] Display unit 703 is used to display the first original image after rendering.

[0187] In one embodiment, the target indication object includes at least one of the following: a user's limb and an indication device;

[0188] The first image rendering instruction also includes at least one of the following rendering methods: resolution adjustment, highlighting, framed display, pointing icon display, and overlaying virtual images;

[0189] The first original image is obtained from either the image distribution device or the first target device.

[0190] The virtual content interaction method, apparatus, electronic device, and storage medium provided in this application embodiment employ a display device to display a first original image; captures an image of the surrounding environment of the display device; identifies the environmental image to obtain a first image indication position of a target object in the surrounding environment relative to the first original image, where the first image indication position is the position of a target element in the first original image pointed to by the target object; and sends a first image rendering instruction containing the first image indication position to at least one virtual content display device, causing the virtual content display device to render and display the first image indication position of the first original image based on the first image rendering instruction. The virtual content display device is a virtual reality head-mounted display device or an augmented reality head-mounted display device. In this way, the virtual content position (i.e., the first original image displayed by the display device) pointed to by a user through a target object (e.g., a finger) can be rendered and presented to other users, guiding other users of VR head-mounted display devices to focus on the rendered position in the virtual content, thus realizing user interaction with the virtual content.

[0191] Figure 8 A schematic diagram of the structure of an electronic device 8000 is shown. (See attached diagram.) Figure 8 As shown, the electronic device 8000 includes a processor 8010 and a memory 8020, and optionally may also include a power supply 8030, a display unit 8040, and an input unit 8050.

[0192] The processor 8010 is the control center of the electronic device 8000. It connects various components through various interfaces and lines, and performs various functions of the electronic device 8000 by running or executing software programs and / or data stored in the memory 8020, thereby performing overall monitoring of the electronic device 8000.

[0193] In this embodiment, when the processor 8010 calls the computer program stored in the memory 8020, it executes the steps in the above embodiments.

[0194] Optionally, the processor 8010 may include one or more processing units; preferably, the processor 8010 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 8010. In some embodiments, the processor and memory may be implemented on a single chip; in some embodiments, they may also be implemented separately on independent chips.

[0195] The memory 8020 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, various applications, etc.; the data storage area may store data created based on the use of the electronic device 8000, etc. In addition, the memory 8020 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0196] The electronic device 8000 also includes a power supply 8030 (such as a battery) that supplies power to various components. The power supply can be logically connected to the processor 8010 through a power management system, thereby enabling the management of charging, discharging, and power consumption.

[0197] The display unit 8040 can be used to display information input by the user or information provided to the user, as well as various menus of the electronic device 8000. In this embodiment of the invention, it is mainly used to display the display interfaces of various applications in the electronic device 8000, as well as text, images, and other objects displayed on the display interfaces. The display unit 8040 may include a display panel 8041. The display panel 8041 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0198] The input unit 8050 can be used to receive information such as numbers or characters input by the user. The input unit 8050 may include a touch panel 8051 and other input devices 8052. The touch panel 8051, also known as a touch screen, can collect touch operations on or near the user (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 8051).

[0199] Specifically, the touch panel 8051 can detect user touch operations and the signals generated by these operations, convert them into touch point coordinates, send them to the processor 8010, and receive and execute commands from the processor 8010. Furthermore, the touch panel 8051 can be implemented using various types of sensors, including resistive, capacitive, infrared, and surface acoustic wave sensors. Other input devices 8052 can include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0200] Of course, the touch panel 8051 can cover the display panel 8041. When the touch panel 8051 detects a touch operation on or near it, it transmits the information to the processor 8010 to determine the type of touch event. Subsequently, the processor 8010 provides corresponding visual output on the display panel 8041 based on the type of touch event. Although in Figure 8 In this embodiment, the touch panel 8051 and the display panel 8041 are two separate components to realize the input and output functions of the electronic device 8000. However, in some embodiments, the touch panel 8051 and the display panel 8041 can be integrated to realize the input and output functions of the electronic device 8000.

[0201] The electronic device 8000 may also include one or more sensors, such as a pressure sensor, a gravity acceleration sensor, a proximity sensor, etc. Of course, depending on the specific application, the electronic device 8000 may also include other components such as a camera. Since these components are not the focus of this application's embodiments, therefore... Figure 8 It is not shown in the text and will not be described in detail here.

[0202] Those skilled in the art will understand that Figure 8 This is merely an example of an electronic device and does not constitute a limitation on the electronic device. It may include more or fewer components than shown, or a combination of certain components, or different components.

[0203] In this embodiment of the application, a computer-readable storage medium stores a computer program thereon. When the computer program is executed by a processor, it enables a communication device to perform the steps in the above embodiments.

[0204] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0205] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0206] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0207] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0208] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0209] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0210] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method of virtual content interaction, characterized by, include: The first target device uses a display device to display the first original image; The surrounding environment of the display device is photographed to obtain an environmental image; The environmental image is identified to obtain the first image indication position of the target indication object in the surrounding environment relative to the first original image. The first image indication position is the position of the target element in the first original image pointed to by the target indication object in the first original image. Send a first image rendering instruction containing the first image indication position to at least one virtual content display device, so that the virtual content display device renders and displays the first image indication position of the first original image based on the first image rendering instruction; The first image rendering instruction further includes at least one of the following rendering methods: resolution adjustment, highlighting, framed display, pointing icon display, and overlaying virtual images; The method further includes: The first target device receives a second image rendering instruction for the second original image sent by the second target device; Obtain the position of the second image indicator contained in the second image rendering indicator; According to the rendering method in the second image rendering instruction, the second original image is rendered at the second image instruction position in the second original image, and the second original image is obtained from the second target device, the image distribution device, or locally; The first target device displays the rendered second original image.

2. The method of claim 1, wherein, The target indication object includes at least one of the following: a user's limbs and an indication device; The first original image in the virtual content display device is obtained from the image distribution device or the first target device.

3. The method as described in claim 1, characterized in that, Before displaying the first original image using a display device, the method further includes: Receive the first original image sent by the image distribution device; Retrieve the first original image stored locally; Alternatively, the first original image can be obtained by acquiring an image using a first image acquisition device.

4. The method as described in claim 1, characterized in that, The step of capturing images of the surrounding environment of the display device to obtain environmental images includes: When the image acquisition triggering conditions are met, a second image acquisition device is used to capture images of the surrounding environment to obtain an environmental image.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: According to the rendering method in the first image rendering instruction, the rendering operation is performed at the first image instruction position of the first original image; Displays the first original image after rendering.

6. A device for virtual content interaction, characterized in that, include: A display unit is configured to display a first original image via a display device using a first target device; The imaging unit is used to capture images of the surrounding environment of the display device to obtain environmental images; The recognition unit is used to recognize the environmental image and obtain the first image indication position of the target indication object in the surrounding environment relative to the first original image. The first image indication position is the position of the target element in the first original image pointed to by the target indication object in the first original image. A rendering unit is configured to send a first image rendering instruction containing the first image indication position to at least one virtual content display device, such that the virtual content display device renders and displays the first image indication position of the first original image based on the first image rendering instruction; The first image rendering instruction further includes at least one of the following rendering methods: resolution adjustment, highlighting, framed display, pointing icon display, and overlaying virtual images; The rendering unit is further used for: Receive a second image rendering instruction for the second original image sent by the second target device; Obtain the position of the second image indicator contained in the second image rendering indicator; According to the rendering method in the second image rendering instruction, the second original image is rendered at the second image instruction position in the second original image, and the second original image is obtained from the second target device, the image distribution device, or locally; Displays the rendered second original image.

7. The apparatus as claimed in claim 6, characterized in that, The target indication object includes at least one of the following: a user's limbs and an indication device; The first original image in the virtual content display device is obtained from the image distribution device or the first target device.

8. The apparatus as claimed in claim 6, characterized in that, The display unit also includes: Receive the first original image sent by the image distribution device; Retrieve the first original image stored locally; Alternatively, the first original image can be obtained by acquiring an image using a first image acquisition device.

9. The apparatus as claimed in claim 6, characterized in that, The imaging unit is used for: When the image acquisition triggering conditions are met, a second image acquisition device is used to capture images of the surrounding environment to obtain an environmental image.

10. The apparatus according to any one of claims 6-9, characterized in that, The rendering unit is also used for: According to the rendering method in the first image rendering instruction, the rendering operation is performed at the first image instruction position of the first original image; Displays the first original image after rendering.

11. An electronic device, characterized in that, It includes a processor and a memory, the memory storing computer-readable instructions that, when executed by the processor, perform the method as described in any one of claims 1-5.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it performs the method as described in any one of claims 1-5.

Images