Extended reality-based control method and apparatus, electronic device, and storage medium

Abstract

The present disclosure provides an extended reality-based control method, device, electronic device and storage medium. In some embodiments, the present disclosure provides an extended reality-based control method, comprising: displaying a virtual environment and a virtual screen located in the virtual environment in an extended reality space; in response to displaying a playing content in the virtual screen, rendering light rays of the playing content into a rendering area of the virtual environment to present a reflection of the playing content in the rendering area of the virtual environment. The reflection effect when playing through a screen in the real world is simulated in the embodiments of the present disclosure, improving the experience of the user.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a control method, apparatus, electronic device and storage medium based on virtual display. Background Technology

[0002] Extended reality technology includes virtual reality, augmented reality, and mixed reality. Extended reality spaces can display images and videos, but the display quality of these images and videos differs from that in the real world, resulting in a less than ideal user experience. Summary of the Invention

[0003] This disclosure provides a control method, apparatus, electronic device, and storage medium based on extended reality.

[0004] The following technical solution is adopted in this disclosure.

[0005] In some embodiments, this disclosure provides a control method based on extended reality, including:

[0006] Displaying a virtual environment and a virtual screen within the extended reality space;

[0007] In response to displaying playback content on the virtual screen, the light from the playback content is rendered onto the rendering area of ​​the virtual environment to present the reflection of the playback content in the rendering area of ​​the virtual environment.

[0008] In some embodiments, this disclosure provides a control device based on extended reality, including:

[0009] The display unit is used to display a virtual environment and a virtual screen located in the extended reality space;

[0010] A control unit is configured to, in response to displaying playback content on the virtual screen, render the light of the playback content onto a rendering area of ​​the virtual environment to present the reflection of the playback content in the rendering area of ​​the virtual environment.

[0011] In some embodiments, this disclosure provides an electronic device, including: at least one memory and at least one processor;

[0012] The memory is used to store program code, and the processor is used to call the program code stored in the memory to execute the above method.

[0013] In some embodiments, this disclosure provides a computer-readable storage medium for storing program code that, when run by a processor, causes the processor to perform the methods described above.

[0014] This disclosure provides a control method based on extended reality. When displaying playback content on a virtual screen, the light from the playback content is rendered onto the rendering area of ​​the virtual environment. The rendering area displays the reflection of the playback content, thereby simulating the reflection effect when playing content through a screen in the real world and improving the user's realistic experience. Attached Figure Description

[0015] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0016] Figure 1 This is a schematic diagram of an embodiment of the present disclosure using an extended reality device.

[0017] Figure 2 This is a flowchart of a control method based on extended reality according to an embodiment of this disclosure.

[0018] Figure 3 This is a schematic diagram of the display of the virtual environment and virtual screen in an embodiment of this disclosure.

[0019] Figure 4 This is a schematic diagram of the reflection effect when no noise map is used in the embodiments of this disclosure.

[0020] Figure 5 This is a schematic diagram of the reflection effect when using a noise map in an embodiment of this disclosure.

[0021] Figure 6 This is a schematic diagram of the trapezoidal shape of the rendering area in an embodiment of this disclosure.

[0022] Figure 7 This is a schematic diagram of light emitted from a virtual screen in an embodiment of this disclosure.

[0023] Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0024] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0025] It should be understood that the various steps described in the method embodiments of this disclosure can be performed in sequence and / or in parallel. Furthermore, method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0026] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0027] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0028] It should be noted that the use of the word "a" in this disclosure is illustrative rather than restrictive, and those skilled in the art should understand that it should be understood as "one or more" unless otherwise expressly indicated in the context.

[0029] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0030] The solutions provided by the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0031] Extended reality can be at least one of virtual reality, augmented reality, or mixed reality. Taking extended reality as virtual reality as an example, such as... Figure 1 As shown, users can enter a virtual reality space through smart terminal devices such as VR headsets, and control their own virtual avatar to interact with other user-controlled virtual avatars for social interaction, entertainment, learning, remote work, and more.

[0032] The virtual reality space can be a simulation of the real world, a semi-simulated / semi-fictional virtual scene, or a purely fictional virtual scene. The virtual scene can be any of a two-dimensional, 2.5-dimensional, or three-dimensional virtual scene; this application does not limit the dimension of the virtual scene. For example, the virtual scene may include the sky, land, ocean, etc., and the land may include environmental elements such as deserts and cities. Users can control virtual objects to move within the virtual scene.

[0033] In one embodiment, in a virtual reality space, a user can perform related interactive operations by operating a device, such as a controller, for example, by pressing buttons on the controller. Alternatively, in another embodiment, gestures, voice, or multimodal control methods can be used to control the target object in the virtual reality device instead of a controller.

[0034] In some embodiments of this disclosure, the proposed control method can be used for extended reality devices, such as virtual reality devices. Virtual reality devices are terminals that realize virtual reality effects and can typically be provided in the form of glasses, head-mounted displays (HMDs), or contact lenses to realize visual perception and other forms of perception. Of course, the form of virtual reality devices is not limited to these and can be further miniaturized or enlarged as needed.

[0035] The virtual reality devices described in this disclosure can include, but are not limited to, the following types:

[0036] PC-based virtual reality (PCVR) devices utilize a PC for calculations and data output related to virtual reality functions. External PC-based virtual reality devices then use the data output from the PC to achieve the virtual reality effect.

[0037] Mobile virtual reality devices support setting up mobile terminals (such as smartphones) in various ways (such as head-mounted displays with dedicated card slots). Through wired or wireless connections with the mobile terminal, the mobile terminal performs calculations related to virtual reality functions and outputs data to the mobile virtual reality device, such as watching virtual reality videos through a mobile terminal's app.

[0038] All-in-one virtual reality devices have a processor for performing virtual functions, thus possessing independent virtual reality input and output capabilities. They do not require connection to a PC or mobile terminal, offering a high degree of freedom of use.

[0039] Virtual reality devices are capable of displaying virtual reality images in virtual reality space. The underlying operating systems of virtual reality devices are often Android, iOS, etc.

[0040] like Figure 2 As shown, Figure 2 This is a flowchart of a control method based on extended reality according to an embodiment of the present disclosure, which includes the following steps.

[0041] S11. Display the virtual environment and the virtual screen located in the virtual environment in the extended reality space.

[0042] In some embodiments of this disclosure, the extended reality-based control method can be used in extended reality devices. The extended reality space can be a purely virtual space or a hybrid space of virtual and reality. A virtual environment is a virtual environment within the extended reality space. A virtual environment can refer to a scene formed by virtual objects, such as walls, furniture, floors, etc., forming a virtual room within the extended reality space. The virtual environment has a virtual screen, which can be a virtual monitor or a virtual projection screen.

[0043] S12. In response to displaying playback content on a virtual screen, the light of the playback content is rendered onto the rendering area of ​​the virtual environment to present the reflection of the playback content in the rendering area of ​​the virtual environment.

[0044] In some embodiments, the virtual screen can display playback content, which can be images, videos, etc. In the real world, if playback content is displayed on a screen, the environment in which the screen is located should have reflectivity. For example, in the dark, if a video is played on a television, the wall next to the television will reflect the video being played. This embodiment takes into account the real-world situation. Therefore, when displaying playback content on the virtual screen, the light from the playback content is rendered into the rendering area of ​​the virtual environment. The rendering area presents the diffuse reflection of the playback content, thereby simulating the effect of playback through a screen in the real world and improving the user's realistic experience.

[0045] In some embodiments, the rendering area in the virtual environment may be dependent on the virtual screen. For example, the rendering area may be a trapezoidal region extending outwards from the four sides of the virtual screen, since light propagates radially outwards. There may be four rendering areas, specifically trapezoidal regions extending outwards from the four sides of the virtual screen. In some embodiments, the light from the playback content may be rendered into the rendering area of ​​the virtual environment when the brightness of the virtual environment is less than a preset value. This is because when the ambient brightness is greater than the preset value, reflections of the playback content should not normally be visible on walls or other areas.

[0046] To better illustrate the methods proposed in the embodiments of this disclosure, the following is in conjunction with the appendix. Figure 3 Explain it. Figure 3 It displays a purely virtual extended reality space, such as Figure 3 As shown, extended reality space displays, as Figure 3The virtual environment shown is a virtual room, including walls, furniture, etc. A virtual screen is displayed within this virtual environment, showing video as playback content. It can be seen that diffuse reflections of the video content are displayed on the rendered areas of the virtual environment above, below, to the left, and to the right of the virtual screen, particularly... Figure 3 On the ground below the virtual screen, diffused light generated by the virtual screen can be clearly seen. In this embodiment, the simulated light emission in the virtual environment not only displays the playback content on the virtual screen but also renders the light of the playback content into the virtual environment. As the playback content changes, reflections of the playback content will appear in the virtual environment around the screen (e.g., on the surfaces of objects in the virtual environment), thus simulating the effect of screen light reflecting into the room in scenarios such as watching a movie in a dark environment. This embodiment improves the user experience when viewing a virtual screen using an extended reality device and enhances realism.

[0047] In some embodiments of this disclosure, rendering the light of the playback content into a virtual environment includes: acquiring the current content frame of the playback content; determining a target content area corresponding to the rendering area in the virtual environment based on the current content frame and the position of the rendering area in the virtual environment; and rendering the color of the target content area into the corresponding rendering area in the virtual environment.

[0048] In some embodiments, the content being played may not remain constant, thus requiring the rendering area to be rendered based on the current content screen. There can be one or more rendering areas; for example, there can be four, extending outwards from the four edges of the virtual screen. For a single rendering area, the reflection it can display is typically the reflection of the area closest to it in the current content screen. Areas farther from the rendering area in the current content screen have almost no impact on the reflection displayed by the rendering area. Calculating the reflection of the rendering area based on the entire current content screen would result in excessive computational load, which extended reality devices often cannot handle. This would cause the displayed reflection to fail to synchronize with the current content screen, ultimately degrading the user experience. Therefore, in this embodiment, a target content area corresponding to the rendering area is determined. The target content area is a part of the current content screen. Then, the target content area is rendered onto the rendering area. Specifically, in some embodiments, a copy image of the target content area may be generated, and then the copy image may be stretched to the same size as the rendering area. Then, the copy image may be overlaid onto the rendering area. For any position in the rendering area, the color of the copy image and the original color of the rendering area at that position may be blended according to a blending coefficient. Alternatively, a copy image of the target content area may be generated, and the copy image may be set to semi-transparent. Then, the semi-transparent copy image may be stretched to the same size as the rendering area before being overlaid onto the rendering area.

[0049] In some embodiments of this disclosure, the target content region corresponding to different rendering regions in the virtual environment is determined based on the current content screen and the position of different regions in the virtual environment, including: determining the target edge of the virtual screen that the rendering region in the virtual environment is close to; and extracting the edge region of the target edge in the current content screen as the corresponding target content region.

[0050] In some embodiments, with Figure 3 For example, Figure 3 The floor area displayed in front of the sofa is a rendering area. The edge of the virtual screen closest to the floor area is the bottom edge of the virtual screen. Therefore, for the floor area, its corresponding target edge is the bottom edge. The bottom edge is then cropped from the current content screen as the target content area. In some embodiments, considering that the current content screen may have black borders, when obtaining the target content area, the black borders in the current content screen can be removed first, and then the target edge area can be cropped from the current content screen after removing the black borders. The length of the target content area can be equal to the length of the corresponding target edge, and its width can be a preset value, for example, the width of the target content area can be set to 10cm.

[0051] In some embodiments of this disclosure, before rendering the color of the target content area to the corresponding rendering area in the virtual environment, the method further includes: generating a noise map, which causes the coordinates of at least some pixels in the target content area to be offset.

[0052] In some embodiments, such as Figure 4 The image shows the display effect of directly rendering the color of the target content area onto the rendering area without using a noise map. As you can see, the light rays projected onto the rendering area are in a straight line. However, in the real world, due to the roughness of object surfaces, the light emitted from the screen hitting a wall or ground should exhibit diffuse reflection; that is, the emitted light rays should not be in a straight line. Therefore, in this embodiment, a noise map is used to process the target content area (here, the target content area is extracted from the current content image for rendering; the current content image displayed on the virtual screen is not processed using a noise map) to shift the coordinates of the pixels in the noise map. Because the pixel coordinates are shifted, rendering the color of the shifted target content image onto the rendering area will produce a diffuse reflection effect, as shown in the image. Figure 5 As shown, the light hitting the rendering area exhibits a certain degree of curvature and dispersion, representing diffuse reflection, resulting in a softer and more realistic effect. Specifically, in some embodiments, different positions on the noise map correspond to offset coefficients (the offset coefficients can be 0 to 1). The noise map is used to shift the coordinates of the reflected light, including multiplying the pixel coordinates of the target content area by the offset coefficient of the corresponding point on the noise map to obtain the shifted pixel coordinates. The positions on the noise map can correspond one-to-one with the positions in the target content area, with each position corresponding to an offset coefficient used to shift the pixel coordinates of the pixel at that position, thus causing the pixel to shift.

[0053] In some embodiments of this disclosure, rendering the color of the target content area onto the corresponding rendering area in the virtual environment includes: determining a fusion coefficient; determining the weights of the color of the target content area and the color of the corresponding rendering area in the virtual environment based on the fusion coefficient; and fusion the color of the target content area and the color of the corresponding rendering area in the virtual environment according to their respective weights to obtain the color displayed by the rendering area in the virtual environment.

[0054] In some embodiments, the proposed method simulates the diffuse reflection of light from the content being played in a rendering area within a virtual environment. While the rendering area needs to display the color (including brightness) of the light from the content being played, it should also display its own color. Therefore, the color of the rendering area (specifically, the surface of the object in the rendering area) needs to be blended with its own color to obtain the color that the rendering area should display. The blending coefficient can be, for example, a coefficient between 0 and 1, and it can be preset. The total weight can be set to 1, and the blending coefficient can be the weight of the target content area's color. For example, if the blending coefficient is 0.6, then the weight of the target content area's color is 0.6, and the weight of the rendering area's own color is 0.4. In some embodiments, the steps of determining the blending coefficient; determining the weight of the target content area's color and the weight of the corresponding rendering area's own color in the virtual environment based on the blending coefficient; and blending the target content area's color with the corresponding rendering area's own color in the virtual environment according to their respective weights to obtain the color displayed by the rendering area in the virtual environment can be processed by the CPU.

[0055] In some embodiments of this disclosure, before projecting light from the target content area onto the corresponding rendering area in the virtual environment, the method further includes setting the area size of the rendering area in the virtual environment that can reflect the playback content in different directions.

[0056] In some embodiments, such as Figure 3 As shown, the virtual screen is rectangular, and different directions can be the directions that the edges of the virtual screen face. Therefore, there can be four rendering areas, corresponding to the top, bottom, left, and right edges of the virtual screen. The reflection effect presented by different rendering areas is affected by the color of the edge area of ​​their corresponding edge. The rendering algorithm for different directions can be the same. In some embodiments, the influence of multiple different target content areas on the reflection presented by the rendering area can be ignored. That is, for a rendering area, the reflection presented is not affected by two or more target content areas, but only by one target content area. This is because if multiple target content areas are considered, the colors of multiple target content areas need to be mixed, and the algorithm complexity will increase significantly. For example, in calculating... Figure 3 When rendering the reflections in the floor area, if the influence of the left and top edges of the virtual screen needs to be considered simultaneously, the algorithm complexity will be greatly increased, which can easily cause stuttering. This will result in the rendered reflections not being synchronized with the playback content displayed on the virtual screen, thus reducing the user experience.

[0057] In some embodiments of this disclosure, the rendering area is trapezoidal, and the length of the upper base of the trapezoid is equal to the length of the side of the playback content in the direction of the rendering area. In some embodiments, such as... Figure 6As shown, it schematically illustrates the rendering area ( Figure 6 The relationship between the trapezoid (drawn in the image) and the edges of the virtual screen is crucial for the rendering area. It's used to represent reflections. In reality, light emitted from the screen diffuses radially outwards; therefore, the rendering area is trapezoidal. When determining the dimensions of the rendering area, the height of the trapezoid and the slopes of its two hypotenuses need to be determined. Because it simulates light emanating from the virtual screen, the upper base of the trapezoid (the shorter side of the parallel sides) has the same length as the side along the direction of the rendering area. Figure 6 In this context, the rendering area corresponding to the floor is located in the direction that the bottom edge of the virtual screen faces. Therefore, the length of the upper base of the trapezoid in this rendering area is taken as the length of the bottom edge of the virtual screen.

[0058] In some embodiments of this disclosure, rendering the color of the target content area onto a corresponding rendering area in the virtual environment includes: simulating diffuse light from the target content area illuminating the corresponding rendering area based on the angle between the virtual screen and the rendering area, and the target content area, and displaying the diffuse light in the rendering area.

[0059] In some embodiments, such as Figure 7 As shown, when rendering the color of the target content area to the rendering area in the virtual environment, the angle of the virtual screen needs to be considered, because the direction of diffuse light will differ at different angles. For example, Figure 7 In this scenario, the virtual screen is perpendicular to the floor rendering area. The light emitted from the virtual screen is directed towards the rendering area at a 45-degree angle. Using this as a reference, as the angle between the virtual screen and the floor increases by 'a', the incident angle of the emitted light increases by 'a / 2'. Thus, when the display screen is parallel to the floor (i.e., the angle between them increases to 90 degrees), the incident angle increases by 45 degrees to reach 90 degrees. Based on the angle between the virtual screen and the rendering area, the system simulates light from the target content area illuminating the rendering area, forming diffused light, which is then displayed on the surface of the object within the rendering area.

[0060] In some embodiments, the method further includes blurring the edges of the light rays rendered into the virtual environment. Light rays rendered into the virtual environment should not have overly distinct boundaries; therefore, blurring the edges of the light rays rendered into the virtual environment, i.e., blurring the boundaries of the light rays rendered into the rendering area, is more consistent with reality and makes the rendering more realistic.

[0061] In some embodiments, the method further includes controlling the intensity of light from the content being played in the virtual environment, rendered to decrease as the distance from the virtual screen increases, according to a pre-set attenuation coefficient. In some embodiments, the brightness of the light should gradually decrease with increasing distance after it is emitted; therefore, an attenuation coefficient is set. This coefficient determines how the emitted light attenuates. For example, the light intensity can be set to linearly or non-linearly attenuate from 100% to 0% after 5 meters from the screen.

[0062] In some embodiments of this disclosure, the method further includes: if there is a shadow area in the rendering area of ​​the virtual environment that is occluded by an object, the light of the playback content projected onto the shadow area is softened.

[0063] In some embodiments, for a rendering area in a virtual environment, if there is an object that blocks the light emitted from the virtual screen, then the area blocked by the object is a shadow area. Calculating the specific value of diffuse reflection in a shadow area would require excessive computing power. Therefore, it can be assumed that there is no shadow area, and the reflected light that should be presented in the area under normal, unobstructed conditions can be calculated. Then, the reflected light in the shadow area can be weakened, for example, by multiplying it by a pre-set coefficient, such as 0.1, which means that the light parameters such as the intensity and color of the light rendered to the shadow area are 0.1 times the normal value. The weakened reflected light is then presented in the shadow area. This achieves both shadow area processing and improved calculation speed.

[0064] Some embodiments of this disclosure also propose a control device based on extended reality, comprising:

[0065] The display unit is used to display a virtual environment and a virtual screen located in the virtual environment in an extended reality space;

[0066] The control unit is used to render the light of the playback content into the virtual environment in response to displaying playback content on the virtual screen, so as to present the reflection of the playback content in the virtual environment.

[0067] In some embodiments, rendering the light of the playback content onto the rendering area of ​​the virtual environment includes: obtaining the current content screen of the playback content;

[0068] Based on the current content screen and the position of the rendering area in the virtual environment, determine the target content area corresponding to the rendering area in the virtual environment;

[0069] The color of the target content area is rendered onto the corresponding rendering area in the virtual environment.

[0070] In some embodiments, determining the target content region corresponding to the rendering region in the virtual environment based on the current content screen and the position of the rendering region in the virtual environment includes:

[0071] Determine the target edge of the virtual screen that the rendering area in the virtual environment is close to;

[0072] The edge region of the target edge in the current content frame is captured as the corresponding target content region.

[0073] In some embodiments, before rendering the color of the target content area onto the corresponding rendering area in the virtual environment, the control unit is further configured to:

[0074] A noise map is generated, which causes the coordinates of at least some pixels in the target content region to shift.

[0075] In some embodiments, rendering the color of the target content area onto the corresponding rendering area in the virtual environment includes:

[0076] Determine the fusion coefficient;

[0077] Based on the fusion coefficient, the weights of the color of the target content area and the color of the corresponding rendering area in the virtual environment are determined;

[0078] The color of the target content area is blended with the color of the corresponding rendering area in the virtual environment according to their respective weights to obtain the color displayed by the rendering area in the virtual environment.

[0079] In some embodiments, before rendering the color of the target content area onto the corresponding rendering area in the virtual environment, the control unit is further configured to:

[0080] The size of the rendering area in the virtual environment that can reflect the playback content, set in different directions.

[0081] In some embodiments, the rendering area is a trapezoid, and the length of the upper base of the trapezoid is equal to the length of the side of the playback content in the direction of the rendering area.

[0082] In some embodiments, rendering the color of the target content area onto a corresponding rendering area in the virtual environment includes: simulating diffuse light from the target content area illuminating the corresponding rendering area based on the angle between the virtual screen and the rendering area, and the target content area, and displaying the diffuse light in the rendering area.

[0083] In some embodiments, the control unit is further configured to: blur the edges of the light rays of the playback content rendered into the virtual environment; and / or,

[0084] Based on a pre-set attenuation coefficient, the intensity of the light emitted from the playback content rendered into the virtual environment is controlled to decrease as the distance from the virtual screen increases.

[0085] In some embodiments, the control unit is further configured to: if there is a shadow area occluded by an object in the rendering area of ​​the virtual environment, to soften the light of the playback content projected onto the shadow area.

[0086] For embodiments of the apparatus, since they basically correspond to the method embodiments, relevant details can be found in the descriptions of the method embodiments. The apparatus embodiments described above are merely illustrative, and the modules described as separate modules may or may not be separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0087] The methods and apparatus of this disclosure have been described above based on embodiments and application examples. Furthermore, this disclosure also provides an electronic device and a computer-readable storage medium, which are described below.

[0088] The following is for reference. Figure 8 The figure illustrates a structural schematic of an electronic device (e.g., a terminal device or server) 800 suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in the figure is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present disclosure.

[0089] Electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 802 or a program loaded from storage device 808 into random access memory (RAM) 803. RAM 803 also stores various programs and data required for the operation of electronic device 800. The processing device 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0090] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although an electronic device 800 with various devices is shown in the figure, it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0091] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of embodiments of this disclosure.

[0092] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0093] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0094] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0095] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods of the present disclosure.

[0096] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0097] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0098] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0099] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0100] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0101] According to one or more embodiments of this disclosure, a control method based on extended reality is provided, comprising:

[0102] Displaying a virtual environment and a virtual screen within the extended reality space;

[0103] In response to displaying playback content on the virtual screen, the light from the playback content is rendered onto the rendering area of ​​the virtual environment to present the reflection of the playback content in the rendering area of ​​the virtual environment.

[0104] According to one or more embodiments of this disclosure, a control method based on extended reality is provided, which renders the light of the played content onto the rendering area of ​​the virtual environment, including:

[0105] Get the current content screen of the playback content;

[0106] Based on the current content screen and the position of the rendering area in the virtual environment, determine the target content area corresponding to the rendering area in the virtual environment;

[0107] The color of the target content area is rendered onto the corresponding rendering area in the virtual environment.

[0108] According to one or more embodiments of this disclosure, a control method based on extended reality is provided, which determines a target content region corresponding to the rendering region in the virtual environment based on the current content screen and the position of the rendering region in the virtual environment, including:

[0109] Determine the target edge of the virtual screen that the rendering area in the virtual environment is close to;

[0110] The edge region of the target edge in the current content frame is captured as the corresponding target content region.

[0111] According to one or more embodiments of this disclosure, a control method based on extended reality is provided, which further includes, before rendering the color of the target content area onto the corresponding rendering area in the virtual environment:

[0112] A noise map is generated, which causes the coordinates of at least some pixels in the target content region to shift.

[0113] According to one or more embodiments of this disclosure, a control method based on extended reality is provided, which renders the color of the target content area onto a corresponding rendering area in the virtual environment, including:

[0114] Determine the fusion coefficient;

[0115] Based on the fusion coefficient, the weights of the color of the target content area and the color of the corresponding rendering area in the virtual environment are determined;

[0116] The color of the target content area is blended with the color of the corresponding rendering area in the virtual environment according to their respective weights to obtain the color displayed by the rendering area in the virtual environment.

[0117] According to one or more embodiments of this disclosure, a control method based on extended reality is provided, which further includes, before rendering the color of the target content area onto the corresponding rendering area in the virtual environment:

[0118] The size of the rendering area in the virtual environment that can reflect the playback content, set in different directions.

[0119] According to one or more embodiments of this disclosure, an extended reality-based control method is provided, wherein the rendering area is a trapezoid, and the length of the upper base of the trapezoid is equal to the length of the side of the playback content in the direction of the rendering area.

[0120] According to one or more embodiments of this disclosure, a control method based on extended reality is provided, which renders the color of the target content area onto a corresponding rendering area in the virtual environment, including: simulating diffuse light from the target content area illuminating the corresponding rendering area based on the angle between the virtual screen and the rendering area, and the target content area, and displaying the diffuse light in the rendering area.

[0121] According to one or more embodiments of this disclosure, a control method based on extended reality is provided, further comprising:

[0122] The edges of the light rays in the playback content rendered into the virtual environment are blurred; and / or,

[0123] Based on a pre-set attenuation coefficient, the intensity of the light emitted from the playback content rendered into the virtual environment is controlled to decrease as the distance from the virtual screen increases.

[0124] According to one or more embodiments of this disclosure, a control method based on extended reality is provided, further comprising:

[0125] If there is a shadow area in the rendering area of ​​the virtual environment that is occluded by an object, the light of the playback content projected onto the shadow area will be softened.

[0126] According to one or more embodiments of this disclosure, a control device based on extended reality is provided, comprising:

[0127] The display unit is used to display a virtual environment and a virtual screen located in the extended reality space;

[0128] A control unit is configured to, in response to displaying playback content on the virtual screen, render the light of the playback content onto a rendering area of ​​the virtual environment to present the reflection of the playback content in the rendering area of ​​the virtual environment.

[0129] According to one or more embodiments of the present disclosure, an electronic device is provided, including: at least one memory and at least one processor;

[0130] The at least one memory is used to store program code, and the at least one processor is used to call the program code stored in the at least one memory to execute the method described in any one of the above.

[0131] According to one or more embodiments of the present disclosure, a computer-readable storage medium is provided for storing program code that, when executed by a processor, causes the processor to perform the methods described above.

[0132] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0133] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0134] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An extended reality-based control method, characterized by, Comprising: displaying a virtual environment and a virtual screen located in the virtual environment in an extended reality space; in response to displaying a playing content in the virtual screen, rendering light rays of a current content frame of the playing content into a rendering area of the virtual environment to present diffuse reflection of the current content frame of the playing content in the rendering area of the virtual environment; wherein rendering the light rays of the current content frame of the playing content into the rendering area of the virtual environment comprises: obtaining the current content frame of the playing content; determining a target content area corresponding to the rendering area in the virtual environment based on the current content frame and a position of the rendering area in the virtual environment; and rendering a color of the target content area into the corresponding rendering area in the virtual environment; wherein rendering the color of the target content area into the corresponding rendering area in the virtual environment comprises: generating a copy picture of the target content area, then stretching the copy picture to have the same size as the rendering area, and then covering the copy picture to the rendering area; wherein determining the target content area corresponding to the rendering area in the virtual environment based on the current content frame and the position of the rendering area in the virtual environment comprises: determining a target edge of the virtual screen close to the rendering area in the virtual environment; and intercepting an edge area of the target edge in the current content frame as the corresponding target content area.

2. The method of claim 1, wherein, Before rendering the color of the target content area into the corresponding rendering area in the virtual environment, further comprising: generating a noise point map to cause a coordinate of at least part of pixels in the target content area to be offset.

3. The method of claim 1, wherein, Rendering the color of the target content area into the corresponding rendering area in the virtual environment comprises: determining a fusion coefficient; determining weights of the color of the target content area and a self color of the corresponding rendering area in the virtual environment based on the fusion coefficient; fusing the color of the target content area and the self color of the corresponding rendering area in the virtual environment according to the respective weights to obtain a displayed color of the rendering area in the virtual environment.

4. The method of claim 1, wherein, Before rendering the color of the target content area into the corresponding rendering area in the virtual environment, further comprising: setting a region size of a rendering area in the virtual environment capable of reflecting the playing content in different directions.

5. The method of claim 4, wherein: the rendering area is a trapezoid, and a length of an upper base of the trapezoid is equal to a length of an edge of the playing content in a direction in which the rendering area is located.

6. The method of claim 1, wherein, further comprising: blurring an edge of the light rays of the current content frame of the playing content rendered into the virtual environment; and / or, controlling an intensity of the light rays of the current content frame of the playing content rendered into the virtual environment to decrease with an increase of a distance from the virtual screen according to a pre-set attenuation coefficient.

7. The method of claim 1, wherein, further comprising: If there is a shadow region occluded by an object in a rendering region of the virtual environment, the light rays of the current content frame of the play content projected to the shadow region are processed by fading.

8. An extended reality-based control device, comprising: Comprise: A display unit configured to display a virtual environment and a virtual screen in the virtual environment in an extended reality space; A control unit configured to, in response to displaying play content in the virtual screen, render light rays of the play content into a rendering region of the virtual environment to present diffuse reflection of a current content frame of the play content in the rendering region of the virtual environment; Wherein rendering light rays of the current content frame of the play content into the rendering region of the virtual environment comprises: obtaining the current content frame of the play content; determining a target content region corresponding to the rendering region in the virtual environment based on the current content frame and the position of the rendering region in the virtual environment; and rendering the color of the target content region to the corresponding rendering region in the virtual environment. Wherein rendering the color of the target content region to the corresponding rendering region in the virtual environment comprises: generating a copy picture of the target content region, then stretching the copy picture to the same size as the rendering region, and then covering the copy picture to the rendering region. Wherein, based on the current content frame and the position of the rendering region in the virtual environment, determining the target content region corresponding to the rendering region in the virtual environment comprises: determining a target edge of the virtual screen close to the rendering region in the virtual environment; and intercepting an edge region of the target edge in the current content frame as a corresponding target content region. 9.An electronic device comprising: at least one memory and at least one processor; Wherein, the at least one memory is configured to store program code, and the at least one processor is configured to call the program code stored in the at least one memory to execute the method of any one of claims 1 to 7. 10.A computer readable storage medium configured to store program code, wherein the program code, when executed by a processor, causes the processor to perform the method of any one of claims 1 to 7.

Images