Method, apparatus, device and medium for rendering virtual objects

Abstract

A method, apparatus, device and medium for rendering virtual objects are provided. In one method, multiple environment images of a real environment are acquired, and the multiple environment images include images of the real environment acquired at multiple acquisition time points respectively. Based on the multiple acquisition time points, an environment image that matches the current time point when an augmented reality application is used is selected from the multiple environment images as a rendering environment image. A virtual object rendered using the rendering environment image is presented in the augmented reality application. Using the exemplary implementation of the present disclosure, a rendering environment image for rendering a virtual object can be selected in real time based on the current time point and rendering can be performed. In this way, the lighting information of the rendering environment image can be made consistent with the lighting information of the real environment at the current time point.

Description

Technical Field

[0001] Exemplary implementations of the present disclosure generally relate to rendering virtual objects, and particularly to methods, apparatuses, devices, and computer-readable storage media for rendering virtual objects based on real-time selected environmental images in augmented reality (AR) applications. Background Art

[0002] Currently, a large number of augmented reality applications have been developed. Users can use image acquisition devices in devices such as mobile terminals to capture scenes in the real environment, and can add virtual objects to the video of the captured real environment. For example, virtual objects can be placed at desired positions, or virtual characters that can move can be added, etc. Since the light in the real environment may change continuously, how to set the light rendering parameters for virtual objects so that the light effect of the rendered virtual objects is consistent with the surrounding real environment has become an urgent problem to be solved. Summary of the Invention

[0003] In a first aspect of the present disclosure, a method for rendering a virtual object in an augmented reality application is provided. In this method, a plurality of environmental images of the real environment are acquired, and the plurality of environmental images include images of the real environment acquired at a plurality of acquisition time points respectively. Based on the plurality of acquisition time points, an environmental image that matches the current time point when the augmented reality application is used is selected from the plurality of environmental images as the rendering environmental image. A virtual object rendered using the rendering environmental image is presented in the augmented reality application.

[0004] In a second aspect of the present disclosure, an apparatus for rendering a virtual object in an augmented reality application is provided. The apparatus includes: an acquisition module configured to acquire a plurality of environmental images of the real environment, the plurality of environmental images including images of the real environment acquired at a plurality of acquisition time points respectively; a selection module configured to select, based on the plurality of acquisition time points, an environmental image that matches the current time point when the augmented reality application is used from the plurality of environmental images as the rendering environmental image; and a presentation module configured to present a virtual object rendered using the rendering environmental image in the augmented reality application.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the device to execute the method according to the first aspect of the present disclosure when executed by the at least one processing unit.

[0006] In a fourth aspect of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program, which when executed by a processor causes the processor to implement the method according to the first aspect of the present disclosure.

[0007] It should be understood that the content described in this part is not intended to limit the key features or important features of the implementation manners of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In the following, with reference to the accompanying drawings and the following detailed description, the above and other features, advantages and aspects of the various implementation manners of the present disclosure will become more apparent. In the drawings, the same or similar reference numerals denote the same or similar elements, where:

[0009] Figure 1 FIG. shows an example of an application environment in which the implementation manners of the present disclosure can be used;

[0010] Figure 2 FIG. shows a block diagram of a virtual object rendering in an augmented reality application according to some implementation manners of the present disclosure;

[0011] Figure 3 FIG. shows a block diagram of a plurality of environmental images respectively collected at a plurality of acquisition time points according to some implementation manners of the present disclosure;

[0012] Figure 4 FIG. shows a block diagram of a plurality of environmental images respectively collected at a plurality of acquisition positions according to some implementation manners of the present disclosure;

[0013] Figure 5 FIG. shows a block diagram of a process for selecting a rendered environmental image based on a comparison between the device position and the acquisition position of a terminal device running an augmented reality application according to some implementation manners of the present disclosure;

[0014] Figure 6 FIG. shows a block diagram of a process for selecting a rendered environmental image based on an occlusion relationship according to some implementation manners of the present disclosure;

[0015] Figure 7 FIG. shows a block diagram of a process for converting an environmental image into a standard environmental image according to some implementation manners of the present disclosure;

[0016] Figure 8 FIG. shows a block diagram of a process for mapping pixels according to some implementation manners of the present disclosure;

[0017] Figure 9 FIG. shows a block diagram of a spherical panoramic image according to some implementation manners of the present disclosure;

[0018] Figure 10 A block diagram showing a process for generating new rendering parameters based on multiple environmental images according to some implementations of the present disclosure;

[0019] Figure 11A and 11B Block diagrams respectively showing the presentation of virtual objects in an augmented reality application according to some implementations of the present disclosure;

[0020] Figure 12 A flowchart showing a method for rendering a virtual object according to some implementations of the present disclosure;

[0021] Figure 13 A block diagram showing an apparatus for rendering a virtual object according to some implementations of the present disclosure; and

[0022] Figure 14 A block diagram showing a device capable of implementing multiple implementations of the present disclosure. Detailed Implementations

[0023] Implementations of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain implementations of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the implementations set forth herein. On the contrary, these implementations are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and implementations of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0024] In the description of the implementations of the present disclosure, the term "including" and its like should be understood as an open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one implementation" or "the implementation" should be understood as "at least one implementation". The term "some implementations" should be understood as "at least some implementations". There may also be other explicit and implicit definitions below. As used herein, the term "model" may represent the association relationship between various data. For example, the above-mentioned association relationship can be obtained based on various technical solutions known currently and / or to be developed in the future.

[0025] It can be understood that the data involved in the technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the corresponding laws, regulations and related provisions.

[0026] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, usage scopes, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the users and the authorization of the users should be obtained in an appropriate manner according to the relevant laws and regulations.

[0027] For example, when a user's active request is received, a prompt message is sent to the user to clearly prompt the user that the operation requested by the user will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, an application program, a server, or a storage medium that performs the operations of the present disclosure's technical solution based on the prompt message.

[0028] As an optional but non-limiting implementation manner, the manner of sending a prompt message to the user in response to receiving the user's active request may be, for example, in the form of a pop-up window, and the prompt message may be presented in text in the pop-up window. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0029] It can be understood that the above process of notifying and obtaining user authorization is only illustrative and does not limit the implementation manner of the present disclosure. Other manners that comply with relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0030] Example environment

[0031] Currently, a large number of AR applications have been developed. See Figure 1 Describe an AR application environment according to an exemplary implementation manner of the present disclosure. Figure 1 FIG. 100 shows an example of an application environment in which the implementation manner of the present disclosure can be used. An AR application can be run at a terminal device, and the user can hold the terminal device and use the augmented reality application 110 running at the terminal device to capture a scene in the real environment, and a virtual object 120 (for example, adding a cube sculpture) can be added to the video of the captured real environment. At this time, since the user holds the terminal device, it can be considered that the device position of the terminal device in the real environment is the same as the user position.

[0032] Generally speaking, the light in the AR application environment will change with time and the movement of the device position. Figure 1 FIG. 100 shows the virtual object 120 rendered using pre-set light rendering data. In Figure 1The lighting effect of the rendered virtual object 120 in [context] is inconsistent with the surrounding real environment. Specifically, the pre-set lighting rendering data is set for the daytime scene, and the lighting effect of the virtual object 120 obtained using this lighting rendering data is relatively bright. When the user uses the augmented reality application 110 at night, the lighting in the real environment is relatively dim. At this time, when adding the virtual object 120 to the AR scene, the overly bright virtual object 120 does not harmonize with the surrounding dim environment. This results in the visual effect of the augmented reality application 110 not being realistic.

[0033] It will be understood that the real environment may have complex lighting information. For example, lighting can be generally divided into direct lighting and ambient lighting, and there may be multiple direct light sources in the real environment. According to one technical solution, the captured image of the real environment can be processed based on real-time lighting estimation technology, that is, lighting analysis can be performed on the image of the real environment to obtain the real lighting. However, this technical solution can only obtain the overall lighting intensity of the real environment or the most significant direct lighting, and the accuracy is not satisfactory. In addition, the real-time lighting estimation technology incurs a large computational resource overhead, and thus it is difficult to implement in portable computing devices such as mobile terminals. At this time, how to determine the surrounding lighting information of the virtual object and perform rendering in a more convenient and effective manner has become an urgent problem to be solved.

[0034] Summary process of rendering virtual objects

[0035] To address the deficiencies in the above technical solution, according to an exemplary implementation of the present disclosure, a method for rendering a virtual object in an augmented reality application is proposed. Generally speaking, multiple environmental images of the real environment can be obtained, and these environmental images can be used as candidate rendering environmental images for rendering the virtual object. Here, the multiple environmental images can be images of the real environment collected at multiple acquisition time points. For example, multiple environmental images can be collected at different time points of a day (such as daytime and night).

[0036] In the context of the present disclosure, the AR application can be run on various terminal devices. For example, the AR application can be run on a conventional mobile terminal device (including but not limited to, mobile phones, mobile tablet computing devices, mobile notebook computing devices, etc.). Also, for example, the AR application can be run on a wearable terminal device (including but not limited to, glasses devices with computing functions, helmet devices, etc.). Again, for example, the AR application can be run on a computing device where the display function is separated from the computing function (for example, using a portable computing device to run the AR application and using a glasses device communicating with the portable computing device to display the interface of the AR application).

[0037] SeeFigure 2 Describes an overview of an exemplary implementation according to the present disclosure, the Figure 2 shows a block diagram 200 for rendering virtual objects in an augmented reality application according to some implementations of the present disclosure. As Figure 2 shown, a plurality of environmental images collected at a plurality of acquisition times can be obtained. For example, the acquisition time point 210 of the environmental image 220 can be during the day,..., and the acquisition time point 212 of the environmental image 222 can be at night.

[0038] The current time point 244 when the user uses the augmented reality application 240 can be determined. Further, based on the comparison between the current time point 244 and the plurality of acquisition time points 210,..., 212, the environmental image 222 that matches the current time point 244 can be selected from the plurality of environmental images 220,..., 222 as the rendering environmental image 230. Specifically, if the user uses the augmented reality application 240 during the day, the environmental image 220 collected during the day can be selected as the rendering environmental image 230 and rendered. At this time, the rendered virtual object 250 will present a bright lighting effect during the day. Again, for example, if the user uses the augmented reality application 240 at night, the environmental image 222 collected at night can be selected as the rendering environmental image 230. At this time, the rendered virtual object 250 will present a dim lighting effect at night.

[0039] As Figure 2 shown, the current time point 244 is at night, so the environmental image 222 at night can be selected as the rendering environmental image 230. The rendering environmental image 230 can be used to render the virtual object 250, and the rendered virtual object 250 is presented in the augmented reality application 240. At this time, the night lighting effect as Figure 2 shown can be presented.

[0040] Using the exemplary implementation of the present disclosure, the rendering environmental image 230 for rendering the virtual object 250 can be selected in real time based on the current time point 244 and rendering can be performed. In this way, the lighting information of the rendering environmental image 230 can be made consistent with the lighting information of the real environment at the current time point 244, and thus the rendered virtual object 250 can be made consistent with the surrounding real environment.

[0041] Detailed process of rendering virtual objects

[0042] has been referred to Figure 2 Describes an overview of an exemplary implementation according to the present disclosure. In the following, more details for rendering virtual objects will be described in detail. Refer to Figure 3 Describes more information about the environmental image, the Figure 3Block diagram 300 shows multiple environmental images collected at multiple acquisition time points according to some implementations of the present disclosure. As Figure 3 shown, multiple environmental images can be collected at different time points of a day. For example, environmental image 220 can be collected at time point T1, …, and environmental image 222 can be collected at time point TN at a predetermined time interval (e.g., 2 hours, 4 hours, etc.).

[0043] According to an exemplary implementation of the present disclosure, multiple environmental images can be directly stored at the terminal device for running the augmented reality application 240. Alternatively and / or additionally, multiple environmental images can be stored at the server providing the AR service so as to obtain the desired environmental image via the network.

[0044] It will be understood that although the process of collecting environmental images at different time points of a day is shown above, alternatively and / or additionally, different environmental images can also be collected in different seasons (e.g., spring, summer, autumn, winter) in an outdoor application scenario. In this way, the environmental image more matching the specific usage time can be selected and rendered based on both the season and the time point when the user uses the augmented reality application 240. Thereby, the rendering effect of the virtual object 250 can be further improved, and the lighting information of the virtual object 250 can be made more matching the surrounding real environment.

[0045] In an augmented reality application, positioning is usually based on a Visual Positioning System (VPS). To improve the positioning accuracy of the VPS, a large number of images of the real environment need to be pre-collected at different time points and different positions. According to an exemplary implementation of the present disclosure, no additional steps are required to collect the multiple environmental images 220, …, 222 described above, but each environmental image pre-collected for VPS purposes can be directly used as the environmental image for rendering purposes. In this way, the workload of data collection is not increased, but the already collected environmental images can be reused to serve the purpose of rendering the virtual object 250.

[0046] It will be understood that the lighting information of the virtual object 250 in the augmented reality application 240 will change with the position of the user in the real environment, so the device position can also be considered when rendering the virtual object 250. According to an exemplary implementation of the present disclosure, a set of environmental images matching the current time point 244 can be first found from multiple environmental images based on the current time point 244. Further, an environmental image more conforming to the device position can be found in this set of environmental images to be used as the rendering environmental image 230. Specifically, multiple environmental images collected at multiple acquisition positions in the real environment can be obtained.

[0047] See Figure 4 for more information regarding the acquisition locations. Figure 4 FIG. 400 is a block diagram showing a plurality of environmental images respectively acquired at a plurality of acquisition locations according to some implementations of the present disclosure. As Figure 4 shown, the real environment 410 may include a plurality of acquisition locations 420, 430, …, and 440, and environmental images may be acquired at each acquisition location. For example, the environmental image 220 may be acquired at the acquisition location 420, and the other environmental images may be respectively acquired at the acquisition locations 430 and 440, and so on.

[0048] According to an exemplary implementation of the present disclosure, at time point T1, a plurality of environmental images may be respectively acquired at a plurality of acquisition locations; …; at time point TN, a plurality of environmental images may be respectively acquired at a plurality of acquisition locations. It will be understood that the present disclosure does not limit the number of environmental images acquired at each time point, nor does it limit whether the acquisition locations selected at each time point are the same, but rather a plurality of environmental images acquired for VPS purposes may be directly used. The above-mentioned plurality of environmental images may be managed according to the acquisition time and acquisition location.

[0049] According to an exemplary implementation of the present disclosure, the device location of the user in the real environment 410 may be determined, and based on the comparison between the device location and the plurality of acquisition locations, a rendered environmental image 230 that matches the device location may be selected from the plurality of environmental images. See Figure 5 for more details on the selection of the rendered environmental image 230, which Figure 5 FIG. 500 is a block diagram showing a process for selecting a rendered environmental image 230 based on the comparison between the device location 510 of a terminal device running an augmented reality application and the acquisition locations according to some implementations of the present disclosure.

[0050] As Figure 5 shown, assuming that the user is located at the device location 510 in the real environment 410, the distances between the device location 510 and each of the acquisition locations 420, 430, …, and 440 may be respectively obtained. Further, based on the comparison of the respective distances, an environmental image acquired at the acquisition location closest to the device location 510 may be selected. In Figure 5 , assuming that the distances between the device location 510 and the acquisition locations 420 and 440 are 520 and 530 respectively, based on the comparison, it can be known that the distance 520 is greater than the distance 530. At this time, the environmental image acquired at the acquisition location 440 may be selected as the rendered environmental image 230.

[0051] Using the exemplary implementation of the present disclosure, by selecting the environmental image collected at the collection location 440 closest to the device location 510, an environmental image most approximating the surrounding environment seen by the user at the device location 510 can be used to render the virtual object 250. In this way, the lighting effect of the virtual object 250 can be made to more closely match the environmental lighting seen at the device location 510, thereby improving the visual effect of the augmented reality application 240.

[0052] It will be understood that in the real environment 410, there may be obstructions such as walls, plants, etc., and these obstructions will affect the lighting effect in the real environment 410. According to an exemplary implementation of the present disclosure, when selecting the rendering environmental image 230, the occlusion relationship can be further considered.

[0053] Figure 6 A block diagram 600 showing the process of selecting the rendering environmental image 230 based on the occlusion relationship according to some implementations of the present disclosure is shown. Specifically, the spatial structure of the real environment 410 can be determined based on multiple environmental images. The spatial structure of the real environment 410 can be obtained based on currently known and / or three-dimensional reconstruction techniques to be developed in the future, which will not be elaborated herein. Further, this spatial structure can be used to determine whether there are obstacles between the collection location of the rendering environmental image and the device location 510. As Figure 6 shown, the spatial structure indicates that there is an obstacle 610 in the real environment 410, and this obstacle 610 is located between the device location 510 and the collection location 440.

[0054] It will be understood that the obstacle 610 may cause the user at the device location 510 not to be able to directly see the collection location 440. Therefore, if the environmental image collected at the collection location 440 is directly used as the rendering environmental image 230, it may result in the lighting information of the rendered virtual object 250 not conforming to the user's real surrounding lighting. At this time, even if the collection location 440 is the closest to the device location 510, the environmental image at the collection location 440 cannot be selected.

[0055] According to an exemplary implementation of the present disclosure, an environmental image collected at another collection location 420 adjacent to the device location 510 can be selected from multiple environmental images. For example, since there is no obstacle between the collection location 420 and the device location 510, the environmental image at the collection location 420 relatively close to the device location 510 can be selected as the rendering environmental image 230. Using the exemplary implementation of the present disclosure, by considering the occlusion relationship, an environmental image more matching the user's surrounding lighting information can be selected. In this way, the incoordination problem that occurs when selecting the rendering environmental image 230 based solely on distance can be avoided.

[0056] It will be understood that during the process of collecting environmental images, an engineer can use a panoramic image collection device to collect multiple panoramic environmental images at different time points and different positions. Generally speaking, the panoramic image collection device is installed on a pan-tilt device, and the engineer can hold the pan-tilt and / or fix the pan-tilt at a device such as a bracket and perform the collection. During the collection process, the engineer needs to move the pan-tilt to collect environmental images at different collection positions. At this time, it cannot always be ensured that the collection angles of the pan-tilt are consistent.

[0057] It will be understood that different collection angles can cause the pixel distribution of the surrounding scenery to be different in the collected panoramic images. The rendering effect of the virtual object 250 depends on the pixel distribution used for rendering the environmental image 230. Therefore, different collection angles will directly result in different rendering lighting effects for the virtual object 250. At this time, it is necessary to standardize the collection angles of the collected panoramic images, that is, to "zero" them to a standard angle (for example, (0, 0, 0)).

[0058] According to an exemplary implementation of the present disclosure, standardization processing can be performed on each of the collected environmental images. Specifically, the collection angles associated with each environmental image can be obtained, and based on the collection angles, the environmental images can be converted to standard environmental images under the standard angle. In this way, each environmental image can be processed in a unified manner under the unified standard angle and an accurate rendering lighting effect can be obtained.

[0059] Figure 7 A block diagram 700 showing a process for converting an environmental image into a standard environmental image according to some implementations of the present disclosure is shown. The purpose of the conversion process is to zero the angles of each panoramic image, that is, to convert the Euler angles of the panoramic image to a predefined standard angle 720 (0, 0, 0) while keeping the position of the panoramic image unchanged. As Figure 7 shown, when the panoramic Figure 3 image collection device operates at the collection angle 710, the environmental image 220 can be obtained.

[0060] According to an exemplary implementation of the present disclosure, the collection angle 710 can be defined based on multiple coordinate systems. For example, the coordinate system 712 can be used as a reference coordinate system. According to an exemplary implementation of the present disclosure, the collection angle 710 can be represented based on Euler angles (roll, pitch, heading). In the VPS technology, the collection position (x, y, z) and collection angle (roll, pitch, heading) of the environmental image can be determined based on known algorithms, so they will not be elaborated here.

[0061] Further, each pixel in the environmental image 220 can be processed one by one to convert the environmental image 220 into a standard environmental image 730. Specifically, based on the acquisition angle 710, the spherical coordinates of the pixel in the environmental image 220, and the standard angle 720, the standard pixel corresponding to the pixel in the standard environmental image 730 can be determined. Hereinafter, more details of the standardization process will be described with reference to Figure 8 which shows a block diagram 800 for a pixel mapping process according to some implementations of the present disclosure. Figure 8

[0062] As Figure 8 shown, the environmental image 220 includes a pixel 810 (e.g., any pixel), and the pixel 810 can be mapped to a pixel 820 in the standard environmental image 730 based on a mathematical transformation. The spherical coordinates of the pixel 820 in the converted standard environmental image 730 can be expressed as (long new , lat new ), and the corresponding quaternion can be expressed as Q new . For the environmental image 220 before conversion, the Euler angles of the environmental image 220 can be expressed as (roll, pitch, heading), and the corresponding quaternion can be expressed as Q pano . The quaternion of the pixel 810 in the environmental image 220 can be expressed as Q new - Q pano , and the spherical coordinates of the pixel 810 can be expressed as (long old , lat old ).

[0063] Based on the mapping relationship, the color of the pixel 820 in the standard environmental image 730 is the color of the pixel 810 in the environmental image 220. It will be understood that the conversion between the image coordinates and spherical coordinates of the panoramic image, as well as the conversion between the Euler angles and quaternions, can be performed based on the coordinate conversion formulas that have been proposed currently and / or will be developed in the future, which will not be elaborated herein. Thus, the color of each pixel in the standard environmental image 730 can be determined one by one, and then the complete standard environmental image 730 can be obtained.

[0064] Using the exemplary implementation of the present disclosure, the standard environmental image 730 can be determined based on a simple mathematical transformation. Although only the environmental image 220 is used as an example above to describe the standardization process, similar processing can be performed in advance for each acquired environmental image, so that in the subsequent rendering process, the processing can be directly based on the standard environmental image of the selected rendering environmental image 230. In this way, when using the standard environmental image 730 for the subsequent rendering process, it can be ensured that the rendering lighting effect of the virtual object 250 is more matched to the user's surrounding real environment. ​

[0065] It will be understood that although the above describes the processing of an environmental image with a panoramic image in a rectangular format as an example, alternatively and / or additionally, the panoramic image may also be stored in the Figure 9 spherical format shown. The Figure 9 block diagram 900 of a spherical panoramic image 910 according to some implementations of the present disclosure is shown. Each pixel in the spherical panoramic image 910 can be processed one by one based on the principles of the normalization process described above, so as to obtain a corresponding standard environmental image stored in a spherical format.

[0066] According to an exemplary implementation of the present disclosure, after the rendered environmental image 230 that best matches the current time and the device location 510 has been selected, the relevant standard environmental image of the rendered environmental image 230 can be used as an environment light map to render the virtual object 250. Using the exemplary implementation of the present disclosure, an environment light map can be input to the renderer, and thus a virtual object 250 that matches the user's surrounding environment can be obtained.

[0067] It will be understood that the rendering efficiency of directly using a panoramic image as an environment light map in an actual rendering process may not be ideal. At this time, based on the spherical harmonic lighting model, the normalized panoramic image can be further converted into a spherical harmonic lighting parameter vector, and this vector can be used to perform the rendering process. Specifically, each normalized standard environmental image can be processed in a similar manner, and a corresponding spherical harmonic lighting parameter vector can be generated for each standard environmental image. Using the exemplary implementation of the present disclosure, in a later rendering process, the corresponding spherical harmonic lighting parameter vector can be directly called, so as to perform the rendering process with higher performance. In this way, the rendering efficiency can be improved and thus the latency of the augmented reality application 240 in rendering virtual objects can be reduced.

[0068] According to an exemplary implementation of the present disclosure, multiple environmental images can be pre-processed to extract the corresponding spherical harmonic lighting parameter vectors. The multiple spherical harmonic lighting parameter vectors can be directly stored at the terminal device for running the augmented reality application 240. Alternatively and / or additionally, the multiple spherical harmonic lighting parameter vectors can be stored at a server so as to obtain a desired spherical harmonic lighting parameter vector via a network.

[0069] According to an exemplary implementation of the present disclosure, multiple environmental images can be indexed respectively according to the acquisition time points and acquisition locations, so as to improve the search efficiency of selecting an environmental image that matches the current time point and the device location from a large number of environmental images. For example, multiple environmental images can be indexed according to the chronological order of the acquisition time points and / or the distances between the acquisition locations. The environmental image that is most similar to the current time and the device location can be directly searched based on this index.

[0070] It will be understood that the number of acquisition positions may be small and / or the distribution of multiple acquisition positions may not evenly cover the real environment 410. At this time, even using the environmental image at the acquisition position closest to the device position 510, sometimes a satisfactory rendered lighting effect may not be obtained. To further improve the rendered lighting effect, new rendering parameters can be generated based on the environmental images at two or more acquisition positions near the device position 510. In the following, more details are described with reference to Figure 10 Describe more details, the Figure 10 FIG. 1000 is a block diagram showing a process for generating new rendering parameters based on multiple environmental images according to some implementations of the present disclosure.

[0071] Figure 10 Generating new rendering parameters based on multiple environmental images is described by way of interpolation based on spatial position. As Figure 10 shown, the acquisition positions 420 and 440 near the device position 510 can be determined based on the index described above. For example, the standard environmental image 730 of the environmental image 220 at the acquisition position 420 can be obtained, and further the corresponding spherical harmonic lighting parameter vector 1010 can be obtained. Similarly, the standard environmental image 1022 of the environmental image 1020 at the acquisition position 440 can be obtained, and further the corresponding spherical harmonic lighting parameter vector 1024 can be obtained. At this time, the interpolation 1030 based on spatial position can be determined based on the spherical harmonic lighting parameter vectors 1010 and 1024. With respect to the spherical harmonic lighting parameter vectors 1010 and 1024, this interpolation 1030 takes into account the lighting information at the two acquisition positions 420 and 440, and thus can more accurately simulate the surrounding lighting information at the device position 510.

[0072] According to an exemplary implementation of the present disclosure, the interpolation 1030 can be determined based on various methods. For example, the interpolation between two or more vectors can be determined based on any one of nearest neighbor interpolation, linear interpolation, and bilinear interpolation. In a simple example, the interpolation 1030 can be determined based on the average of each vector. Alternatively and / or additionally, the distance between each acquisition position and the device position 510 can be utilized, and the interpolation 1030 can be determined based on a weighted average method. Subsequently, the interpolation 1030 can be used as the new spherical harmonic lighting parameter vector and as the rendering parameter of the renderer.

[0073] It will be understood that Figure 10Only one example of generating new rendering parameters based on multiple environmental images is schematically shown. Alternatively and / or additionally, occlusion relationships may be further considered. For example, acquisition positions where there are no obstacles between the acquisition position and the device position may be selected based on the method described above. Assuming there is an obstacle between the acquisition position 440 and the device position 510, environmental images at other acquisition positions may be selected.

[0074] It will be understood that Figure 10 The process of generating new rendering parameters using multiple environmental images is only described by taking the spatial-based interpolation 1030 as an example. Alternatively and / or additionally, time-based interpolation may be obtained in a similar manner. For example, for the spherical harmonic lighting parameter vectors at different time points, interpolation may be performed again in the time dimension (e.g., using linear interpolation) to simulate the surrounding lighting information at a time point closer to the current time point. Assuming that there are only environmental images at 8 am and 12 pm currently, and the current time point is 10 am. At this time, interpolation may be determined based on the relevant spherical harmonic lighting parameter vectors of the environmental images at 8 am and 12 pm, and this interpolation may be used as the new rendering parameters for rendering the virtual object.

[0075] Using the exemplary implementation of the present disclosure, interpolation may be determined based on the spherical harmonic lighting parameter vectors related to multiple environmental images respectively within the time and / or space range. In this way, the interpolation may consider more environmental images within the time and / or space range, and thus obtain surrounding lighting information that is more matched to the current time and / or device position. When rendering is performed using the interpolation, the rendering lighting effect of the virtual object 250 may be more matched to the real environment around the user.

[0076] The specific process for rendering the virtual object 250 in the augmented reality application 240 has been described above. In the following, refer to Figure 11A and 11B to provide specific rendering effects. The Figure 11A and 11B respectively show block diagrams 1100A and 1100B of presenting the virtual object 250 in the augmented reality application 240 according to some implementations of the present disclosure. Specifically, FIG. 1100A schematically shows the effect of a user using the augmented reality application 240 during the day. At this time, the virtual object 250 may be rendered based on the environmental images acquired during the day at the acquisition positions close to the device position. As Figure 11A shown, at this time, the surface of the virtual object 250 is consistent with the daytime lighting of the real environment and exhibits a relatively bright lighting effect.

[0077] Further, FIG. 1100B schematically shows the effect of a user using the augmented reality application 240 at night. At this time, the virtual object 250 can be rendered based on the night environment image collected at the collection position near the device location. As Figure 11B shown, at this time, the surface of the virtual object 250 is consistent with the night lighting of the real environment and exhibits a relatively dim lighting effect.

[0078] It will be understood that although the rendering process has been described above with a cube as a specific example of the virtual object. According to an exemplary implementation of the present disclosure, the virtual object can represent other objects. For example, in an AR-based street view navigation application, the virtual object can represent virtual signs, virtual mascots, etc. of street stores; in an AR-based game application, the virtual object can represent virtual props, virtual characters, etc.; in an AR-based shopping application, the virtual object can represent virtual clothes being tried on, etc.

[0079] Using the exemplary implementation of the present disclosure, multiple environment images (i.e., panoramic images) collected for VPS purposes can be reused, and without additional data collection overhead, the visual effect of virtual object rendering can be improved. Further, by normalizing the collected panoramic images, the panoramic images collected at different angles can be converted into standard environment images in the standard direction (i.e., (0, 0, 0)). Thus, the standard environment images can be directly used for rendering without further coordinate transformation.

[0080] Alternatively and / or additionally, in order to further improve the rendering performance, the spherical harmonic lighting parameter vector can be extracted from the standardized standard environment image. Alternatively and / or additionally, interpolation processing can be performed on the spherical harmonic lighting parameter vectors related to multiple environment images within a time and / or space range to obtain lighting parameters that more match the current time and / or device location. In this way, the lighting information around the current time point and the current device location can be simulated, and thus the rendered virtual object can be more consistent with the surrounding real environment.

[0081] Example process

[0082] Figure 12 FIG. shows a flowchart of a method 1200 for rendering a virtual object according to some implementations of the present disclosure. Specifically, at block 1210, multiple environment images of the real environment are obtained, and the multiple environment images include images of the real environment collected at multiple collection time points. At block 1220, based on the multiple collection time points, an environment image that matches the current time point when the augmented reality application is used is selected from the multiple environment images as the rendering environment image. At block 1230, a virtual object rendered using the rendering environment image is presented in the augmented reality application.

[0083] According to an exemplary implementation of the present disclosure, multiple environmental images are respectively collected at multiple acquisition positions in the real environment, and selecting a rendering environmental image further includes: determining the device position of the terminal device running the augmented reality application in the real environment; and based on the multiple acquisition positions, selecting a rendering environmental image that matches the device position from the multiple environmental images.

[0084] According to an exemplary implementation of the present disclosure, the rendering environmental image that matches the device position includes: respectively determining the corresponding distances between the multiple acquisition positions of the multiple environmental images and the device position; and selecting the rendering environmental image based on the comparison of the corresponding distances.

[0085] According to an exemplary implementation of the present disclosure, selecting the rendering environmental image based on the comparison of the corresponding distances further includes: determining the spatial structure of the real environment based on the multiple environmental images; determining whether there is an obstacle between the acquisition position of the rendering environmental image and the device position based on the spatial structure; and in response to determining that there is no obstacle between the acquisition position of the rendering environmental image and the device position, selecting the rendering environmental image.

[0086] According to an exemplary implementation of the present disclosure, the method further includes: in response to determining that there is an obstacle between the acquisition position of the rendering environmental image and the device position, selecting another environmental image whose acquisition position is adjacent to the device position from the multiple environmental images as the rendering environmental image.

[0087] According to an exemplary implementation of the present disclosure, presenting a virtual object includes: obtaining the acquisition angle associated with the rendering environmental image; based on the acquisition angle, converting the rendering environmental image to a standard environmental image under a standard angle; and using the standard environmental image as an environment light map to render the virtual object.

[0088] According to an exemplary implementation of the present disclosure, converting the rendering environmental image to a standard environmental image includes: for a pixel in the rendering environmental image, determining a standard pixel corresponding to the pixel in the standard environmental image based on the acquisition angle, the spherical coordinates of the pixel in the rendering environmental image, and the standard angle.

[0089] According to an exemplary implementation of the present disclosure, rendering a virtual object includes: determining a spherical harmonic lighting parameter vector associated with the standard environmental image based on the spherical harmonic lighting model; and rendering the virtual object based on the spherical harmonic lighting parameter vector.

[0090] According to an exemplary implementation of the present disclosure, rendering a virtual object using a spherical harmonic lighting parameter vector includes: selecting another rendering environment image from a plurality of environment images based on at least any one of a device location and a current time; determining another spherical harmonic lighting parameter vector of the another rendering environment image; and rendering a virtual image based on an interpolation between the spherical harmonic lighting parameter vector and the another spherical harmonic lighting parameter vector.

[0091] According to an exemplary implementation of the present disclosure, the interpolation includes at least any one of the following: position-based interpolation, time-based interpolation.

[0092] Example devices and equipment

[0093] Figure 13 A block diagram of an apparatus 1300 for rendering a virtual object according to some implementations of the present disclosure is shown. The apparatus 1300 includes: an acquisition module 1310 configured to acquire a plurality of environment images of a real environment, the plurality of environment images including images of the real environment respectively acquired at a plurality of acquisition time points; a selection module 1320 configured to select, based on the plurality of acquisition time points, an environment image that matches a current time point at which an augmented reality application is used from the plurality of environment images as a rendering environment image; and a presentation module 1330 configured to present a virtual object rendered using the rendering environment image in the augmented reality application.

[0094] According to an exemplary implementation of the present disclosure, the plurality of environment images are respectively acquired at a plurality of acquisition positions in the real environment, and the selection module 1320 further includes: a position determination module configured to determine a device location of a terminal device running the augmented reality application in the real environment; and an image selection module configured to select, based on the plurality of acquisition positions, a rendering environment image that matches the device location from the plurality of environment images.

[0095] According to an exemplary implementation of the present disclosure, the image selection module includes: a distance determination module configured to respectively determine corresponding distances between the plurality of acquisition positions of the plurality of environment images and the device location; and a comparison module configured to select a rendering environment image based on a comparison of the corresponding distances.

[0096] According to an exemplary implementation of the present disclosure, the comparison module further includes: a structure determination module configured to determine a spatial structure of the real environment based on the plurality of environment images; a detection module configured to determine whether there is an obstacle between an acquisition position of the rendering environment image and the device location based on the spatial structure; and a selection module based on the obstacle configured to select the rendering environment image in response to determining that there is no obstacle between the acquisition position of the rendering environment image and the device location.

[0097] According to an exemplary implementation of the present disclosure, the obstacle-based selection module is further configured to: in response to determining that there is an obstacle between the acquisition position of the rendered environmental image and the device position, select another environmental image adjacent to the device position from the multiple environmental images as the rendered environmental image.

[0098] According to an exemplary implementation of the present disclosure, the presentation module 1330 includes: an angle acquisition module configured to acquire the acquisition angle associated with the rendered environmental image; a conversion module configured to convert the rendered environmental image to a standard environmental image under a standard angle based on the acquisition angle; and a rendering module configured to use the standard environmental image as an environment light map to render the virtual object.

[0099] According to an exemplary implementation of the present disclosure, the conversion module includes: a pixel determination module configured to, for a pixel in the rendered environmental image, determine a standard pixel corresponding to the pixel in the standard environmental image based on the acquisition angle, the spherical coordinates of the pixel in the rendered environmental image, and the standard angle.

[0100] According to an exemplary implementation of the present disclosure, the rendering module includes: a vector determination module configured to determine a spherical harmonic lighting parameter vector associated with the standard environmental image based on the spherical harmonic lighting model; and a virtual object rendering module configured to render the virtual object based on the spherical harmonic lighting parameter vector.

[0101] According to an exemplary implementation of the present disclosure, the selection module 1320 is further configured to select another rendered environmental image from the multiple environmental images based on at least any one of the device position and the current time; the vector determination module is further configured to determine another spherical harmonic lighting parameter vector of the another rendered environmental image; and the virtual object rendering module further includes: an interpolation-based rendering module configured to render the virtual image based on the interpolation of the spherical harmonic lighting parameter vector and the another spherical harmonic lighting parameter vector.

[0102] According to an exemplary implementation of the present disclosure, the interpolation includes at least any one of the following: position-based interpolation, time-based interpolation.

[0103] Figure 14 The block diagram of the device 1400 capable of implementing multiple implementations of the present disclosure is shown. It should be understood that Figure 14 The shown computing device 1400 is merely exemplary and should not constitute any limitation to the functions and scopes of the implementations described herein. Figure 14 The shown computing device 1400 can be used to implement the methods described above.

[0104] As Figure 14As shown, computing device 1400 is in the form of a general-purpose computing device. The components of computing device 1400 may include, but are not limited to, one or more processors or processing units 1410, a memory 1420, a storage device 1430, one or more communication units 1440, one or more input devices 1450, and one or more output devices 1460. The processing unit 1410 may be an actual or virtual processor and be capable of performing various processes according to programs stored in the memory 1420. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to enhance the parallel processing ability of computing device 1400.

[0105] Computing device 1400 generally includes multiple computer storage media. Such media can be any accessible media available to computing device 1400, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 1420 may be volatile memory (such as registers, caches, random access memory (RAM)), non-volatile memory (such as read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 1430 may be removable or non-removable media and may include machine-readable media, such as a flash drive, a magnetic disk, or any other media that can be used to store information and / or data (such as training data for training) and can be accessed within computing device 1400.

[0106] Computing device 1400 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in Figure 14 a disk drive for reading from or writing to a removable, non-volatile magnetic disk (such as a "floppy disk") and an optical disk drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. The memory 1420 may include a computer program product 1425 having one or more program modules that are configured to perform various methods or actions of various implementations of the present disclosure.

[0107] The communication unit 1440 enables communication with other computing devices through a communication medium. Additionally, the functions of the components of computing device 1400 may be implemented in a single computing cluster or multiple computer machines that are capable of communicating through a communication connection. Thus, computing device 1400 may operate in a networked environment using a logical connection to one or more other servers, network personal computers (PCs), or another network node.

[0108] The input device 1450 can be one or more input devices, such as a mouse, a keyboard, a trackball, etc. The output device 1460 can be one or more output devices, such as a display, a speaker, a printer, etc. The computing device 1400 can also communicate with one or more external devices (not shown) as needed through the communication unit 1440. The external devices such as a storage device, a display device, etc., communicate with one or more devices that enable a user to interact with the computing device 1400, or communicate with any device that enables the computing device 1400 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication can be performed via an input / output (I / O) interface (not shown).

[0109] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, and the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided. The computer program product is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0110] Aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and the combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0111] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is produced that implements the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause a computer, a programmable data processing device, and / or other devices to work in a specific manner. Thus, the computer-readable medium storing the instructions includes a manufacture, which includes instructions for implementing various aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0112] Computer-readable program instructions may be loaded onto a computer, other programmable data processing apparatus, or other device, causing a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to generate a computer-implemented process, such that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0113] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various implementations of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two consecutive blocks may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or acts, or by a combination of dedicated hardware and computer instructions.

[0114] The various implementations of the present disclosure have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The choice of terms used herein is intended to best explain the principles of the implementations, the practical application, or the improvement of technologies in the market, or to enable other ordinary skilled persons in the art to understand the various implementations disclosed herein.

Claims

1. A method for rendering a virtual object in an augmented reality application, comprising: Acquire a plurality of environment images of a real environment, wherein the plurality of environment images include images of the real environment acquired at a plurality of acquisition time points, and the plurality of environment images are acquired at a plurality of acquisition positions in the real environment, respectively; Based on the multiple acquisition time points, selecting, from the multiple environment images, the multiple environment images that match a current time point at which the augmented reality application is used; Based on the multiple acquisition positions, determining a rendering environment image that matches the device position includes: Determining a device position of a terminal device running the augmented reality application in the real environment; and respectively determining corresponding distances between a plurality of acquisition locations of the plurality of environment images matching a current time point at which the augmented reality application is used and the device location; selecting an environment image based on the comparison of the corresponding distances; determining a spatial structure of the real environment based on the multiple environment images; Determining whether there is an obstacle between a capture location of the selected environment image and a location of the device based on the spatial structure; and In response to determining that there is no obstacle between the acquisition location of the selected environment image and the device location, selecting the environment image as the rendered environment image; and The virtual object rendered using the rendered environment image is presented in the augmented reality application.

2. The method according to claim 1, further comprising: In response to determining that there is an obstacle between the acquisition position of the selected environment image and the device position, another environment image whose acquisition position is adjacent to the device position among the multiple environment images is selected as the rendered environment image.

3. The method of claim 1 , wherein presenting the virtual object comprises: Acquiring a capture angle associated with the rendering environment image; Based on the acquisition angle, converting the rendering environment image to a standard environment image at a standard angle; as well as The standard environment image is used as an ambient light map to render the virtual object.

4. The method according to claim 3, wherein converting the rendered environment image to the standard environment image comprises: For each pixel in the rendered environment image, Based on the acquisition angle, the spherical coordinates of the pixel in the rendering environment image, and the standard angle, a standard pixel in the standard environment image corresponding to the pixel is determined.

5. The method of claim 3, wherein rendering the virtual object comprises: Determine a spherical harmonic illumination parameter vector associated with the standard environment image based on a spherical harmonic illumination model; as well as The virtual object is rendered based on the spherical harmonic lighting parameter vector.

6. The method according to claim 5, wherein rendering the virtual object using the spherical harmonic lighting parameter vector comprises: selecting another rendered environment image from the plurality of environment images based on at least any one of the device location and the current time; Determine another spherical harmonic lighting parameter vector of the another rendered environment image; as well as The virtual object is rendered based on an interpolation of the spherical harmonic lighting parameter vector and the another spherical harmonic lighting parameter vector. The method according to claim 6 , wherein the interpolation comprises at least any one of the following: position-based interpolation, time-based interpolation.

8. An apparatus for rendering a virtual object in an augmented reality application, comprising: An acquisition module is configured to acquire a plurality of environment images of a real environment, wherein the plurality of environment images include images of the real environment acquired at a plurality of acquisition time points, and the plurality of environment images are acquired at a plurality of acquisition positions in the real environment, respectively; A selection module configured to select, from the plurality of environment images, the plurality of environment images that match a current time point at which the augmented reality application is used, based on the plurality of acquisition time points; A determination module is configured to determine a rendering environment image matching a device position based on the multiple acquisition positions, including: a position determination module, configured to determine a device position of a terminal device running the augmented reality application in the real environment; and a distance determination module configured to respectively determine corresponding distances between a plurality of acquisition locations of the plurality of environment images matching a current time point at which the augmented reality application is used and the device location; A comparison module is configured to select an environment image based on the comparison of the corresponding distances; the comparison module comprises: a structure determination module, configured to determine the spatial structure of the real environment based on the multiple environment images; a detection module configured to determine whether there is an obstacle between a capture position of the selected environment image and a position of the device based on the spatial structure; and An obstacle-based selection module configured to select the environment image as the rendered environment image in response to determining that there is no obstacle between the acquisition location of the selected environment image and the device location; and A rendering module is configured to present the virtual object rendered using the rendering environment image in the augmented reality application.

9. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 7 when executed by the at least one processing unit.

10. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the processor is caused to implement the method according to any one of claims 1 to 7.

Images