Image processing method and electronic device

Abstract

The embodiment of the present application provides an image processing method and an electronic device. The method includes: in response to a received first user operation, mapping a panoramic image of a visiting point corresponding to the first user operation to a first model, wherein the panoramic image is rendered based on a second model, and the fineness of the second model is higher than that of the first model; then rendering is performed based on the first model after mapping the panoramic image to obtain an image of the visiting point and an image showing the visiting point. In this way, through the mapping of the panoramic image, the image rendered by the first model can have the same image quality as the panoramic image rendered by the second model, and then the image rendered based on the first model mapping the panoramic image also has high image quality, and can also provide users with a strong immersive experience.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing, and in particular to an image processing method and electronic device. Background Art

[0002] With the vigorous development of information technology and the digital economy, many traditional offline businesses are gradually transforming into digital ones. For example, traditional exhibition halls are gradually transforming into digital cloud exhibition halls.

[0003] One existing digital technology is to switch and display high-definition exhibition hall pictures according to the user's position during the exhibition hall visit, but the pictures are two-dimensional and cannot create a three-dimensional feeling for the user, and the user's sense of immersion is poor. Another digital technology is to download the three-dimensional model on the terminal side, and then render the corresponding image according to the user's position during the exhibition hall visit; however, the computing power and bandwidth on the terminal side are limited, so only relatively simple three-dimensional models can be downloaded on the terminal side, and the image quality (brightness and darkness changes of the exhibition hall, light and shadow effects of the exhibition hall, etc.) of the image rendered by the simple three-dimensional model is poor. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides an image processing method and electronic device. In the method, a panoramic image is fused with a three-dimensional model to render an image, which can obtain an image with high image quality and can bring a strong immersive experience to the user.

[0005] In the first aspect, the embodiment of the present application provides an image processing method, which is applied to an electronic device, and the method includes: in response to a received first user operation, mapping a panoramic image of a visit point corresponding to the first user operation to a first model, wherein the panoramic image is rendered based on a second model, and the fineness of the second model is higher than that of the first model; then rendering is performed based on the first model after mapping the panoramic image, to obtain an image of the visit point and an image showing the visit point. In this way, through the mapping of the panoramic image, the image rendered by the first model can be made to have the same image quality as the panoramic image rendered by the second model, and then the image rendered based on the first model of the mapped panoramic image also has high image quality, and can also bring a strong immersive experience to the user. In addition, the first model is a relatively simple three-dimensional model, so the amount of calculation required for rendering based on the first model is small, thereby reducing the performance requirements for the client. And the time required for rendering based on the simple three-dimensional model is short, thereby shortening the delay of the client displaying the exhibition hall image, and improving the user's online experience of visiting the exhibition hall.

[0006] According to the first aspect, when the first user operation is an operation of entering a tour interface, and the tour point is the first tour point, in response to the received first user operation, the panorama of the tour point corresponding to the first user operation is mapped to the first model, including: in response to the received first user operation, downloading the first model and the panorama of the first tour point from the server; creating a three-dimensional scene, loading the first model into the three-dimensional scene, and setting the camera at the first tour point; mapping the panorama of the first tour point to the sky box of the three-dimensional scene; taking the location of the camera as the endpoint, respectively drawing rays to multiple vertices P1 in the first model, intersecting with the sky box at P2; mapping the rendering and shading information of P2 to P1 belonging to the same ray as P2. Since the panorama is rendered based on the second model, each pixel in the panorama corresponds to a vertex in the second model. In this way, the rendering and shading information of the vertex in the second model can be mapped to the corresponding vertex in the first model, so that the rendering and shading information of the vertex in the first model and the corresponding vertex in the second model are the same.

[0007] According to the first aspect, or any implementation of the first aspect above, rendering is performed based on the first model after mapping the panorama to obtain an image of the visit point, including: setting the direction of the camera to the preset visit direction of the first visit point, and rendering is performed based on the direction of the camera and the first model after mapping the panorama to obtain an image of the first visit point in the preset visit direction. In this way, after entering the visit interface, the user can view the image of the first visit point in the preset visit direction.

[0008] According to the first aspect, or any implementation method of the first aspect above, with the camera position as the endpoint, rays are respectively drawn to multiple vertices P1 in the first model, intersecting with the sky box at P2, including: with the camera position as the endpoint, rays are respectively drawn to multiple vertices P1 in the first model located in the observation area, intersecting with the sky box at P2; wherein the vertices in the observation area can be observed by the camera.

[0009] Exemplarily, the observation area may refer to an area that can be observed by a camera. Alternatively, the observation area may be an area between a near plane and a far plane in a viewing cone.

[0010] According to the first aspect, or any implementation method of the first aspect above, with the camera position as the endpoint, rays are drawn to multiple vertices P1 in the observation area in the first model, and intersect with the sky box at P2, including: with the camera position as the endpoint, rays are drawn to multiple vertices P1 in the observation area corresponding to each visiting direction of the camera position in the first model, and intersect with the sky box at P2.

[0011] According to the first aspect, or any implementation of the first aspect above, with the location of the camera as an endpoint, rays are respectively drawn to the multiple vertices P1 in the observation area in the first model, and intersect with the sky box at P2, including: with the location of the camera as an endpoint, rays are respectively drawn to the multiple vertices P1 in the observation area corresponding to the preset visiting direction at the location of the camera in the first model, and intersect with the sky box at P2. Compared with drawing rays to the multiple vertices P1 in the observation area corresponding to each visiting direction at the location of the camera in the first model, in this way, rays are not drawn to the multiple vertices P1 in the observation area corresponding to other visiting directions at the location of the camera in the first model, so that the image in the preset visiting direction of the first visiting point can be rendered faster later.

[0012] According to the first aspect, or any implementation method of the first aspect above, when the first user operation is a visiting point switching operation, and the visiting point is the Kth visiting point other than the first visiting point, in response to the received first user operation, the panoramic image of the visiting point corresponding to the first user operation is mapped to the first model, including: in response to the received first user operation, downloading the panoramic image of the Kth visiting point from the server; moving the camera from the Jth visiting point to the Kth visiting point, and performing position correction on the vertices in the first model, the Jth visiting point is a visiting point other than the Kth visiting point; mapping the panoramic image of the Kth visiting point to the sky box of the three-dimensional scene; taking the camera position as the endpoint, respectively making rays to multiple vertices P1 in the first model, intersecting with the sky box at P2; mapping the rendering shading information of P2 to P1 belonging to the same ray as P2.

[0013] According to the first aspect, or any implementation method of the first aspect above, rendering is performed based on the first model after mapping the panorama to obtain an image of the visiting point, including: setting the direction of the camera to a preset visiting direction corresponding to the Kth visiting point, and rendering is performed based on the direction of the camera and the first model after mapping the panorama to obtain an image of the Kth visiting point in the preset visiting direction.

[0014] According to the first aspect, or any implementation method of the first aspect above, the positions of the vertices in the first model are corrected, including: determining the camera motion offset information corresponding to moving the camera from the Jth visiting point to the Kth visiting point based on the coordinate information of the Jth visiting point and the coordinate information of the Kth visiting point; based on the camera motion offset information, reversely correcting the coordinate information of the vertices in the first model when the camera is located at the Jth visiting point.

[0015] According to the first aspect, or any implementation of the first aspect above, after displaying the image of the visiting point, the method also includes: in response to a received second user operation, rendering based on the user visiting direction corresponding to the second user operation and the first model after mapping the panoramic image, to obtain an image of the visiting point in the user visiting direction.

[0016] In a second aspect, an embodiment of the present application provides an image processing method, which is applied to an electronic device, and the method includes: in response to a received first user operation, downloading a tile image in the visiting direction of a visiting point corresponding to the first user operation from a server, the tile image is obtained by tile segmentation of a panoramic image of the visiting point obtained by rendering based on a second model, and the fineness of the second model is higher than that of the first model; then mapping the tile image to the first model; rendering based on the first model after mapping the tile image, obtaining an image in the visiting direction of the visiting point, and displaying the image in the visiting direction of the visiting point. In this way, through the mapping of the tile image, the image of a certain visiting direction rendered by using the first model can be made to have the same image quality as the image in the corresponding visiting direction in the panoramic image rendered by using the second model, and then the image rendered by the first model based on the mapping tile image also has high image quality, and can also bring a strong immersive experience to the user. In addition, the first model is a relatively simple three-dimensional model, so the amount of calculation required for rendering based on the first model is small, thereby reducing the performance requirements for the client. And the time required for rendering based on the simple three-dimensional model is short, thereby shortening the delay of the client displaying the exhibition hall image, and improving the user's online experience of visiting the exhibition hall. In addition, compared with downloading the entire panorama, it takes less time to download some tiles in the panorama, which can reduce the delay in displaying images to users and improve user experience.

[0017] According to the second aspect, when the first user operation is an operation to enter a visiting interface and the visiting point is the first visiting point, in response to the received first user operation, a tile map in a visiting direction of the visiting point corresponding to the first user operation is downloaded from a server, including: in response to the received first user operation, a first model is downloaded from a server; a three-dimensional scene is created, the first model is loaded into the three-dimensional scene, and a camera is set at the visiting point; a reference cube is created with the camera position as the center; according to the preset visiting direction and the reference cube corresponding to the visiting point, the coordinate information of multiple tile maps to be downloaded in the panoramic map corresponding to the visiting point is determined; according to the coordinate information, the tile map in the visiting direction of the visiting point is downloaded from the server.

[0018] According to the second aspect, or any implementation method of the second aspect above, when the first user operation is a visiting point switching operation, and the visiting point is a visiting point other than the first visiting point, in response to the received first user operation, a tile image in the visiting direction of the visiting point corresponding to the first user operation is downloaded from the server, including: setting the camera at the visiting point; creating a reference cube with the camera position as the center; determining the coordinate information of multiple tile images to be downloaded in the panoramic map corresponding to the visiting point according to the preset visiting direction and the reference cube corresponding to the visiting point; and downloading the tile image in the visiting direction of the visiting point from the server according to the coordinate information.

[0019] According to the second aspect, or any implementation method of the second aspect above, according to the preset visiting direction and the reference cube corresponding to the visiting point, the coordinate information of the multiple tile images to be downloaded in the panoramic map corresponding to the visiting point is determined, including: according to the preset visiting direction and the reference cube corresponding to the visiting point, the cross-section of the reference cube and the camera's viewing cone is determined; and the vertices in the first model that are located between the cross-section inside the viewing cone and the far plane of the viewing cone are projected and transformed to obtain the coordinate information of the multiple tile images to be downloaded in the panoramic map corresponding to the visiting point.

[0020] According to the second aspect, or any implementation method of the second aspect above, when the visiting point is the first visiting point, the tile image is mapped to the first model, including: mapping the tile image in the visiting direction of the first visiting point to the sky box of the three-dimensional scene; taking the camera position as the endpoint, making rays to multiple vertices P1 in the first model respectively, intersecting the sky box at P2; mapping the rendering shading information of P2 to P1 belonging to the same ray as P2.

[0021] According to the second aspect, or any implementation method of the second aspect above, when the visiting point is the Kth visiting point other than the first visiting point, mapping the tile image to the first model includes: moving the camera from the Jth visiting point to the Kth visiting point, and performing position correction on the vertices in the first model, the Jth visiting point being a visiting point other than the Kth visiting point; mapping the tile image in the visiting direction of the Kth visiting point to the sky box of the three-dimensional scene; taking the position of the camera as the endpoint, respectively making rays to multiple vertices P1 in the first model, intersecting the sky box at P2; mapping the rendering shading information of P2 to P1 belonging to the same ray as P2.

[0022] According to the second aspect, or any implementation method of the second aspect above, the positions of the vertices in the first model are corrected, including: determining the camera motion offset information corresponding to moving the camera from the Jth visiting point to the Kth visiting point based on the coordinate information of the Jth visiting point and the coordinate information of the Kth visiting point; based on the camera motion offset information, reversely correcting the coordinate information of the vertices in the first model when the camera is located at the Jth visiting point.

[0023] According to the second aspect, or any implementation method of the second aspect above, with the camera position as the endpoint, rays are drawn to multiple vertices P1 in the first model, intersecting the sky box at P2, including: with the camera position as the endpoint, rays are drawn to multiple vertices P1 in the first model in the viewing direction corresponding to the observation area located at the camera position, and intersecting the sky box at P2.

[0024] According to the second aspect, or any implementation method of the second aspect above, rendering is performed based on the first model after mapping the panorama to obtain an image of the visiting point, including: setting the direction of the camera to the visiting direction of the visiting point corresponding to the first user operation, and rendering is performed based on the direction of the camera and the first model after mapping the panorama to obtain an image in the preset visiting direction of the visiting point.

[0025] According to the second aspect, or any implementation of the second aspect above, after rendering based on the first model after mapping the panorama to obtain the image of the visiting point, the method also includes: downloading tile images of other visiting directions of the visiting point from the server.

[0026] According to the second aspect, or any implementation of the second aspect above, after downloading tile images of other visiting directions of the visiting point from the server, the method also includes: mapping the tile images of other visiting directions of the visiting point to the sky box of the three-dimensional scene; taking the camera position as the endpoint, making rays to multiple vertices P1 in the observation area corresponding to other visiting directions located at the camera position in the first model, and intersecting with the sky box at P2; mapping the rendering shading information of P2 to P1 belonging to the same ray as P2.

[0027] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0028] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the memory is coupled to the processor; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the image processing method in the first aspect or any possible implementation of the first aspect.

[0029] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0030] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the memory is coupled to the processor; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the image processing method in the second aspect or any possible implementation of the second aspect.

[0031] The fourth aspect and any implementation of the fourth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, which will not be repeated here.

[0032] In a fifth aspect, an embodiment of the present application provides a chip comprising one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send a signal to the processor, the signal comprising a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the image processing method in the first aspect or any possible implementation of the first aspect.

[0033] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0034] In a sixth aspect, an embodiment of the present application provides a chip comprising one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send a signal to the processor, the signal comprising a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the image processing method in the second aspect or any possible implementation of the second aspect.

[0035] The sixth aspect and any implementation of the sixth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the sixth aspect and any implementation of the sixth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, which will not be repeated here.

[0036] In the seventh aspect, an embodiment of the present application provides a computer storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the image processing method in the first aspect or any possible implementation of the first aspect.

[0037] The seventh aspect and any implementation of the seventh aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the seventh aspect and any implementation of the seventh aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0038] In an eighth aspect, an embodiment of the present application provides a computer storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the image processing method in the second aspect or any possible implementation of the second aspect.

[0039] The eighth aspect and any implementation of the eighth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the eighth aspect and any implementation of the eighth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a schematic diagram of an application scenario shown as an example;

[0041] Figure 2 is a schematic diagram of a three-dimensional model shown as an example;

[0042] Figure 3 A rendering diagram is shown for exemplary purposes;

[0043] Figure 4 A storage schematic diagram is shown as an example;

[0044] Figure 5 A schematic diagram of a processing process is shown as an example;

[0045] Figure 6 is a schematic diagram of a viewing cone shown as an example;

[0046] Figure 7 is a schematic diagram of rays shown as an example;

[0047] Figure 8 is a schematic diagram of rays shown as an example;

[0048] Fig. 9 is a schematic diagram of an exhibition hall image shown as an example;

[0049] Fig.10 A schematic diagram of a processing process is shown as an example;

[0050] Fig.11 A schematic diagram of a processing process is shown as an example;

[0051] Fig.12 is a schematic diagram of an exhibition hall image shown as an example;

[0052] Fig.13 A rendering diagram is shown for exemplary purposes;

[0053] Fig.14 A storage schematic diagram is shown as an example;

[0054] Fig.15 A schematic diagram of a processing process is shown as an example;

[0055] Fig.16 is a schematic diagram of a reference cube shown as an example;

[0056] Fig.17 is a schematic diagram showing the positional relationship between the face of a cube and the near plane;

[0057] Fig.18 A schematic diagram of a processing process is shown as an example;

[0058] Fig.19 A schematic diagram of a processing process is shown as an example;

[0059] Fig. 20 Schematic diagram of the structure of the device shown as an example. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0061] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0062] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0063] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0064] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0065] The embodiments of the present application can be applied to various digital scenarios, including but not limited to: digital exhibition halls (also known as cloud exhibition halls), online home improvement, digital twins, 5GtoB (5th Generation Mobile Communication Technology to Business, the fifth generation of mobile communication technology for the industry) solutions (for example, 5G steel automation, 5G remote assisted assembly, 5G steel surface quality inspection, 5G+ smart ports, 5G+ smart coal mines, etc.), etc. This application takes a digital exhibition hall as an example for illustration.

[0066] Figure 1 The figure is a schematic diagram of an exemplary application scenario.

[0067] For example, Figure 1 (1) in (a) is an exhibition hall, that is, a real exhibition hall that users can visit. For example, the exhibition hall can be composed of exhibition areas, exhibition islands, exhibition stands and exhibition screens ( Figure 1 (not shown in (a)). Among them, an exhibition hall may contain several exhibition areas, each of which corresponds to a theme. An exhibition island is a subset of different exhibitors in an exhibition area. An exhibition area will contain several exhibition islands, and each exhibition island represents an exhibitor. A booth is a subset of different theme contents exhibited by the same exhibitor in an exhibition island, and an exhibition island contains multiple booths. The display screen is the smallest exhibition unit of the exhibition hall, and each booth may contain multiple display screens. In view of the structural characteristics of the exhibition hall, several visiting points can be set up for the exhibition hall, and each visiting point can be used to visit nearby booths. Several visiting points can be strung together to form a tour route of the exhibition hall. Users can visit all the booths in the exhibition hall by following the tour route.

[0068] Exemplarily, corresponding digital data of the exhibition hall is created in advance based on the locations of exhibition areas, exhibition islands, booths and exhibition screens in the exhibition hall, the visiting route of the exhibition hall, and the objects that can be visited at each visiting point in the visiting route.

[0069] For example, Figure 1(2) in (a) is a server. Exemplarily, after obtaining the exhibition hall digitized data, first association information between the exhibition hall digitized data and the corresponding exhibition hall can be established, wherein the first association information can include the relationship between the data identifier of the exhibition hall digitized data and the exhibition hall identifier of the corresponding exhibition hall. The exhibition hall identifier is used to uniquely identify the exhibition hall, and the data identifier is used to uniquely identify the exhibition hall digitized data of an exhibition hall. Then, the first association information and the exhibition hall digitized data can be stored in the server.

[0070] For example, Figure 1 (3) in (b) is a client, which is provided with a browser or a cloud exhibition hall application, and the browser or the cloud exhibition hall application is used to provide an exhibition hall visit interface. The present application embodiment takes a browser as an example for illustrative description. For example, a user can enter the exhibition hall visit interface from a browser and visit the exhibition hall online.

[0071] For example, after receiving the user's operation of entering the visit interface, the browser may execute the following S101 to S103 in response to the user's operation behavior:

[0072] S101, the browser receives a user operation.

[0073] For example, the user can start the browser in the client, and the client starts the browser in response to the received user operation. After the browser is started, the user can enter the exhibition hall's visit website in the browser and enter the visit interface, and the browser can receive the user operation.

[0074] S102: The browser sends a request message to the server in response to the user operation.

[0075] Exemplarily, after receiving the user operation, the browser may generate a request message in response to the user operation. The request message is used to instruct to download the digital data of the exhibition hall from the server, and the request message includes the exhibition hall identifier. Then the browser sends the request message to the corresponding server according to the URL of the cloud exhibition hall.

[0076] S103, the browser downloads the digital data of the exhibition hall.

[0077] Exemplarily, after receiving the request information, the server can parse the request information and extract the exhibition hall identifier from the request information. Then, the server searches for the data identifier corresponding to the exhibition hall identifier in the request information based on the pre-stored first association information, and then determines the exhibition hall digitized data corresponding to the data identifier as the exhibition hall digitized data requested by the request information. Subsequently, the server sends the exhibition hall digitized data requested by the request information to the browser, and the browser can then download the exhibition hall digitized data.

[0078] Exemplarily, the client is also provided with a three-dimensional rendering engine. Exemplarily, after the browser downloads the digital data of the exhibition hall, the three-dimensional rendering engine can be called to process the digital data of the exhibition hall to generate a corresponding exhibition hall image; then the browser displays the exhibition hall image on the visit interface. Then, the user can view the exhibition hall image through the browser and realize online visit to the exhibition hall.

[0079] For example, later, the user can switch the visiting points in the browser to move from one visiting point to another for visiting, and can change the visiting direction in the browser to visit in different visiting directions at the same visiting point.

[0080] For example, if the browser has downloaded the digital data of the exhibition hall corresponding to the switched visiting point or the changed visiting direction from the server before the user executes the operation of switching visiting points or changing the visiting direction, then after receiving the user operation, the browser can respond to the user operation and directly call the three-dimensional rendering engine to process based on the digital data of the exhibition hall, generate an exhibition hall image corresponding to the switched visiting point or the changed visiting direction by the user, and then display the exhibition hall image corresponding to the switched visiting point or the changed visiting direction by the user on the visiting interface.

[0081] For example, if the browser has not downloaded the digital data of the exhibition hall corresponding to the switched viewing point or the changed viewing direction from the server before the user switches the viewing point or changes the viewing direction, the browser can execute S102 to S103 in response to the user operation after receiving the user operation, and then call the 3D rendering engine to process based on the digital data of the exhibition hall to generate the exhibition hall image corresponding to the switched viewing point or the changed viewing direction. Then the browser displays the exhibition hall image corresponding to the switched viewing point or the changed viewing direction on the viewing interface.

[0082] The following pairs are Figure 1 The process of creating the digital data of the corresponding exhibition hall for the exhibition hall (1) in (a) is explained.

[0083] Figure 2 Schematic diagram of a three-dimensional model is shown as an example. Exemplarily, the three-dimensional model includes a plurality of vertices and data associated with the vertices, such as vertex coordinates, texture mapping, etc.

[0084] Reference Figure 2 , exemplarily, a three-dimensional model corresponding to the exhibition hall is constructed. In one possible approach, two three-dimensional models are constructed, such as Figure 2 (1) and Figure 2 (2) shows the two models. For the convenience of subsequent explanation, Figure 2 The three-dimensional model in (1) is called the first model. Figure 2The three-dimensional model in (2) is called the second model. Exemplarily, the first model is used to render an exhibition hall image for display in a browser. Exemplarily, the second model is used to generate a panoramic image.

[0085] For example, Figure 2 The first model in (1) may refer to a model whose model data has a precision less than a first preset precision, wherein the first preset precision may be set as required, and the present application embodiment does not limit this. Exemplarily, the model precision may include horizontal precision and vertical precision, wherein the horizontal precision may refer to the number of vertices. The vertical precision may refer to the precision corresponding to the data associated with the vertex, for example, the precision of the vertex coordinates, the resolution of the texture map, etc.

[0086] For example, Figure 2 The second model in (2) may refer to a model whose model fineness is greater than a second preset fineness, wherein the second preset fineness may be set as required, and the embodiments of the present application are not limited thereto. Optionally, the model fineness of the second model may be the highest fineness supported by the client. Exemplarily, the second preset fineness may be greater than or equal to the first preset fineness. For example, the number of vertices in the first model is less than the number of vertices in the second model. For another example, the precision of the vertex coordinates in the first model is less than the precision of the vertex coordinates in the second model. For another example, the resolution of the texture map in the first model is less than the resolution of the texture map in the second model, and so on.

[0087] That is, the fineness of the second model is greater than that of the first model. Figure 2 It can be seen that Figure 2 The first model (1) is relative to Figure 2 The second model (2) is relatively rough. For example, Figure 2 The first model (1) has no Figure 2 In the second model (2), the exhibition hall wall 201. For another example, Figure 2 The exhibition hall pillar 203 of the first model (1) is relative to Figure 2 The exhibition hall pillars 202 of the second model (2) are relatively rough.

[0088] It should be noted that the greater the precision, the larger the data volume of the model, that is, the data volume of the second model is greater than that of the first model.

[0089] In one possible approach, a second model corresponding to the exhibition hall may be constructed first, and then based on the second model, the second model may be simplified according to the first preset precision and the second preset precision to obtain the first model.

[0090] For example, a second model corresponding to the exhibition hall may be constructed based on information such as the location, size, shape, color, etc. of the exhibition areas, exhibition islands, exhibition stands, and exhibition screens in the exhibition hall.

[0091] Exemplarily, 3DMAX software (three-dimensional animation rendering and production software) can be used to construct the second model and the first model.

[0092] Exemplarily, during the process of constructing the second model and the first model, the visiting points may be identified in the second model and the first model.

[0093] Figure 3 A rendering diagram is shown for exemplary purposes.

[0094] Reference Figure 3 , exemplarily, based on the second model (such as Figure 3 (1)), a panoramic image corresponding to each visiting point in the exhibition hall is rendered. The panoramic image of each visiting point may refer to a 360° image generated around the exhibition hall with the visiting point as the center. Figure 3 As shown in (2), Figure 3 (2) is a panoramic view corresponding to a certain visiting point.

[0095] Exemplarily, the coordinates of each pixel point in the panoramic image are coordinates in a spherical coordinate system (two-dimensional coordinates).

[0096] Exemplarily, a new three-dimensional scene can be created, and the second model can be loaded into the three-dimensional model, and a light source and a camera can be added to the three-dimensional scene. Exemplarily, a light source can be added to the three-dimensional scene, that is, lighting information such as the light source position, lighting intensity, lighting direction, and light color can be added to the second model of the three-dimensional scene. Exemplarily, a camera can be added to the three-dimensional scene, that is, the camera position, viewing angle, near plane, far plane, and aspect ratio and other parameters can be set in the second model of the three-dimensional scene. Exemplarily, the camera can be set at each visiting point in the second model in turn. After the camera is set at each visiting point, rendering based on the second model can be performed to obtain a panoramic view corresponding to the visiting point.

[0097] In a possible implementation, after the camera is set at each visiting point, the camera can be rotated to render a panoramic view of each visiting point.

[0098] In a possible implementation, after the camera is set at each visiting point, the second model can be rotated to render a panoramic view of each visiting point.

[0099] Figure 4 FIG. 1 is a schematic diagram of an exemplary storage system. Figure 4 For example, after obtaining the panorama corresponding to each visiting point, the panorama corresponding to each visiting point (such as Figure 4(2), where N is the number of visiting points included in the exhibition hall tour route, and N is a positive integer) and the first model (such as Figure 4 (1)), the digital data of the exhibition hall corresponding to the exhibition hall is stored in the server (as shown in Figure 4 (3) shown).

[0100] Exemplarily, after obtaining the panorama corresponding to each visiting point, second association information between the visiting point and the corresponding panorama can be established. Exemplarily, second association information between the point identifier of the visiting point and the image identifier of the corresponding panorama can be established, and the second association information can be stored in the server. The point identifier is used to uniquely identify a visiting point, and the image identifier is used to uniquely identify a panorama.

[0101] Exemplarily, different exhibition halls correspond to different first models. The first model has a corresponding model identifier, and the model identifier is used to uniquely identify the model.

[0102] In one possible approach, panoramic images and three-dimensional models are integrated to realize the digitization of the exhibition hall, so as to show users high-quality exhibition hall images during their online visit to the exhibition hall and increase the user's sense of immersion in the online visit to the exhibition hall.

[0103] Figure 5 FIG. 1 is a schematic diagram of an exemplary processing process. Figure 5 , Figure 5 These are the steps that the browser performs in response to the user's operation after the user enters the viewing interface, and the steps that the 3D rendering engine performs after receiving the browser call.

[0104] S501, the browser receives an operation of entering a visit interface.

[0105] For example, the user can execute an operation of starting a browser in the client, and the client can start the browser in response to the received user operation. After the browser is started, the user can enter the exhibition hall's visiting URL in the browser and execute an operation of entering the visiting interface, and then the browser can receive the operation of entering the visiting interface.

[0106] S502, the browser downloads the first model and the panoramic image of the first visiting point.

[0107] For example, after entering the exhibition hall, the user usually starts to visit the exhibition hall from the first viewing point of the exhibition hall, so the browser can first show the user the exhibition hall image corresponding to the first viewing point. The embodiment of the present application renders the exhibition hall image based on the fusion of the panoramic image and the three-dimensional model. Therefore, after the browser receives the operation of entering the viewing interface, it can generate the first download request information in response to the operation of entering the viewing interface, wherein the first download request information is used to instruct to download the first model and the panoramic image of the first viewing point. For the sake of convenience, the first viewing point will be referred to as the first viewing point in the following.

[0108] Exemplarily, the first download request information includes a model identifier of the first model and a point identifier of the first visiting point.

[0109] Exemplarily, the browser may send the first download request information to the server. After receiving the first download request information, the server may parse the first download request information and extract the model identifier and the point identifier from the first download request information. The server then searches for the first model that matches the extracted model identifier, and determines the found first model as the first model requested by the first download request information, and based on the pre-established second association information, searches for the image identifier corresponding to the extracted point identifier, and determines the panoramic image corresponding to the image identifier as the panoramic image requested by the first download request information (that is, the panoramic image of the first visiting point). Subsequently, the server returns the found first model and the panoramic image of the first visiting point to the browser, and the browser can receive the first model and the panoramic image of the first visiting point.

[0110] Exemplarily, after the browser completes downloading the first model and the panoramic image of the first visiting point, it can load the first model and the panoramic image of the first visiting point into the graphics card memory (hereinafter referred to as video memory), and call the three-dimensional rendering engine to render the exhibition hall image corresponding to the first visiting point.

[0111] Exemplarily, when calling the 3D rendering engine, the browser may pass the storage addresses of the first model and the panoramic image of the first visiting point in the video memory, as well as the point identifier of the first visiting point, to the 3D rendering engine.

[0112] Exemplarily, after receiving the call from the browser, the 3D rendering engine may execute S503 to S509:

[0113] S503, the 3D rendering engine creates a 3D scene.

[0114] S504: The 3D rendering engine loads the first model into the 3D model.

[0115] S505: The 3D rendering engine sets the camera at the first visiting point in the first model.

[0116] Exemplarily, a 3D rendering engine can create a 3D scene.

[0117] Exemplarily, a 3D rendering engine may create a 3D scene in a world space.

[0118] Exemplarily, the 3D rendering engine can determine the coordinates corresponding to the first visiting point based on the point identifier transmitted by the browser, and then set the camera to the location of the first visiting point, as well as set the camera's viewing angle, near plane, far screen, aspect ratio and other parameters.

[0119] Exemplarily, since the user usually starts to visit the exhibition hall from the first visiting point, the coordinates of the first visiting point can be set as the origin coordinates of the world space. In other words, the camera is set at the origin of the world space.

[0120] Exemplarily, since the influence of the light source in the 3D scene on the objects in the 3D model is considered in advance during the process of rendering the panorama based on the second model, it is not necessary to add a light source to the 3D scene after the 3D scene is created this time.

[0121] Exemplarily, the 3D rendering engine can load the first model from the video memory into the 3D scene according to the storage address of the first model in the video memory transmitted by the browser. Exemplarily, since the 3D models are all created in the model space, and the 3D scenes are created in the world space, the 3D models and the 3D scenes belong to different spaces, so the 3D rendering engine can perform coordinate transformation (such as rotation, translation, scaling, etc.) on the first model based on the origin of the world space, and transform each object in the first model from the model space to the world space, so as to load the first model into the 3D scene.

[0122] S506, the 3D rendering engine maps the panoramic image of the first visit point to the sky box of the 3D scene.

[0123] For example, the skybox may refer to a cube or sphere that surrounds a three-dimensional scene and is infinitely far from the origin of the world space. When a user stands in the three-dimensional scene and looks around, what he sees is the inner wall of the skybox. For example, the coordinates of each point corresponding to the skybox are coordinates in a polar coordinate system (two-dimensional coordinates).

[0124] It should be noted that the embodiment of the present application maps the panorama to the skybox of the three-dimensional scene, which does not create a skybox for the three-dimensional scene and map the panorama to the inner wall of the skybox, but maps the coordinates of each pixel of the panorama. Exemplarily, the three-dimensional rendering engine can load the panorama from the video memory according to the storage address of the panorama in the video memory transmitted by the browser, and then align the position of the camera in the second model when generating the panorama with the position of the camera in the first model (that is, the position of the first visit point), and convert the coordinates of each pixel of the panorama into coordinates under polar coordinates through the conversion of spherical coordinates to polar coordinates, thus completing the step of mapping the panorama to the skybox of the three-dimensional scene. In other words, the skybox in S506 is actually a virtual skybox.

[0125] S507, the 3D rendering engine uses the location of the camera as an endpoint, and respectively makes rays to the multiple vertices P1 in the first model, intersecting with the sky box at P2.

[0126] For example, the first model includes multiple vertices P1, and rays can be drawn to the multiple vertices P1 in the first model with the camera position as the endpoint, and each ray intersects with the sky box at point P2 at infinity (although the sky box is virtual, each ray has an intersection with the virtual sky box in space). In other words, each ray starts from the camera position, intersects with the first model at P1, and intersects with the sky box at P2.

[0127] In a possible implementation, the camera position can be used as an endpoint to draw rays to multiple vertices P1 in the first model that are located in the observation area corresponding to each viewing direction of the camera position, and each ray intersects the sky box at P2 at infinity.

[0128] Figure 6 Schematic diagram of a viewing cone shown as an example. Figure 6 , exemplarily, 601 is a camera in a three-dimensional scene, the camera is located at the top of the cone, 602 is a near plane, and 603 is a far plane. θ1, θ2 and θ3 are viewing angles, θ1 is a vertical viewing angle, θ2 is a horizontal viewing angle, and θ3 is a diagonal viewing angle.

[0129] Reference Figure 6 For example, only the vertices between the near plane and the far plane of the viewing cone in the three-dimensional model can be observed by the camera, such as Figure 6The vertices located in the observation area 604. Among the vertices located in the observation area 604, some vertices can be observed by the camera, but some vertices will be blocked by other vertices and cannot be observed by the camera; and only the points that can be observed by the camera can be rendered to the screen. In a possible implementation, the vertices that can be observed by the camera in the corresponding observation area when the camera is directed in each viewing direction can be determined according to the camera's viewing angle, near plane, and far plane. Then, with the camera's location as the endpoint, rays are drawn to the multiple vertices P1 in each observation area corresponding to each viewing direction located at the camera's location, and intersect with the sky box at P2.

[0130] Figure 7 is a schematic diagram of rays shown as an example. Figure 7 The sky box 701 of the three-dimensional scene is shown (each point of the sky box corresponds to each pixel point of the panoramic image, which is only for the convenience of explanation). Figure 7 702 is the first model in the three-dimensional scene, and 703 is a ray. O is the location of the camera, P1 is the vertex in the first model, and P2 is the intersection of the ray and the skybox. It should be understood that Figure 7 Only some of the rays are shown.

[0131] Since the user can only see the objects in a certain visiting direction of the visiting point after arriving at the visiting point, the three-dimensional model can render the exhibition hall image of the visiting direction and display it to the user. Exemplarily, the corresponding preset visiting direction can be set for each visiting point in advance. Exemplarily, each visiting point can be associated with the corresponding preset visiting direction in advance to generate third association information. Exemplarily, the point identification of each visiting point and the corresponding preset visiting direction can be used to generate the third association information. Exemplarily, the third association information can be associated with the first model, and then when the browser requests to download the first model, the first model and the third association information associated with the first model are sent to the browser. Exemplarily, the browser can also load the third association information into the video memory, and when calling the three-dimensional rendering engine, the storage address of the third association information is passed to the three-dimensional rendering engine. Exemplarily, when the three-dimensional rendering engine loads the first model from the video memory, the third association information can be loaded according to the storage address of the third association information.

[0132] In one possible implementation, the 3D rendering engine can determine the preset visiting direction of the first visiting point based on the point identifier and the third associated information transmitted by the browser, and then use the camera position as the endpoint to make rays to multiple vertices P1 in the first model that are located in the observation area corresponding to the preset visiting direction of the first visiting point, and each ray intersects the sky box at P2 at infinity.

[0133] For example, the vertices that can be observed by the camera in the observation area corresponding to the preset viewing direction of the first viewing point can be determined according to the camera's viewing angle, near plane, and far plane. Then, with the camera's location as the endpoint, rays are drawn to the multiple vertices P1 that can be observed by the camera in the observation area corresponding to the preset viewing direction of the first viewing point, and intersect with the skybox at P2.

[0134] Figure 8 is a schematic diagram of rays shown as an example. Figure 8 The sky box 801 of the three-dimensional scene is shown (each point of the sky box corresponds to each pixel point of the panoramic image, which is only for the convenience of explanation. Figure 8 802 is the first model in the three-dimensional scene, and 803 is a ray. O is the location of the camera, P1 is the vertex in the first model, and P2 is the intersection of the ray and the skybox. It should be understood that Figure 8 Only some of the rays are shown.

[0135] S508: The 3D rendering engine maps the rendering and shading information of P2 to P1 that belongs to the same ray as P2.

[0136] For example, the 3D model rendering process involves projection transformation, that is, projecting the vertices in the 3D model onto the screen (i.e., the transformation process from 3D space to 2D space). When performing projection transformation, after the 3D rendering engine projects a vertex that can be observed by the camera onto the screen, it can store the rendering information corresponding to the vertex in the 3D model (such as coordinates, rendering shading information, texture information, etc.), as well as the rendering information of other vertices in the 3D model that are blocked by the vertex, as the rendering information of the fragment corresponding to the vertex.

[0137] Exemplarily, for each P2 on the skybox, the rendering information of the corresponding fragment of P2 can be obtained from the panoramic view of the first visit point according to the coordinates of P2. Then, the rendering shading information corresponding to P2 (such as RGB (R (Red, red) G (Green, green) B (Blue, blue)))) is obtained from the rendering information of the corresponding fragment of P2. Then, the rendering shading information corresponding to P2 is mapped to P1 belonging to the same ray as P2. Exemplarily, the rendering shading information corresponding to P2 can be used to update the rendering shading information of the vertex P1 belonging to the same ray as P2. Figure 7As shown, the rendering shading information of P2 on ray 703 is used to update the rendering shading information of P1 on ray 703. For example, if the rendering shading information of P2 on ray 703 is RGB1 and the rendering shading information of P1 on ray 703 is RGB2, the rendering shading information of P1 on ray 703 can be updated from RGB2 to RGB1.

[0138] S509, the 3D rendering engine sets the direction of the camera to the preset visiting direction of the first visiting point, and performs rendering based on the first model to obtain an exhibition hall image in the preset visiting direction of the first visiting point.

[0139] Exemplarily, after mapping the rendering shading information of each P2 on the sky box to P1 on the corresponding ray, the three-dimensional rendering engine can set the direction of the camera to the preset visiting direction, and then render based on the first model to obtain a rendered image, that is, the exhibition hall image in the preset visiting direction of the first visiting point.

[0140] For example, when the 3D rendering engine is based on Figure 8 When drawing a ray from the camera position in a manner, after obtaining the exhibition hall image rendered in the preset visiting direction of the first visiting point, the 3D rendering engine can also use the camera position as an endpoint to draw a ray to the vertex P1 in the observation area corresponding to other visiting directions located at the first visiting point in the first model, and intersect with the sky box at P2. Among them, other visiting directions may refer to directions other than the preset visiting directions. Exemplarily, after drawing a ray from the camera position as an endpoint to the vertex P1 in the observation area corresponding to other visiting directions located at the first visiting point in the first model, and intersecting with the sky box at P2, the 3D rendering model can execute S508.

[0141] Exemplarily, after obtaining the exhibition hall image in the preset visiting direction of the first visiting point, the 3D rendering engine may send the exhibition hall image in the preset visiting direction of the first visiting point to the browser.

[0142] S510, the browser displays the exhibition hall image in the preset visiting direction of the first visiting point.

[0143] Exemplarily, after the browser receives the exhibition hall image in the preset visiting direction of the first visiting point sent by the three-dimensional rendering engine, the exhibition hall image in the preset visiting direction of the first visiting point can be displayed; and then the user can see the exhibition hall image in the preset visiting direction of the first visiting point.

[0144] Fig. 9 Schematic diagram of an exhibition hall image is shown as an example. Fig. 9 , Fig. 9 The exhibition hall image in the preset visiting direction of the first visiting point rendered based on the first model.

[0145] When visiting each visiting point, the user can also change the visiting direction to browse objects in other visiting directions of the visiting point.

[0146] Fig.10 FIG. 1 is a schematic diagram of an exemplary processing process. Fig.10 , Fig.10 These are the steps that the browser executes in response to the user's operation after the user performs a visiting direction change operation at the first visiting point, and the steps that the 3D rendering engine executes after receiving the browser call.

[0147] S1001, the browser receives a visit direction change operation.

[0148] For example, after the browser displays the exhibition hall image in the preset viewing direction of the first viewing point, the user can perform a viewing direction change operation to view the exhibition hall images in other viewing directions of the first viewing point. After the user performs the viewing direction change operation as required, the browser can receive the viewing direction change operation, and then call the 3D rendering engine in response to the viewing direction change operation.

[0149] Exemplarily, the browser can determine the changed visiting direction of the user (hereinafter referred to as the user visiting direction) according to the visiting direction change operation. Exemplarily, when calling the 3D rendering engine, the browser can pass the user visiting direction corresponding to the visiting direction change operation to the 3D rendering engine.

[0150] S1002, the 3D rendering engine sets the direction of the camera to the user's visiting direction corresponding to the visiting direction change operation, and performs rendering based on the first model to obtain an exhibition hall image in the user's visiting direction of the first visiting point.

[0151] Exemplarily, after receiving a call from the browser, the 3D rendering engine can set the direction of the camera to the user's visiting direction corresponding to the visiting direction change operation, and then render based on the first model to obtain a rendered image, that is, an image of the exhibition hall in the user's visiting direction of the first visiting point.

[0152] For example, when the 3D rendering engine is based on Figure 7 When a ray is drawn from the position of the camera in the manner of FIG. 1 , the 3D engine directly executes S1002 after receiving the browser call.

[0153] For example, when the 3D rendering engine is based on Figure 8When making a ray from the camera position, after obtaining the exhibition hall image in the preset visiting direction of the first visiting point, the 3D rendering engine takes the camera position as the endpoint and makes a ray to the vertex P1 in the observation area corresponding to other visiting directions of the first visiting point in the first model, and intersects the sky box at P2, and executes S508; then when the 3D rendering engine receives the call executed by the browser in response to the visiting direction change operation, the 3D rendering engine directly executes S1002.

[0154] For example, when the three-dimensional rendering is performed according to Figure 8 When the 3D rendering engine makes a ray from the position of the camera in a manner, if the 3D rendering engine does not make a ray from the position of the camera to the vertex P1 in the observation area corresponding to other visiting directions of the first visiting point in the first model after obtaining the exhibition hall image rendered in the preset visiting direction of the first visiting point, the 3D rendering engine can first execute S507 to S508 and then execute S1002 after receiving the call executed by the browser in response to the visiting direction change operation. Among them, when the 3D rendering engine executes S507, the position of the camera can be used as an endpoint to make rays to the multiple vertices P1 in the observation area corresponding to the user visiting direction of the first visiting point in the first model, and each ray intersects with the sky box at P2 at infinity. Exemplarily, the vertices that can be observed by the camera in the corresponding observation area when the camera direction is the user visiting direction of the first visiting point can be determined according to the camera's viewing angle, near plane and far plane. Then, the position of the camera is used as an endpoint, and rays are made to the multiple vertices P1 that can be observed by the camera in the observation area corresponding to the user visiting direction, and intersect with the sky box at P2.

[0155] S1003, the browser displays the exhibition hall image in the user's visiting direction of the first visiting point.

[0156] Exemplarily, after the browser receives the user's visiting direction of the first visiting point sent by the 3D rendering engine, the browser can display the image in the user's visiting direction of the first visiting point; and then the user can see the exhibition hall image in the user's visiting direction of the first visiting point.

[0157] It should be noted that the user can continue to perform the operation of changing the visiting direction at the first visiting point. The corresponding operations performed by the browser and the 3D rendering engine can be referred to in Fig.10 As well as the above Fig.10 The description will not be repeated here.

[0158] After visiting one sightseeing spot, users can move to other sightseeing spots for further visits.

[0159] Fig.11FIG. 1 is a schematic diagram of an exemplary processing process. Fig.11 , Fig.11 These are the steps that the browser executes in response to the user's operation after the user executes the operation of switching the visiting points, and the steps that the 3D rendering engine executes after receiving the browser call.

[0160] S1101, the browser receives a visit point switching operation.

[0161] S1102, the browser downloads the panoramic image of the Kth sightseeing point corresponding to the sightseeing point switching operation.

[0162] Exemplarily, after the user completes the visit to the first sightseeing point, the user can perform a sightseeing point switching operation and select other sightseeing points on the sightseeing route for visiting. After the user performs the sightseeing point switching operation, the browser can receive the sightseeing point switching operation and generate a second download request information in response to the sightseeing point switching operation. The second download request information is used to indicate the download of a panoramic view of the Kth sightseeing point corresponding to the sightseeing point switching operation (that is, the sightseeing point switched by the user, K is an integer greater than 1, ranging from 2 to N, and the Kth sightseeing point is a sightseeing point other than the first sightseeing point), and the second download request information includes the point identifier of the Kth sightseeing point.

[0163] Exemplarily, the browser may send the second download request information to the server. After receiving the second download request information, the server may parse the second download request information and extract the point identifier from the second download request information. The server then searches for the image identifier corresponding to the extracted point identifier based on the pre-established second association information, and then determines the panoramic image corresponding to the image identifier (that is, the panoramic image of the Kth visiting point) as the panoramic image requested by the second download request information. Subsequently, the server returns the found panoramic image of the Kth visiting point to the browser, and the browser can receive the panoramic image of the Kth visiting point.

[0164] Exemplarily, after the browser finishes downloading the panorama, it can load the panorama into the video memory and call the three-dimensional rendering engine to render the exhibition hall image corresponding to the Kth visiting point.

[0165] Exemplarily, when calling the 3D rendering engine, the browser may pass the storage address of the panoramic image in the video memory and the point identifier of the Kth visiting point to the 3D rendering engine.

[0166] S1103: The 3D rendering engine moves the camera to the Kth viewing point in the first model.

[0167] S1104: The 3D rendering engine corrects the positions of the vertices in the first model.

[0168] S1105, the 3D rendering engine maps the panoramic image of the Kth visiting point to the sky box of the 3D scene.

[0169] Exemplarily, after the 3D rendering engine receives the call from the browser, in one possible manner, the 3D rendering engine may execute in the order of S1103→S1104→S1105. In one possible manner, the 3D rendering engine may execute in the order of S1105→S1104→S1103. In one possible manner, the 3D rendering engine may execute S1103 and S1105 simultaneously, and execute S1104 after executing S1103, which is not limited in the embodiment of the present application. The embodiment of the present application takes the sequential execution of S1103→S1104→S1105 as an example for exemplary description.

[0170] Exemplarily, after receiving the call from the browser, the 3D rendering engine can determine the coordinates of the Kth viewing point according to the point identifier, and then move the camera from the first viewing point in the first model to the Kth viewing point.

[0171] Exemplarily, during the rendering process, the three-dimensional model is rendered with the position of the camera as the origin. Therefore, after the camera position is moved, the positions of the vertices in the three-dimensional model need to be corrected to ensure that the position of the camera is at the origin of the world space relative to each vertex. Exemplarily, the positions of the vertices in the first model can be corrected according to the camera motion offset trajectory vector. Exemplarily, assuming that the origin position of the world space is O(X, Y, Z), the camera position is also O(X, Y, Z), assuming that the position of the Kth visit point is O'(X', Y', Z'), that is, the position after the camera moves, then the camera motion offset trajectory information can be obtained as (X'-X, Y'-Y, Z'-Z). Let the position of a vertex P1 in the first model when the camera is at the Jth visiting point (the Jth visiting point is the position set for the camera last time in the 3D rendering model, J is greater than a positive integer, the value range is 1 to N, the Jth visiting point is a visiting point other than the Kth visiting point, and the Jth visiting point can be the first visiting point) be z(x, y, z), and the position of the model vertex is reversely corrected according to the camera motion offset trajectory information, and the corrected position of the vertex P1 is z'(x-X'+X, y-Y'+Y, z-Z'+Z).

[0172] Illustratively, S1105 may refer to the description of S506 above, which will not be repeated here.

[0173] S1106, the 3D rendering engine uses the location of the camera as an endpoint, and respectively makes rays to the multiple vertices P1 in the first model, intersecting with the sky box at P2.

[0174] S1107, the 3D rendering engine maps the rendering and shading information of P2 to P1 that belongs to the same ray as P2.

[0175] S1108, the 3D rendering engine sets the direction of the camera to the preset visiting direction of the Kth visiting point, and performs rendering based on the first model to obtain an exhibition hall image in the preset visiting direction of the Kth visiting point.

[0176] Illustratively, S1106 to S1108 may refer to the description of S507 to S509 above, which will not be repeated here.

[0177] S1109, the browser displays the exhibition hall image in the preset visiting direction of the Kth visiting point.

[0178] Exemplarily, after the browser receives the image in the preset visiting direction of the Kth visiting point sent by the 3D rendering engine, the image in the preset visiting direction of the Kth visiting point can be displayed; and then the user can see the exhibition hall image in the preset visiting direction of the Kth visiting point.

[0179] For example, when the user visits the Kth viewing point, the user may also change the viewing direction to observe objects in other viewing directions of the Kth viewing point. After the user performs the viewing direction change operation at the Kth viewing point, the steps executed by the browser in response to the user operation, and the image processing process executed by the 3D rendering engine after receiving the browser call, can be referred to as Fig.10 As well as the above Fig.10 The description will not be repeated here.

[0180] It should be noted that after visiting the first visiting point, the user can perform a visiting point switching operation to switch to any other visiting point for visiting, and the embodiment of the present application does not limit this.

[0181] It should also be noted that after the user has visited the Kth sightseeing point, he can perform the sightseeing point switching operation again to switch to any other sightseeing point for visiting. For example, to switch to sightseeing point M (M is a positive integer ranging from 1 to N, and sightseeing point M is a sightseeing point other than the Kth sightseeing point) for visiting, you can refer to Fig.11 As well as the above Fig.11 The description is not limited here.

[0182] In one possible manner, after the browser displays the exhibition hall image in the preset viewing direction of the first viewing point in response to the operation of entering the viewing interface, if the browser does not receive the user's viewing direction change operation within the first set time, the browser can call the three-dimensional rendering engine to render and display the exhibition hall image in any other viewing direction of the first viewing point. The first set time can be set as required, and the embodiment of the present application does not limit this. In this way, the user can visit multiple viewing directions of a viewing point without any operation.

[0183] In one possible manner, after the browser displays the exhibition hall image of the preset visiting direction of the first visiting point in response to the operation of entering the visiting interface, if the browser does not receive the user's visiting point switching operation within the second set time, the browser can call the three-dimensional rendering engine to render and display the exhibition hall image of the preset visiting direction of the Kth visiting point. The second set time can be set as required, and the embodiment of the present application does not limit this. In this way, the user can visit multiple visiting points in the exhibition hall without any operation.

[0184] Fig.12 The figure is a schematic diagram of an exhibition hall image shown as an example.

[0185] Reference Fig.12 , Fig.12 (a1), Fig.12 (a2) and Fig.12 (a3) is a case where the panoramic image rendered based on the second model is not mapped into the first model, but the exhibition hall image in a certain visiting direction of the three visiting points rendered by the first model is directly used. Fig.12 (b1), Fig.12 (b2) and Fig.12 (b3) is to map the panoramic image rendered based on the second model to the first model, and then use the first model after mapping the panoramic image to render the exhibition hall image in a certain visiting direction of the three visiting points (that is, the exhibition hall image rendered using the method of the present application). Fig.12 The visiting points and directions corresponding to (a1) and 12(b1) are the same. Fig.12 The visiting points and parties corresponding to (a2) and 12(b2) are the same. Fig.12 The visiting points and parties corresponding to (a3) ​​and 12(b3) are the same. By comparison, we can see that Fig.12 (b1) has higher quality than 12(a1). Fig.12 (b2) has higher quality than 12(a2), and Fig.12 (b3) has higher image quality than 12(a3), that is, after the panoramic image rendered based on the second model is mapped to the first model, the embodiment of the present application can render a high-quality image using the first model that maps the panoramic image.

[0186] As described above, the exhibition hall image is obtained by mapping the panorama to a three-dimensional model and then rendering based on the three-dimensional model of the mapped panorama. Since the second model has a higher degree of precision than the first model, the panorama rendered based on the second model has a high image quality. After the panorama is mapped to the first model, the rendering and shading information of each vertex in the first model is the same as the rendering and shading information of each vertex in the second model. Therefore, the image rendered using the first model with lower precision has the same image quality as the panorama rendered using the second model with higher precision. Furthermore, the image rendered based on the first model that maps the panorama also has high image quality and can also provide users with a strong immersive experience.

[0187] In addition, the first model is a relatively simple 3D model. When the browser calls the 3D rendering engine to render based on the first model, the 3D rendering engine needs to perform less calculations, thereby reducing the performance requirements on the client. The 3D rendering engine takes less time to render based on a simple 3D model, thereby shortening the latency of the client displaying the exhibition hall image and improving the user's online exhibition hall visit experience.

[0188] In addition, compared with the use of Figure 7 The 3D rendering engine uses Figure 8 As rays are drawn in the way of , there is no need to draw rays in all directions of the first visit point, nor is there any need to map the rendering shading information of P2 of rays in all directions of the visit to P1, so Figure 8 By using the method of using the ray function, the exhibition hall image in the preset visiting direction of the first visiting point can be quickly rendered, which can shorten the delay of the browser displaying the exhibition hall image and improve the user's online exhibition hall visiting experience.

[0189] Since the image quality of the panorama is relatively high, the amount of data in the panorama is also large. In order to reduce the delay of users' online visit to the exhibition hall, the panorama can be transmitted to the browser in blocks. After the browser downloads multiple blocks corresponding to a certain visiting direction, it can call the 3D rendering engine to render the exhibition hall image corresponding to the visiting direction.

[0190] Fig.13 A rendering diagram is shown for exemplary purposes.

[0191] Reference Fig.13 , exemplarily, based on the second model (such as Fig.13 (1)), and render the panoramic view corresponding to each visiting point in the exhibition hall, such as Fig.13 As shown in (2), Fig.13 (2) is a panoramic view of a certain sightseeing point, which can be referred to Figure 3 As well as the above Figure 3 The description will not be repeated here.

[0192] For example, after obtaining the panorama corresponding to each sightseeing point, the panorama of each sightseeing point can be tiled as follows: Fig.13 As shown in (3), multiple tile images of the panoramic image are obtained. Optionally, the tile image can be a square. The method of dividing the panoramic image into tiles can be set according to requirements, and the embodiment of the present application does not limit this.

[0193] Fig.14 FIG. 1 is a schematic diagram of an exemplary storage system. Fig.14 For example, after obtaining the tile images of the panorama corresponding to each visiting point, the tile images of the panorama corresponding to each visiting point (such as Fig.14 (2)) and the first model (as shown in Fig.14 (1)), the digital data of the exhibition hall corresponding to the exhibition hall is stored in the server (as shown in Fig.14 (3) shown).

[0194] Exemplarily, after obtaining the panorama corresponding to each visiting point, the second association information between the visiting point and the corresponding panorama can be established. Exemplarily, the second association information between the point identifier of the visiting point and the image identifier of the corresponding panorama can be established, and the second association information can be stored in the server.

[0195] Exemplarily, after the panoramic image is divided into a plurality of tile images, a tile identifier may be added to each tile image; and then fourth association information between the tile image and the panoramic image to which the tile image belongs is established. Exemplarily, fourth association information between the tile identifier of the tile image and the image identifier of the panoramic image to which the tile image belongs may be established, and the fourth association information may be stored in the server. The tile identifier is used to uniquely identify a tile image.

[0196] Exemplarily, different exhibition halls correspond to different first models. The first model has a corresponding model identifier, and the model identifier is used to uniquely identify a first model.

[0197] Fig.15 FIG. 1 is a schematic diagram of an exemplary processing process. Fig.15 , Fig.15 These are the steps that the browser performs in response to the user's operation after the user enters the viewing interface, and the steps that the 3D rendering engine performs after receiving the browser call.

[0198] S1501, the browser receives the operation of entering the visit interface.

[0199] S1501 can refer to the above S501 and will not be described in detail here.

[0200] S1502: The browser downloads the first model.

[0201] Exemplarily, after receiving the operation of entering the visit interface, the browser may generate a third download request message in response to the operation of entering the visit interface, wherein the third download request message is used to instruct to download the first model, and the third download request message includes the model identifier of the first model.

[0202] Exemplarily, the browser may send the third download request information to the server. After receiving the third download request information, the server may parse the third download request information and extract the model identifier from the first download request information. The server then searches for a first model that matches the proposed model identifier and determines the found first model as the first model requested by the third download request information. Subsequently, the server returns the found first model to the browser, and the browser may receive the first model.

[0203] Exemplarily, after the browser finishes downloading the first model, it can load the first model into the video memory and call the three-dimensional rendering engine to render the exhibition hall image corresponding to the first visiting point.

[0204] Exemplarily, when calling the 3D rendering engine, the browser may pass the storage address of the first model and the point identifier of the first visiting point to the 3D rendering engine.

[0205] S1503, the 3D rendering engine creates a 3D scene.

[0206] S1504: The 3D rendering engine loads the first model into the 3D model.

[0207] S1505: The 3D rendering engine sets the camera at the first visit point in the first model.

[0208] S1503 to S1505 may refer to the above S503 to S505 and will not be described in detail here.

[0209] S1506, the 3D rendering engine creates a reference cube with the camera position as the center.

[0210] S1507, the 3D rendering engine determines the coordinate information of the plurality of target tile images to be downloaded in the panoramic image corresponding to the first viewing point according to the preset viewing direction and the reference cube.

[0211] After entering the exhibition hall, the user visits the exhibition hall in the preset visiting direction at the first visiting point, so the browser can first download the tile image corresponding to the preset visiting direction of the first visiting point in the panoramic image from the server, and then the 3D rendering engine renders the image based on the first model and the tile image corresponding to the preset visiting direction of the first visiting point, so as to obtain the exhibition hall image in the preset visiting direction of the first visiting point. Compared with the entire panoramic image, the data volume of some tile images in the panoramic image is smaller, which can save the browser time to download the panoramic image, thereby reducing the delay of the browser showing the exhibition hall image to the user.

[0212] Exemplarily, the 3D rendering engine creates a reference cube with the camera position as the center, wherein the reference cube is located in the world space, and each point of the reference cube is a point in the 3D space.

[0213] It should be noted that, in the actual processing process of the embodiment of the present application, no reference cube is created in the first model, but only the coordinate information corresponding to the reference cube is generated; that is, the reference cube is a virtual cube.

[0214] Fig.16 FIG. 1 is a schematic diagram of a reference cube shown as an example. Fig.16 , Fig.16 1601 in FIG. 1 is the first model, 1602 is the reference cube, and 1603 is the position of the camera, which is also the center of the reference cube 1602. It should be noted that, for the sake of convenience, Fig.16 The first model shows a reference cube.

[0215] Exemplarily, the center of the reference cube coincides with the location of the camera, and each face of the reference cube is between the near plane and the far plane of the camera's viewing cone, that is, the near plane of the camera when it is in different viewing directions is wrapped by the reference cube.

[0216] Fig.17 FIG. 1 is a schematic diagram showing the positional relationship between the face of a cube and the near plane. Fig.17 , 1701 is the camera, located at the top of the cone, 1702 is a face of the reference cube, 1703 is the near plane, and 1704 is the far plane.

[0217] For example, the side length of each face of the reference cube can be slightly larger than the maximum value of the length and width of the near plane, that is, the reference cube can just cover the near plane when the camera is in different viewing directions. For example, the difference between the side length of each face of the reference cube and the maximum value of the length and width of the near plane can be set as required, and the embodiments of the present application are not limited to this.

[0218] Exemplarily, when the direction of the camera is the preset viewing direction, the cross section of the reference cube and the cone is determined, and then the vertex between the cross section of the cone and the far plane in the first model is determined. Since each pixel point in the panorama is obtained by projection transformation of the vertex in the three-dimensional model, after determining the vertex between the cross section of the cone and the far plane in the first model, these vertices can be projected and transformed, that is, the coordinates of the vertices in the three-dimensional model (three-dimensional) are converted from the world space to the observation space, and the coordinate information (two-dimensional) of the pixel points projected onto the screen corresponding to each vertex can be obtained. These pixel point coordinate information is the coordinate information of the tile map that the user can view in the preset viewing direction of the first viewing point. Among them, for the sake of convenience, the tile map that the user can see in the preset viewing direction of the first viewing point is called the target tile map to be downloaded in the panorama of the first viewing point. Furthermore, through projection transformation, the coordinate information of multiple target tile maps to be downloaded in the panorama of the first viewing point can be obtained. Then the three-dimensional rendering engine can return the coordinate information of multiple target tile maps to be downloaded in the panorama of the first viewing point to the browser.

[0219] S1508: The browser downloads multiple target tile images based on the coordinate information of the multiple target tile images.

[0220] Exemplarily, after the browser receives the coordinate information of multiple target tile images to be downloaded in the panoramic map of the first visiting point, it can generate a fourth download request information; wherein the fourth download request information includes the coordinate information of multiple target tile images to be downloaded in the panoramic map of the first visiting point and the point identifier of the first visiting point, and the fourth download request information is used to indicate the download target tile image.

[0221] Exemplarily, the browser may send the fourth download request information to the server. After receiving the fourth download request information, the server may analyze the fourth download request information and extract the point identifier and coordinate information from the fourth download request information. Then, the server determines the map identifier corresponding to the extracted point identifier based on the second association relationship, and then determines the tile identifier corresponding to the map identifier based on the fourth association information. Subsequently, the server may determine multiple target tile images corresponding to the extracted coordinate information from the tile image corresponding to the tile identifier (that is, the tile image included in the panoramic image corresponding to the first visiting point), and return the multiple target tile images to the browser.

[0222] Exemplarily, for each coordinate information in the fourth download request information, the server may compare the coordinate information with the coordinate information of each tile image included in the panoramic image corresponding to the first visit point. When a certain coordinate information of a certain tile image matches a coordinate information extracted from the fourth download request, the tile image may be determined as the target tile image.

[0223] Exemplarily, after the browser finishes downloading the multiple target tile images, the browser may load the multiple target tile images into the video memory, and transmit the storage addresses of the multiple target tile images in the video memory to the 3D rendering engine.

[0224] S1509, the 3D rendering engine maps the multiple target tile images to the sky box of the 3D scene.

[0225] Exemplarily, the 3D rendering engine can load multiple target tiles from the video memory according to the storage addresses of the multiple target tile images in the video memory transmitted by the browser, and then map the multiple target tile images to the sky box of the 3D scene. Please refer to S506 above and will not be repeated here.

[0226] It should be noted that, different from S506, S1509 converts the coordinates of multiple target tile images into polar coordinates by converting spherical coordinates into polar coordinates, so as to map the multiple target tile images to the sky box of the three-dimensional scene.

[0227] S1510, the 3D rendering engine uses the location of the camera as an endpoint, and respectively makes rays to the multiple vertices P1 in the first model, intersecting with the sky box at P2.

[0228] For example, since the browser only downloads the tile map in the preset viewing direction of the first viewing point, the 3D rendering engine uses the camera location as the endpoint and makes rays to multiple vertices P1 in the first model that are located in the observation area corresponding to the preset viewing direction of the first viewing point. Each ray intersects the skybox at P2 at infinity. For details, please refer to Figure 8 And the corresponding description will not be repeated here.

[0229] S1511, the 3D rendering engine maps the rendering and shading information of P2 to P1 which belongs to the same ray as P2.

[0230] S1512, the 3D rendering engine sets the direction of the camera to the preset visiting direction of the first visiting point, and performs rendering based on the first model to obtain an exhibition hall image in the preset visiting direction of the first visiting point.

[0231] S1513, the browser displays the exhibition hall image in the preset visiting direction of the first visiting point.

[0232] S1511~S1513 can refer to the above S508~S510 and will not be described in detail here.

[0233] Exemplarily, after the browser completes downloading multiple target tiles, it can generate a fifth download request message, the fifth download request message is used to instruct the download of other tiles in the panorama of the first visiting point, the fifth download request message includes the point identifier of the first visiting point, and other tiles refer to tiles in the panorama of the first visiting point except for the multiple target tiles. Exemplarily, the browser sends the fifth download request message to the server, and the server can return the other tiles in the panorama of the first visiting point to the browser, which is similar to the above description of the fourth download request message and will not be repeated here. Exemplarily, after downloading other tiles in the panorama of the first visiting point, the browser can call a three-dimensional rendering engine to map the other tiles to the sky box of the three-dimensional scene, and then execute S1510 to S1511.

[0234] Exemplarily, if the browser has downloaded other tile images of the panoramic image of the first visiting point before receiving the visiting direction change operation, and called the 3D rendering engine to map the other tile images to the sky box of the 3D scene and execute S1510-S1511, then after receiving the visiting direction change operation, the browser can respond to the received visiting direction change operation and call the 3D rendering engine to set the direction of the camera to the user visiting direction corresponding to the visiting direction change operation, and render based on the first model to obtain a rendered image, that is, the exhibition hall image in the user visiting direction of the first visiting point; it can refer to Fig.10 And the corresponding description, the embodiments of the present application are not limited to this.

[0235] Exemplarily, if the browser has not completed at least one of the following operations before receiving the visiting direction change operation: downloading other tile images of the panorama of the first visiting point, calling the three-dimensional rendering engine to map the other tile images of the panorama of the first visiting point to the sky box of the three-dimensional scene, executing S1510 and executing S1511, then after receiving the visiting direction change operation, the browser can wait for all the above operations to be completed, and then respond to the received visiting direction change operation, call the three-dimensional rendering engine to set the direction of the camera to the user visiting direction corresponding to the visiting direction change operation, and render based on the first model to obtain a rendered image, that is, the exhibition hall image in the user visiting direction of the first visiting point; it can refer to Fig.10 And the corresponding description, the embodiments of the present application are not limited to this.

[0236] Fig.18 FIG. 1 is a schematic diagram of an exemplary processing process. Fig.18 , Fig.18It is the image processing process performed by the browser in response to the user's operation after the user performs a visiting direction change operation at the first visiting point, and the three-dimensional rendering engine receives the browser call and executes the steps.

[0237] S1801, the browser receives a visit direction change operation.

[0238] S1802, the 3D rendering engine determines the coordinate information of multiple target tile images to be downloaded in the panoramic image of the first visiting point according to the visiting direction transformation operation corresponding to the user's visiting direction and the reference cube.

[0239] S1803: The browser downloads multiple target tile images based on the coordinate information of the multiple target tile images.

[0240] Exemplarily, after receiving the coordinate information of multiple target tiles, the browser may generate a sixth download request message, the sixth download request message is used to instruct to download tiles in the user's visiting direction in the panoramic view of the first visiting point, and the sixth download request message includes the point identifier of the first visiting point. Exemplarily, the browser sends the sixth download request message to the server, and the server may return other tiles in the user's visiting direction in the panoramic view to the browser, which is similar to the above description of the fourth download request message and will not be repeated here.

[0241] S1804, the 3D rendering engine maps the multiple target tile images to the sky box of the 3D scene.

[0242] S1805, the 3D rendering engine uses the location of the camera as an endpoint and respectively makes rays to the multiple vertices P1 in the first model, intersecting with the sky box at P2.

[0243] S1806: The 3D rendering engine maps the rendering and shading information of P2 to P1 that belongs to the same ray as P2.

[0244] S1807, the 3D rendering engine sets the direction of the camera to the user's visiting direction of the first visiting point, and performs rendering based on the first model to obtain an exhibition hall image in the user's visiting direction of the first visiting point.

[0245] S1808, the browser displays the exhibition hall image in the user's visiting direction of the first visiting point.

[0246] S1802~S1808 can refer to the above S1507~S1513 and will not be repeated here.

[0247] Fig.19 FIG. 1 is a schematic diagram of an exemplary processing process. Fig.19 , Fig.19 These are the steps that the browser executes in response to the user's operation after the user executes the operation of switching the visiting points, and the steps that the 3D rendering engine executes after receiving the browser call.

[0248] S1901, the browser receives a visit point switching operation.

[0249] S1902: The 3D rendering engine moves the camera to the Kth viewing point in the first model.

[0250] S1903: The 3D rendering engine corrects the positions of the vertices in the first model.

[0251] S1904, the 3D rendering engine creates a reference cube with the camera position as the center.

[0252] Exemplarily, after the 3D rendering engine receives the call from the browser, in one possible manner, the 3D rendering engine may execute in the order of S1902→S1903→S1904. In one possible manner, the 3D rendering engine may execute in the order of S1904→S1903→S1902. In one possible manner, the 3D rendering engine may execute S1902 and S1904 simultaneously, and execute S1903 after executing S1902, which is not limited in the embodiment of the present application. The embodiment of the present application takes the sequential execution of S1902→S1903→S1904 as an example for exemplary description.

[0253] Exemplarily, S1902 to S1903 may refer to the description of S1103 to S1104 above, which will not be repeated here.

[0254] Exemplarily, S1904 may refer to the description of S1506 above, which will not be repeated here.

[0255] S1905, the 3D rendering engine determines the coordinate information of multiple target tile images to be downloaded in the panoramic image corresponding to the Kth visiting point according to the preset visiting direction and the reference cube corresponding to the Kth visiting point in the visiting point switching operation.

[0256] S1906: The browser downloads multiple target tile images based on the coordinate information of the multiple target tile images.

[0257] S1907, the 3D rendering engine maps the multiple target tile images to the sky box of the 3D scene.

[0258] S1908, the 3D rendering engine uses the location of the camera as an endpoint and respectively makes rays to the multiple vertices P1 in the first model, intersecting with the sky box at P2.

[0259] S1909, the 3D rendering engine maps the rendering and shading information of P2 to P1 which belongs to the same ray as P2.

[0260] S1910, the 3D rendering engine sets the direction of the camera to the preset visiting direction of the Kth visiting point, and performs rendering based on the first model to obtain an exhibition hall image in the preset visiting direction of the Kth visiting point.

[0261] S1911, the browser displays the exhibition hall image in the preset visiting direction of the Kth visiting point.

[0262] Illustratively, S1905 to S1911 may refer to the description of S1507 to S1515 above, which will not be repeated here.

[0263] Exemplarily, after the browser receives the image in the preset visiting direction of the Kth visiting point sent by the 3D rendering engine, the image in the preset visiting direction of the Kth visiting point can be displayed; and then the user can see the exhibition hall image in the preset visiting direction of the Kth visiting point.

[0264] For example, when the user visits the Kth viewing point, he can also change the viewing direction to observe objects in other viewing directions of the viewing point. After the user performs the viewing direction change operation at the Kth viewing point, the steps executed by the browser in response to the user operation, and the image processing process executed by the 3D rendering engine after receiving the browser call, can be referred to Fig.10 As well as the above Fig.10 The description will not be repeated here.

[0265] It should be noted that after visiting the first visiting point, the user can perform a visiting point switching operation to switch to any other visiting point for visiting, and the embodiment of the present application does not limit this.

[0266] It should also be noted that after the user has visited the Kth sightseeing spot, the user can perform the sightseeing spot switching operation again to switch to any other sightseeing spot for visiting, for example, switching to sightseeing spot M for visiting.

[0267] In one possible manner, after the browser displays the exhibition hall image of the preset viewing direction of the first viewing point in response to the operation of entering the viewing interface, if the browser does not receive the user's viewing direction change operation within the first set time, the browser can call the three-dimensional rendering engine to render and display the exhibition hall image of any other viewing direction of the first viewing point. The first set time can be set as required, and the embodiment of the present application does not limit this. In this way, the user can visit multiple viewing directions of a viewing point without any operation.

[0268] In one possible manner, after the browser displays the exhibition hall image of the preset visiting direction of the first visiting point in response to the operation of entering the visiting interface, if the browser does not receive the user's visiting point switching operation within the second set time, the browser can call the three-dimensional rendering engine to render and display the exhibition hall image of the second preset visiting direction of the second visiting point. The second set time can be set as required, and the embodiment of the present application does not limit this. In this way, the user can visit multiple visiting points in the exhibition hall without any operation.

[0269] In one example, Fig. 20 A schematic block diagram of a device 2000 according to an embodiment of the present application is shown. The device 2000 may include: a processor 2001 and a transceiver / transceiver pin 2002 , and optionally, a memory 2003 .

[0270] The components of the device 2000 are coupled together via a bus 2004, wherein the bus 2004 includes a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, all buses are referred to as bus 2004 in the figure.

[0271] Optionally, the memory 2003 may be used for instructions in the aforementioned method embodiment. The processor 2001 may be used to execute instructions in the memory 2003, and control the receiving pin to receive a signal, and control the sending pin to send a signal.

[0272] The apparatus 2000 may be the electronic device or a chip of the electronic device in the above method embodiment.

[0273] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0274] This embodiment further provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the image processing method in the above-mentioned embodiment.

[0275] This embodiment further provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the image processing method in the above-mentioned embodiment.

[0276] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory so that the chip executes the image processing method in the above-mentioned method embodiments.

[0277] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above and will not be repeated here.

[0278] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0279] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0280] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0281] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0282] Any content of each embodiment of the present application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of the present application.

[0283] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0284] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0285] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (ReadOnly Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disk (CD-ROM) or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0286] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0287] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method comprises: In response to a received first user operation, mapping a panoramic image of a visit point corresponding to the first user operation to a first model, wherein the panoramic image is rendered based on a second model, and the fineness of the second model is higher than that of the first model; Rendering is performed based on the first model after mapping the panoramic image to obtain an image of the visiting spot, and the image of the visiting spot is displayed; Wherein, when the first user operation is an operation of entering a tour interface and the tour point is the first tour point, in response to receiving the first user operation, mapping the panoramic image of the tour point corresponding to the first user operation to the first model includes: In response to the received first user operation, downloading the first model and the panoramic image of the first visiting point from the server; Creating a three-dimensional scene, loading the first model into the three-dimensional scene, and setting a camera at the first visiting point; Mapping the panoramic image of the first visit point to the skybox of the three-dimensional scene; Taking the position of the camera as the endpoint, drawing rays to the multiple vertices P1 in the first model respectively, and intersecting with the sky box at P2; Map the rendering shading information of P2 to P1 which belongs to the same ray as P2; The rendering based on the first model after mapping the panorama to obtain the image of the visit point includes: The direction of the camera is set to the preset visiting direction of the first visiting point, and rendering is performed based on the direction of the camera and the first model after mapping the panoramic image to obtain an image of the first visiting point in the preset visiting direction.

2. The method according to claim 1, characterized in that The method of taking the position of the camera as an endpoint and drawing rays to the multiple vertices P1 in the first model respectively, and intersecting with the sky box at P2, includes: Taking the position of the camera as an endpoint, drawing rays to the multiple vertices P1 in the first model within the observation area, intersecting with the sky box at P2; The vertices within the observation area can be observed by the camera.

3. The method according to claim 2, characterized in that The method of taking the position of the camera as an endpoint and drawing rays to a plurality of vertices P1 in the observation area of ​​the first model respectively, and intersecting with the sky box at P2, comprises: With the camera position as the endpoint, rays are drawn to the multiple vertices P1 in the observation area corresponding to each viewing direction at the camera position in the first model, and intersect with the sky box at P2.

4. The method according to claim 2, characterized in that: The method of taking the position of the camera as an endpoint and drawing rays to a plurality of vertices P1 in the observation area of ​​the first model respectively, and intersecting with the sky box at P2, comprises: With the camera position as the endpoint, rays are drawn to the plurality of vertices P1 in the observation area corresponding to the preset viewing direction at the camera position in the first model, and intersect with the sky box at P2.

5. The method according to claim 1, characterized in that When the first user operation is a sightseeing point switching operation, and the sightseeing point is the Kth sightseeing point other than the first sightseeing point, in response to receiving the first user operation, mapping the panoramic image of the sightseeing point corresponding to the first user operation to the first model includes: In response to the received first user operation, downloading the panoramic image of the Kth sightseeing point from the server; The camera is moved from the Jth visiting point to the Kth visiting point, and positions of vertices in the first model are corrected, wherein the Jth visiting point is a visiting point other than the Kth visiting point; Mapping the panoramic image of the Kth visit point to the skybox of the three-dimensional scene; Taking the position of the camera as the endpoint, rays are drawn to the multiple vertices P1 in the first model respectively, and intersect with the skybox at P2; Map the rendering shading information of P2 to P1 which belongs to the same ray as P2.

6. The method according to claim 5, characterized in that The rendering based on the first model after mapping the panorama to obtain an image of the visit point includes: The direction of the camera is set to a preset visiting direction corresponding to the Kth visiting point, and rendering is performed based on the direction of the camera and the first model after mapping the panoramic image to obtain an image in the preset visiting direction of the Kth visiting point.

7. The method according to claim 5, characterized in that The step of correcting the positions of the vertices in the first model comprises: Determine, according to the coordinate information of the Jth visiting point and the coordinate information of the Kth visiting point, the camera motion offset information corresponding to moving the camera from the Jth visiting point to the Kth visiting point; Based on the camera motion offset information, the coordinate information of the vertices in the first model when the camera is located at the Jth visiting point is reversely corrected.

8. The method according to claim 1, characterized in that After displaying the image of the visit point, the method further comprises: In response to the received second user operation, rendering is performed based on the user visiting direction corresponding to the second user operation and the first model after mapping the panoramic image, so as to obtain an image of the visiting point in the user visiting direction.

9. An image processing method, characterized in that: Applied to electronic equipment, the method comprises: In response to the received first user operation, downloading from the server a tile image in a visiting direction of a visiting point corresponding to the first user operation, wherein the tile image is obtained by tile segmenting a panoramic image of the visiting point rendered based on a second model, and the second model has a higher degree of precision than the first model; Mapping the tile map to a first model; Rendering is performed based on the first model after mapping the tile map to obtain an image in the visiting direction of the visiting point, and the image in the visiting direction of the visiting point is displayed; Wherein, when the visiting point is the first visiting point, mapping the tile map to the first model includes: In response to a received first user operation, downloading the first model from the server; Creating a three-dimensional scene, loading the first model into the three-dimensional scene, and setting a camera at the first visiting point; Mapping the tile map in the visiting direction of the first visiting point to the sky box of the three-dimensional scene; Taking the position of the camera as the endpoint, drawing rays to the multiple vertices P1 in the first model respectively, and intersecting with the sky box at P2; Map the rendering shading information of P2 to P1 which belongs to the same ray as P2; The rendering based on the first model after mapping the tile map to obtain an image in the reference direction of the visit point includes: The direction of the camera is set to the visiting direction of the visiting point corresponding to the first user operation, and rendering is performed based on the direction of the camera and the first model after mapping the tile image to obtain an image in the preset visiting direction of the visiting point.

10. The method according to claim 9, characterized in that When the first user operation is an operation of entering a tour interface and the tour point is the first tour point, the step of downloading a tile map in a tour direction of the tour point corresponding to the first user operation from a server in response to the received first user operation includes: placing the camera at the visit point; Creating a reference cube with the camera position as the center; Determining coordinate information of a plurality of tile images to be downloaded in the panoramic image corresponding to the visiting point according to a preset visiting direction and a reference cube corresponding to the visiting point; According to the coordinate information, a tile map in the visiting direction of the visiting point is downloaded from the server.

11. The method according to claim 10, characterized in that When the first user operation is a sightseeing point switching operation, and the sightseeing point is a sightseeing point other than the first sightseeing point, the step of downloading a tile map in a sightseeing direction of the sightseeing point corresponding to the first user operation from a server in response to the received first user operation includes: placing the camera at the visit point; Creating a reference cube with the camera position as the center; Determining coordinate information of a plurality of tile images to be downloaded in the panoramic image corresponding to the visiting point according to a preset visiting direction and a reference cube corresponding to the visiting point; According to the coordinate information, a tile map in the visiting direction of the visiting point is downloaded from the server.

12. The method according to claim 10 or 11, characterized in that: The step of determining coordinate information of a plurality of tile images to be downloaded in the panoramic image corresponding to the visiting point according to the preset visiting direction and the reference cube corresponding to the visiting point includes: Determining cross sections of the reference cube and the camera viewing cone according to a preset viewing direction and a reference cube corresponding to the viewing point; Projection transformation is performed on the vertices in the first model that are located between the cross section in the viewing cone and the far plane of the viewing cone to obtain coordinate information of a plurality of tile images to be downloaded in the panoramic image corresponding to the visiting point.

13. The method according to claim 11, characterized in that When the visiting point is the Kth visiting point other than the first visiting point, mapping the tile map to the first model includes: The camera is moved from the Jth visiting point to the Kth visiting point, and positions of vertices in the first model are corrected, wherein the Jth visiting point is a visiting point other than the Kth visiting point; Mapping the tile image in the visiting direction of the Kth visiting point to the sky box of the three-dimensional scene; Taking the position of the camera as the endpoint, rays are drawn to the multiple vertices P1 in the first model respectively, and intersect with the skybox at P2; Map the rendering shading information of P2 to P1 which belongs to the same ray as P2.

14. The method according to claim 13, characterized in that The step of correcting the positions of the vertices in the first model comprises: Determine, according to the coordinate information of the Jth visiting point and the coordinate information of the Kth visiting point, the camera motion offset information corresponding to moving the camera from the Jth visiting point to the Kth visiting point; Based on the camera motion offset information, the coordinate information of the vertices in the first model when the camera is located at the Jth visiting point is reversely corrected.

15. The method according to claim 9 or 13, characterized in that: Taking the position of the camera as the endpoint, rays are drawn to the multiple vertices P1 in the first model respectively, and intersect with the skybox at P2, including: Taking the position of the camera as the endpoint, rays are drawn to the multiple vertices P1 in the observation area corresponding to the viewing direction at the position of the camera in the first model, and intersect with the sky box at P2.

16. The method according to claim 9, characterized in that After rendering the first model based on the mapped tile image to obtain the image of the visit point, the method further includes: Tile maps of other visiting directions of the visiting point are downloaded from the server.

17. The method according to claim 16, characterized in that After downloading the tile images of other visiting directions of the visiting point from the server, the method further comprises: Mapping the tile images in the other visiting directions of the visiting point to the sky box of the three-dimensional scene; Taking the position of the camera as the endpoint, rays are drawn to the multiple vertices P1 in the observation area corresponding to the other viewing directions located at the position of the camera in the first model, and intersect with the sky box at P2; Map the rendering shading information of P2 to P1 which belongs to the same ray as P2.

18. An electronic device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the image processing method according to any one of claims 1 to 8.

19. An electronic device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the image processing method according to any one of claims 9 to 17.

20. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send the signal to the processor, the signal comprising a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the image processing method according to any one of claims 1 to 8.

21. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send the signal to the processor, the signal comprising a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the image processing method described in any one of claims 9 to 17.

22. A computer storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the image processing method according to any one of claims 1 to 8.

23. A computer storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the image processing method according to any one of claims 9 to 17.

Images