Method, system, storage medium and electronic device for generating virtual reality scene

Abstract

This application discloses a method, system, storage medium, and electronic device for generating a virtual reality scene, relating to the field of smart homes. The method comprises: capturing an image of the real space without any home appliances placed in it to obtain a first panoramic image; determining, based on an operation instruction of a target object, virtual home appliances corresponding to the home appliances to be placed and the location information of the virtual home appliances in the virtual space, wherein the virtual space is a virtualized space of the real space; and generating a target virtual reality scene based on the first panoramic image, the virtual home appliances, and the location information. The above technical solution solves the problem of how to improve the efficiency of generating virtual reality scenes.

Description

Technical Field

[0001] The present application relates to the field of smart home, and more specifically, to a method, system, storage medium, and electronic device for generating a virtual reality scene. Background Art

[0002] With the development of VR (Virtual Reality) technology, VR scenes are gradually being applied in various fields. Currently, in the smart home sector, VR scenes are often used to render interior fully furnished model rooms. Traditionally, there are two rendering methods. One method is to render all objects in the entire interior scene into a panoramic image to construct the VR scene. This method makes it difficult to replace home appliances in the VR scene in real time, as each replacement requires re-rendering the panoramic image. The other method is to render every object in the entire interior scene using a 3D model. Although this method can replace home appliances in real time, the excessive number of 3D models in the VR scene consumes a lot of resources, leading to problems such as slow VR scene loading. Therefore, the related art faces the problem of how to improve the efficiency of VR scene generation.

[0003] Regarding the problem of how to improve the efficiency of generating virtual reality scenes in related technologies, no effective solution has been proposed yet. Summary of the Invention

[0004] The embodiments of the present application provide a method, system, storage medium, and electronic device for generating a virtual reality scene, so as to at least solve the problem of how to improve the generation efficiency of the virtual reality scene in the related art.

[0005] According to one embodiment of the present application, a method for generating a virtual reality scene is provided, including: when no household appliances are placed in the real space, capturing images of the real space to obtain a first panoramic image; determining, according to an operation instruction of a target object, a virtual household appliance corresponding to the household appliance to be placed and position information of the virtual household appliance in the virtual space, wherein the virtual space is a space obtained by virtualizing the real space; and generating a target virtual reality scene based on the first panoramic image, the virtual household appliance and the position information.

[0006] In an exemplary embodiment, a target virtual reality scene is generated based on the first panoramic image, the virtual home appliance and the position information, including: determining an initial virtual reality scene, the initial virtual reality scene being constructed based on a spherical model, the spherical model being a three-dimensional carrier model of the first panoramic image, and the pixels of the inner surface of the spherical model corresponding one-to-one with the pixels of the first panoramic image; determining the position of the virtual home appliance in the initial virtual reality scene according to the position information to obtain a first virtual reality scene; and updating first spatial visual information within the first virtual reality scene to obtain the target virtual reality scene, wherein the first spatial visual information represents visual information formed by the virtual home appliance occluding objects in the first panoramic image from an observation perspective facing the virtual home appliance.

[0007] In an exemplary embodiment, determining an initial virtual reality scene includes: establishing a spherical coordinate system with the geometric center of the spherical model as a first origin; establishing a three-dimensional rectangular coordinate system with the image acquisition position of the first panoramic image as a second origin; obtaining a coordinate correspondence between the spherical coordinate system and the three-dimensional rectangular coordinate system, wherein the first origin corresponds to the second origin; and mapping the first panoramic image onto the inner surface of the spherical model according to the coordinate correspondence to obtain the initial virtual reality scene.

[0008] In an exemplary embodiment, the position of the virtual home appliance in the first virtual reality scene is determined based on the virtual home appliance and the position information to obtain the first virtual reality scene, including: determining a first coordinate point based on the position information of the virtual home appliance in the virtual space, wherein the first coordinate point represents the coordinate point of the virtual home appliance in the three-dimensional rectangular coordinate system; determining a second coordinate point based on the first coordinate point and the coordinate correspondence, wherein the second coordinate point represents the coordinate point of the virtual home appliance in the spherical coordinate system; and determining the position of the virtual home appliance in the initial virtual reality scene based on the second coordinate point to obtain the first virtual reality scene.

[0009] In an exemplary embodiment, updating the first spatial visual information within the first virtual reality scene to obtain the target virtual reality scene includes: acquiring images in different directions based on a first preset point of the first virtual reality scene to obtain multiple images, wherein the first preset point corresponds to the geometric center of the spherical model, the different directions include the direction from the first preset point to a second preset point, and the second preset point includes the coordinate point corresponding to the center point of each surface in the real space in the first virtual reality scene; performing image recognition on the multiple images to obtain image recognition results; and when it is determined that the image recognition result is used to indicate the presence of a target virtual home appliance in the target direction, updating the first spatial visual information to obtain the target virtual reality scene.

[0010] In an exemplary embodiment, the first spatial visual information is updated to obtain the target virtual reality scene, including: determining a field of view corresponding to the target direction, and determining an obstacle object corresponding to the target virtual home appliance in the field of view; determining that the obstacle object is used to block the occlusion portion of the target virtual home appliance; determining a first image corresponding to the obstacle object from a first panoramic image, and determining an occlusion image corresponding to the occlusion portion in the first image from the first image; placing the occlusion image within the field of view to update the first spatial visual information to second spatial visual information to obtain a second virtual reality scene, wherein the second spatial visual information represents visual information formed when the occlusion image covers the target virtual home appliance from an observation perspective facing the target virtual home appliance; and adding a shadow image of the virtual home appliance to the second virtual reality scene to obtain the target virtual reality scene.

[0011] In an exemplary embodiment, determining the obstacle object corresponding to the target virtual home appliance in the field of view includes: determining a third preset point in the virtual space corresponding to the first preset point, wherein the first preset point and the third preset point correspond to the same position in the real space; determining the obstacle object based on a comparison result of a first distance and a second distance, wherein the first distance represents the distance between the target virtual home appliance in the virtual space and the third preset point, and the second distance represents the distance between the object in the target direction in the virtual space and the third preset point, and the distance between the obstacle object and the third preset point is less than the first distance.

[0012] In an exemplary embodiment, the shadow image of the virtual home appliance is added to the second virtual reality scene to obtain the target virtual reality scene, including: when the home appliance has been placed in the real space, image capture is performed on the real space to obtain a second panoramic image; a shadow image of the target home appliance is determined from the second panoramic image, wherein the target home appliance is the home appliance corresponding to the target virtual home appliance; and an image of a target area in the second virtual reality scene is replaced according to the shadow image of the target home appliance to obtain the target virtual reality scene, wherein the target area represents the shadow area corresponding to the target home appliance.

[0013] According to another aspect of an embodiment of the present application, a device for generating a virtual reality scene is also provided, including: an acquisition module, used to acquire images of the real space when no home appliance is placed in the real space, so as to obtain a first panoramic image; a determination module, used to determine, according to an operation instruction of a target object, a virtual home appliance corresponding to the home appliance to be placed and position information of the virtual home appliance in the virtual space, wherein the virtual space is a space obtained by virtualizing the real space; and a generation module, used to generate a target virtual reality scene based on the first panoramic image, the virtual home appliance and the position information.

[0014] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the above-mentioned method for generating a virtual reality scene when running.

[0015] According to another aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the method for generating a virtual reality scene through the computer program.

[0016] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, which implements the steps of any of the above method embodiments when executed by a processor.

[0017] In an embodiment of the present application, when no home appliances are placed in the real space, an image of the real space is captured to obtain a first panoramic image; virtual home appliances corresponding to the home appliances to be placed and position information of the virtual home appliances in the virtual space are determined according to the operation instructions of the target object, wherein the virtual space is a space obtained by virtualizing the real space; and a target virtual reality scene is generated based on the first panoramic image, the virtual home appliances, and the position information. By adopting the above technical solution, a panoramic image can be captured when no home appliances are placed in the model room, and then a virtual reality scene can be generated based on the captured panoramic image combined with the home appliance model, thereby solving the problem of how to improve the generation efficiency of virtual reality scenes, thereby achieving the effect of improving the generation efficiency of virtual reality scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 This is a schematic diagram of the hardware environment of a method for generating a virtual reality scene according to an embodiment of the present application;

[0021] Figure 2 is a flowchart of a method for generating a virtual reality scene according to an embodiment of the present application;

[0022] Figure 3 is a platform architecture diagram of a method for generating a virtual reality scene according to an embodiment of the present application;

[0023] Figure 4 is a data structure diagram of a home decoration design scheme according to an embodiment of the present application;

[0024] Figure 5 is a schematic diagram of a method for generating a virtual reality scene according to an embodiment of the present application;

[0025] Figure 6 is a schematic diagram of an occlusion relationship according to an embodiment of the present application;

[0026] Figure 7 is a schematic flow chart of a process for calling a cutout service according to an embodiment of the present application;

[0027] Figure 8is a schematic flow chart of a cutout algorithm according to an embodiment of the present application;

[0028] Figure 9 This is a structural block diagram of a device for generating a virtual reality scene according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] According to one aspect of the embodiment of the present application, a method for generating a virtual reality scene is provided. The method for generating a virtual reality scene is widely used in application scenarios such as software development. Optionally, in this embodiment, the method for generating a virtual reality scene can be applied to Figure 1 In the hardware environment shown in FIG. 1 , which is composed of a terminal device 102 and a server 104. Figure 1 As shown, the server 104 is connected to the terminal device 102 via a network, and can be used to provide services (such as application services, etc.) for the terminal or a client installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for the server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data computing services for the server 104.

[0032] The aforementioned network may include, but is not limited to, at least one of the following: a wired network and a wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: a wide area network, a metropolitan area network, and a local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity) and Bluetooth. The terminal device 102 may include, but is not limited to, a PC, a mobile phone, a tablet computer, a smart air conditioner, a smart range hood, a smart refrigerator, a smart oven, a smart stove, a smart washing machine, a smart water heater, a smart washing machine, a smart dishwasher, a smart projector, a smart TV, a smart clothes drying rack, smart curtains, a smart audio / video system, a smart socket, a smart speaker, a smart fresh air system, smart kitchen and bathroom equipment, smart bathroom equipment, a smart robot vacuum, a smart window cleaning robot, a smart robot mop, a smart air purifier, a smart steamer, a smart microwave oven, a smart kitchen appliance, a smart purifier, a smart water dispenser, a smart door lock, and the like.

[0033] In this embodiment, a method for generating a virtual reality scene is provided. Figure 2 4 is a flow chart of a method for generating a virtual reality scene according to an embodiment of the present application, the flow comprising the following steps:

[0034] Step S202: When no household appliance is placed in the real space, an image of the real space is captured to obtain a first panoramic image;

[0035] Optionally, in the above-mentioned step S202, for example, the above-mentioned real space can be a house model room, and the above-mentioned first panoramic image can fully present the image of the 360° environment of the house model room, wherein the above-mentioned first panoramic image includes the sofa, cabinet, bed and other furniture items in the house model room.

[0036] Step S204, determining, according to the operation instruction of the target object, a virtual home appliance corresponding to the home appliance to be placed and location information of the virtual home appliance in a virtual space, wherein the virtual space is a space obtained by virtualizing the real space;

[0037] Optionally, in the above-mentioned step S204, the above-mentioned virtual space can be the three-dimensional floor plan corresponding to the above-mentioned house model room on the home decoration design platform. On the home decoration design platform, the designer can select a three-dimensional home appliance model from the model library and add it to the three-dimensional floor plan (equivalent to the above-mentioned virtual home appliance equipment), and adjust the position of the three-dimensional home appliance model in the floor plan.

[0038] Step S206: Generate a target virtual reality scene based on the first panoramic image, the virtual home appliance and the location information.

[0039] Through the above steps, when no home appliances are placed in the real space, the real space is imaged and a first panoramic image is obtained; the virtual home appliance corresponding to the home appliance to be placed and the position information of the virtual home appliance in the virtual space are determined according to the operation instructions of the target object, wherein the virtual space is a space obtained by virtualizing the real space; and the target virtual reality scene is generated based on the first panoramic image, the virtual home appliance, and the position information. Using the above technical solution, a panoramic image can be captured when no home appliances are placed in the model room, and then a virtual reality scene can be generated based on the captured panoramic image and the home appliance model, solving the problem of how to improve the generation efficiency of virtual reality scenes, thereby achieving the effect of improving the generation efficiency of virtual reality scenes.

[0040] In an exemplary embodiment, a target virtual reality scene is generated based on the first panoramic image, the virtual home appliance and the position information, including: determining an initial virtual reality scene, the initial virtual reality scene being constructed based on a spherical model, the spherical model being a three-dimensional carrier model of the first panoramic image, and the pixels of the inner surface of the spherical model corresponding one-to-one with the pixels of the first panoramic image; determining the position of the virtual home appliance in the initial virtual reality scene according to the position information to obtain a first virtual reality scene; and updating first spatial visual information within the first virtual reality scene to obtain the target virtual reality scene, wherein the first spatial visual information represents visual information formed by the virtual home appliance occluding objects in the first panoramic image from an observation perspective facing the virtual home appliance.

[0041] Optionally, in the above embodiment, for example, the first panoramic image is obtained by photographing the model room using a 360-degree panoramic camera. Mapping the first panoramic image onto the inner surface of a spherical model can be accomplished using 3D modeling software. A spherical model is created in the 3D modeling software as a carrier model for the panoramic image. The first panoramic image is then imported and mapped onto the interior of the sphere. Looking around from the center of the sphere from a first-person perspective, a seamless panorama of the model room is seen, i.e., the initial virtual reality scene.

[0042] Based on the position of the 3D home appliance model in the floor plan, the distance between the 3D home appliance model and the first panoramic image acquisition point in the model room can be determined. Based on this distance, the distance between the 3D home appliance model and the center point of the sphere in the spherical model (initial virtual reality scene) can be determined. Furthermore, the position of the 3D home appliance model in the initial virtual reality scene can be determined to obtain the first virtual reality scene.

[0043] In the first virtual display scene, the first panoramic image is rendered at the farthest end, closely attached to the inner surface of the sphere, while the 3D appliance model is rendered in front of the first panoramic image. Therefore, from the first perspective (equivalent to the first spatial visual information) facing the 3D appliance model from the center of the sphere, the 3D appliance model will inevitably block objects in the first panoramic image. In actual design, objects in the first panoramic image may obstruct the 3D appliance model. For example, if the air conditioner is located between the sofa and the wall, the correct visual information should be that the sofa blocks the air conditioner, and the air conditioner blocks the wall. However, in the first virtual display scene, the sofa and the wall are rendered together at the farthest end, and the air conditioner model blocks both the sofa and the wall. Therefore, the first spatial visual information in the first virtual display scene may be incorrect and need to be updated.

[0044] In an exemplary embodiment, determining an initial virtual reality scene includes: establishing a spherical coordinate system with the geometric center of the spherical model as a first origin; establishing a three-dimensional rectangular coordinate system with the image acquisition position of the first panoramic image as a second origin; obtaining a coordinate correspondence between the spherical coordinate system and the three-dimensional rectangular coordinate system, wherein the first origin corresponds to the second origin; and mapping the first panoramic image onto the inner surface of the spherical model according to the coordinate correspondence to obtain the initial virtual reality scene.

[0045] In an exemplary embodiment, the position of the virtual home appliance in the first virtual reality scene is determined based on the virtual home appliance and the position information to obtain the first virtual reality scene, including: determining a first coordinate point based on the position information of the virtual home appliance in the virtual space, wherein the first coordinate point represents the coordinate point of the virtual home appliance in the three-dimensional rectangular coordinate system; determining a second coordinate point based on the first coordinate point and the coordinate correspondence, wherein the second coordinate point represents the coordinate point of the virtual home appliance in the spherical coordinate system; and determining the position of the virtual home appliance in the initial virtual reality scene based on the second coordinate point to obtain the first virtual reality scene.

[0046] Optionally, in the above embodiment, the surface of the sphere model is composed of countless points. In computer graphics, a spherical coordinate system (polar coordinate system) can be used to define each point on its surface. The spherical coordinate system consists of two angle parameters (polar angle θ and azimuth angle φ) and a radius r. The following formula can be used to map a point (X, Y, Z) in the rectangular coordinate system to a point (r, θ, ϕ) in the spherical coordinate system:

[0047] .

[0048] In an exemplary embodiment, updating the first spatial visual information within the first virtual reality scene to obtain the target virtual reality scene includes: acquiring images in different directions based on a first preset point of the first virtual reality scene to obtain multiple images, wherein the first preset point corresponds to the geometric center of the spherical model, the different directions include the direction from the first preset point to a second preset point, and the second preset point includes the coordinate point corresponding to the center point of each surface in the real space in the first virtual reality scene; performing image recognition on the multiple images to obtain image recognition results; and when it is determined that the image recognition result is used to indicate the presence of a target virtual home appliance in the target direction, updating the first spatial visual information to obtain the target virtual reality scene.

[0049] Optionally, in the above embodiment, after generating the first virtual reality scene, it is necessary to monitor whether the visual relationship in the first virtual reality scene is correct. The actual model room is composed of six faces, corresponding to the six directions of up, down, left, right, front and back, and respectively identifying whether there are home appliances in each direction. If there are home appliances, the visual relationship in that direction needs to be adjusted.

[0050] In the above recognition process, it is necessary to determine the recognition perspective. First, the center point of the first virtual reality scene (that is, the geometric center of the spherical model) is determined as the position of the first-person camera. Then, the center point of a surface of the model room is aligned from this position for observation to obtain a visual image in that direction, and then the presence of home appliances in the image is identified.

[0051] In an exemplary embodiment, the first spatial visual information is updated to obtain the target virtual reality scene, including: determining a field of view corresponding to the target direction, and determining an obstacle object corresponding to the target virtual home appliance in the field of view; determining that the obstacle object is used to block the occlusion portion of the target virtual home appliance; determining a first image corresponding to the obstacle object from a first panoramic image, and determining an occlusion image corresponding to the occlusion portion in the first image from the first image; placing the occlusion image within the field of view to update the first spatial visual information to second spatial visual information to obtain a second virtual reality scene, wherein the second spatial visual information represents visual information formed when the occlusion image covers the target virtual home appliance from an observation perspective facing the target virtual home appliance; and adding a shadow image of the virtual home appliance to the second virtual reality scene to obtain the target virtual reality scene.

[0052] Optionally, in the above embodiment, for example, when looking at a certain surface of the model room from the center point of the first virtual reality scene, the air-conditioning model in the model room should be blocked by a corner of the sofa, but the perspective effect of the first virtual reality scene is incorrect, and the air-conditioning model blocks the sofa. At this time, it is necessary to crop the image of the sofa from the first panoramic image, and then further crop the sofa image to the part that blocks the air-conditioning model to obtain the blocked image, and then paste the blocked image in front of the air-conditioning model. In this way, when looking at the air-conditioning model from the center point of the virtual reality scene, you can correctly see that the sofa blocks part of the air-conditioning model.

[0053] In an exemplary embodiment, determining the obstacle object corresponding to the target virtual home appliance in the field of view includes: determining a third preset point in the virtual space corresponding to the first preset point, wherein the first preset point and the third preset point correspond to the same position in the real space; determining the obstacle object based on a comparison result of a first distance and a second distance, wherein the first distance represents the distance between the target virtual home appliance in the virtual space and the third preset point, and the second distance represents the distance between the object in the target direction in the virtual space and the third preset point, and the distance between the obstacle object and the third preset point is less than the first distance.

[0054] Optionally, in the above embodiment, the designer can already determine the positional relationship between the 3D appliance model and other items in the model room when the design is completed on the model room floor plan on the design platform. Therefore, the positional relationship between the 3D appliance model and other items in the model room on the floor plan can be used to determine which obstacles will likely block the 3D appliance model in the target direction. The obstacles and the 3D appliance model are located in the same direction, and the distance between the obstacles and the camera's observation point is less than the distance between the 3D appliance model and the camera's observation point.

[0055] In an exemplary embodiment, the shadow image of the virtual home appliance is added to the second virtual reality scene to obtain the target virtual reality scene, including: when the home appliance has been placed in the real space, image capture is performed on the real space to obtain a second panoramic image; a shadow image of the target home appliance is determined from the second panoramic image, wherein the target home appliance is the home appliance corresponding to the target virtual home appliance; and an image of a target area in the second virtual reality scene is replaced according to the shadow image of the target home appliance to obtain the target virtual reality scene, wherein the target area represents the shadow area corresponding to the target home appliance.

[0056] Optionally, in the above embodiment, although the spatial visual information in the second VR scene has been adjusted to obtain the correct occlusion relationship, the background image in the second VR scene is the first panoramic image. The first panoramic image was captured in the model room without any appliances. Now that the scene has newly added appliances, the background image lacks the corresponding shadow effects for the appliances. Therefore, it is necessary to add shadow effects for the appliances. Specifically, a panoramic image can be captured with the appliances already in the model room. The shadow image corresponding to the placed appliances can then be cropped from the captured panoramic image. This shadow image can then be pasted to the corresponding position in the second VR scene to obtain the final target VR scene.

[0057] Through the above embodiments, the use of panoramic images combined with three-dimensional models for hybrid rendering of VR scenes can reduce the memory usage of the three-dimensional models in the scene, and the home appliances in the scene can be replaced at any time, which greatly improves the generation efficiency of the VR scene and ensures that the objects in the scene have the correct visual display effect.

[0058] In order to better understand the process of the above-mentioned virtual reality scene generation method, the implementation method flow of the above-mentioned virtual reality scene generation is described below in combination with an optional embodiment, but it is not intended to limit the technical solution of the embodiment of this application.

[0059] In an optional embodiment, Figure 3 is a platform architecture diagram of a method for generating a virtual reality scene according to an embodiment of the present application, such as Figure 3 As shown, this method mainly involves the following platforms.

[0060] Designer Platform: A 3D interior design platform where interior designers can create complete home renovation plans. The design process is visualized in 3D, and the 3D floor plans on the platform are rendered using 3D models. The platform also offers panoramic rendering capabilities and an open API (Application Programming Interface) through which users can access relevant information about design plans.

[0061] Server / Database: This is the data server used to acquire, store, and distribute data. By calling the designer platform's open API, it determines relevant data for the design plan, such as room lists and appliance lists. It also provides interfaces for other clients to use, such as querying design plan data.

[0062] Mobile page: Users view the VR scene on this page. The VR scene is displayed on the mobile terminal after mixing and rendering the panoramic image of the sample room without products (products here refer to home appliances, not including furniture and other items) and the three-dimensional home appliance models. It is easy to share and disseminate. In addition, the home appliances in the VR scene can be replaced in real time. Users can easily and intuitively see the effects of different home appliances placed in the VR scene. During the rendering process, the mobile page will initiate a cutout request to the algorithm engine.

[0063] Algorithm Engine: Provides appliance occlusion cutout services and algorithms. This solves the problem of incorrect front-to-back occlusion relationships between furniture or walls in background panoramas of VR scenes and 3D appliance models. The occlusion cutout algorithm generates one or more foreground occlusion images for the 3D appliance model. When rendering the VR scene, the foreground occlusion images are placed in front of the appliance model to achieve the correct occlusion relationship.

[0064] In an optional embodiment, Figure 4 This is a data structure diagram of a home decoration design scheme according to an embodiment of the present application, such as Figure 4 As shown, the home decoration design plan specifically includes the following data.

[0065] Basic information: plan name, plan ID, unit area, unit specifications, unit floor plan, community name, unit label, etc.

[0066] Room information list: room name, room ID, panoramic color image with products, panoramic channel image with products, panoramic color image without products, panoramic channel image without products, room coordinates (i.e. camera coordinates), room area, home appliance list, etc.

[0067] Home appliance information list: home appliance ID, home appliance name, home appliance 3D model download address, home appliance pose (including model placement, rotation angle, and scaling), home appliance size (width, height, and depth), home appliance foreground occlusion image list, and home appliance replaceable list.

[0068] In an optional embodiment, Figure 5 is a schematic diagram of a method for generating a virtual reality scene according to an embodiment of the present application, such as Figure 5 As shown, the specific process includes the following:

[0069] 1. After completing the home design plan on the designer platform, the interior designer renders a panoramic image without products (equivalent to the first panoramic image) and a panoramic image with products (equivalent to the second panoramic image) for each room. The camera positions of the two panoramic images are the same. The panoramic image with products includes home appliances, while the panoramic image without products does not. It should be noted that the panoramic image without products can include room furniture, such as beds and sofas.

[0070] 2. The server / database obtains relevant data of the home decoration design plan by calling the API provided by the designer platform, including basic information of the plan, room information, home appliance information, etc.

[0071] 3. Render the VR scene on a mobile page (such as an H5 page). The VR scene rendering process includes determining the background panorama rendering, 3D model rendering, and adding the foreground occlusion image of home appliances.

[0072] The background panorama rendering method is to stick the panorama on the inner surface of a sphere, and then set the camera at the center of the sphere, then the most basic panoramic VR scene is formed.

[0073] 3D model rendering is to render the 3D home appliance model in front of the background panorama. The 3D home appliance model is closer to the camera at the center point than the panorama. The front-to-back relationship between the 3D home appliance model and the furniture in the panorama may be incorrect, which leads to an incorrect occlusion relationship between the 3D home appliance model and the furniture in the panorama, forming an incorrect spatial visual effect.

[0074] By adding appliance foreground occlusion images, you can correct incorrect occlusion relationships between the 3D appliance model and the furniture in the panorama, restoring the spatial visual effect to normal. When a 3D appliance model is obscured by other furniture or walls, a separate appliance foreground occlusion image is generated using the appliance occlusion cutout algorithm. This image is then placed in front of the 3D appliance model during 3D rendering. From the camera's perspective, the 3D appliance model appears to be properly obscured by the furniture. The appliance foreground occlusion image is generated by calling the cutout service when you first enter the VR scene.

[0075] Specifically, such as Figure 6 As shown in the example, in a real-world room, there's a standing air conditioner with a sofa next to it. Looking at the air conditioner from a preset position in the room, one corner is obscured by the sofa. If a panorama is rendered at this position, the entire room, including the sofa, is rendered. Then, using the 3D model to render the air conditioner in VR, the entire air conditioner appears in front of the sofa, creating an incorrect occlusion relationship. The correct occlusion relationship should be one corner of the air conditioner behind the sofa. By invoking the algorithm engine's cutout algorithm, the sofa can be cut out from the panorama as a foreground occlusion image and placed in front of the air conditioner. This way, when looking at the air conditioner from the preset position in VR, one corner of the air conditioner is obscured by the sofa, creating the correct occlusion relationship. It's important to note that the foreground occlusion image doesn't necessarily have to be the entire sofa. In some cases, only a portion of the sofa obscures the air conditioner, and in this case, only that portion of the sofa is cut out as the foreground occlusion image.

[0076] 4. The cutout algorithm engine starts cutting out the image after receiving the cutout request. After the cutout is successful, it will produce the foreground occlusion image and shadow image of the home appliance.

[0077] The above process of calling the cutout service to generate the appliance foreground occlusion image and shadow image when the mobile page enters the VR scene for the first time is as follows: Figure 7 As shown, the following steps are included:

[0078] Step S701: Entering a VR scene for the first time.

[0079] Step S702: traverse the six surfaces of the VR scene: top, bottom, front, back, left, and right.

[0080] Step S703: Determine whether there is any home appliance on the surface. If yes, execute step S704; otherwise, execute step S705.

[0081] Step S704: No home appliances, no need to cut out.

[0082] Step S705: There are home appliances that need to be cut out. The panoramic image data and home appliance data are sent to the cutout service to initiate a cutout request.

[0083] Step S706: The cutout service performs cutout processing based on the received data to generate a foreground occlusion image and a shadow image of the home appliance.

[0084] Step S707: Save the generated image to the database.

[0085] Step S708: Cutting out is completed.

[0086] In an optional embodiment, the process of the above step S806 of the cutout service performing cutout processing based on the received data to generate the foreground occlusion image and the shadow image of the home appliance is as follows: Figure 8 As shown, the specific steps include:

[0087] Step S801: Input the appliance model outline, room information, appliance information, product panorama, product-less panorama and other data to the cutout service.

[0088] Step S802: Check whether the input data is qualified, if so, execute step S804, otherwise execute step S803.

[0089] Step S803: Return to cutout failure.

[0090] Step S804: Identify home appliances.

[0091] Step S805: When a home appliance is identified, determine whether the occlusion relationship of the home appliance is correct. If so, execute steps S807 and S808; otherwise, execute step S806.

[0092] Step S806: No processing is required, and the cutout process ends.

[0093] Step S807: Perform synthesis processing based on the panoramic image with products and the panoramic image without products to generate a shadow image.

[0094] Step S808: Determine the obstruction that blocks the home appliance, and extract the obstruction from the panoramic image as a foreground obstruction image.

[0095] Step S809: Output the shadow map and foreground occlusion image.

[0096] Through the above embodiments, it is possible to achieve mixed rendering of panoramas and 3D appliance models in mobile VR showrooms, and then accurately process the front-to-back occlusion relationship between 3D appliance models and panoramic furniture through the appliance occlusion cutout algorithm, ensuring the authenticity and consistency of VR scene rendering, while retaining the high-quality visual effects of offline rendered panoramas and supporting the real-time replacement function of appliances. In addition, by optimizing the rendering method of VR scenes, the resource usage and loading delay of the mobile terminal are significantly reduced, thereby improving the smoothness of user operations and interactive experience, and ultimately achieving efficient, lightweight and immersive VR home improvement plan display and dissemination.

[0097] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0098] Figure 9 is a structural block diagram of a device for generating a virtual reality scene according to an embodiment of the present application; Figure 9 Shown, including:

[0099] An acquisition module 92 is configured to acquire an image of the real space without placing any household appliances in the real space to obtain a first panoramic image;

[0100] a determination module 94 for determining, based on the operation instruction of the target object, a virtual home appliance corresponding to the home appliance to be placed and location information of the virtual home appliance in a virtual space, wherein the virtual space is a space obtained by virtualizing the real space;

[0101] The generating module 96 is configured to generate a target virtual reality scene based on the first panoramic image, the virtual home appliance and the location information.

[0102] By means of the above-mentioned device, when no household appliances are placed in the real space, an image of the real space is captured to obtain a first panoramic image; the virtual household appliance corresponding to the household appliance to be placed and the position information of the virtual household appliance in the virtual space are determined according to the operation instructions of the target object, wherein the virtual space is a space obtained by virtualizing the real space; and a target virtual reality scene is generated based on the first panoramic image, the virtual household appliance, and the position information. By adopting the above-mentioned technical solution, a panoramic image can be captured when no household appliances are placed in the model room, and then a virtual reality scene can be generated based on the captured panoramic image combined with the household appliance model, thus solving the problem of how to improve the generation efficiency of virtual reality scenes, thereby achieving the effect of improving the generation efficiency of virtual reality scenes.

[0103] In an exemplary embodiment, the generation module 96 is further used to determine an initial virtual reality scene, where the initial virtual reality scene is constructed based on a spherical model, which is a three-dimensional carrier model of the first panoramic image, and the pixels on the inner surface of the spherical model correspond one-to-one to the pixels of the first panoramic image; determine the position of the virtual home appliance in the initial virtual reality scene based on the position information to obtain a first virtual reality scene; and update the first spatial visual information in the first virtual reality scene to obtain the target virtual reality scene, wherein the first spatial visual information represents visual information formed by the virtual home appliance occluding objects in the first panoramic image from an observation perspective facing the virtual home appliance.

[0104] In an exemplary embodiment, the generation module 96 is further configured to establish a spherical coordinate system with the geometric center of the spherical model as a first origin; establish a three-dimensional rectangular coordinate system with the image acquisition position of the first panoramic image as a second origin; obtain a coordinate correspondence between the spherical coordinate system and the three-dimensional rectangular coordinate system, wherein the first origin corresponds to the second origin; and map the first panoramic image onto the inner surface of the spherical model according to the coordinate correspondence to obtain the initial virtual reality scene.

[0105] In an exemplary embodiment, the generation module 96 is further used to determine a first coordinate point based on the position information of the virtual home appliance in the virtual space, wherein the first coordinate point represents the coordinate point of the virtual home appliance in the three-dimensional rectangular coordinate system; determine a second coordinate point based on the first coordinate point and the coordinate correspondence, wherein the second coordinate point represents the coordinate point of the virtual home appliance in the spherical coordinate system; determine the position of the virtual home appliance in the initial virtual reality scene based on the second coordinate point to obtain the first virtual reality scene.

[0106] In an exemplary embodiment, the generation module 96 is further used to collect images in different directions based on a first preset point of the first virtual reality scene to obtain multiple images, wherein the first preset point corresponds to the geometric center of the spherical model, the different directions include the direction from the first preset point to a second preset point, and the second preset point includes the coordinate point corresponding to the center point of each surface in the real space in the first virtual reality scene; perform image recognition on the multiple images to obtain image recognition results; and when it is determined that the image recognition result is used to indicate the existence of a target virtual home appliance in the target direction, update the first spatial visual information to obtain the target virtual reality scene.

[0107] In an exemplary embodiment, the generation module 96 is further used to determine the field of view corresponding to the target direction, and determine the obstacle object corresponding to the target virtual home appliance in the field of view; determine that the obstacle object is used to block the blocked part of the target virtual home appliance; determine the first image corresponding to the obstacle object from the first panoramic image, and determine the blocking image corresponding to the blocked part in the first image from the first image; place the blocking image in the field of view to update the first spatial visual information to the second spatial visual information to obtain a second virtual reality scene, wherein the second spatial visual information represents the visual information formed by the blocking image covering the target virtual home appliance under the observation perspective facing the target virtual home appliance; and add the shadow image of the virtual home appliance to the second virtual reality scene to obtain the target virtual reality scene.

[0108] In an exemplary embodiment, the generation module 96 is further used to determine a third preset point in the virtual space corresponding to the first preset point, wherein the first preset point and the third preset point correspond to the same position in the real space; and determine the obstacle object based on the comparison result of the first distance and the second distance, wherein the first distance represents the distance between the target virtual home appliance in the virtual space and the third preset point, and the second distance represents the distance between the object in the target direction in the virtual space and the third preset point, and the distance between the obstacle object and the third preset point is less than the first distance.

[0109] In an exemplary embodiment, the generation module 96 is further configured to, when a household appliance has been placed in the real space, capture an image of the real space to obtain a second panoramic image; determine a shadow image of a target household appliance from the second panoramic image, wherein the target household appliance is the household appliance corresponding to the target virtual household appliance; and replace an image of a target area in the second virtual reality scene according to the shadow image of the target household appliance to obtain the target virtual reality scene, wherein the target area represents a shadow area corresponding to the target household appliance.

[0110] An embodiment of the present application further provides a storage medium, which includes a stored program, wherein the program executes any of the above methods when it is run.

[0111] Optionally, in this embodiment, the storage medium may be configured to store program codes for executing the following steps:

[0112] S1, when no household appliances are placed in the real space, capturing images of the real space to obtain a first panoramic image;

[0113] S2, determining, according to the operation instruction of the target object, a virtual home appliance corresponding to the home appliance to be placed and location information of the virtual home appliance in a virtual space, wherein the virtual space is a space obtained by virtualizing the real space;

[0114] S3: Generate a target virtual reality scene based on the first panoramic image, the virtual home appliance and the location information.

[0115] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0116] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0117] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0118] S1, when no household appliances are placed in the real space, capturing images of the real space to obtain a first panoramic image;

[0119] S2, determining, according to the operation instruction of the target object, a virtual home appliance corresponding to the home appliance to be placed and location information of the virtual home appliance in a virtual space, wherein the virtual space is a space obtained by virtualizing the real space;

[0120] S3: Generate a target virtual reality scene based on the first panoramic image, the virtual home appliance and the location information.

[0121] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0122] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0123] An embodiment of the present application further provides another computer program product, comprising a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above method embodiments are implemented.

[0124] An embodiment of the present application also provides a computer program, which includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any of the above method embodiments.

[0125] Optionally, specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0126] Obviously, those skilled in the art should understand that the modules or steps of the present application described above can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices. Alternatively, they can be implemented using program code executable by the computing device, so that they can be stored in a storage device and executed by the computing device. In some cases, the steps shown or described can be performed in a different order than herein, or they can be made into separate integrated circuit modules, or multiple modules or steps can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0127] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for generating a virtual reality scene, characterized in that: include: When no household appliances are placed in the real space, image acquisition is performed on the real space to obtain a first panoramic image; Determining, according to the operation instruction of the target object, a virtual home appliance corresponding to the home appliance to be placed and position information of the virtual home appliance in a virtual space, wherein the virtual space is a space obtained by virtualizing the real space; generating a target virtual reality scene in the real space based on the first panoramic image, the virtual home appliance, and the position information; Among them, generating a target virtual reality scene based on the first panoramic image, the virtual home appliance and the position information includes: determining an initial virtual reality scene, the initial virtual reality scene is constructed based on a spherical model, the spherical model is a three-dimensional carrier model of the first panoramic image, and the pixel points of the inner surface of the spherical model correspond one-to-one with the pixel points of the first panoramic image; determining the position of the virtual home appliance in the initial virtual reality scene according to the position information to obtain a first virtual reality scene; updating first spatial visual information in the first virtual reality scene to obtain the target virtual reality scene, wherein the first spatial visual information represents visual information formed by the virtual home appliance occluding objects in the first panoramic image from an observation perspective facing the virtual home appliance.

2. The method for generating a virtual reality scene according to claim 1, wherein: Determine the initial VR scene, including: Establishing a spherical coordinate system with the geometric center of the spherical model as the first origin; Establishing a three-dimensional rectangular coordinate system with the image acquisition position of the first panoramic image as a second origin; Acquire a coordinate correspondence between the spherical coordinate system and the three-dimensional rectangular coordinate system, wherein the first origin corresponds to the second origin; The first panoramic image is mapped to the inner surface of the spherical model according to the coordinate correspondence to obtain the initial virtual reality scene.

3. The method for generating a virtual reality scene according to claim 2, wherein: Determining the position of the virtual home appliance in the first virtual reality scene according to the virtual home appliance and the position information to obtain the first virtual reality scene includes: determining a first coordinate point according to the position information of the virtual home appliance in the virtual space, wherein the first coordinate point represents the coordinate point of the virtual home appliance in the three-dimensional rectangular coordinate system; determining a second coordinate point according to the first coordinate point and the coordinate correspondence, wherein the second coordinate point represents the coordinate point of the virtual home appliance in the spherical coordinate system; The position of the virtual home appliance in the initial virtual reality scene is determined according to the second coordinate point to obtain the first virtual reality scene.

4. The method for generating a virtual reality scene according to claim 1, wherein: Updating the first spatial visual information in the first virtual reality scene to obtain the target virtual reality scene includes: Based on a first preset point in the first virtual reality scene, image acquisition is performed in different directions to obtain a plurality of images, wherein the first preset point corresponds to the geometric center of the spherical model, the different directions include directions from the first preset point to a second preset point, and the second preset point includes coordinates corresponding to the center point of each surface in the real space in the first virtual reality scene; performing image recognition on the multiple images to obtain image recognition results; When it is determined that the image recognition result indicates that a target virtual home appliance exists in a target direction, the first spatial visual information is updated to obtain the target virtual reality scene.

5. The method for generating a virtual reality scene according to claim 4, wherein: Updating the first spatial visual information to obtain the target virtual reality scene includes: Determine a visual field area corresponding to the target direction, and determine an obstacle object corresponding to the target virtual home appliance in the visual field area; Determine an obstruction portion of the target virtual home appliance that is obstructed by the obstacle object; Determining a first image corresponding to the obstacle object from the first panoramic image, and determining an occlusion image corresponding to the occluded portion in the first image from the first image; Placing the occlusion image in the field of view to update the first spatial visual information to second spatial visual information, thereby obtaining a second virtual reality scene, wherein the second spatial visual information represents visual information formed by the occlusion image covering the target virtual home appliance from an observation perspective facing the target virtual home appliance; The shadow image of the virtual home appliance is added to the second virtual reality scene to obtain the target virtual reality scene.

6. The method for generating a virtual reality scene according to claim 5, characterized in that: Determining an obstacle object corresponding to the target virtual home appliance in the field of view includes: Determining a third preset point in the virtual space corresponding to the first preset point, wherein the first preset point and the third preset point correspond to the same position in the real space; The obstacle object is determined based on the comparison result of the first distance and the second distance, wherein the first distance represents the distance between the target virtual home appliance in the virtual space and the third preset point, and the second distance represents the distance between the object in the target direction in the virtual space and the third preset point, and the distance between the obstacle object and the third preset point is less than the first distance.

7. The method for generating a virtual reality scene according to claim 5, wherein: Adding the shadow image of the virtual home appliance to the second virtual reality scene to obtain the target virtual reality scene includes: When the home appliance is placed in the real space, performing image acquisition on the real space to obtain a second panoramic image; determining a shadow image of a target home appliance from the second panoramic image, wherein the target home appliance is the home appliance corresponding to the target virtual home appliance; The image of the target area in the second virtual reality scene is replaced according to the shadow image of the target home appliance to obtain the target virtual reality scene, wherein the target area represents the shadow area corresponding to the target home appliance.

8. A device for generating a virtual reality scene, characterized in that: include: an acquisition module, configured to acquire an image of the real space without placing any household appliances in the real space, to obtain a first panoramic image; a determination module, configured to determine, based on an operation instruction of a target object, a virtual home appliance corresponding to the home appliance to be placed and location information of the virtual home appliance in a virtual space, wherein the virtual space is a space obtained by virtualizing the real space; a generating module, configured to generate a target virtual reality scene based on the first panoramic image, the virtual home appliance, and the location information; The generation module is further configured to determine an initial virtual reality scene, where the initial virtual reality scene is constructed based on a spherical model, which is a three-dimensional carrier model of the first panoramic image, and where pixels on an inner surface of the spherical model correspond one-to-one to pixels on the first panoramic image; determine a position of the virtual home appliance in the initial virtual reality scene based on the position information to obtain a first virtual reality scene; and update first spatial visual information within the first virtual reality scene to obtain a target virtual reality scene, where the first spatial visual information represents visual information formed by the virtual home appliance occluding objects in the first panoramic image from an observation perspective facing the virtual home appliance.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 7 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 7 through the computer program.

Images