Virtual reality scene generation method and system, storage medium and electronic device

Abstract

The invention discloses a virtual reality scene generation method and system, a storage medium and an electronic device, and relates to the field of smart home, and the virtual reality scene generation method comprises the steps: carrying out the image collection of a real space under the condition that no household appliance is placed in the real space, and obtaining a first panoramic image; according to an operation instruction of a target object, determining a virtual household appliance corresponding to a household appliance to be placed and position information of the virtual household appliance in a virtual space, the virtual space being a space obtained by virtualizing the real space; and generating a target virtual reality scene based on the first panoramic image, the virtual household appliance and the position information. By adopting the technical scheme, the problem of how to improve the generation efficiency of the virtual reality scene is solved.

Description

Technical Field

[0001] The present application relates to the field of smart home, and in particular, to a method, a system, a storage medium, and an electronic device for generating a virtual reality scene. Background Art

[0002] With the development of VR (Virtual Reality) technology, VR scenes have gradually been applied in various fields. Currently, in the field of smart home, VR scenes are usually used to render interior fine decoration model rooms. There are two traditional rendering methods. One method is to render all objects in an entire indoor scene in a single panoramic image to construct a VR scene. This method is very difficult to achieve real-time replacement of home appliances in the VR scene, and each time a home appliance is replaced, the panoramic image needs to be re-rendered. Another method is to render each object in the entire indoor scene through a 3D model. Although this method can replace home appliances in real time, due to the excessive number of 3D models in the VR scene, it occupies a large amount of resources, resulting in problems such as slow loading of the VR scene. Therefore, there is a problem in the related art of how to improve the generation efficiency of virtual reality scenes.

[0003] In view of the problem in the related art of how to improve the generation efficiency of virtual reality scenes, no effective solution has been proposed yet. Summary of the Invention

[0004] Embodiments of the present application provide a method, a system, a storage medium, and an electronic device for generating a virtual reality scene, so as to at least solve the problem in the related art of how to improve the generation efficiency of virtual reality scenes.

[0005] According to an embodiment of the embodiments of the present application, a method for generating a virtual reality scene is provided, including: when no home appliance device is placed in the real space, collecting an image of the real space to obtain a first panoramic image; determining a virtual home appliance device corresponding to the home appliance device to be placed and position information of the virtual home appliance device in the virtual space according to an operation instruction of a target object, where the virtual space is a space obtained by virtualizing the real space; generating a target virtual reality scene based on the first panoramic image, the virtual home appliance device, and the position information.

[0006] In an exemplary embodiment, generating a target virtual reality scene based on the first panoramic image, the virtual home appliance device, and the location information includes: determining an initial virtual reality scene, which is constructed based on a sphere model, where the sphere model is a three-dimensional carrier model of the first panoramic image, and the pixel points on the inner surface of the sphere model correspond one-to-one with the pixel points of the first panoramic image; determining the position of the virtual home appliance device in the initial virtual reality scene according to the location information to obtain a first virtual reality scene; updating first spatial visual information in the first virtual reality scene to obtain the target virtual reality scene, where the first spatial visual information represents the visual information formed by the virtual home appliance device blocking an object in the first panoramic image from an observation perspective facing the virtual home appliance device.

[0007] In an exemplary embodiment, determining the initial virtual reality scene includes: establishing a spherical coordinate system with the geometric center of the sphere model as the first origin; establishing a three-dimensional rectangular coordinate system with the image acquisition position of the first panoramic image as the second origin; obtaining the coordinate correspondence between the spherical coordinate system and the three-dimensional rectangular coordinate system, where the first origin corresponds to the second origin; mapping the first panoramic image to the inner surface of the sphere model according to the coordinate correspondence to obtain the initial virtual reality scene.

[0008] In an exemplary embodiment, determining the position of the virtual home appliance device in the first virtual reality scene according to the virtual home appliance device and the location information to obtain a first virtual reality scene includes: determining a first coordinate point according to the position information of the virtual home appliance device in the virtual space, where the first coordinate point represents the coordinate point of the virtual home appliance device in the three-dimensional rectangular coordinate system; determining a second coordinate point according to the first coordinate point and the coordinate correspondence, where the second coordinate point represents the coordinate point of the virtual home appliance device in the spherical coordinate system; determining the position of the virtual home appliance device in the initial virtual reality scene according to the second coordinate point to obtain the first virtual reality scene.

[0009] In an exemplary embodiment, updating the first spatial visual information in the first virtual reality scene to obtain the target virtual reality scene includes: performing image acquisition in different directions respectively based on a first preset point in the first virtual reality scene to obtain a plurality of images, where the first preset point corresponds to the geometric center of the sphere model, and the different directions include the direction from the first preset point to a second preset point, and the second preset point includes the coordinate points corresponding in the first virtual reality scene to the center points of each surface in the real space; performing image recognition on the plurality of images to obtain an image recognition result; and when it is determined that the image recognition result is used to indicate the existence of a target virtual household appliance device in a target direction, updating the first spatial visual information to obtain the target virtual reality scene.

[0010] In an exemplary embodiment, updating the first spatial visual information to obtain the target virtual reality scene includes: determining the visual field area corresponding to the target direction and determining the obstacle object corresponding to the target virtual household appliance device in the visual field area; determining the occlusion part of the obstacle object for occluding the target virtual household appliance device; determining a first image corresponding to the obstacle object from a first panoramic image, and determining an occlusion image corresponding to the occlusion part in the first image from the first image; placing the occlusion image in the visual field area to update the first spatial visual information to second spatial visual information, obtaining a second virtual reality scene, where the second spatial visual information represents the visual information formed by the occlusion image covering the target virtual household appliance device under the observation perspective facing the target virtual household appliance device; and adding a shadow image of the virtual household appliance device in the second virtual reality scene to obtain the target virtual reality scene.

[0011] In an exemplary embodiment, determining the obstacle object corresponding to the target virtual household appliance device in the visual field area includes: determining a third preset point corresponding to the first preset point in the virtual space, where the first preset point and the third preset point correspond to the same position in the real space; and determining the obstacle object according to the comparison result of a first distance and a second distance, where the first distance represents the distance between the target virtual household appliance device and the third preset point in the virtual space, the second distance represents the distance between an object in the target direction and the third preset point in the virtual space, and the distance between the obstacle object and the third preset point is less than the first distance.

[0012] In an exemplary embodiment, adding a shadow image of the virtual household appliance device in the second virtual reality scene to obtain the target virtual reality scene includes: when a household appliance device has been placed in the real space, performing image acquisition on the real space to obtain a second panoramic image; determining a shadow image of a target household appliance device from the second panoramic image, where the target household appliance device is the household appliance device corresponding to the target virtual household appliance device; and replacing an image of a target area in the second virtual reality scene with the shadow image of the target household appliance device to obtain the target virtual reality scene, where the target area represents the shadow area corresponding to the target household appliance.

[0013] According to another aspect of the embodiments of the present application, there is also provided a device for generating a virtual reality scene, including: an acquisition module, configured to perform image acquisition on the real space to obtain a first panoramic image when no household appliance device is placed in the real space; a determination module, configured to determine a virtual household appliance device corresponding to a household appliance device to be placed and position information of the virtual household appliance device in a virtual space according to an operation instruction of a target object, where the virtual space is a space obtained by virtualizing the real space; and a generation module, configured to generate a target virtual reality scene based on the first panoramic image, the virtual household appliance device, and the position information.

[0014] According to yet another aspect of the embodiments of the present application, there is also provided a computer-readable storage medium storing a computer program, where the computer program is configured to execute the above method for generating a virtual reality scene when running.

[0015] According to yet another aspect of the embodiments of the present application, there is also provided an electronic device including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the above processor executes the above method for generating a virtual reality scene through the computer program.

[0016] According to yet another aspect of the embodiments of the present application, there is also provided a computer program product including a computer program, where the computer program implements the steps in any one of the above method embodiments when executed by a processor.

[0017] In an embodiment of the present application, when no household electrical appliances are placed in the real space, an image of the real space is collected to obtain a first panoramic image; the virtual household electrical appliance corresponding to the household electrical appliance to be placed and the position information of the virtual household electrical appliance in the virtual space are determined according to the operation instruction of the target object, where the virtual space is a space obtained by virtualizing the real space; a target virtual reality scene is generated based on the first panoramic image, the virtual household electrical appliance, and the position information. By adopting the above technical solution, a panoramic image can be collected when no household electrical appliances are placed in the show flat, and then a virtual reality scene can be generated according to the collected panoramic image in combination with the household electrical appliance model, solving the problem of how to improve the generation efficiency of the virtual reality scene, and thus achieving the effect of improving the generation efficiency of the virtual reality scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present application and used together with the specification to explain the principles of the present application.

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0020] Figure 1 It is a schematic diagram of the hardware environment of a method for generating a virtual reality scene according to an embodiment of the present application;

[0021] Figure 2 It is a flowchart of a method for generating a virtual reality scene according to an embodiment of the present application;

[0022] Figure 3 It is a platform architecture diagram of a method for generating a virtual reality scene according to an embodiment of the present application;

[0023] Figure 4 It is a data structure diagram of a home decoration design scheme according to an embodiment of the present application;

[0024] Figure 5 It is a schematic diagram of a method for generating a virtual reality scene according to an embodiment of the present application;

[0025] Figure 6 It is a schematic diagram of an occlusion relationship according to an embodiment of the present application;

[0026] Figure 7 It is a schematic flowchart of a matte service call process according to an embodiment of the present application;

[0027] Figure 8It is a schematic flowchart of a matting algorithm according to an embodiment of the present application;

[0028] Figure 9 It is a structural block diagram of a virtual reality scene generation device according to an embodiment of the present application. Detailed implementation manners

[0029] In order to enable those skilled in the art to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application described here can be implemented in an order different from those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily need to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0031] According to one aspect of the embodiments of the present application, a method for generating a virtual reality scene is provided. The method for generating a virtual reality scene is widely applied to application scenarios such as software development. Optionally, in this embodiment, the above method for generating a virtual reality scene can be applied to, for example, Figure 1 the hardware environment composed of the terminal device 102 and the server 104 as shown. As Figure 1 shown, the server 104 is connected to the terminal device 102 through a network, and can be used to provide services (such as application services, etc.) for the terminal or the client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data operation services for the server 104.

[0032] The above network may include, but is not limited to, at least one of the following: wired network, wireless network. The above wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The above wireless network may include, but is not limited to, at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal device 102 may not be limited to PC, mobile phone, tablet computer, smart air conditioner, smart range hood, smart refrigerator, smart oven, smart stove, smart washing machine, smart water heater, smart washing equipment, smart dishwasher, smart projection device, smart TV, smart clothes hanger, smart curtain, smart audio and video, smart socket, smart speaker, smart sound box, smart fresh air equipment, smart kitchen and bathroom equipment, smart bathroom equipment, smart floor sweeping robot, smart window cleaning robot, smart mopping robot, smart air purification equipment, smart steam box, smart microwave oven, smart kitchen water heater, smart purifier, smart water dispenser, smart door lock, etc.

[0033] In this embodiment, a method for generating a virtual reality scene is provided. Figure 2 It is a flowchart of the method for generating a virtual reality scene according to an embodiment of the present application. The process includes the following steps:

[0034] Step S202, when there is no household appliance device placed in the real space, perform image acquisition on the real space to obtain a first panoramic image;

[0035] Optionally, in the above step S202, for example, the above real space may be a house show flat, and the above first panoramic image is an image that can completely present the 360° environment of the house show flat. Among them, the above first panoramic image includes furniture items such as sofas, cabinets, and beds in the house show flat.

[0036] Step S204, determine the virtual household appliance device corresponding to the household appliance device to be placed and the position information of the virtual household appliance device in the virtual space according to the operation instruction of the target object, where the virtual space is a space obtained by virtualizing the real space;

[0037] Optionally, in the above step S204, the above virtual space may be a three-dimensional house type drawing corresponding to the above house show flat on the home decoration design platform. On the home decoration design platform, a designer can select a three-dimensional household appliance model from the model library and add it to the three-dimensional house type drawing (equivalent to the above virtual household appliance device), and adjust the position of the three-dimensional household appliance model in the house type drawing.

[0038] Step S206, generate a target virtual reality scene based on the first panoramic image, the virtual household appliance device, and the position information.

[0039] Through the above steps, when there is no household appliance placed in the real space, the real space is imaged to obtain a first panoramic image; according to the operation instruction of the target object, the virtual household appliance corresponding to the household appliance to be placed and the position information of the virtual household appliance in the virtual space are determined, wherein the virtual space is a space obtained by virtualizing the real space; based on the first panoramic image, the virtual household appliance and the position information, a target virtual reality scene is generated. By adopting the above technical solution, a panoramic image can be collected when there is no household appliance in the show flat, and then a virtual reality scene can be generated according to the collected panoramic image and the household appliance model, solving the problem of how to improve the generation efficiency of the virtual reality scene, and thus achieving the effect of improving the generation efficiency of the virtual reality scene.

[0040] In an exemplary embodiment, generating a target virtual reality scene based on the first panoramic image, the virtual household appliance and the position information includes: determining an initial virtual reality scene, the initial virtual reality scene being constructed based on a sphere model, the sphere model being a three-dimensional carrier model of the first panoramic image, and the pixel points on the inner surface of the sphere model corresponding one-to-one to the pixel points of the first panoramic image; determining the position of the virtual household appliance in the initial virtual reality scene according to the position information to obtain a first virtual reality scene; updating the first spatial visual information in the first virtual reality scene to obtain the target virtual reality scene, wherein the first spatial visual information represents the visual information formed by the virtual household appliance blocking the object in the first panoramic image from the observation perspective facing the virtual household appliance.

[0041] Optionally, in the above embodiment, for example, the first panoramic image is an image obtained by photographing the show flat with a 360-degree panoramic camera. Mapping the first panoramic image to the inner surface of the sphere model can be completed by 3D modeling software. In the 3D modeling software, a sphere model is created as the carrier model of the panoramic image, and then the first panoramic image is imported and mapped to the inside of the sphere. Looking around from the center point of the sphere at the first perspective, a seamless panoramic view of the show flat, that is, the initial virtual reality scene, can be seen.

[0042] According to the position of the three-dimensional household appliance model in the house type plan, the distance between the three-dimensional household appliance model and the first panoramic image acquisition point in the show flat can be determined. According to this distance, the distance between the three-dimensional household appliance model and the center point of the sphere in the sphere model (initial virtual reality scene) can be determined, and then the position of the three-dimensional household appliance model in the initial virtual reality scene can be determined to obtain the first virtual reality scene.

[0043] In the first virtual display scenario, the first panoramic image is rendered at the farthest end, close to the inner surface of the sphere, while the 3D home appliance model is rendered in front of the first panoramic image. Therefore, from the visual effect of the first perspective facing the 3D home appliance model from the center of the sphere (equivalent to the first spatial visual information), the 3D home appliance model will necessarily block the items in the first panoramic image. In actual design, it is possible that the items in the first panoramic image will block the 3D home appliance model. For example, the air conditioner is located between the sofa and the wall. In the correct visual information, the sofa should block the air conditioner, and the air conditioner should block the wall. However, in the first virtual display scenario, the sofa and the wall are rendered together at the farthest end, and the air conditioner model blocks the sofa and the wall. Therefore, the first spatial visual information in the first virtual display scenario may be incorrect and needs to be updated.

[0044] In an exemplary embodiment, determining the initial virtual reality scenario includes: establishing a spherical coordinate system with the geometric center of the sphere model as the first origin; establishing a three-dimensional rectangular coordinate system with the image acquisition position of the first panoramic image as the second origin; obtaining the coordinate correspondence between the spherical coordinate system and the three-dimensional rectangular coordinate system, where the first origin corresponds to the second origin; and mapping the first panoramic image to the inner surface of the sphere model according to the coordinate correspondence to obtain the initial virtual reality scenario.

[0045] In an exemplary embodiment, determining the position of the virtual home appliance device in the first virtual reality scenario according to the virtual home appliance device and the position information to obtain the first virtual reality scenario includes: determining a first coordinate point according to the position information of the virtual home appliance device in the virtual space, where the first coordinate point represents the coordinate point of the virtual home appliance device in the three-dimensional rectangular coordinate system; determining a second coordinate point according to the first coordinate point and the coordinate correspondence, where the second coordinate point represents the coordinate point of the virtual home appliance device in the spherical coordinate system; and determining the position of the virtual home appliance device in the initial virtual reality scenario according to the second coordinate point to obtain the first virtual reality scenario.

[0046] Optionally, in the above embodiment, the surface of the sphere model is composed of countless points. In computer graphics, a spherical coordinate system (polar coordinate system) can be used to define each point on its surface. The spherical coordinate system consists of two angular parameters (polar angle θ and azimuth angle φ) and a radius r. The point (X, Y, Z) in the rectangular coordinate system can be mapped to the point (r, θ, ϕ) in the spherical coordinate system through the following formula:

[0047] 。

[0048] In an exemplary embodiment, updating the first spatial visual information in the first virtual reality scene to obtain the target virtual reality scene includes: performing image acquisition in different directions respectively based on a first preset point in the first virtual reality scene to obtain a plurality of images, where the first preset point corresponds to the geometric center of the sphere model, and the different directions include the direction from the first preset point to a second preset point, and the second preset point includes the coordinate points corresponding in the first virtual reality scene to the center points of each face in the real space; performing image recognition on the plurality of images to obtain an image recognition result; and when it is determined that the image recognition result is used to indicate that there is a target virtual household appliance device in the target direction, updating the first spatial visual information to obtain the target virtual reality scene.

[0049] Optionally, in the above embodiment, after generating the first virtual reality scene, it is necessary to monitor whether the visual relationship in the first virtual reality scene is correct. The actual show flat consists of six faces, corresponding to the six directions of up, down, left, right, front, and back respectively. Identify whether there are household appliance devices in each direction respectively. If there are household appliance devices, the visual relationship in that direction needs to be adjusted.

[0050] In the above recognition process, it is necessary to determine the recognition perspective. First, determine the center point of the first virtual reality scene (i.e., the geometric center of the sphere model) as the position of the first-person camera, and then observe from this position by aligning with the center point of a face of the show flat to obtain the visual image in this direction, and further identify whether there are household appliance devices in the image.

[0051] In an exemplary embodiment, updating the first spatial visual information to obtain the target virtual reality scene includes: determining the visual field area corresponding to the target direction, and determining the obstacle object corresponding to the target virtual household appliance device in the visual field area; determining the occlusion part of the obstacle object for occluding the target virtual household appliance device; determining the first image corresponding to the obstacle object from the first panoramic image, and determining the occlusion image corresponding to the occlusion part in the first image from the first image; placing the occlusion image in the visual field area to update the first spatial visual information to the second spatial visual information to obtain the second virtual reality scene, where the second spatial visual information represents the visual information formed by the occlusion image covering the target virtual household appliance device under the observation perspective facing the target virtual household appliance device; and adding the shadow image of the virtual household appliance device in the second virtual reality scene to obtain the target virtual reality scene.

[0052] Optionally, in the above embodiments, for example, when looking at a certain surface of the model apartment from the center point of the first virtual reality scene, a corner of the air conditioner model in the model apartment should be blocked by the sofa, but the perspective effect of the first virtual reality scene is incorrect, and the air conditioner model blocks the sofa. At this time, it is necessary to crop the image of the sofa from the first panoramic image, and then further crop the sofa image to the part that blocks the air conditioner model to obtain a blocking image, and then paste the blocking image in front of the air conditioner model, so that when looking at the air conditioner model from the center point of the virtual reality scene, it can be correctly seen that the sofa blocks a part of the air conditioner model.

[0053] In an exemplary embodiment, determining the obstacle object corresponding to the target virtual home appliance device in the visual field area includes: determining a third preset point corresponding to the first preset point in the virtual space, where the first preset point and the third preset point correspond to the same position in the real space; determining the obstacle object according to the comparison result of the first distance and the second distance, where the first distance represents the distance between the target virtual home appliance device and the third preset point in the virtual space, the second distance represents the distance between the object in the target direction in the virtual space and the third preset point, and the distance between the obstacle object and the third preset point is less than the first distance.

[0054] Optionally, in the above embodiments, when the designer has completed the design on the floor plan of the model apartment on the design platform, the positional relationship between the three-dimensional home appliance model and other items in the model apartment can already be determined. Therefore, based on the positional relationship between the three-dimensional home appliance model and other items in the floor plan, it can be determined which obstacle objects should block the three-dimensional home appliance model in the target direction. Among them, the obstacle object and the three-dimensional home appliance model are in the same direction, and the distance between the obstacle object and the camera observation point is less than the distance between the three-dimensional home appliance model and the camera observation point.

[0055] In an exemplary embodiment, adding a shadow image of the virtual home appliance device to the second virtual reality scene to obtain the target virtual reality scene includes: when a home appliance device has been placed in the real space, performing image acquisition on the real space to obtain a second panoramic image; determining a shadow image of the target home appliance device from the second panoramic image, where the target home appliance device is the home appliance device corresponding to the target virtual home appliance device; replacing the image of the target area in the second virtual reality scene with the shadow image of the target home appliance device to obtain the target virtual reality scene, where the target area represents the shadow area corresponding to the target home appliance.

[0056] Optionally, in the above embodiments, although the spatial visual information has been adjusted in the second virtual reality scene to obtain the correct occlusion relationship, the background image in the second virtual reality scene is the first panoramic image, which was captured in the show flat without household appliances. Now, household appliances have been added to the scene, but there is no shadow effect corresponding to the household appliances in the background image. Therefore, it is necessary to add the shadow effect of the household appliances. Specifically, panoramic image acquisition can be performed with household appliances already placed in the show flat, and then the shadow image corresponding to the placed household appliances can be cropped from the captured panoramic image. Then, the above shadow image can be pasted at the corresponding position in the second virtual reality scene to obtain the final target virtual reality scene.

[0057] Through the above embodiments, the VR scene hybrid rendering using the combination of panoramic images and 3D models can reduce the memory occupancy of the 3D models in the scene, and can also replace the household appliances in the scene at any time, greatly improving the generation efficiency of the VR scene, and at the same time ensuring that the items in the scene have the correct visual display effect.

[0058] To better understand the process of the above virtual reality scene generation method, the following further describes the implementation method flow of the above virtual reality scene generation in combination with optional embodiments, but it is not used to limit the technical solutions of the embodiments of the present application.

[0059] In an optional embodiment, Figure 3 is the platform architecture diagram of the virtual reality scene generation method according to the embodiments of the present application. As Figure 3 shown, this method mainly involves the following platforms.

[0060] Designer platform: The design platform for indoor 3D home improvement solutions. Indoor designers can design complete home improvement solutions here, and the design process is all three-dimensional visual. The 3D house type diagrams in the platform are all designed and rendered using 3D models. The platform also provides the function of rendering panoramic images, and also provides an open API (Application Programming Interface). Users can obtain relevant information about the design solution through the API interface.

[0061] Server / database: The data server side, used to obtain, store, and distribute data. By calling the open API of the designer platform, the relevant data of the design solution is determined, such as the room list, household appliance list, etc. At the same time, relevant interfaces are provided for other ends to use. For example, other ends can query the design solution data through the relevant interfaces.

[0062] Mobile page: Users view the VR scene on this page. The panoramic image of the sample room without goods (the goods here refer to household appliances, excluding furniture and other items) and the 3D household appliance models are mixed and rendered to display the VR scene on the mobile device, which is convenient for sharing and dissemination. Moreover, the household appliances in the VR scene can be replaced in real time, allowing users to conveniently and intuitively see the effects of different household appliances placed in the VR scene. During the rendering process, the mobile page sends a matte extraction request to the algorithm engine.

[0063] Algorithm engine: Provides household appliance occlusion matte extraction service and algorithms. It solves the incorrect front-back occlusion relationship between the furniture or walls in the background panoramic image and the 3D household appliance models in the VR scene. Through the occlusion matte extraction algorithm, one or more foreground occlusion images can be generated for the 3D household appliance models. When rendering the VR scene, placing the foreground occlusion images in front of the household appliance models can form the correct occlusion relationship.

[0064] In an optional embodiment, Figure 4 is the data structure diagram of the home decoration design solution according to the embodiment of the present application, as Figure 4 shown, and the home decoration design solution specifically includes the following data.

[0065] Basic information: Solution name, solution ID, apartment area, apartment specification, apartment floor plan, community name, apartment label, etc.

[0066] Room information list: Room name, room ID, color picture of the panoramic image with goods, channel map of the panoramic image with goods, color picture of the panoramic image without goods, channel map of the panoramic image without goods, room coordinates (i.e., camera coordinates), room area, household appliance list, etc.

[0067] Household appliance information list: Household appliance ID, household appliance name, download address of the 3D household appliance model, pose of the household appliance (including the placement position, rotation angle, and scaling ratio of the model), dimensions of the household appliance (width, height, depth), list of foreground occlusion images of the household appliance, replaceable list of the household appliance, etc.

[0068] In an optional embodiment, Figure 5 is the schematic diagram of the method for generating a virtual reality scene according to the embodiment of the present application, as Figure 5 shown, and specifically includes the following processes:

[0069] 1. After the interior designer completes the home decoration design solution on the designer platform, render a panoramic image without goods (equivalent to the first panoramic image) and a panoramic image with goods (equivalent to the second panoramic image) for each room. The camera positions of the two panoramic images are the same. The panoramic image with goods includes household appliances, and the panoramic image without goods does not include household appliances. It should be noted that the panoramic image without goods can include furniture in the room, such as beds, sofas, and other items.

[0070] 2. The server / database obtains relevant data of the home decoration design plan by calling the API provided by the designer platform, including the basic information of the plan, room information, home appliance information, etc.

[0071] 3. Render a VR scene on the mobile page (such as an H5 page). The process of VR scene rendering includes determining the background panoramic image rendering, three-dimensional model rendering, and adding foreground occlusion images for home appliances.

[0072] The background panoramic image rendering method is to paste the panoramic image on the inner surface of a sphere, and then set the camera at the center of the sphere. At this time, the most basic panoramic VR scene is formed.

[0073] The three-dimensional model rendering is to render the three-dimensional home appliance model in front of the background panoramic image. The three-dimensional home appliance model is closer to the central camera than the panoramic image. The front-back relationship between the three-dimensional home appliance model and the furniture in the panoramic image may be incorrect, resulting in an incorrect occlusion relationship between the three-dimensional home appliance model and the furniture in the panoramic image, forming an incorrect spatial visual effect.

[0074] By adding foreground occlusion images for home appliances, the incorrect occlusion relationship between the three-dimensional home appliance model and the furniture in the panoramic image can be corrected, and the spatial visual effect can be restored to normal. When the three-dimensional home appliance model is blocked by other furniture or walls, through the home appliance occlusion matte algorithm, a separate foreground occlusion image for the home appliance is generated. Then, during three-dimensional rendering, the foreground occlusion image is placed in front of the three-dimensional home appliance model. From the camera angle, the three-dimensional home appliance model is normally blocked by the furniture. The foreground occlusion image for the home appliance is generated by calling the matte service when first entering the VR scene.

[0075] Specifically, as Figure 6 shown, in a real room, there is a floor-standing air conditioner, and there is a sofa beside the air conditioner. When looking at the air conditioner from a preset position in the room, a corner of the air conditioner is blocked by the sofa. At this time, if the panoramic image is rendered at this position, the entire room scene including the sofa will be rendered into the panoramic image. Then, when using a three-dimensional model to render the air conditioner in the VR scene, it can be seen that the entire air conditioner is in front of the sofa, thus forming an incorrect occlusion relationship. The correct occlusion relationship should be that a corner of the air conditioner is behind the sofa. By calling the matte algorithm of the algorithm engine, the sofa can be separately cut out from the panoramic image as a foreground occlusion image and placed in front of the air conditioner. In this way, when looking at the air conditioner from the preset position in the VR scene, a corner of the air conditioner is blocked by the sofa, forming a correct occlusion relationship. It should be noted that the foreground occlusion image is not necessarily a complete sofa. In some cases, only a part of the sofa blocks the air conditioner. At this time, only this part of the sofa is cut out as the foreground occlusion image.

[0076] 4. After receiving the matte painting request, the matte painting algorithm engine starts matte painting. After successful matte painting, it will generate the foreground occlusion picture and shadow picture of the household appliance.

[0077] The process of calling the matte painting service to generate the foreground occlusion picture and shadow picture of the household appliance when the mobile page first enters the VR scene is as Figure 7 shown and includes the following steps:

[0078] Step S701: First enter the VR scene.

[0079] Step S702: Traverse the six faces of the VR scene, namely up, down, front, back, left, and right.

[0080] Step S703: Determine whether there is a household appliance on this face. If so, execute Step S704; otherwise, execute Step S705.

[0081] Step S704: There is no household appliance, so no matte painting is required.

[0082] Step S705: There is a household appliance, so matte painting is required. Send the panoramic image data and household appliance data to the matte painting service to initiate a matte painting request.

[0083] Step S706: The matte painting service performs matte painting based on the received data to generate the foreground occlusion picture and shadow picture of the household appliance.

[0084] Step S707: Save the generated pictures to the database.

[0085] Step S708: Matte painting is completed.

[0086] In an optional embodiment, the process of the matte painting service performing matte painting based on the received data to generate the foreground occlusion picture and shadow picture of the household appliance in Step S806 is as Figure 8 shown and specifically includes the following steps:

[0087] Step S801: Input data such as the contour map of the household appliance model, room information, household appliance information, panoramic picture with goods, and panoramic picture without goods into the matte painting service.

[0088] Step S802: Check whether the input data is qualified. If so, execute Step S804; otherwise, execute Step S803.

[0089] Step S803: Return matte painting failure.

[0090] Step S804: Identify the household appliance device.

[0091] Step S805: In the case of identifying the household appliance device, determine whether the occlusion relationship of the household appliance device is correct. If so, execute Steps S807 and S808; otherwise, execute Step S806.

[0092] Step S806: No processing is required, and the matte extraction process ends.

[0093] Step S807: Perform compositing processing based on whether there is a panoramic view of the commodity to generate a shadow map.

[0094] Step S808: Determine the occluder that occludes the home appliance device, and extract the occluder from the panoramic view as the foreground occlusion picture.

[0095] Step S809: Output the shadow map and the foreground occlusion picture.

[0096] Through the above embodiments, it is possible to achieve hybrid rendering of panoramic views and three-dimensional home appliance models in a mobile VR show flat, and then accurately process the front and rear occlusion relationships between three-dimensional home appliance models and panoramic view furniture through the home appliance occlusion matte extraction algorithm, ensuring the authenticity and coherence of VR scene rendering, achieving high-quality visual effects while retaining the offline-rendered panoramic views, and supporting the real-time replacement function of home appliances. In addition, by optimizing the rendering method of the VR scene, the resource occupancy and loading latency on the mobile device are significantly reduced, thereby improving the fluency of user operations and the interaction experience, and finally realizing the display and dissemination of an efficient, lightweight and immersive VR home improvement solution.

[0097] From the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc), and includes several instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of the various embodiments of the present application.

[0098] Figure 9 is a structural block diagram of a virtual reality scene generation device according to an embodiment of the present application; as Figure 9 shown, it includes:

[0099] An acquisition module 92, configured to collect an image of the real space to obtain a first panoramic image when no home appliance device is placed in the real space;

[0100] A determination module 94, configured to determine a virtual home appliance device corresponding to the home appliance device to be placed and position information of the virtual home appliance device in the virtual space according to an operation instruction of a target object, where the virtual space is a space obtained by virtualizing the real space;

[0101] A generation module 96, configured to generate a target virtual reality scene based on the first panoramic image, the virtual home appliance device, and the location information.

[0102] With the above device, in the case where no home appliance device is placed in the real space, an image of the real space is collected to obtain a first panoramic image; the virtual home appliance device corresponding to the home appliance device to be placed and the location information of the virtual home appliance device in the virtual space are determined according to the operation instruction of the target object, where the virtual space is a space obtained by virtualizing the real space; a target virtual reality scene is generated based on the first panoramic image, the virtual home appliance device, and the location information. By adopting the above technical solution, a panoramic image can be collected when no home appliances are placed in the show flat, and then a virtual reality scene can be generated according to the collected panoramic image combined with the home appliance model, solving the problem of how to improve the generation efficiency of the virtual reality scene, and thus achieving the effect of improving the generation efficiency of the virtual reality scene.

[0103] In an exemplary embodiment, the generation module 96 is further configured to determine an initial virtual reality scene, where the initial virtual reality scene is constructed based on a sphere model, the sphere model is a three-dimensional carrier model of the first panoramic image, and the pixel points on the inner surface of the sphere model correspond one-to-one with the pixel points of the first panoramic image; determine the position of the virtual home appliance device in the initial virtual reality scene according to the location information to obtain a first virtual reality scene; update the first spatial visual information in the first virtual reality scene to obtain the target virtual reality scene, where the first spatial visual information represents the visual information formed by the virtual home appliance device blocking the object in the first panoramic image under the observation view facing the virtual home appliance device.

[0104] In an exemplary embodiment, the generation module 96 is further configured to establish a spherical coordinate system with the geometric center of the sphere model as the first origin; establish a three-dimensional rectangular coordinate system with the image acquisition position of the first panoramic image as the second origin; obtain the coordinate correspondence between the spherical coordinate system and the three-dimensional rectangular coordinate system, where the first origin corresponds to the second origin; map the first panoramic image to the inner surface of the sphere model according to the coordinate correspondence to obtain the initial virtual reality scene.

[0105] In an exemplary embodiment, the generating module 96 is further configured to determine a first coordinate point according to the position information of the virtual home appliance device in the virtual space, where the first coordinate point represents the coordinate point of the virtual home appliance device in the three-dimensional rectangular coordinate system; determine a second coordinate point according to the first coordinate point and the coordinate correspondence relationship, where the second coordinate point represents the coordinate point of the virtual home appliance device in the spherical coordinate system; determine the position of the virtual home appliance device in the initial virtual reality scene according to the second coordinate point, so as to obtain the first virtual reality scene.

[0106] In an exemplary embodiment, the generating module 96 is further configured to perform image acquisition in different directions based on a first preset point of the first virtual reality scene to obtain a plurality of images, where the first preset point corresponds to the geometric center of the sphere model, and the different directions include the direction from the first preset point to a second preset point, and the second preset point includes the coordinate points corresponding to the center points of each surface in the real space in the first virtual reality scene; perform image recognition on the plurality of images to obtain an image recognition result; in the case that it is determined that the image recognition result is used to indicate the existence of a target virtual home appliance device in the target direction, update the first spatial visual information to obtain the target virtual reality scene.

[0107] In an exemplary embodiment, the generating module 96 is further configured to determine the visual field area corresponding to the target direction, and determine the obstacle object corresponding to the target virtual home appliance device in the visual field area; determine the occlusion part of the obstacle object for occluding the target virtual home appliance device; determine a first image corresponding to the obstacle object from the first panoramic image, and determine an occlusion image corresponding to the occlusion part in the first image from the first image; place the occlusion image in the visual field area to update the first spatial visual information to second spatial visual information, so as to obtain a second virtual reality scene, where the second spatial visual information represents the visual information formed by the occlusion image covering the target virtual home appliance device under the observation perspective facing the target virtual home appliance device; add a shadow image of the virtual home appliance device in the second virtual reality scene to obtain the target virtual reality scene.

[0108] In an exemplary embodiment, the generating module 96 is further configured to determine a third preset point in the virtual space corresponding to the first preset point, where the first preset point and the third preset point correspond to the same position in the real space; determine the obstacle object according to the comparison result of the first distance and the second distance, where the first distance represents the distance between the target virtual home appliance device in the virtual space and the third preset point, the second distance represents the distance between the object in the target direction in the virtual space and the third preset point, and the distance between the obstacle object and the third preset point is less than the first distance.

[0109] In an exemplary embodiment, when a home appliance device has been placed in the real space, the generating module 96 is further configured to collect an image of the real space to obtain a second panoramic image; determine a shadow image of the target home appliance device from the second panoramic image, where the target home appliance device is the home appliance device corresponding to the target virtual home appliance device; replace the image of the target area in the second virtual reality scene with the shadow image of the target home appliance device to obtain the target virtual reality scene, where the target area represents the shadow area corresponding to the target home appliance.

[0110] An embodiment of the present application further provides a storage medium, which includes a stored program, where the above program executes the method of any one of the above when running.

[0111] Optionally, in this embodiment, the above storage medium may be set to store program codes for executing the following steps:

[0112] S1, when no home appliance device is placed in the real space, collect an image of the real space to obtain a first panoramic image;

[0113] S2, determine the virtual home appliance device corresponding to the home appliance device to be placed and the position information of the virtual home appliance device in the virtual space according to the operation instruction of the target object, where the virtual space is a space obtained by virtualizing the real space;

[0114] S3, generate a target virtual reality scene based on the first panoramic image, the virtual home appliance device, and the position information.

[0115] An embodiment of the present application further provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0116] Optionally, the above electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the above processor, and the input / output device is connected to the above processor.

[0117] Optionally, in this embodiment, the above processor may be configured to execute the following steps through a computer program:

[0118] S1, when there is no household appliance placed in the real space, perform image acquisition on the real space to obtain a first panoramic image;

[0119] S2, determine the virtual household appliance corresponding to the household appliance to be placed and the position information of the virtual household appliance in the virtual space according to the operation instruction of the target object, wherein the virtual space is a space obtained by virtualizing the real space;

[0120] S3, generate a target virtual reality scene based on the first panoramic image, the virtual household appliance and the position information.

[0121] Optionally, in this embodiment, the above storage medium may include but is not limited to: USB flash drive, read-only memory (ROM for short), random access memory (RAM for short), mobile hard disk, magnetic disk or optical disc and other various media that can store program codes.

[0122] The embodiment of the present application also provides a computer program product, the above computer program product includes a computer program, and when the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0123] The embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0124] The embodiment of the present application also provides a computer program, the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium; the processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps in any one of the above method embodiments.

[0125] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiment and optional implementation manners, and this embodiment will not be elaborated here.

[0126] Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present application can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. Optionally, they can be implemented by program codes executable by the computing device. Thus, they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described can be executed in a sequence different from that here, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module for implementation. In this way, the present application is not limited to any specific combination of hardware and software.

[0127] The above are only the preferred embodiments of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can still be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A method for generating a virtual reality scene, characterized in that, Including: When there is no household appliance device placed in the real space, image acquisition is performed on the real space to obtain a first panoramic image; Determine the virtual household appliance device corresponding to the household appliance device to be placed and the position information of the virtual household appliance device in the virtual space according to the operation instruction of the target object, wherein the virtual space is a space obtained by virtualizing the real space; Generate a target virtual reality scene of the real space based on the first panoramic image, the virtual household appliance device, and the position information.

2. The method for generating a virtual reality scene according to claim 1, wherein Generating a target virtual reality scene based on the first panoramic image, the virtual household appliance device, and the position information includes: Determine an initial virtual reality scene, which is constructed based on a sphere model. The sphere model is a three-dimensional carrier model of the first panoramic image, and the pixel points on the inner surface of the sphere model correspond one-to-one with the pixel points of the first panoramic image; Determine the position of the virtual household appliance device in the initial virtual reality scene according to the position information to obtain a first virtual reality scene; Update the first spatial visual information in the first virtual reality scene to obtain the target virtual reality scene, wherein the first spatial visual information represents the visual information formed by the virtual household appliance device blocking the object in the first panoramic image under the observation perspective facing the virtual household appliance device.

3. The method for generating a virtual reality scene according to claim 2, wherein Determining the initial virtual reality scene includes: Taking the geometric center of the sphere model as the first origin to establish a spherical coordinate system; Taking the image acquisition position of the first panoramic image as the second origin to establish a three-dimensional rectangular coordinate system; Obtain the coordinate correspondence between the spherical coordinate system and the three-dimensional rectangular coordinate system, wherein the first origin corresponds to the second origin; Map the first panoramic image to the inner surface of the sphere model according to the coordinate correspondence to obtain the initial virtual reality scene.

4. The method for generating a virtual reality scene according to claim 3, wherein Determining the position of the virtual household appliance device in the first virtual reality scene according to the virtual household appliance device and the position information to obtain a first virtual reality scene includes: Determine a first coordinate point according to the position information of the virtual household appliance device in the virtual space, wherein the first coordinate point represents the coordinate point of the virtual household appliance device in the three-dimensional rectangular coordinate system; Determine a second coordinate point according to the first coordinate point and the coordinate correspondence, wherein the second coordinate point represents the coordinate point of the virtual household appliance device in the spherical coordinate system; Determine the position of the virtual household appliance device in the initial virtual reality scene according to the second coordinate point to obtain the first virtual reality scene.

5. The method for generating a virtual reality scene according to claim 2, wherein, Updating the first spatial visual information in the first virtual reality scene to obtain the target virtual reality scene includes: Performing image acquisition in different directions from a first preset point based on the first virtual reality scene to obtain a plurality of images, wherein the first preset point corresponds to the geometric center of the sphere model, and the different directions include the direction from the first preset point to a second preset point, and the second preset point includes the coordinate points corresponding in the first virtual reality scene to the center points of each surface in the real space; Performing image recognition on the plurality of images to obtain an image recognition result; When it is determined that the image recognition result is used to indicate the existence of a target virtual household appliance device in a target direction, updating the first spatial visual information to obtain the target virtual reality scene.

6. The method for generating a virtual reality scene according to claim 5, wherein Updating the first spatial visual information to obtain the target virtual reality scene includes: Determining the visual field area corresponding to the target direction and determining the obstacle object corresponding to the target virtual household appliance device in the visual field area; Determining the occlusion part of the obstacle object for occluding the target virtual household appliance device; Determining a first image corresponding to the obstacle object from a first panoramic image and determining an occlusion image corresponding to the occlusion part in the first image from the first image; Placing the occlusion image within the visual field area to update the first spatial visual information to second spatial visual information to obtain a second virtual reality scene, wherein the second spatial visual information represents the visual information formed by the occlusion image covering the target virtual household appliance device from the observation perspective facing the target virtual household appliance device; Adding a shadow image of the virtual household appliance device in the second virtual reality scene to obtain the target virtual reality scene.

7. The method for generating a virtual reality scene according to claim 6, wherein Determining the obstacle object corresponding to the target virtual household appliance device in the visual field area includes: Determining a third preset point corresponding to the first preset point in the virtual space, wherein the first preset point and the third preset point correspond to the same position in the real space; Determining the obstacle object according to the comparison result of a first distance and a second distance, wherein the first distance represents the distance between the target virtual household appliance device in the virtual space and the third preset point, the second distance represents the distance between an object in the target direction in the virtual space and the third preset point, and the distance between the obstacle object and the third preset point is less than the first distance.

8. The method for generating a virtual reality scene according to claim 6, wherein, Adding a shadow image of the virtual household appliance device in the second virtual reality scene to obtain the target virtual reality scene includes: When a household appliance device has been placed in the real space, performing image acquisition on the real space to obtain a second panoramic image; Determining a shadow image of a target household appliance device from the second panoramic image, wherein the target household appliance device is the household appliance device corresponding to the target virtual household appliance device; Replacing the image of a target area in the second virtual reality scene according to the shadow image of the target household appliance device to obtain the target virtual reality scene, wherein the target area represents the shadow area corresponding to the target household appliance.

9. A generating device for a virtual reality scene, characterized in that, including: An acquisition module, configured to acquire an image of the real space to obtain a first panoramic image when no household electrical appliance is placed in the real space; A determination module, configured to determine a virtual household electrical appliance corresponding to the household electrical appliance to be placed and position information of the virtual household electrical appliance in a virtual space according to an operation instruction of a target object, where the virtual space is a space obtained by virtualizing the real space; A generation module, configured to generate a target virtual reality scene based on the first panoramic image, the virtual household electrical appliance, and the position information.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, where the program, when running, executes the method described in any one of claims 1 to 8 above.

11. An electronic device, comprising a memory and a processor, characterized in that, A computer program is stored in the memory, and the processor is configured to execute the method described in any one of claims 1 to 8 through the computer program.

Images