Home scene generation method and device, electronic equipment and readable storage medium

Abstract

The application discloses a home scene generation method and device, electronic equipment and a readable storage medium, comprising: obtaining a target video to be reconstructed; performing scene reconstruction based on indoor image frames in the target video to obtain an initial home decoration scene, wherein the indoor image frames contain home decoration commodities, and the initial scene contains commodity models corresponding to the home decoration commodities; and associating commodity information of the home decoration commodities with the commodity models to obtain a target home decoration scene. The method can automatically filter indoor image frames containing indoor scenes in a video, reconstruct a home decoration scene contained in the indoor scene, and also associate commodity information of home decoration commodities to the home decoration scene. When displaying the target home decoration scene obtained by reconstruction, detailed commodity information of the home decoration commodities is provided for the user, so that the user does not need to search for the commodity information of the home decoration commodities in the home decoration scene, and the operation of the user to obtain the home decoration scene is simplified.

Description

Technical Field

[0001] This application relates to the field of home scene generation technology, specifically to a home scene generation method, device, electronic device, and readable storage medium. Background Technology

[0002] With the development of technology, smart TVs have become a standard feature in every household. The hardware capabilities of smart TVs are becoming increasingly powerful, and users are experiencing a wider variety of products on their TVs. Video playback is the most frequently used function on TVs. When users watch TV series, movies, and other videos, they sometimes want to obtain information about the home decoration layout and items within the on-screen content in order to make adjustments to their own room's layout. However, currently, users can only record this information by taking photos or other methods and then searching for it, which is a very cumbersome process. Summary of the Invention

[0003] This application provides a method, apparatus, electronic device, and readable storage medium for generating home scenes, aiming to solve the technical problem that existing methods for generating home scenes are relatively cumbersome to operate.

[0004] Firstly, this application provides a method for generating a home scene, including:

[0005] Obtain the target video to be reconstructed;

[0006] Scene reconstruction is performed based on indoor image frames in the target video to obtain an initial home decoration scene, wherein the indoor image frames contain home decoration products, and the initial scene contains product models corresponding to the home decoration products;

[0007] By associating the product information of the home decoration products with the product model, the target home decoration scene is obtained.

[0008] In one possible implementation of this application, after associating the product information of the home decoration product with the product model to obtain the target home decoration scene, the method further includes:

[0009] In response to a trigger command on the target product model in the target home decoration scenario, the target product information of the target home decoration product associated with the target product model is displayed.

[0010] In one possible implementation of this application, before reconstructing the scene based on the indoor image frames in the target video to obtain the initial home decoration scene, the method further includes:

[0011] Obtain candidate image frames containing the ceiling from each image frame in the target video;

[0012] The walls in each of the candidate image frames are identified;

[0013] For each candidate image frame, based on the relative positional relationship between the wall and the ceiling, it is determined whether each candidate image frame is an indoor image frame, and the determination result is obtained.

[0014] In one possible implementation of this application, the step of reconstructing the scene based on indoor image frames in the target video to obtain an initial home decoration scene includes:

[0015] The home decoration products contained in the indoor image frames of the target video are modeled in 3D to obtain the product models corresponding to the home decoration products, the product dimensions of each product model, and the positional relationships between each product model;

[0016] Based on the product size and positional relationship of each product model, a room model for placing each product model is constructed, wherein the room model is composed of rendered building material models, including wall models, ceiling models and / or floor models;

[0017] Based on the aforementioned positional relationships, each of the product models is placed within the room model to obtain an initial home decoration scene.

[0018] In one possible implementation of this application, constructing a room model for placing each of the product models based on their dimensions and positional relationships includes:

[0019] Based on the product size and positional relationship of each product model, generate the minimum bounding box of each product model;

[0020] Based on the bounding box size of the minimum bounding box, a model to be rendered is generated, wherein the model to be rendered is composed of a building material model to be rendered;

[0021] The model material of the building material model to be rendered is identified from the indoor image frame;

[0022] The building material model to be rendered is rendered based on the model material to obtain the rendered building material model and the room model composed of the rendered building material model.

[0023] In one possible implementation of this application, the product information includes a product purchase link, product model, and / or brand.

[0024] In one possible implementation of this application, obtaining the target video to be reconstructed includes:

[0025] In response to the scene generation command, the currently playing video is used as the target video to be reconstructed.

[0026] Secondly, this application provides a home scene generation device, comprising:

[0027] The acquisition unit is used to acquire the target video to be reconstructed.

[0028] The reconstruction unit is used to reconstruct the scene based on the indoor image frames in the target video to obtain an initial home decoration scene, wherein the indoor image frames contain home decoration products, and the initial scene contains product models corresponding to the home decoration products.

[0029] The association unit is used to associate the product information of the home decoration product with the product model to obtain the target home decoration scene.

[0030] In one possible implementation of this application, the associated unit is further used for:

[0031] In response to a trigger command on the target product model in the target home decoration scenario, the target product information of the target home decoration product associated with the target product model is displayed.

[0032] In one possible implementation of this application, the reconstruction unit is used for:

[0033] Obtain candidate image frames containing the ceiling from each image frame in the target video;

[0034] The walls in each of the candidate image frames are identified;

[0035] For each candidate image frame, based on the relative positional relationship between the wall and the ceiling, it is determined whether each candidate image frame is an indoor image frame, and the determination result is obtained.

[0036] In one possible implementation of this application, the reconstruction unit is used for:

[0037] The home decoration products contained in the indoor image frames of the target video are modeled in 3D to obtain the product models corresponding to the home decoration products, the product dimensions of each product model, and the positional relationships between each product model;

[0038] Based on the product size and positional relationship of each product model, a room model for placing each product model is constructed, wherein the room model is composed of rendered building material models, including wall models, ceiling models and / or floor models;

[0039] Based on the aforementioned positional relationships, each of the product models is placed within the room model to obtain an initial home decoration scene.

[0040] In one possible implementation of this application, the reconstruction unit is used for:

[0041] Based on the product size and positional relationship of each product model, generate the minimum bounding box of each product model;

[0042] Based on the bounding box size of the minimum bounding box, a model to be rendered is generated, wherein the model to be rendered is composed of a building material model to be rendered;

[0043] The model material of the building material model to be rendered is identified from the indoor image frame;

[0044] The building material model to be rendered is rendered based on the model material to obtain the rendered building material model and the room model composed of the rendered building material model.

[0045] In one possible implementation of this application, the product information includes a product purchase link, product model, and / or brand.

[0046] In one possible implementation of this application, the acquiring unit is further configured to:

[0047] In response to the scene generation command, the currently playing video is used as the target video to be reconstructed.

[0048] Thirdly, this application also provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor calls the computer program in the memory, it executes the steps in any of the home scene generation methods provided in this application.

[0049] Fourthly, this application also provides a readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the home scene generation methods provided in this application.

[0050] In summary, the home decoration scene generation method provided in this application includes: acquiring a target video to be reconstructed; reconstructing a scene based on indoor image frames in the target video to obtain an initial home decoration scene, wherein the indoor image frames contain home decoration products, and the initial scene contains a product model corresponding to the home decoration products; associating the product information of the home decoration products with the product model to obtain a target home decoration scene.

[0051] As can be seen, the home decoration scene generation method provided in this application can automatically filter out indoor image frames containing indoor scenes in videos, reconstruct the home decoration scene contained in the indoor scene, simplify the user's operation of obtaining home decoration scenes, and can also associate the product information of home decoration products with the home decoration scene. When displaying the reconstructed target home decoration scene, it provides the user with detailed product information of home decoration products. The user no longer needs to search for product information of home decoration products in the home decoration scene, further simplifying the user's operation of obtaining home decoration scenes. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a schematic diagram illustrating an application scenario of the home scene generation method provided in this application embodiment;

[0054] Figure 2 This is a schematic flowchart of a home scene generation method provided in the embodiments of this application;

[0055] Figure 3 This is a schematic diagram of a home scene provided in the embodiments of this application;

[0056] Figure 4 This is a schematic diagram of a process for acquiring indoor image frames provided in an embodiment of this application;

[0057] Figure 5 This is a schematic diagram of a process for obtaining an initial home decoration scene provided in an embodiment of this application;

[0058] Figure 6 This is a schematic diagram of a module of the home scene generation system provided in the embodiments of this application;

[0059] Figure 7 This is a schematic diagram of an embodiment of the home scene generation device provided in this application.

[0060] Figure 8 This is a schematic diagram of an embodiment of the electronic device provided in this application. Detailed Implementation

[0061] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0062] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0063] To enable any person skilled in the art to implement and use this application, the following description is provided. In this description, details are set forth for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be implemented without using these specific details. In other instances, well-known processes will not be described in detail to avoid obscuring the description of the embodiments of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in the embodiments of this application.

[0064] This application provides a method, apparatus, electronic device, and readable storage medium for generating a home scene. The home scene generation apparatus can be integrated into an electronic device, which can be a server or a terminal, etc.

[0065] The execution subject of the home scene generation method in this application embodiment can be the home scene generation device provided in this application embodiment, or different types of electronic devices such as server equipment, physical host, or special effects production personnel equipment (User Equipment, UE) that integrate the home scene generation device. The home scene generation device can be implemented in hardware or software. The UE can specifically be a terminal device such as a smartphone, tablet computer, laptop computer, handheld computer, desktop computer, or personal digital assistant (PDA).

[0066] The electronic device can operate independently or in a cluster.

[0067] See Figure 1 , Figure 1This is a schematic diagram of a home scene generation system provided in an embodiment of this application. The home scene generation system may include an electronic device 100, which integrates a home scene generation device.

[0068] In addition, such as Figure 1 As shown, the home scene generation system may also include a memory 200 for storing data, such as text data.

[0069] It should be noted that, Figure 1 The scene diagram of the home scene generation system shown is merely an example. The home scene generation system and scene described in this application embodiment are for the purpose of more clearly illustrating the technical solutions of this application embodiment and do not constitute a limitation on the technical solutions provided in this application embodiment. As those skilled in the art will know, with the evolution of home scene generation systems and the emergence of new business scenarios, the technical solutions provided in the embodiments of this invention are also applicable to similar technical problems.

[0070] The following describes the home scene generation method provided in this application embodiment. In this application embodiment, an electronic device is used as the execution subject. For simplification and ease of description, the execution subject will be omitted in subsequent method embodiments. The home scene generation method includes: acquiring a target video to be reconstructed; reconstructing a scene based on indoor image frames in the target video to obtain an initial home decoration scene, wherein the indoor image frames contain home decoration products, and the initial scene contains product models corresponding to the home decoration products; associating the product information of the home decoration products with the product models to obtain a target home decoration scene.

[0071] Reference Figure 2 , Figure 2 This is a flowchart illustrating a home scene generation method provided in an embodiment of this application. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here. Specifically, the home scene generation method may include the following steps 201-203, wherein:

[0072] 201. Obtain the target video to be reconstructed.

[0073] To facilitate understanding, the application background of this application will first be explained, but this should not be construed as an admission of prior art: With the development of technology, smart TVs have become standard equipment in every household. The hardware capabilities of smart TVs are becoming more and more powerful, and users are experiencing more and more products on TVs. Video playback is the most commonly used function of TVs. When users watch TV series, movies and other videos on TVs, they sometimes want to obtain information about the home decoration layout and the home decoration items in the content on the screen, so as to make it easier to adjust the home decoration layout of their own rooms. However, at present, users can only record the home decoration layout and home decoration items by taking pictures or other methods, and then search for them, which is very cumbersome.

[0074] Therefore, in order to solve the above problems, this application provides a method for reconstructing home decoration scenes in videos that can be applied to electronic devices such as televisions, projectors, and mobile phones that can play videos.

[0075] The target video refers to the video containing the home decoration scene to be reconstructed. For example, it could be a TV series, movie, short video, etc., containing the home decoration scene. For instance, a user can select the video being played as the target video by inputting a command while the electronic device is playing a video. For example, if a user is watching a TV series and sees a desired home decoration layout in the series, they can input a command using the TV remote. Upon receiving the command, the TV will use the currently playing TV series as the target video to be reconstructed. Therefore, the step "obtaining the target video to be reconstructed" includes:

[0076] In response to the scene generation command, the currently playing video is used as the target video to be reconstructed.

[0077] Alternatively, the television can acquire the image frames currently playing in the TV series, and based on a preset time range and the corresponding playback time of the image frame in the TV series, select the image frames used for reconstruction from the TV series, and use the video composed of the image frames used for reconstruction as the target video.

[0078] 202. Reconstruct the scene based on the indoor image frames in the target video to obtain an initial home decoration scene, wherein the indoor image frames contain home decoration products, and the initial scene contains product models corresponding to the home decoration products.

[0079] An indoor image frame refers to an image frame in the target video that contains an indoor scene. Since the target video may contain both indoor and outdoor image frames, and the outdoor image frames do not contain the home layout the user wants to reconstruct, the electronic device can color-code each image frame of the target video to obtain the indoor image frames within each frame. For example, a trained indoor object recognition model can be used to identify whether common indoor items such as teacups, home appliances, sofas, and beds exist in each image frame, and the image frame containing these items is designated as an indoor image frame. The indoor object recognition model can be trained using open-source object recognition / detection models such as AlexNet, and this application does not impose any limitations on this.

[0080] The initial home decoration scene refers to the scene obtained after scene reconstruction, which is composed of the three-dimensional models (i.e., product models) corresponding to each home decoration product in the indoor image frames. The initial home decoration scene includes the three-dimensional models corresponding to each home decoration product, as well as the positional relationship between each three-dimensional model.

[0081] This application does not limit the method of scene reconstruction. For example, point clouds corresponding to indoor image frames can be generated using the SFM (Structure From Motion) algorithm and the MVS (Multiple View Stereo) algorithm. Then, a 3D model corresponding to each home decoration product can be generated based on the point cloud corresponding to each home decoration product. It is understood that the surface of the reconstructed 3D model is rendered with the surface material of the corresponding home decoration product.

[0082] In some embodiments, to avoid inaccurate initial home decoration scenes reconstructed from indoor image frames in the target video that only capture a partial view of the interior, after identifying the indoor image frames in the target video, multi-view image frames corresponding to the indoor image frames can be generated and added to the indoor image frames to obtain new indoor image frames. For example, multi-view image frames corresponding to indoor image frames can be generated using Generative Adversarial Networks (GANs).

[0083] In other embodiments, to avoid the impact of people captured in indoor image frames on the accuracy of scene reconstruction, the image region corresponding to the person in the indoor image frame can be identified, the image content (i.e., the person) in that region can be deleted, and then the image content in that region can be re-filled. For example, a content-aware fill algorithm can be used to fill in the image content; the specific process will not be elaborated further.

[0084] 203. Associate the product information of the home decoration products with the product model to obtain the target home decoration scene.

[0085] Product information for home improvement products can refer to at least one of the following: brand, model, and purchase link. For example, an electronic device can identify each home improvement product in an indoor image frame and retrieve its product information using the image search function of a pre-set search engine.

[0086] When performing step 203, association can refer to displaying the product information of home furnishing products in the device screen area corresponding to the product model. For ease of understanding, please refer to... Figure 3 , Figure 3 A target home decoration scenario 301 is shown, which includes multiple product models 303 associated with product information 302 (not all product information and product models are shown).

[0087] After obtaining the target home decoration scene, the electronic device can save the corresponding file in a preset database. Users can open the file corresponding to the target home decoration scene at any time to view the target home decoration scene.

[0088] In summary, the home decoration scene generation method provided in this application includes: acquiring a target video to be reconstructed; reconstructing a scene based on indoor image frames in the target video to obtain an initial home decoration scene, wherein the indoor image frames contain home decoration products, and the initial scene contains a product model corresponding to the home decoration products; associating the product information of the home decoration products with the product model to obtain a target home decoration scene.

[0089] As can be seen, the home decoration scene generation method provided in this application can automatically filter out indoor image frames containing indoor scenes in videos, reconstruct the home decoration scene contained in the indoor scene, simplify the user's operation of obtaining home decoration scenes, and can also associate the product information of home decoration products with the home decoration scene. When displaying the reconstructed target home decoration scene, it provides the user with detailed product information of home decoration products. The user no longer needs to search for product information of home decoration products in the home decoration scene, further simplifying the user's operation of obtaining home decoration scenes.

[0090] In some embodiments, to reduce the amount of information displayed in the target home decoration scene, avoid confusing users with too much information, and prevent the displayed product information from obscuring product models in the target home decoration scene or obscuring other product information, when associating product information and product models, it can be set that the product information associated with the product model is only displayed when the user triggers the product model in the target home decoration scene. That is, after the step "associating the product information of the home decoration product with the product model to obtain the target home decoration scene", it further includes:

[0091] In response to a trigger command on the target product model in the target home decoration scenario, the target product information of the target home decoration product associated with the target product model is displayed.

[0092] The trigger command can be input by the user through a remote control or other peripheral device, or through a touch screen, and this application embodiment does not impose any restrictions on this. For example, when the electronic device is a television, after opening the file corresponding to the target home decoration scene to display the target home decoration scene on the television screen, the user can use the movement buttons on the remote control to select the target product model from the various product models included in the target home decoration scene, and input the trigger command for the target product model by pressing the confirmation button on the remote control. After receiving the trigger command for the target product model, the television can display the target product information associated with the target product model on the television screen.

[0093] While steps 201-203 provide a method for obtaining indoor image frames from a target video, this method has certain limitations and cannot rule out the possibility that outdoor scenes may also contain home furnishing products. Therefore, to improve the accuracy of obtaining indoor image frames, this application also provides a method, referencing... Figure 4 Before the step "reconstructing the scene based on the indoor image frames in the target video to obtain the initial home decoration scene", the following steps are also included:

[0094] 401. Obtain candidate image frames containing the ceiling from each image frame in the target video.

[0095] For example, an open-source object recognition / object detection model such as AlexNet can be trained to obtain a building material detection model, and the building material detection model can be used to detect whether each image frame contains a ceiling in order to obtain candidate image frames containing a ceiling.

[0096] It should be noted that, since the specific location of the ceiling in the candidate image frame needs to be obtained in the embodiments of this application, the target detection model can be selected as the model to be trained to obtain the building material detection model. Furthermore, the subsequent steps of the embodiments of this application also need to obtain the location of the wall in the candidate image frame. Therefore, when training the target detection model, its ability to detect the wall can be trained at the same time to obtain a building material detection model that can detect both the ceiling and the wall.

[0097] 402. Identify the walls in each of the candidate image frames.

[0098] For example, the wall in the candidate image frame can be detected by the building material detection model mentioned in step 401, and the specific details will not be elaborated.

[0099] 403. For each candidate image frame, determine whether each candidate image frame is an indoor image frame based on the relative positional relationship between the wall and the ceiling, and obtain the determination result.

[0100] In this embodiment, the reason for determining the indoor image frame based on the relative positional relationship between the wall and the ceiling is that if the candidate image frame is an indoor image frame, the wall and the ceiling will present a specific structural positional relationship. For example, in an indoor image frame, since the wall and the ceiling usually belong to the same room, they will intersect, and the angle they present is usually close to a right angle. In an outdoor image frame, since the wall and the ceiling usually belong to different buildings and not to the same room, they usually do not intersect. Even if the wall and the ceiling intersect in an outdoor image frame due to the viewing angle, the probability that the angle they present is close to a right angle is not high. Therefore, the method in step 403 can accurately identify the indoor image frame in the candidate image frame.

[0101] The target home decoration scene obtained in steps 201-203 only includes product information and the positional relationships between home decoration items. However, users cannot obtain the visual effect of the home decoration items actually placed in a room based on the target home decoration scene obtained in steps 201-203, making it not very practical. To solve this problem, this application provides a method that, after modeling the home decoration items, constructs a room model consisting of rendered wall models and rendered ceiling models, and then places the modeled product models into the room model to obtain a target home decoration scene composed of both product models and room models. (Reference) Figure 5 The step "reconstructing the scene based on the indoor image frames in the target video to obtain the initial home decoration scene" includes:

[0102] 501. Perform 3D modeling on the home decoration products contained in the indoor image frames of the target video to obtain the product models corresponding to the home decoration products, the product dimensions of each product model, and the positional relationships between each product model.

[0103] The method for 3D modeling can be found in the description of step 202. For example, the SFM algorithm and MVS algorithm can be combined to obtain the product models corresponding to home decoration products, as well as the positional relationships between the product models.

[0104] 502. Based on the product size and positional relationship of each product model, construct a room model for placing each product model, wherein the room model is composed of rendered building material models, and the building material models include wall models, ceiling models and / or floor models.

[0105] A room model refers to a model depicting the outline of a room used to house various product models. It consists of at least one of the following: a rendered wall model, a rendered ceiling model, and a rendered floor model. For example, in... Figure 3 In the model, the room model consists of the rendered wall model and the rendered floor model.

[0106] Step 502 aims to build a room model to provide users with the visual effect of home decoration products when they are actually placed in a room. It is understandable that since the room model is composed of rendered building material models, that is, the building material models have been given corresponding materials, when users open the file corresponding to the target home decoration scene to display the target home decoration scene on the screen of electronic devices, they can see the visual matching effect between the textures of the home decoration products and the building material models.

[0107] For example, a room model to be rendered can be constructed first, and then the material of the building material model can be identified from the interior image frame. The room model to be rendered can then be rendered to obtain the rendered room model. For example, the step "constructing a room model for placing each of the commodity models based on the said positional relationship" can be performed in the following way:

[0108] (1.1) Based on the positional relationship, generate the minimum bounding box of each of the commodity models.

[0109] The minimum bounding box refers to the smallest three-dimensional volume that can enclose each product model. For example, the shape of the minimum bounding box can be set to a three-dimensional rectangle. Taking any product model as a reference point, based on the positional relationship between other product models (excluding the stated product model) and any product model, as well as the product size of the product models, the minimum three-dimensional rectangle that can enclose each product model is determined, and this minimum three-dimensional rectangle is used as the minimum bounding box of each product model.

[0110] This application does not limit the specific algorithm for generating the minimum bounding box. For example, the axis-aligned bounding box algorithm, the oriented bounding box algorithm, etc., can be used.

[0111] (1.2) Based on the bounding box size of the minimum bounding box, generate a model to be rendered, wherein the model to be rendered is composed of a building material model to be rendered.

[0112] The model to be rendered refers to the room model to be rendered.

[0113] For ease of understanding, this application embodiment takes a three-dimensional rectangle as the smallest bounding box as an example. The bounding box size may include the length, width and height of the three-dimensional rectangle.

[0114] Understandably, the location of the building materials to be generated can be determined based on the bounding box size of the minimum bounding box. For example, the location of the ceiling model can be determined based on the area of ​​the top of the minimum bounding box, which is the product of the length and width in the bounding box size, and the height in the bounding box size. The same applies to the wall model and the ground model, so I will not go into details.

[0115] (1.3) Identify the model material of the building material model to be rendered from the indoor image frame.

[0116] This application does not limit the method for identifying materials. For example, a trained material recognition model can be used to identify the material of a building material model in an indoor image frame. For instance, the open-source SENet model can be trained to obtain a trained material recognition model; the specific process will not be elaborated further.

[0117] (1.4) Render the building material model to be rendered based on the model material to obtain the rendered building material model and the room model composed of the rendered building material model.

[0118] After rendering, the materials can be assigned to the building material model to be rendered, resulting in the rendered building material model and the room model composed of the rendered building material model.

[0119] 503. Based on the aforementioned positional relationship, place each of the aforementioned product models within the room model to obtain an initial home decoration scene.

[0120] This application also provides a home scene generation system that can be used to generate target home scenes. (Reference) Figure 6 , Figure 6 The diagram shows a schematic of the modules of a home scene generation system 600, which includes a scene recognition module 601, a 3D modeling module 602, a home item recognition module 603, a layout recognition module 604, and a model saving module 605.

[0121] The scene recognition module 601 is used to identify indoor image frames from the image frames contained in the target video.

[0122] The 3D modeling module 602 is used to perform 3D modeling of home decoration products contained in indoor image frames, obtain the product model corresponding to each home decoration product, and construct the model to be rendered.

[0123] The home furnishing item identification module 603 is used to obtain product information of home furnishing products.

[0124] The layout recognition module 604 is used to identify the model material of the building material model to be rendered from the indoor image frame, render the model material to the building material model to be rendered, obtain the rendered room model, and generate the initial home decoration scene based on the room model and each product model.

[0125] The model storage module 605 is used to associate the product information of home decoration products with the product models corresponding to the home decoration products in the initial home decoration scene to obtain the target home decoration scene, and store the target home decoration scene in a preset database. The database can refer to the backend database of electronic devices. For example, when the electronic device refers to a television, the database can refer to the backend database of the television.

[0126] In practical applications, the home scene generation system 600 can operate according to the following process:

[0127] (A) When a user watches a video and sees the home decoration scene he wants to obtain, the user inputs a scene generation command into the home scene generation system 600, which triggers the scene recognition module 601.

[0128] (B) The scene recognition module 601 identifies indoor image frames from the currently playing video and inputs the indoor image frames into the 3D modeling module 602;

[0129] (C) The 3D modeling module 602 performs 3D modeling on the home decoration products contained in the indoor image frame, obtains the product model corresponding to each home decoration product, and constructs the model to be rendered, and inputs the product model corresponding to each home decoration product and the model to be rendered into the layout recognition module 604.

[0130] (D) The layout recognition module 604 identifies the model material of the building material model to be rendered from the indoor image frame, renders the model material to the building material model to be rendered, obtains the rendered room model, and generates an initial home decoration scene based on the room model and each product model, and inputs the initial home decoration scene into the model saving module 605.

[0131] (E) The home furnishing item recognition module 603 searches for the product information of home furnishing products and inputs the product information into the model storage module 605;

[0132] (F) The model saving module 605 associates the product information of the home decoration products with the product model corresponding to the home decoration products in the initial home decoration scene to obtain the target home decoration scene, and stores the target home decoration scene in the preset database.

[0133] To better implement the home scene generation method in this application embodiment, based on the home scene generation method, this application embodiment also provides a home scene generation device, such as... Figure 7The diagram shown is a structural schematic of one embodiment of the home scene generation device in this application. The home scene generation device 700 includes:

[0134] Acquisition unit 701 is used to acquire the target video to be reconstructed;

[0135] The reconstruction unit 702 is used to reconstruct the scene based on the indoor image frames in the target video to obtain an initial home decoration scene, wherein the indoor image frames contain home decoration products, and the initial scene contains product models corresponding to the home decoration products.

[0136] The association unit 703 is used to associate the product information of the home decoration product with the product model to obtain the target home decoration scene.

[0137] In one possible implementation of this application, the associated unit 703 is further configured to:

[0138] In response to a trigger command on the target product model in the target home decoration scenario, the target product information of the target home decoration product associated with the target product model is displayed.

[0139] In one possible implementation of this application, the reconstruction unit 702 is used for:

[0140] Obtain candidate image frames containing the ceiling from each image frame in the target video;

[0141] The walls in each of the candidate image frames are identified;

[0142] For each candidate image frame, based on the relative positional relationship between the wall and the ceiling, it is determined whether each candidate image frame is an indoor image frame, and the determination result is obtained.

[0143] In one possible implementation of this application, the reconstruction unit 702 is used for:

[0144] The home decoration products contained in the indoor image frames of the target video are modeled in 3D to obtain the product models corresponding to the home decoration products, the product dimensions of each product model, and the positional relationships between each product model;

[0145] Based on the product size and positional relationship of each product model, a room model for placing each product model is constructed, wherein the room model is composed of rendered building material models, including wall models, ceiling models and / or floor models;

[0146] Based on the aforementioned positional relationships, each of the product models is placed within the room model to obtain an initial home decoration scene.

[0147] In one possible implementation of this application, the reconstruction unit 702 is used for:

[0148] Based on the product size and positional relationship of each product model, generate the minimum bounding box of each product model;

[0149] Based on the bounding box size of the minimum bounding box, a model to be rendered is generated, wherein the model to be rendered is composed of a building material model to be rendered;

[0150] The model material of the building material model to be rendered is identified from the indoor image frame;

[0151] The building material model to be rendered is rendered based on the model material to obtain the rendered building material model and the room model composed of the rendered building material model.

[0152] In one possible implementation of this application, the product information includes a product purchase link, product model, and / or brand.

[0153] In one possible implementation of this application, the acquiring unit 701 is further configured to:

[0154] In response to the scene generation command, the currently playing video is used as the target video to be reconstructed.

[0155] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of the above modules, please refer to the previous method implementation examples, which will not be repeated here.

[0156] Since the home scene generation device can execute the steps in the home scene generation method in any embodiment, it can achieve the beneficial effects that the home scene generation method in any embodiment of this application can achieve, as detailed in the preceding description, and will not be repeated here.

[0157] Furthermore, to better implement the home scene generation method in this application embodiment, based on the home scene generation method, this application embodiment also provides an electronic device, see below. Figure 8 , Figure 8 This illustration shows a structural diagram of an electronic device according to an embodiment of this application. Specifically, the electronic device provided in this embodiment includes a processor 801. The processor 801 is used to execute a computer program stored in a memory 802 to implement each step of the home scene generation method in any embodiment; or, the processor 801 is used to execute a computer program stored in a memory 802 to implement, for example... Figure 7 The functions of each module in the corresponding embodiment.

[0158] For example, a computer program may be divided into one or more modules / units, one or more of which are stored in memory 802 and executed by processor 801 to complete the embodiments of this application. One or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in a computer device.

[0159] The electronic device may include, but is not limited to, processor 801 and memory 802. Those skilled in the art will understand that the illustrations are merely examples of an electronic device and do not constitute a limitation on the device. It may include more or fewer components than illustrated, or combine certain components, or use different components.

[0160] The processor 801 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the electronic device, connecting various parts of the electronic device through various interfaces and lines.

[0161] The memory 802 can be used to store computer programs and / or modules. The processor 801 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 802 and by calling the data stored in the memory 802. The memory 802 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device (such as audio data, video data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0162] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the home scene generation device, electronic device and its corresponding modules described above can be referred to the description of the home scene generation method in any embodiment, and will not be repeated here.

[0163] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a readable storage medium and loaded and executed by a processor.

[0164] Therefore, embodiments of this application provide a readable storage medium storing a computer program. When the computer program is executed by a processor, it performs the steps in the home scene generation method in any embodiment of this application. For specific operations, please refer to the description of the home scene generation method in any embodiment.

[0165] The readable storage medium may include: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0166] Since the instructions stored in the readable storage medium can execute the steps in the home scene generation method in any embodiment of this application, the beneficial effects that the home scene generation method in any embodiment of this application can achieve can be realized, as detailed in the preceding description, and will not be repeated here.

[0167] The foregoing has provided a detailed description of a home scene generation method, apparatus, storage medium, and electronic device provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A home scene generation method, characterized by, The method comprises the following steps: obtaining a target video to be reconstructed; performing scene reconstruction based on indoor image frames in the target video to obtain an initial home furnishing scene, wherein the indoor image frames contain home furnishing commodities, and the initial home furnishing scene contains commodity models corresponding to the home furnishing commodities; associating commodity information of the home furnishing commodities with the commodity models to obtain a target home furnishing scene; before the step of performing scene reconstruction based on the indoor image frames in the target video to obtain the initial home furnishing scene, the method further comprises the following steps: obtaining candidate image frames containing a ceiling from each image frame in the target video; identifying walls in each of the candidate image frames; for each of the candidate image frames, judging whether each of the candidate image frames is an indoor image frame according to a relative positional relationship between the walls and the ceiling to obtain a judgment result.

2. The home scene generation method of claim 1, wherein, after the step of associating the commodity information of the home furnishing commodities with the commodity models to obtain the target home furnishing scene, the method further comprises the following step: in response to a triggering instruction of a target commodity model in the target home furnishing scene, displaying target commodity information of a target home furnishing commodity associated with the target commodity model.

3. The home scene generation method of claim 1, wherein, the step of performing scene reconstruction based on the indoor image frames in the target video to obtain the initial home furnishing scene comprises the following steps: performing three-dimensional modeling on the home furnishing commodities contained in the indoor image frames in the target video to obtain the commodity models corresponding to the home furnishing commodities, commodity sizes of the commodity models, and positional relationships between the commodity models; based on the commodity sizes of the commodity models and the positional relationships, constructing a room model for placing the commodity models, wherein the room model is composed of rendered building material models, and the building material models include wall models, ceiling models, and / or floor models; based on the positional relationships, placing the commodity models in the room model to obtain the initial home furnishing scene.

4. The home scene generation method according to claim 3, characterized by, the step of constructing the room model for placing the commodity models based on the commodity sizes of the commodity models and the positional relationships comprises the following steps: based on the commodity sizes of the commodity models and the positional relationships, generating minimum bounding boxes of the commodity models; based on bounding box sizes of the minimum bounding boxes, generating a to-be-rendered model, wherein the to-be-rendered model is composed of to-be-rendered building material models; identifying model materials of the to-be-rendered building material models from the indoor image frames; based on the model materials, rendering the to-be-rendered building material models to obtain rendered building material models, and a room model composed of the rendered building material models.

5. The home scene generation method of claim 1, wherein, The commodity information includes a commodity purchase link, a commodity model, and / or a brand.

6. The home scene generation method according to any one of claims 1 to 5, characterized by, The step of obtaining the target video to be reconstructed comprises the following step: in response to a scene generation instruction, taking a currently played video as the target video to be reconstructed.

7. A home scene generation apparatus characterized by comprising: The method comprises the following steps: an obtaining unit is configured to obtain a target video to be reconstructed; a reconstructing unit is configured to perform scene reconstruction based on indoor image frames in the target video to obtain an initial home furnishing scene, wherein the indoor image frames contain home furnishing commodities, and the initial home furnishing scene contains commodity models corresponding to the home furnishing commodities; An association unit is configured to associate the product information of the home furnishing product with the product model to obtain a target home furnishing scene; The reconstruction unit is further configured to: obtain candidate image frames containing a ceiling from each image frame in the target video; identify a wall in each of the candidate image frames; for each of the candidate image frames, determine whether each of the candidate image frames is an indoor image frame according to a relative position relationship between the wall and the ceiling to obtain a determination result.

8. An electronic device, comprising: The electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps in the home scene generation method of any one of claims 1 to 6.

9. A readable storage medium, characterized by, The readable storage medium has a computer program stored thereon, and the computer program is executable by a processor to implement the steps in the home scene generation method of any one of claims 1 to 6.

Images