Housing information display method, device and electronic equipment

Abstract

The disclosed embodiments disclose a method, device and electronic device for displaying house information. A specific implementation of the method includes: obtaining target panoramic images corresponding to multiple rooms of a target house; for each room, constructing a three-dimensional virtual space of the room according to the target panoramic image corresponding to the room; extracting image features corresponding to each of the multiple rooms according to the target panoramic image; determining the relative position relationship between the multiple rooms based on the image features corresponding to each of the multiple rooms; splicing and displaying the three-dimensional virtual spaces corresponding to each of the multiple rooms based on the relative position relationship, realizing the splicing of three-dimensional virtual real scenes of each house, and realizing the real scene restoration of the house information. It is convenient for users to browse the overall view of the house through the above solution.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a housing information display method, device and electronic device. Background Art

[0002] Virtual Reality (VR) technology is an emerging practical technology. VR technology uses computer simulation to generate a virtual scene in a three-dimensional space, providing users with simulations of vision, hearing, touch and other senses, allowing users to observe things in the three-dimensional space in an immersive way. With the continuous development of social productivity and science and technology, the demand for VR technology in various fields is gradually increasing. For example, VR technology can be applied to space (for example, buildings, etc.) display systems. Based on VR technology, a realistic three-dimensional model of a building can be constructed to fully display the internal and external structure and functions of the building, so that users can roam in the three-dimensional model of the building through the network, experience the building and its surrounding environment in an immersive way, and quickly and conveniently realize remote house viewing and house selection. Summary of the invention

[0003] This disclosure section is provided to introduce concepts in a brief form, which will be described in detail in the detailed description section below. This disclosure section is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] Embodiments of the present disclosure provide a housing information display method, device, and electronic device.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for displaying house information, the method comprising: obtaining a target panoramic image corresponding to each of a plurality of rooms of a target house; for each room, constructing a three-dimensional virtual space of the room according to the target panoramic image corresponding to the room; extracting image features corresponding to each of the plurality of rooms according to the target panoramic image; determining a relative position relationship between the plurality of rooms based on the image features corresponding to each of the plurality of rooms; and splicing and displaying the three-dimensional virtual spaces corresponding to each of the plurality of rooms based on the relative position relationships.

[0006] In a second aspect, an embodiment of the present disclosure provides a house information display device, comprising: an acquisition unit, used to acquire a target panoramic image corresponding to each of the multiple rooms of a target house; a construction unit, used to construct, for each room, a three-dimensional virtual space of the room according to the panoramic image corresponding to the room; an extraction unit, used to extract image features corresponding to each of the multiple rooms according to the target panoramic image; a determination unit, used to determine a relative position relationship between the multiple rooms based on the image features corresponding to each of the multiple rooms; and a splicing unit, used to splice the three-dimensional virtual spaces corresponding to each of the multiple rooms based on the relative position relationship.

[0007] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the housing information display method as described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the housing information display method as described in the first aspect.

[0009] The housing information display method, device and electronic device provided by the embodiments of the present disclosure obtain target panoramic images corresponding to multiple rooms of the target house; for each room, construct the three-dimensional virtual space of the room according to the target panoramic image corresponding to the room; extract the image features corresponding to the multiple rooms according to the target panoramic image; determine the relative position relationship between the multiple rooms based on the image features corresponding to the multiple rooms; splice and display the three-dimensional virtual spaces corresponding to the multiple rooms based on the relative position relationship, thereby realizing the splicing of three-dimensional virtual scenes of each house and restoring the house information to the real scene. It is convenient for users to browse the overall view of the house through the above scheme. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0011] Figure 1 is a flow chart of an embodiment of a housing information display method according to the present disclosure;

[0012] Figure 2 is a flow chart of another embodiment of the housing information display method according to the present disclosure;

[0013] Figure 3is a structural schematic diagram of an embodiment of a housing information display device according to the present disclosure;

[0014] Figure 4 is an exemplary system architecture to which the housing information display method of an embodiment of the present disclosure may be applied;

[0015] Figure 5 It is a schematic diagram of the basic structure of an electronic device provided according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0017] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0018] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0019] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0020] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0021] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0022] Please refer to Figure 1 , which shows the process of an embodiment of the housing information display method according to the present disclosure. Figure 1 The housing information display method shown comprises the following steps:

[0023] Step 101 , obtaining target panoramic images corresponding to multiple rooms of a target house.

[0024] In this embodiment, the electronic device that executes the housing information display method may be a server or a terminal device.

[0025] In some embodiments, for each room, the user may use an ordinary camera to take multiple two-dimensional images of the room at various angles, and then stitch the multiple two-dimensional images into a panoramic image of the room.

[0026] In some other embodiments, for each room, the user may also use two 180-degree wide-angle cameras to capture a two-dimensional image of the room, and stitch the two two-dimensional images into a panoramic image of the room.

[0027] In some other embodiments, for each room, the user can also use a 360-degree panoramic camera to capture a panoramic image of the room.

[0028] In some embodiments, the panoramic image of the room obtained by the above embodiments may be used as the target panoramic image of the room.

[0029] In some optional implementations of this embodiment, the above step 101 may further include the following sub-steps:

[0030] Sub-step 1011, for each room, displaying the acquired original panoramic image corresponding to the room in the display interface of the room image, and displaying the automatically identified reference geometric features of the room in the original panoramic image.

[0031] Sub-step 1012, receiving an adjustment operation of the user on the reference geometric features of the room in the display interface, and obtaining a target panoramic image of the room after the reference geometric features are adjusted.

[0032] In these optional implementations, the above-mentioned reference geometric features may include one of the following: the intersection of three planes of the room (corner vertices), the intersection line of two walls of the room, and the four vertices of the door.

[0033] In some embodiments, the room image may be displayed in a user's terminal device. The original panoramic image of the room may be displayed in the display interface of the room image. The automatically identified reference geometric features may be displayed in the original panoramic image. For example, a colored marking point may be used to mark the identified corner vertex, and a colored marking line may be used to mark the intersection of the two identified walls.

[0034] The user can perform adjustment operations on the above-mentioned reference geometric features in the above-mentioned display interface. For example, drag the above-mentioned colored marking point to adjust the position of the corner vertex. After the user adjusts the marking of the above-mentioned reference geometric features displayed in the display interface, the target panoramic image of the room corrected by the user is obtained.

[0035] In these optional implementations, an interactive entry is provided in the display interface for users to adjust the reference geometric features identified by automatic image recognition, so that users can manually correct the automatically identified reference geometric features, eliminating errors that may occur in image recognition, and making the three-dimensional virtual space of the room obtained according to the corrected target panorama more closely matched with the real space of the room.

[0036] Step 102: for each room, construct a three-dimensional virtual space of the room according to the target panoramic image corresponding to the room.

[0037] In this embodiment, the three-dimensional virtual space of the room may be constructed using the target panoramic image corresponding to the room according to various existing methods for constructing a three-dimensional image using panoramic images.

[0038] In some embodiments, the reference geometric features of the room may be identified from the target panoramic image, where the reference geometric features may include, for example, the intersection of three adjacent surfaces of the room, the intersection of two adjacent surfaces of the room, and the door fixed point.

[0039] Specifically, the above step 102 may further include: constructing a three-dimensional virtual space of the room based on the reference geometric features of the room.

[0040] In some application scenarios, after determining the above-mentioned reference geometric features, the above-mentioned panoramic image can be pre-processed, such as setting a first color for pixels corresponding to the ground and ceiling, and setting a second color for vertical planes such as walls, to obtain superpixels. A panoramic depth map of the room is obtained based on the superpixels. The three-dimensional virtual space of the room is constructed based on the panoramic depth map and the above-mentioned reference geometric features.

[0041] It should be noted that the method of constructing a three-dimensional virtual space of a house based on a panoramic image of the room is a technology that is currently widely studied and applied, and will not be described in detail here.

[0042] Step 103: extracting image features corresponding to each of the multiple rooms according to the target panoramic image.

[0043] In this embodiment, the features of the doors corresponding to the multiple rooms can be extracted from the target panoramic image. The features of the door may include, for example, the outline of the door, the position of the door in the target panoramic image, the size of the door, and the type of the door.

[0044] In practice, for each room, the outline of the door appearing in the target panoramic image can be detected in the target panoramic image, and the size of the door can be estimated based on the outline of the door; the position of the door appearing in the target panoramic image. The position of the door appearing in the target panoramic image here can include, for example: the door appears in the wall of the room in the target panoramic image, and the room is not in the wall of the room.

[0045] The type of the door in the target panoramic image of the room can be determined by the position of the door in the target panoramic image. The type of the door may include: the door of the room, the door of the room adjacent to the room, and the door of the room connected to the room by a public space.

[0046] Step 104: Determine the relative position relationship between the multiple rooms based on the image features corresponding to each of the multiple rooms.

[0047] The relative position relationship here includes adjacency, connection by a public space, etc. For example, it may include: room A is adjacent to living room B, and room C and room D are connected by a passage.

[0048] The above step 104 may include: determining the relative position relationship of the multiple rooms according to the characteristics of the doors corresponding to the multiple rooms respectively.

[0049] For multiple rooms, the relative position relationship between the multiple rooms is determined by the door types determined from the respective rooms and the sizes of the doors determined from the target panoramic image.

[0050] For example, all possible door correspondences are automatically enumerated and attempted to be spliced ​​together. The relative position relationship between the rooms is determined by eliminating impossible splicing such as differences in door size, door outline, and overlap between the two rooms after splicing.

[0051] Step 105 , splicing and displaying the three-dimensional virtual spaces corresponding to the multiple rooms based on the relative position relationship.

[0052] After the relative positional relationship of the multiple rooms is obtained in step 104, the three-dimensional virtual spaces corresponding to the multiple rooms can be arranged in the three-dimensional space according to the relative positional relationship, so as to obtain a reconstructed three-dimensional image of the target house.

[0053] The method provided by the above-mentioned embodiment of the present disclosure obtains the target panoramic images corresponding to each of the multiple rooms of the target house; for each room, constructs the three-dimensional virtual space of the room according to the target panoramic image corresponding to the room; extracts the image features corresponding to each of the multiple rooms according to the target panoramic image; determines the relative position relationship between the multiple rooms based on the image features corresponding to each of the multiple rooms; and splices and displays the three-dimensional virtual spaces corresponding to each of the multiple rooms based on the relative position relationship. The three-dimensional virtual scene splicing of each house is realized, and the house information is restored to the real scene. It is convenient for users to browse the full view of the house through the above-mentioned scheme.

[0054] Please refer to Figure 2 , which shows the process of an embodiment of the housing information display method according to the present disclosure. Figure 2 The housing information display method shown comprises the following steps:

[0055] Step 201 , obtaining target panoramic images corresponding to multiple rooms of a target house.

[0056] Step 202: for each room, construct a three-dimensional virtual space of the room according to the target panoramic image corresponding to the room.

[0057] The above steps 201 to 202 can be Figure 1 Steps 101 and 102 of the illustrated embodiment are the same and will not be described in detail here.

[0058] Step 203: extracting image features corresponding to each of the multiple rooms according to the target panoramic image, wherein the image features of each room include external image features and internal image features of the room.

[0059] The target panoramic image of each room may include the image inside the room and the images of other rooms captured through the door of the room or the connection between the room and other rooms. For example, the target panoramic image of the living room may include the images of other rooms in the room (such as bedrooms and kitchens). The target panoramic image of a bedroom may include the image of the interior of the bedroom and the image of the living room captured through the door of the bedroom, or images of other bedrooms, etc.

[0060] In some embodiments, for each target panoramic image of a room, the outline of the door may be detected first. Then, the image area inside the door and the image area outside the door are determined in the target panoramic image of the room. Finally, image features are extracted from the image area inside the door and the image area outside the door respectively, thereby obtaining the internal image features and external image features of the room.

[0061] In some other embodiments, for each target panoramic image of a room, the image features of the target panoramic image may be first extracted. Then, the image area inside the door and the image area outside the door are determined from the target panoramic image. Finally, the boundary between the image area inside the door and the image area outside the door is used to determine the internal image features and external image features of the room from the image features.

[0062] When extracting the image features of a room, various existing image feature extraction methods may be used. For each room, the image features of the room are extracted based on the target panoramic image of the room.

[0063] In some embodiments, the image features of each room can be extracted according to at least one of the following image feature extraction methods: an image feature extraction method based on image feature descriptors, a structural feature extraction method based on contour detection, and an image feature extraction method based on neural networks.

[0064] The above-mentioned image feature descriptors may include, for example, but are not limited to: Scale-invariant feature transform (SIFT) descriptor; Binary Robust Independent Elementary Features (BRIEF) descriptor; Speeded Up Robust Features (SURF) descriptor.

[0065] The structural feature extraction method based on contour detection here may include, for example, detecting structural feature information of windows, doors, walls, etc. of the room according to the contour detection method. The structural feature information of windows, doors, and walls may include, for example, the width, height, quantity information, etc. corresponding to the above windows, doors, and walls.

[0066] The neural network-based image feature extraction method may include, for example, extracting features of the target panoramic image using a neural network model. The neural network model may include, but is not limited to, an artificial neural network model, a convolutional neural network model, and a deep learning model.

[0067] It should be noted that the above-mentioned image feature extraction method is a well-known technology that has been widely studied and applied, and will not be described in detail here.

[0068] Step 204: determine the relative positional relationship between the multiple rooms based on the external image features and the internal image features corresponding to the multiple rooms respectively.

[0069] The relative positional relationship between rooms may include, for example, an adjacency relationship and a connection relationship through a public space.

[0070] For each room, by comparing the external image features of the room with the internal image features of other rooms, the room adjacent to it or the room connected to it by a common space can be determined, thereby further determining the relative position relationship between multiple rooms.

[0071] In some embodiments, step 204 may include the following sub-steps:

[0072] Sub-step 2041, for each of the multiple rooms, matching the external features of the room with the internal features of other rooms one by one.

[0073] In some application scenarios, when matching the external features of the room with the internal features of other rooms, the similarity between the external features and the internal features of other rooms can be directly calculated, and whether the external features of the room match the other rooms can be determined based on the similarity.

[0074] In some other application scenarios, a bag of words model can be used to statistically analyze the similar features between the external image features of the room and the internal image features of other rooms based on the bag of words model, and determine the matching results of the room with other rooms based on the proportion of similar features in the internal image features of other rooms. In other words, the external image features of the room are statistically analyzed with the internal image features of other rooms, and scene matching is performed based on the proportion of similar features in a certain scene to obtain a matching result.

[0075] It should be noted that the bag-of-words model here is a well-known technology that is currently widely used and studied, and will not be described in detail here.

[0076] In some other application scenarios, the structural features in the external image features of the room are matched with the internal features of other rooms to determine the matching results of the external image features of the room and the internal image features of other rooms. That is, the structural features in the external image features of the room can be used to match the internal image features of other rooms including sparse structural features to obtain matching results. The sparse structural features here include the number of walls in the room, the width of the walls, the number of doors, the width of the doors, the number of windows, the width of the windows, etc.

[0077] In some other application scenarios, neural network features of the image or statistical features of image pixels may be used for matching to determine the matching results of the external image features of the room and the internal image features of other rooms.

[0078] In these application scenarios, a neural network model can be used to determine the internal neural network image features corresponding to the image area inside the door in the target panoramic image of each room, and the external neural network features corresponding to the image area outside the door. For one of the multiple rooms, the internal neural network features of the room can be matched with the external neural network features of the other rooms. In addition, an image processing method can be used to determine the internal image pixel statistical features of the image area outside the door in the target panoramic images of each of the multiple rooms, and the external image pixel statistical features corresponding to the image area outside the door. For one of the multiple rooms, the internal image pixel statistical features of the room can be matched with the external image pixel statistical features of the other rooms.

[0079] Sub-step 2042, determining a matching room that matches the external features from a plurality of other rooms according to the matching result.

[0080] Sub-step 2043, determining the relative positional relationship between the matching room and the room according to the positional relationship between the matching room and the room indicated by the external features.

[0081] The external features of the room may be features of the door of the room, or images of one or more other rooms captured in the room. In the panoramic image of the room, the images of other rooms may include doors of other rooms and partial images of other rooms.

[0082] The external image features of the room may be matched with the internal image features of other rooms. In practice, the similarity between the external image features of the room and the internal image features of each other room may be calculated. The other rooms indicated by the external image features having a similarity threshold greater than a preset similarity threshold are determined as matching rooms that match the room.

[0083] Usually, the images of other rooms captured through the door of the room are related to the relative positions of other rooms and the room. For example, for two adjacent rooms, or two rooms connected by a common space, the image captured through the door can be the same as the view that the user sees directly from the door. For two non-adjacent rooms, the doors of the two rooms are not directly opposite. In this way, for such two rooms, the image of the other room appears in the panoramic image of one of the rooms, which can be the same as the view that the user sees when viewing the other room from a line of sight that is at a certain angle to the direct line of sight from the door. Therefore, the external features of the room can indicate the positional relationship between the matching room and the room. That is, the relative positional relationship between the matching room and the room is determined based on the area of ​​the internal image area corresponding to the internal image features of the matching room that match the external features of the room in the matching room.

[0084] For example, the internal image area corresponding to the internal image feature of the matching room that matches the external image feature is the area opposite the door of the matching room. It can be considered that the matching room is adjacent to the room or connected through a public space.

[0085] Furthermore, the positional relationship between the matching room indicated by the above-mentioned external features and the room includes: the field of view angle of the camera device shooting the image area including the external image features in the room; and for each of the multiple rooms, matching the external features of the room with the internal features of other rooms one by one, including: selecting the target image area from the internal images of other rooms according to the above-mentioned field of view, and calculating the similarity between the target image area and the above-mentioned external image features.

[0086] Here, the above-mentioned field of view angle is the field of view angle when the camera device captures the image area corresponding to the above-mentioned external image features inside the room. That is, the field of view angle corresponding to the external image captured through the door. The field of view angle of the camera device can be 360°. As a schematic illustration, the total width of the image captured by the camera device occupies 360 pixels, and the width of the door in the image occupies 10 pixels. It can be determined that the field of view angle of the image area including the external image features captured in the room is 10°. The door here can be the door of the room, or it can be the door of other rooms. In some embodiments, the room is separated from other rooms by the door of the room, and the above-mentioned door is the door of the room. In other embodiments, the room and other rooms are each equipped with a door, and the above-mentioned door refers to the door of other rooms.

[0087] Furthermore, the range of other rooms (e.g., adjacent rooms in space, or rooms connected by a common space) opposite to the room observed through the door (when the door is opened) can be determined based on the distance between the placement position of the camera device in the room and the door. The target image area is determined in the other room based on the range.

[0088] For example, the door of room B is in the north, and it is assumed that room A and room B are opposite rooms. The image of room B can be taken from room A through the door of room B in the north. Assume that the original feature points of room B are distributed on the four walls of the southeast, northwest, and northeast. According to the field of view corresponding to the image of room B taken from room A through the door of room B, and the distance between the camera device in room A and the door of room B, determine the range of room B observed from room A through the above door (when it is opened). Then determine the target image area based on the above range.

[0089] Theoretically, the target image area can be determined in other rooms based on the field of view of the camera through the door of the room and the distance between the camera in the room and the door of the room.

[0090] In these optional implementations, the matching relationship between the room and other rooms is determined by extracting the target image region from the internal image of the other room and calculating the similarity between the image features of the target image region and the external features of the room, so that the calculated matching result is more accurate, which helps to improve the accuracy of the determined matching relationship between the room and other rooms.

[0091] Step 205 , splicing and displaying the three-dimensional virtual spaces corresponding to the multiple rooms based on the relative position relationship.

[0092] and Figure 1 Compared with the illustrated embodiment, the house information display method provided in this embodiment highlights the step of determining the relative position relationship of multiple rooms according to the internal image features and external image features of each room, which can speed up the speed of determining the relative position relationship of each room from the panoramic images of each room, thereby speeding up the speed of generating a panoramic virtual three-dimensional space of the house.

[0093] In some optional implementations of this embodiment, for each room, the target panoramic image corresponding to the room includes a first target panoramic image taken from a door corresponding to the room.

[0094] In these optional implementations, the step 203 of extracting image features corresponding to the multiple rooms according to the target panoramic image includes the following sub-steps:

[0095] Sub-step 2031 : for each room, determine a first target panoramic image from the target panoramic images of the room.

[0096] It should be noted here that the panoramic image of the room fixes the panoramic camera at a shooting position in the room. The shooting position can be located at the geometric center of the room, or at any other position in the room, such as under the door of the room. The above shooting position can be a preset distance from the horizontal plane. The preset distance here can be, for example, 1.2 meters. The panoramic camera collects a 360-degree environmental image at the shooting position to obtain a panoramic image of the room.

[0097] The target panoramic image captured under the room door is the first target panoramic image.

[0098] In these optional implementations, the first target panoramic image may be manually annotated or recognized by an image recognition method.

[0099] For multiple panoramic images of the room, all panoramic images can be traversed to determine whether the panoramic image is taken under the door through door frame detection. The detection method includes but is not limited to fitting the door frame through edge detection, detecting the door frame position using a neural network model, etc.

[0100] It can be understood that the panoramic image of the room taken under the door has the following characteristics: in the surrounding 360-degree images, for example, there is a door frame in the image corresponding to 60 degrees, there is a door frame in the image corresponding to 240 degrees, and there is a door frame in the top image. It can be determined that the panoramic image is the first target panoramic image taken at the position of the door.

[0101] Sub-step 2032, performing region segmentation on the first target panoramic image, and segmenting the image into the region inside the door and the region outside the door.

[0102] In sub-step 2031, after determining the door frame from the first target panoramic image, other target panoramic images of the room can be combined to identify the area inside the door and the area outside the door in the first target panoramic image. The area inside the door and the area outside the door are segmented using a segmentation line.

[0103] Sub-step 2033: determining a first external image feature according to the outer region of the first target panoramic image, and determining a first internal image feature according to the inner region of the first target panoramic image.

[0104] Sub-step 2034, integrating the first internal image feature and the internal image feature corresponding to the target panoramic image other than the first target panoramic image of the room to obtain the internal image feature of the room, and using the first external image feature as the external image feature of the room.

[0105] In these optional implementations, the image features corresponding to the area outside the door in the first target panoramic image are used as the external image features of the room. That is, the external image features determined from the target panoramic images taken from other positions of the room can be removed when determining the external image features of the room. Since the image information of other rooms included in the first target panoramic image taken at the position under the door of the room is more than the image information of other rooms in the target panoramic image of the room. Removing the external image features corresponding to other target panoramic images can reduce the amount of calculation required to match the external image features of the room with the internal image features of other rooms.

[0106] In these optional implementations, by adding the first target panoramic image taken at the door position of the room to the target panoramic image of the room, and determining the image features outside the door of the room from the first target panoramic image, the determined image features outside the door can reflect the image information of other rooms to a large extent. The accuracy of the matching results between the image features outside the door of the room and the internal image features of other rooms can be further improved. In turn, the accuracy of the generated three-dimensional virtual reality image of the house is improved.

[0107] Further references Figure 3 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a housing information display device. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0108] like Figure 3 As shown, the housing information display device of this embodiment includes: an acquisition unit 301, a construction unit 302, an extraction unit 303, a determination unit 304 and a splicing unit 305. The acquisition unit 301 is used to acquire the target panoramic images corresponding to the multiple rooms of the target house; the construction unit 302 is used to construct the three-dimensional virtual space of each room according to the panoramic image corresponding to the room; the extraction unit 303 is used to extract the image features corresponding to the multiple rooms according to the target panoramic image; the determination unit 304 is used to determine the relative position relationship between the multiple rooms based on the image features corresponding to the multiple rooms; the splicing unit 305 is used to splice the three-dimensional virtual spaces corresponding to the multiple rooms based on the relative position relationship.

[0109] In this embodiment, the specific processing of the acquisition unit 301, the construction unit 302, the extraction unit 303, the determination unit 304 and the splicing unit 305 of the housing information display device and the technical effects thereof can be referred to respectively. Figure 1 The relevant descriptions of step 101, step 102, step 103, step 104 and step 105 in the corresponding embodiments are not repeated here.

[0110] In some optional implementations of the present embodiment, the acquisition unit 301 is further used to: for each room, display the acquired original panoramic image corresponding to the room in the display interface of the room image, and display the automatically identified baseline geometric features of the room in the original panoramic image; receive the user's adjustment operation on the baseline geometric features of the room in the display interface, and obtain the target panoramic image of the room after adjusting the baseline geometric features.

[0111] In some optional implementations of this embodiment, the construction unit 302 is further used to: construct a three-dimensional virtual space of the room based on the reference geometric features of the room.

[0112] In some optional implementations of this embodiment, the extraction unit 303 is further used to extract image features of each room according to at least one of the following image feature extraction methods: an image feature extraction method based on image feature descriptors, a structural feature extraction method based on contour detection, and an image feature extraction method based on neural networks.

[0113] In some optional implementations of this embodiment, the image features of each room include external image features and internal image features of the room, and the determination unit 304 is further used to determine the relative position relationship between multiple rooms based on the external image features and internal image features corresponding to the multiple rooms respectively.

[0114] In some optional implementations of the present embodiment, for each room, the target panoramic image corresponding to the room includes a first target panoramic image taken from the door corresponding to the room; the extraction unit 303 is further used to: for each room, determine the first target panoramic image from the target panoramic image of the room; perform region segmentation on the first target panoramic image, dividing the area inside the door and the area outside the door; determine a first external image feature based on the area outside the door, and determine a first internal image feature based on the area inside the door; and integrate the first internal image feature and the internal image feature corresponding to the target panoramic image of the room other than the first target panoramic image to obtain the internal image feature of the room, and use the first external image feature as the external image feature of the room.

[0115] In some optional implementations of this embodiment, the determination unit 304 is further used to: for each of the multiple rooms, match the external features of the room with the internal features of the other rooms one by one; determine a matching room that matches the external features from the multiple other rooms based on the matching results; and determine the relative position relationship between the matching room and the room based on the position relationship between the matching room indicated by the external features and the room.

[0116] In some optional implementations of this embodiment, the positional relationship between the matching room indicated by the external feature and the room includes the field of view angle of the camera device capturing the image area including the external image feature in the room; the determination unit 304 is further used to: select the target image area from the internal images of other rooms based on the field of view, and calculate the similarity between the image features of the target image area and the external features.

[0117] In some optional implementations of the present embodiment, the determination unit 304 is further used to: statistically analyze similar features between the external image features of the room and the internal image features of other rooms based on a bag-of-words model, and determine the matching results between the external image features of the room and the internal image features of other rooms according to the proportion of similar features in the internal image features of other rooms; match the structural features in the external image features of the room with the internal features of other rooms to determine the matching results between the external image features of the room and the internal image features of other rooms; match using the neural network features of the image or the statistical features of the image pixels to determine the matching results between the external image features of the room and the internal image features of other rooms.

[0118] In some optional implementations of this embodiment, the extraction unit 303 is further used to: extract the features of the doors corresponding to the multiple rooms respectively according to the target panoramic image; and the determination unit 304 is further used to: determine the relative position relationship of the multiple rooms according to the features of the doors corresponding to the multiple rooms respectively.

[0119] Please refer to Figure 4 , Figure 4 An exemplary system architecture is shown in which the housing information display method according to an embodiment of the present disclosure can be applied.

[0120] like Figure 4 As shown, the system architecture may include terminal devices 401, 402, 403, a network 404, and a server 405. The network 404 is used to provide a medium for communication links between the terminal devices 401, 402, 403 and the server 405. The network 404 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0121] The terminal devices 401, 402, and 403 can interact with the server 405 through the network 404 to receive or send messages, etc. Various client applications can be installed on the terminal devices 401, 402, and 403, such as web browser applications, search applications, news and information applications, and house information browsing applications. The client applications in the terminal devices 401, 402, and 403 can receive user instructions and perform corresponding functions according to the user instructions, such as displaying a panoramic image of a target house according to the user instructions.

[0122] Terminal devices 401, 402, 403 can be hardware or software. When terminal devices 401, 402, 403 are hardware, they can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) players, laptop computers and desktop computers, etc. When terminal devices 401, 402, 403 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0123] Server 405 can be a server that provides various services, such as receiving target panoramic image information sent by terminal devices 401, 402, and 403, analyzing and processing the target panoramic image, and sending the analysis and processing results (such as information on the three-dimensional virtual image of the house) to terminal devices 401, 402, and 403.

[0124] It should be noted that the housing information display method provided in the embodiment of the present disclosure can be executed by the server 405, and accordingly, the housing information display device can be set in the server 405. In addition, the housing information display method provided in the embodiment of the present disclosure can also be executed by the terminal devices 401, 402, and 403, and accordingly, the housing information display device can also be set in the terminal devices 401, 402, and 403.

[0125] It should be understood that Figure 4 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0126] Reference below Figure 5 , which shows an electronic device (eg, Figure 3 The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0127] like Figure 5 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0128] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0129] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0130] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0131] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0132] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0133] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains the target panoramic images corresponding to each of the multiple rooms of the target house; for each room, constructs the three-dimensional virtual space of the room according to the target panoramic image corresponding to the room; extracts the image features corresponding to each of the multiple rooms according to the target panoramic image; determines the relative position relationship between the multiple rooms based on the image features corresponding to each of the multiple rooms; and splices and displays the three-dimensional virtual spaces corresponding to each of the multiple rooms based on the relative position relationship.

[0134] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0135] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0136] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, an acquisition unit may also be described as a "unit for acquiring target panoramic images corresponding to multiple rooms of a target house".

[0137] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0138] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0139] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0140] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0141] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A housing information display method, characterized in that: include: Obtain target panoramic images corresponding to multiple rooms of the target house; For each room, construct a three-dimensional virtual space of the room according to the target panoramic image corresponding to the room; Extracting image features corresponding to each of the multiple rooms according to the target panoramic image; Determining a relative positional relationship between the multiple rooms based on image features corresponding to each of the multiple rooms; splicing and displaying the three-dimensional virtual spaces corresponding to the multiple rooms based on the relative position relationship to obtain a reconstructed three-dimensional image of the target house; The step of extracting image features corresponding to each of the multiple rooms according to the target panoramic image includes: Extracting features of doors corresponding to each of the multiple rooms according to the target panoramic image; and The determining the relative position relationship between the multiple rooms based on the image features corresponding to the multiple rooms respectively includes: The relative position relationship of the multiple rooms is determined according to the characteristics of the doors corresponding to the multiple rooms.

2. The method according to claim 1, characterized in that The step of obtaining target panoramic images corresponding to the multiple rooms of the target house includes: For each room, the acquired original panoramic image corresponding to the room is displayed in the display interface of the room image, and the automatically identified reference geometric features of the room are displayed in the original panoramic image; An adjustment operation of the user on the reference geometric features of the room in the display interface is received, and a target panoramic image of the room after the reference geometric features are adjusted is obtained.

3. The method according to claim 2, characterized in that For each room, constructing the three-dimensional virtual space of the room according to the target panoramic image corresponding to the room includes: A three-dimensional virtual space of the room is constructed based on the reference geometric features of the room.

4. The method according to claim 1, characterized in that: The step of extracting image features corresponding to the plurality of rooms respectively according to the panoramic image comprises: The image features of each room are extracted according to at least one of the following image feature extraction methods: Image feature extraction method based on image feature descriptor, structural feature extraction method based on contour detection, and image feature extraction method based on neural network.

5. The method according to claim 1, characterized in that The image features of each room include the external image features and the internal image features of the room; as well as The determining the relative position relationship between the multiple rooms based on the image features corresponding to the multiple rooms respectively includes: The relative positional relationship between the multiple rooms is determined based on the external image features and the internal image features respectively corresponding to the multiple rooms.

6. The method according to claim 5, characterized in that For each room, the target panoramic image corresponding to the room includes a first target panoramic image taken from the door corresponding to the room; as well as The step of extracting image features corresponding to each of the plurality of rooms according to the target panoramic image comprises: For each room, determining the first target panoramic image from the target panoramic image of the room; Performing region segmentation on the first target panoramic image to segment the region inside the door and the region outside the door; determining a first external image feature according to an area outside the door of the first target panoramic image, and determining a first internal image feature according to an area inside the door of the first target panoramic image; and The first internal image feature and the internal image feature corresponding to the target panoramic image of the room other than the first target panoramic image are integrated to obtain the internal image feature of the room, and the first external image feature is used as the external image feature of the room.

7. The method according to claim 5 or 6, characterized in that: The determining the relative position relationship between the multiple rooms based on the internal image features and the external image features respectively corresponding to the multiple rooms includes: For each of the multiple rooms, matching the external features of the room with the internal features of the other rooms one by one; Determine a matching room that matches the external feature from a plurality of other rooms according to the matching result; The relative positional relationship between the matching room and the room is determined according to the positional relationship between the matching room indicated by the external feature and the room.

8. The method according to claim 7, characterized in that The positional relationship between the matching room indicated by the external feature and the room, including the field of view of the camera device capturing the image area including the external image feature in the room; and For each of the multiple rooms, matching the external features of the room with the internal features of the other rooms one by one includes: A target image region is selected from the internal images of other rooms based on the field of view, and a similarity between an image feature of the target image region and the external feature is calculated.

9. The method according to claim 7, characterized in that: For each of the multiple rooms, matching the external features of the room with the internal features of the other rooms one by one includes at least one of the following: Based on the bag-of-words model, the similarity between the external image features of the room and the internal image features of other rooms is statistically analyzed, and the matching result between the external image features of the room and the internal image features of other rooms is determined according to the proportion of similar features between the internal image features of other rooms; Using structural features in the external image features of the room to match internal features of other rooms, determining a matching result between the external image features of the room and internal image features of other rooms; The neural network features of the image or the statistical features of the image pixels are used for matching to determine the matching results of the external image features of the room and the internal image features of other rooms.

10. A housing information display device, characterized in that: include: An acquisition unit, used for acquiring target panoramic images corresponding to each of the multiple rooms of the target house; A construction unit, configured to construct, for each room, a three-dimensional virtual space of the room according to the panoramic image corresponding to the room; An extraction unit, configured to extract image features corresponding to each of the plurality of rooms according to the target panoramic image; a determining unit, configured to determine a relative position relationship between the plurality of rooms based on image features corresponding to each of the plurality of rooms; A splicing unit, configured to splice the three-dimensional virtual spaces corresponding to the plurality of rooms based on the relative position relationship to obtain a reconstructed three-dimensional image of the target house; The extraction unit is specifically configured to extract features of doors corresponding to the plurality of rooms according to the target panoramic image; and The determining unit is specifically configured to determine the relative position relationship of the multiple rooms according to the features of the doors corresponding to the multiple rooms respectively.

11. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9.

12. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images