Image matching method, device and computer readable storage medium

Abstract

The embodiment of the present invention provides an image matching method, device and computer-readable storage medium. The image matching method according to the embodiment of the present invention includes: obtaining a panoramic image of at least one subspace in a three-dimensional space, and a two-dimensional image of the three-dimensional space, wherein the three-dimensional space is composed of at least one subspace, and when the three-dimensional space is composed of multiple subspaces, the subspaces do not overlap with each other; according to the two-dimensional image of the three-dimensional space, obtaining a two-dimensional image of at least one subspace in the three-dimensional space; performing three-dimensional reconstruction on the panoramic image of at least one subspace, and obtaining projection images corresponding to the panoramic images of at least one subspace; according to the projection image of at least one subspace and the two-dimensional image of at least one subspace, obtaining a matching relationship between the panoramic image of at least one subspace and the two-dimensional image of at least one subspace, and establishing an association relationship between the panoramic image of the subspace and the two-dimensional image of the subspace that generate the matching relationship.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image matching method, device and computer-readable storage medium. Background Art

[0002] Virtual roaming is an important branch of virtual reality (VR) technology, involving a variety of industries such as architecture, tourism, games, aerospace, and medicine. Through the combination of virtual scene creation technology and virtual roaming technology, users can autonomously roam in three-dimensional scenes such as buildings, cities, or game scenes, thereby gaining an intuitive understanding of the above roaming scenes.

[0003] In the interactive process of virtual roaming, it is necessary to establish a matching relationship for subspaces between the 3D scene that the user is roaming and the 2D image that describes the structure of the 3D scene, so as to guide the user to autonomously control the subspace area that the user wants to roam. However, in general, the subspace matching of the 3D scene and the 2D image can be performed manually to establish the association relationship between the 3D scene and each subspace in the 2D image. For example, the mutual matching and mutual jump between the subspace areas of the 3D scene and the 2D image can be achieved by manually establishing navigation points.

[0004] It can be seen that manually matching the subspaces of the three-dimensional scene and the two-dimensional image is both time-consuming and labor-intensive. Therefore, it is necessary to construct a method and device that can automatically match the subspace regions of the three-dimensional scene and the two-dimensional image. Summary of the invention

[0005] To solve the above technical problems, according to one aspect of the present invention, an image matching method is provided, comprising: obtaining a panoramic image of at least one subspace in a three-dimensional space, and a two-dimensional image of the three-dimensional space; obtaining a two-dimensional image of at least one subspace in the three-dimensional space according to the two-dimensional image of the three-dimensional space, wherein the three-dimensional space is composed of at least one subspace, and when the three-dimensional space is composed of multiple subspaces, the subspaces do not overlap with each other; performing three-dimensional reconstruction on the panoramic image of the at least one subspace, and obtaining projection images corresponding to the panoramic images of the at least one subspace; obtaining a matching relationship between the panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace according to the projection image of the at least one subspace and the two-dimensional image of the at least one subspace, and establishing an association relationship between the panoramic image of the subspace and the two-dimensional image of the subspace that generate the matching relationship.

[0006] According to another aspect of the present invention, there is provided an image matching device, comprising: an image acquisition unit, configured to acquire a panoramic image of at least one subspace in a three-dimensional space, and a two-dimensional image of the three-dimensional space, wherein the three-dimensional space is composed of at least one subspace, and when the three-dimensional space is composed of multiple subspaces, the subspaces do not overlap with each other; a subspace acquisition unit, configured to acquire a two-dimensional image of at least one subspace in the three-dimensional space based on the two-dimensional image of the three-dimensional space; a reconstruction unit, configured to perform three-dimensional reconstruction on the panoramic image of the at least one subspace, and acquire projection images corresponding to the panoramic images of the at least one subspace; a matching unit, configured to acquire a matching relationship between the panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace based on the projection image of the at least one subspace and the two-dimensional image of the at least one subspace, and to establish an association relationship between the panoramic image of the subspace and the two-dimensional image of the subspace that generate the matching relationship.

[0007] According to another aspect of the present invention, there is provided an image matching device, comprising: a processor; and a memory, in which computer program instructions are stored, wherein when the computer program instructions are executed by the processor, the processor is caused to perform the following steps: obtaining a panoramic image of at least one subspace in a three-dimensional space, and a two-dimensional image of the three-dimensional space, wherein the three-dimensional space is composed of at least one subspace, and when the three-dimensional space is composed of multiple subspaces, the subspaces do not overlap with each other; obtaining a two-dimensional image of at least one subspace in the three-dimensional space based on the two-dimensional image of the three-dimensional space; performing three-dimensional reconstruction on the panoramic image of the at least one subspace, and obtaining projection images corresponding to the panoramic images of the at least one subspace; obtaining a matching relationship between the panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace based on the projection image of the at least one subspace and the two-dimensional image of the at least one subspace, and establishing an association relationship between the panoramic image of the subspace and the two-dimensional image of the subspace that generate the matching relationship.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the following steps: obtaining a panoramic image of at least one subspace in a three-dimensional space, and a two-dimensional image of the three-dimensional space, wherein the three-dimensional space is composed of at least one subspace, and when the three-dimensional space is composed of multiple subspaces, the subspaces do not overlap with each other; obtaining a two-dimensional image of at least one subspace in the three-dimensional space based on the two-dimensional image of the three-dimensional space; performing three-dimensional reconstruction on the panoramic image of the at least one subspace, and obtaining projection images corresponding to the panoramic images of the at least one subspace; obtaining a matching relationship between the panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace based on the projection image of the at least one subspace and the two-dimensional image of the at least one subspace, and establishing an association relationship between the panoramic image of the subspace and the two-dimensional image of the subspace that generate the matching relationship.

[0009] According to the above-mentioned image matching method, device and computer-readable storage medium of the present invention, it is possible to match the projection image obtained by projecting the panoramic image of the subspace of the three-dimensional space with the two-dimensional image of the subspace, thereby obtaining the matching relationship between the panoramic image of the subspace and the two-dimensional image of the subspace, and constructing an association relationship accordingly. This method, device and computer-readable storage medium for automatic image matching can save costs and effectively improve the efficiency of image matching while ensuring the accuracy of image matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other objects, features and advantages of the present invention will become more apparent by describing in detail the embodiments of the present invention in conjunction with the accompanying drawings.

[0011] Figure 1 A flowchart of an image matching method according to an embodiment of the present invention is shown;

[0012] Figure 2 Showing panoramic images corresponding to multiple rooms in three-dimensional space;

[0013] Figure 3 A two-dimensional floor plan showing a three-dimensional space;

[0014] Figure 4 A schematic diagram showing classification of panoramic images of rooms with different first label information is shown;

[0015] Figure 5 The results of different semantic type recognition for each pixel in the two-dimensional floor plan are shown;

[0016] Figure 6The two-dimensional images of each room obtained by segmenting the two-dimensional floor plan and the respective room types corresponding to the two-dimensional images of each room are shown;

[0017] Figure 7 A schematic diagram showing a panoramic image of a bedroom that was not successfully matched;

[0018] Figure 8 Show Figure 7 Schematic diagram of three-dimensional reconstruction of panoramic images;

[0019] Fig. 9 Show Figure 7 Schematic diagram of target detection and positioning in panoramic images;

[0020] Fig.10 Show Figure 7 A schematic diagram of a projected image obtained by projecting a panoramic image in;

[0021] Fig.11 A block diagram of an image matching device according to an embodiment of the present invention is shown;

[0022] Fig.12 A block diagram of an image matching apparatus according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0023] The image matching method, device and computer readable storage medium according to the embodiments of the present invention will be described below with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same elements from beginning to end. It should be understood that the embodiments described herein are merely illustrative and should not be interpreted as limiting the scope of the present invention.

[0024] The following will refer to Figure 1 An image matching method according to an embodiment of the present invention is described. Figure 1 A flow chart of the image matching method 100 is shown.

[0025] like Figure 1 As shown, in step S101, a panoramic image of at least one subspace in a three-dimensional space and a two-dimensional image of the three-dimensional space are obtained, wherein the three-dimensional space is composed of at least one subspace, and when the three-dimensional space is composed of multiple subspaces, the subspaces do not overlap with each other.

[0026] In this step, various methods can be used to obtain a panoramic image of at least one subspace in the three-dimensional space. For example, a panoramic camera (such as a fisheye camera, a binocular camera, etc.) can be used to obtain a panoramic image (such as an equirectangular projection image) of each subspace in the at least one subspace. For another example, a normal camera can also be used to obtain multiple perspective images of each subspace in the at least one subspace, and these multiple perspective images are spliced ​​to obtain panoramic images of each subspace respectively. Among them, the three-dimensional space can be composed of one or more subspaces, and these subspaces can be spaces obtained by dividing the three-dimensional space in various ways, and when the three-dimensional space is composed of multiple subspaces, each subspace can be non-overlapping and non-nested. For example, when the three-dimensional space is an indoor space of a building, the indoor space of the building can be divided into multiple independent rooms as subspaces by objects such as walls, doors or windows in the indoor space of the building.

[0027] In addition, in this step, a two-dimensional image of the three-dimensional space may also be obtained. The two-dimensional image of the three-dimensional space is used to describe the plan layout of each subspace of the three-dimensional space. For example, when the three-dimensional space is an indoor space of a building, the two-dimensional image may describe the plan layout of each room in the building, such as a two-dimensional floor plan.

[0028] In step S102, a two-dimensional image of at least one subspace in the three-dimensional space is acquired according to the two-dimensional image of the three-dimensional space.

[0029] In this step, the two-dimensional image of the three-dimensional space can be segmented to obtain the two-dimensional images of each subspace in the three-dimensional space. In one example, a pre-trained subspace two-dimensional image recognition model can be used to perform semantic recognition on each pixel of the two-dimensional image of the three-dimensional space to obtain the semantic information of each pixel. Subsequently, the two-dimensional image of the three-dimensional space can be segmented according to the semantic information of each pixel in the above two-dimensional image to obtain the two-dimensional images of each subspace. Optionally, the subspace two-dimensional image recognition model for identifying the semantic information of the pixel can be a deep neural network model. For example, when the two-dimensional image is the aforementioned two-dimensional floor plan, the DeepFloorplan deep neural network model can be used to perform semantic recognition on the two-dimensional floor plan, such as identifying the semantic information of each pixel of the wall, window, door, room type (such as bedroom, kitchen, bathroom) of the two-dimensional floor plan, so as to segment the two-dimensional floor plan according to these semantic information, thereby obtaining the two-dimensional floor plan of each subspace (i.e., each room). During the segmentation process, optionally, based on the same semantic information, the pixel points (such as the pixel points whose semantic information is a certain type of room) can be expanded to the surrounding until the pixel points with different semantic information (such as the pixel points whose semantic information is a wall, door or window) are encountered, and the expanded image is used as the two-dimensional floor plan of the subspace. In this example, the DeepFloorplan deep neural network can be pre-trained using a set of two-dimensional floor plan images and floor plan semantic labels.

[0030] In step S103, three-dimensional reconstruction is performed on the panoramic image of the at least one subspace, and projection images respectively corresponding to the panoramic image of the at least one subspace are acquired.

[0031] In this step, the spatial features of at least one object forming the subspace can be obtained according to the panoramic image of the subspace, and three-dimensional reconstruction can be performed according to the spatial features of the at least one object obtained. Optionally, the spatial features of at least one object forming the subspace can include the geometric shape of the object, the spatial position coordinates, the relative geometric relationship between the objects, etc. After obtaining the spatial features of the at least one object, the panoramic image of the subspace can be three-dimensionally reconstructed. Optionally, the model used for three-dimensional reconstruction can be a deep neural network model. In one example, when the three-dimensional space is an indoor space of a building, the subspace of the three-dimensional space can be a room, and the at least one object forming the subspace can be a wall, a floor, a ceiling, etc. forming the room, and its spatial features can include the spatial position coordinates of the wall, the floor, and the ceiling forming the room; the vertical or parallel geometric relationship between the wall-wall, the wall-floor, and the wall-ceiling, and the edge position of the junction, etc. After obtaining the above spatial features, the room can be three-dimensionally reconstructed to obtain a three-dimensional reconstruction model of the room. In this example, the HorizonNet deep neural network model can be used for three-dimensional reconstruction, and optionally, the deep neural network model can be trained using a panoramic picture set of the room and the room layout label.

[0032] Subsequently, after the three-dimensional reconstruction, the target in the panoramic image of the subspace can be detected and located; according to the three-dimensional reconstruction result of the panoramic image of the subspace, and the detection and positioning results of the target in the panoramic image, the projection image of the subspace on the plane where the two-dimensional image of the three-dimensional space is located is obtained. Optionally, the detection and positioning results of the target in the panoramic image of the subspace may include the geometric shape, spatial position coordinates, etc. of the target. After obtaining the above-mentioned three-dimensional reconstruction results and the detection and positioning results of the target, the three-dimensional reconstruction model of the panoramic image of the subspace can be projected, and its projection plane can be the plane where the two-dimensional image of the three-dimensional space is located, or a plane parallel to this plane. In addition, the shape, position and other information of the detected target in the projection image can also be marked in the projection image. Optionally, the model used for target detection can be a deep neural network model. In one example, when the three-dimensional space is the indoor space of a building, the subspace of the three-dimensional space can be a room, and the target in the panoramic image of the subspace can be objects such as doors, windows, wardrobes, etc. in the room, so that the detection and positioning results such as the geometric shapes and spatial position coordinates of these targets can be obtained based on the detection of objects such as doors, windows, wardrobes, etc. in the room. After obtaining the above results, the three-dimensional reconstructed model of the panoramic image of the room can be projected on the ground, for example, to obtain a projected image of the room, and the positions of objects such as doors, windows, wardrobes, etc. in the room are marked in the projected image. In this example, a YOLOv3 deep neural network model can be used for target detection, and optionally, this deep neural network model can be trained using a panoramic picture set of the room and target detection labels.

[0033] In step S104, based on the projection image of the at least one subspace and the two-dimensional image of the at least one subspace, a matching relationship between the panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace is obtained, and an association relationship is established between the panoramic image of the subspace and the two-dimensional image of the subspace that generate the matching relationship.

[0034] Optionally, the matching relationship between the projection image of the at least one subspace and the two-dimensional image of the at least one subspace can be obtained according to the layout of the objects and / or targets in the projection image of the at least one subspace and the two-dimensional image of the at least one subspace. Subsequently, according to the matching relationship between the projection image of the at least one subspace and the two-dimensional image of the at least one subspace, the matching relationship between the corresponding panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace is obtained. For example, when the subspace is a room, the matching relationship between the panoramic image and the two-dimensional image of the subspace can be determined according to factors such as the shape, quantity, position, geometric relationship, etc. of the objects forming the room such as the wall, the ground, the ceiling, etc. in the room. For another example, the matching relationship between the panoramic image and the two-dimensional image of the subspace can also be determined according to the shape, quantity, position, geometric relationship, etc. of the objects such as the door, window, wardrobe, etc. in the room.

[0035] Optionally, the projection image of the at least one subspace and the two-dimensional image of the at least one subspace may be shape matched, and a matching relationship between the projection image of the at least one subspace and the two-dimensional image of the at least one subspace may be established based on the matching result. Subsequently, based on the matching relationship between the projection image of the at least one subspace and the two-dimensional image of the at least one subspace, a matching relationship between the corresponding panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace may be obtained.

[0036] After obtaining the matching relationship between the panoramic image of the subspace and the two-dimensional image of the subspace, a link can be constructed between the panoramic image of the subspace and the two-dimensional image of the subspace to establish an association relationship between the panoramic image of the subspace and the two-dimensional image of the subspace, thereby enabling the user to jump display between the panoramic image of the subspace and the corresponding two-dimensional image of the subspace through various methods such as clicking and touching.

[0037] In this embodiment, further, after obtaining a panoramic image of at least one subspace in the three-dimensional space and a two-dimensional image of the three-dimensional space, the method may also include: obtaining first label information corresponding to the panoramic image of the at least one subspace; obtaining second label information corresponding to the two-dimensional image of the at least one subspace in the three-dimensional space.

[0038] Optionally, the panoramic images of at least one subspace in the three-dimensional space can be classified, and the first label information corresponding to the panoramic images of at least one subspace can be obtained according to the classification result. For example, the panoramic images of at least one subspace can be classified using a deep neural network model, where the deep neural network model can be a VGG16 deep network model. When the subspace is a room, the panoramic images of each room can be classified using panoramic images of different room types and room type labels, and the classification results can be obtained.

[0039] In addition, optionally, the two-dimensional images of at least one subspace in the three-dimensional space can be classified, and the second label information corresponding to the two-dimensional images of at least one subspace in the three-dimensional space can be obtained according to the classification result. For example, the second label information corresponding to the two-dimensional image of at least one subspace can be obtained in combination with the semantic recognition result of each pixel point of the two-dimensional image in the three-dimensional space.

[0040] Subsequently, after obtaining the above-mentioned first tag information and second tag information, the matching relationship between the panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace can be obtained according to the first tag information and the second tag information. For example, the panoramic image and the two-dimensional image of a certain subspace can be judged whether they match by comparing the first tag information and the second tag information. Optionally, when the first tag information and the second tag information are the same, it can be considered that the panoramic image of a certain subspace and the two-dimensional image of the subspace match (hereinafter referred to as a rough match). Optionally, when the first tag information and the second tag information are similar, or belong to the same category, it can also be considered that the panoramic image of the subspace and the two-dimensional image of the subspace match. The above-mentioned comparison method for the first tag information and the second tag information is only an example. In actual applications, any comparison method can be adopted according to the specific scenario, and there is no limitation here.

[0041] After roughly matching the panoramic image of at least one subspace and the two-dimensional image of the at least one subspace, it can be determined whether to execute steps S103-S104 according to the rough matching result. For example, if the rough matching result indicates that the panoramic image of a specific subspace is not able to generate a matching relationship with the two-dimensional image of at least one subspace, a three-dimensional reconstruction is performed on the panoramic image of the specific subspace that fails to generate a matching relationship, and a corresponding projection image is obtained. Subsequently, step S104 can be executed for the projection image of the panoramic image of the subspace that fails to generate a matching relationship and the two-dimensional image of at least one subspace that fails to generate a matching relationship with the panoramic image of at least one subspace. In one example, when the first label information and / or the second label information corresponding to the panoramic image of a specific subspace or its two-dimensional image is detected incorrectly, steps S103-S104 may be executed for the panoramic image of the subspace and the two-dimensional image of the subspace that cannot be matched due to incorrect label information. In another example, although a matching relationship can be established between the first label information and the second label information, when at least one of the first label information and the second label information corresponds to multiple panoramic images / two-dimensional images of the subspace, it is also possible that a matching relationship cannot be established because there is no one-to-one correspondence between the panoramic image of the subspace and the two-dimensional image of the subspace, thereby continuing to execute steps S103-S104.

[0042] By first comparing the matching method of the first label information and the second label information, the resource occupation of the subsequent three-dimensional reconstruction and projection process in the embodiment of the present invention can be reduced, the image matching time can be saved, and the efficiency of image matching can be improved. In addition, in the operation of performing the subsequent step S104 when the panoramic image of the subspace and the two-dimensional image of the subspace cannot be one-to-one corresponding, the matching process of the projection image of the subspace and the two-dimensional image of the subspace can also first group the projection image of the subspace and the two-dimensional image of the subspace that correspond to the first label information and the second label information that are the same, similar or belong to the same category and can establish a matching relationship, and in the same group, the matching relationship between the projection image and the two-dimensional image of the subspace is established for the subspace that does not generate a matching relationship in the rough matching to further reduce the amount of calculation, thereby further improving the image matching efficiency and improving the image matching accuracy.

[0043] The above description of the image matching method of the embodiment of the present invention is only an example. In the specific implementation process of the present invention, the various steps in the above image matching method can be arranged in any manner, or two or more of the steps can be performed simultaneously, which is not limited here. In addition, the above label information acquisition and matching method for the panoramic image of the subspace and the two-dimensional image of the subspace in the embodiment of the present invention can also be performed independently, or after the method of image matching through three-dimensional reconstruction and projection is completed. For example, the image matching method of the embodiment of the present invention can also only include: acquiring a panoramic image of at least one subspace in a three-dimensional space, and a two-dimensional image of the three-dimensional space, wherein the three-dimensional space is composed of at least one subspace, and when the three-dimensional space is composed of multiple subspaces, the subspaces do not overlap with each other; acquiring the first label information corresponding to the panoramic image of the at least one subspace; acquiring the second label information corresponding to the two-dimensional image of the at least one subspace in the three-dimensional space; according to the first label information and the second label information, acquiring the matching relationship between the panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace. Of course, all the deformations, modifications and optimizations of the above method steps for the embodiment of the present invention are included in the scope of the embodiment of the present invention.

[0044] According to the above-mentioned image matching method of the embodiment of the present invention, the projection image obtained by projecting the panoramic image of the subspace of the three-dimensional space can be matched with the two-dimensional image of the subspace, thereby obtaining the matching relationship between the panoramic image of the subspace and the two-dimensional image of the subspace, and building an association relationship accordingly. This automatic image matching method can save costs and effectively improve the image matching efficiency while ensuring the image matching accuracy.

[0045] An example of an image matching method according to an embodiment of the present invention is shown below. In this example, the three-dimensional space is the indoor space of a building, the subspace of the three-dimensional space is each room of the indoor space of the building, and the two-dimensional image of the three-dimensional space can be a two-dimensional floor plan of each room of the building.

[0046] In this example, first, a panoramic image of at least one room in a three-dimensional space may be acquired. Figure 2 The panoramic images corresponding to multiple rooms in the three-dimensional space are shown. These panoramic images can be obtained by taking pictures of each room by a panoramic camera. Figure 2 As shown, the acquired panoramic images of each room may include panoramic images corresponding to different room types such as a living room, a kitchen, a bathroom, and multiple bedrooms. Subsequently, a two-dimensional floor plan showing the plane layout of this three-dimensional space may also be acquired. Figure 3 A two-dimensional floor plan of the three-dimensional space is shown, which shows the geometric shape, arrangement and position relationship of each room in the three-dimensional space.

[0047] After obtaining the panoramic images of each room in the three-dimensional space and the two-dimensional floor plan of the three-dimensional space, the first label information corresponding to the panoramic images of each room and the second label information corresponding to the two-dimensional floor plan of each room can be obtained respectively. Specifically, after obtaining the panoramic images of each room, the VGG16 deep network model can be used to classify these panoramic images to obtain panoramic images corresponding to different room types. The room type here can be the first label information corresponding to each panoramic image. Figure 4 A schematic diagram of panoramic image classification of rooms with different first label information is shown.

[0048] In addition, in getting Figure 3 After the two-dimensional floor plan shown in the figure, the pre-trained DeepFloorplan deep neural network model can be used to perform semantic recognition on the two-dimensional floor plan and obtain the semantic information of each pixel point of the two-dimensional floor plan. Figure 5 The following figure shows the recognition results of different semantic types for each pixel in the two-dimensional floor plan. Figure 5 As shown, the semantic information of rooms of different types (represented by different grayscales), walls, doors, and windows can be identified in the two-dimensional floor plan, and the corresponding pixels can be classified according to the semantic information to obtain a two-dimensional image of at least one room. On this basis, the room type corresponding to the two-dimensional image of each room can be further obtained. Figure 6 The two-dimensional images of the rooms obtained by segmenting the two-dimensional floor plan are shown, and the two-dimensional image of each room corresponds to the respective room type, that is, the second label information, such as living room, bedroom, balcony, bathroom, etc.

[0049] In getting Figure 4 The panoramic images of each room and the first tag information corresponding to each room, and Figure 6 After the two-dimensional floor plans of each room and the corresponding second label information are shown, the panoramic image of each room and the two-dimensional floor plan of each room can be matched according to their respective first label information and second label information. Optionally, when the first label information and the second label information are the same, that is, when the panoramic image of a room and the room type corresponding to the two-dimensional floor plan of a room are the same, it can be considered that there is a matching relationship between the panoramic image of the room and the two-dimensional floor plan of the room. In the current step, only the panoramic images and two-dimensional floor plans whose corresponding first label information and second label information are the same and there is only one can be matched. For example, a matching relationship can be established between the panoramic image of the living room and the two-dimensional floor plan of the living room corresponding to the two respectively when and only when the first label information and the second label information are for the living room and there is only one of each. Similarly, at this time, matching can also be performed for bathrooms, kitchens, balconies, etc., and as Figure 6 The second tag information shown is a bedroom and the situation where there are three bedrooms may not be matched at this stage.

[0050] After the above-mentioned step of matching the first tag information and the second tag information is completed, when the matching relationship between the panoramic image of at least one room and the two-dimensional image of at least one room cannot be obtained according to the first tag information and the second tag information, three-dimensional reconstruction can be performed based on the panoramic image of the room that cannot generate a matching relationship, and the corresponding projection image can be obtained. For example, three-dimensional reconstruction and target detection and positioning can be performed for the panoramic image of the room type, that is, the first tag information is a bedroom. In one example, Figure 7 A schematic diagram showing a panoramic image of a bedroom for which matching is not successful. Figure 8 Shows the Figure 7 Schematic diagram of 3D reconstruction of a panoramic image in , where the edge positions of the wall-wall, wall-floor, and wall-ceiling junctions in the bedroom can be obtained by using the HorizonNet deep neural network model for 3D reconstruction, as shown by the white lines. Of course, other geometric spatial features of these objects forming the bedroom can also be obtained during the 3D reconstruction process. In addition, Fig. 9 Shown for Figure 7 Schematic diagram of target detection and positioning in a panoramic image. The YOLOv3 deep neural network model is used for Figure 7 Door and window detection is performed on the panoramic image of the bedroom in Figure 1, thereby obtaining the geometric shapes and positions of the doors and windows shown in the black frames.

[0051] After completing the above three-dimensional reconstruction and target detection and positioning steps, the panoramic image of the bedroom can be projected on the ground according to the obtained results to obtain the following: Fig.10 The projected image shown. Fig.10 As can be seen from the figure, it shows the geometric shape and size of the room corresponding to the panoramic image of the bedroom, and further shows the specific positions of the door and windows in the bedroom with thick black lines.

[0052] Finally, according to the projection images of each room obtained above and the two-dimensional floor plans of each room, the matching relationship between the projection images of each room and the two-dimensional floor plans of each room can be constructed by comparing the layout of objects and targets and / or by shape recognition methods, and the matching relationship between the panoramic images of each room and the two-dimensional floor plans of each room can be further constructed. Optionally, at this time, the panoramic images of each room and the first label information and the second label information corresponding to the two-dimensional floor plans of each room can be combined to classify these images, and a matching relationship can be constructed on the basis of the classification. For example, the panoramic images of the bedrooms and the two-dimensional floor plans of the bedrooms, which have both the first label information and the second label information, can be classified into one category, and the panoramic images of the bedrooms and the two-dimensional floor plans of the bedrooms can be matched in this category. In addition, a link can be constructed between the panoramic images of each room with a matching relationship and the two-dimensional images of each room, so that the user can realize the mutual jump display between the panoramic images of each room and the corresponding two-dimensional images of each room by clicking, touching, and other methods.

[0053] Below, refer to Fig.11 The image matching device according to an embodiment of the present invention is described. Fig.11 FIG. 1 is a block diagram of an image matching device 1100 according to an embodiment of the present invention. Fig.11 As shown, the image matching device 1100 includes an image acquisition unit 1110, a subspace acquisition unit 1120, a reconstruction unit 1130, and a matching unit 1140. In addition to these units, the image matching device 1100 may also include other components. However, since these components are irrelevant to the content of the embodiment of the present invention, their illustration and description are omitted here. In addition, since the specific details of the following operations performed by the image matching device 1100 according to the embodiment of the present invention are the same as those described above with reference to Figure 1-Figure 10 The details described are the same, so repeated description of the same details is omitted here to avoid repetition.

[0054] Fig.11 The image acquisition unit 1110 of the image matching device 1100 acquires a panoramic image of at least one subspace in a three-dimensional space, and a two-dimensional image of the three-dimensional space, wherein the three-dimensional space is composed of at least one subspace, and when the three-dimensional space is composed of multiple subspaces, the subspaces do not overlap with each other.

[0055] The image acquisition unit 1110 can use various methods to acquire a panoramic image of at least one subspace in the three-dimensional space. For example, a panoramic camera (such as a fisheye camera, a binocular camera, etc.) can be used to acquire a panoramic image (such as an equirectangular projection image) of each subspace in the at least one subspace. For another example, a common camera can also be used to acquire multiple perspective images of each subspace in the at least one subspace, and these multiple perspective images are spliced ​​to respectively acquire a panoramic image of each subspace. Among them, the three-dimensional space can be composed of one or more subspaces, and these subspaces can be spaces obtained by dividing the three-dimensional space in various ways, and when the three-dimensional space is composed of multiple subspaces, each subspace can be non-overlapping and non-nested. For example, when the three-dimensional space is an indoor space of a building, the indoor space of the building can be divided into multiple independent rooms as subspaces by objects such as walls, doors or windows in the indoor space of the building.

[0056] In addition, the image acquisition unit 1110 can also acquire a two-dimensional image of the three-dimensional space. The two-dimensional image of the three-dimensional space is used to describe the plan layout of each subspace of the three-dimensional space. For example, when the three-dimensional space is an indoor space of a building, the two-dimensional image can describe the plan layout of each room in the building, such as a two-dimensional floor plan.

[0057] The subspace acquisition unit 1120 acquires a two-dimensional image of at least one subspace in the three-dimensional space according to the two-dimensional image of the three-dimensional space.

[0058] The subspace acquisition unit 1120 can segment the two-dimensional image of the three-dimensional space to obtain the two-dimensional images of each subspace in the three-dimensional space. In one example, a pre-trained subspace two-dimensional image recognition model can be used to perform semantic recognition on each pixel of the two-dimensional image of the three-dimensional space to obtain the semantic information of each pixel. Subsequently, the two-dimensional image of the three-dimensional space can be segmented according to the semantic information of each pixel in the above two-dimensional image to obtain the two-dimensional images of each subspace. Optionally, the subspace two-dimensional image recognition model for identifying the semantic information of the pixel point can be a deep neural network model. For example, when the two-dimensional image is the aforementioned two-dimensional floor plan, the DeepFloorplan deep neural network model can be used to perform semantic recognition on the two-dimensional floor plan, such as the semantic information of each pixel of the wall, window, door, room type (such as bedroom, kitchen, bathroom) of the two-dimensional floor plan, so as to segment the two-dimensional floor plan according to these semantic information, thereby obtaining the two-dimensional floor plan of each subspace (i.e., each room). During the segmentation process, optionally, based on the same semantic information, the pixel points (such as the pixel points whose semantic information is a certain type of room) can be expanded to the surrounding until the pixel points with different semantic information (such as the pixel points whose semantic information is a wall, door or window) are encountered, and the expanded image is used as the two-dimensional floor plan of the subspace. In this example, the DeepFloorplan deep neural network can be pre-trained using a set of two-dimensional floor plan images and floor plan semantic labels.

[0059] The reconstruction unit 1130 performs three-dimensional reconstruction on the panoramic image of the at least one subspace, and obtains projection images respectively corresponding to the panoramic image of the at least one subspace.

[0060] The reconstruction unit 1130 can obtain the spatial features of at least one object forming the subspace according to the panoramic image of the subspace, and perform three-dimensional reconstruction according to the spatial features of the at least one object obtained. Optionally, the spatial features of at least one object forming the subspace may include the geometric shape of the object, the spatial position coordinates, the relative geometric relationship between the objects, etc. After obtaining the spatial features of the at least one object, the panoramic image of the subspace can be three-dimensionally reconstructed. Optionally, the model used for three-dimensional reconstruction can be a deep neural network model. In one example, when the three-dimensional space is an indoor space of a building, the subspace of the three-dimensional space can be a room, and the at least one object forming the subspace can be a wall, a floor, a ceiling, etc. forming the room, and its spatial features can include the spatial position coordinates of the wall, the floor, and the ceiling forming the room; the vertical or parallel geometric relationship between the wall-wall, the wall-floor, and the wall-ceiling, and the edge position of the junction, etc. After obtaining the above spatial features, the room can be three-dimensionally reconstructed to obtain a three-dimensional reconstruction model of the room. In this example, the HorizonNet deep neural network model can be used for three-dimensional reconstruction, and optionally, the deep neural network model can be trained using a panoramic picture set of the room and a room layout label.

[0061] Subsequently, after the three-dimensional reconstruction, the target in the panoramic image of the subspace can be detected and located; according to the three-dimensional reconstruction result of the panoramic image of the subspace, and the detection and positioning results of the target in the panoramic image, the projection image of the subspace on the plane where the two-dimensional image of the three-dimensional space is located is obtained. Optionally, the detection and positioning results of the target in the panoramic image of the subspace may include the geometric shape, spatial position coordinates, etc. of the target. After obtaining the above-mentioned three-dimensional reconstruction results and the detection and positioning results of the target, the three-dimensional reconstruction model of the panoramic image of the subspace can be projected, and its projection plane can be the plane where the two-dimensional image of the three-dimensional space is located, or a plane parallel to this plane. In addition, the shape, position and other information of the detected target in the projection image can also be marked in the projection image. Optionally, the model used for target detection can be a deep neural network model. In one example, when the three-dimensional space is the indoor space of a building, the subspace of the three-dimensional space can be a room, and the target in the panoramic image of the subspace can be objects such as doors, windows, wardrobes, etc. in the room, so that the detection and positioning results such as the geometric shapes and spatial position coordinates of these targets can be obtained based on the detection of objects such as doors, windows, wardrobes, etc. in the room. After obtaining the above results, the three-dimensional reconstructed model of the panoramic image of the room can be projected on the ground, for example, to obtain a projected image of the room, and the positions of objects such as doors, windows, wardrobes, etc. in the room are marked in the projected image. In this example, a YOLOv3 deep neural network model can be used for target detection, and optionally, this deep neural network model can be trained using a panoramic picture set of the room and target detection labels.

[0062] The matching unit 1140 obtains a matching relationship between the panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace based on the projection image of the at least one subspace and the two-dimensional image of the at least one subspace, and establishes an association relationship between the panoramic image of the subspace and the two-dimensional image of the subspace that generate the matching relationship.

[0063] Optionally, the matching unit 1140 may obtain the matching relationship between the projection image of the at least one subspace and the two-dimensional image of the at least one subspace according to the layout of the objects and / or targets in the projection image of the at least one subspace and the two-dimensional image of the at least one subspace. Subsequently, according to the matching relationship between the projection image of the at least one subspace and the two-dimensional image of the at least one subspace, the matching relationship between the corresponding panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace is obtained. For example, when the subspace is a room, the matching relationship between the panoramic image and the two-dimensional image of the subspace may be determined according to factors such as the shape, quantity, position, geometric relationship, etc. of the objects forming the room such as the wall, the ground, the ceiling, etc. in the room. For another example, the matching relationship between the panoramic image and the two-dimensional image of the subspace may also be determined according to the shape, quantity, position, geometric relationship, etc. of the objects such as the door, the window, the wardrobe, etc. in the room.

[0064] Optionally, the matching unit 1140 may also perform shape matching on the projection image of the at least one subspace and the two-dimensional image of the at least one subspace, and establish a matching relationship between the projection image of the at least one subspace and the two-dimensional image of the at least one subspace according to the matching result. Subsequently, according to the matching relationship between the projection image of the at least one subspace and the two-dimensional image of the at least one subspace, a matching relationship between the corresponding panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace is obtained.

[0065] After obtaining the matching relationship between the panoramic image of the subspace and the two-dimensional image of the subspace, the matching unit 1140 can construct a link between the panoramic image of the subspace and the two-dimensional image of the subspace to establish an association relationship between the panoramic image of the subspace and the two-dimensional image of the subspace, thereby enabling the user to jump display between the panoramic image of the subspace and the corresponding two-dimensional image of the subspace through various methods such as clicking and touching.

[0066] In this embodiment, further, after obtaining a panoramic image of at least one subspace in the three-dimensional space and a two-dimensional image of the three-dimensional space, the device may further include a coarse matching unit (not shown), configured to obtain first label information corresponding to the panoramic image of the at least one subspace; and obtain second label information corresponding to the two-dimensional image of the at least one subspace in the three-dimensional space.

[0067] Optionally, the rough matching unit may classify the panoramic images of at least one subspace in the three-dimensional space, and obtain first label information corresponding to the panoramic images of at least one subspace according to the classification result. For example, a deep neural network model may be used to classify the panoramic images of at least one subspace, and the deep neural network model here may be a VGG16 deep network model. When the subspace is a room, panoramic images of different room types and room type labels may be used to classify the panoramic images of each room, and obtain the classification results.

[0068] In addition, optionally, the rough matching unit may further classify the two-dimensional images of at least one subspace in the three-dimensional space, and obtain second label information corresponding to the two-dimensional images of at least one subspace in the three-dimensional space according to the classification result. For example, the second label information corresponding to the two-dimensional images of at least one subspace may be obtained in combination with the semantic recognition result of each pixel point of the two-dimensional image in the three-dimensional space.

[0069] Subsequently, after obtaining the above-mentioned first label information and second label information, the rough matching unit can obtain the matching relationship between the panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace according to the first label information and the second label information. For example, the panoramic image and the two-dimensional image of a certain subspace can be judged whether they match by comparing the first label information and the second label information. Optionally, when the first label information and the second label information are the same, the panoramic image of a certain subspace and the two-dimensional image of the subspace can be considered to match (hereinafter referred to as rough matching). Optionally, when the first label information and the second label information are similar, or belong to the same category, the panoramic image of the subspace and the two-dimensional image of the subspace can also be considered to match. The above-mentioned comparison method for the first label information and the second label information is only an example. In actual applications, any comparison method can be adopted according to the specific scenario, and there is no limitation here.

[0070] After the rough matching unit roughly matches the panoramic image of at least one subspace and the two-dimensional image of the at least one subspace, the rough matching unit can determine whether to enter the reconstruction unit and the matching unit for further matching according to the rough matching result. For example, if the rough matching result indicates that the panoramic image of the specific subspace cannot generate a matching relationship with the two-dimensional image of at least one subspace, the panoramic image of the specific subspace that cannot generate a matching relationship is three-dimensionally reconstructed, and the corresponding projection image is obtained. Subsequently, the projection image of the panoramic image of the subspace that cannot generate a matching relationship and the two-dimensional image of at least one subspace that cannot generate a matching relationship with the panoramic image of at least one subspace can be made to enter the reconstruction unit and the matching unit for further matching. In one example, when the first label information and / or the second label information corresponding to the panoramic image of the specific subspace or its two-dimensional image are detected incorrectly, the panoramic image of the subspace that cannot be matched due to the wrong label information and the two-dimensional image of the subspace may be made to enter the reconstruction unit and the matching unit for further matching. In another example, when a matching relationship can be established between the first label information and the second label information, but at least one of the first label information and the second label information corresponds to multiple panoramic images / two-dimensional images of the subspace, it is also possible that a matching relationship cannot be established because there is no one-to-one correspondence between the panoramic image of the subspace and the two-dimensional image of the subspace, thereby continuing to enter the reconstruction unit and the matching unit for further matching.

[0071] By first comparing the first label information and the second label information, the resource usage of the subsequent three-dimensional reconstruction and projection process in the embodiment of the present invention can be reduced, the image matching time can be saved, and the efficiency of image matching can be improved. In addition, when the panoramic image of the subspace and the two-dimensional image of the subspace cannot be one-to-one matched and enter the reconstruction unit and the matching unit for further matching, the matching process of the projection image of the subspace and the two-dimensional image of the subspace can also first group the projection image of the subspace and the two-dimensional image of the subspace that correspond to the first label information and the second label information that are the same, similar or belong to the same category and can establish a matching relationship, and in the same group, the matching relationship between the projection image and the two-dimensional image of the subspace is established for the subspace that does not generate a matching relationship in the rough matching to further reduce the amount of calculation, thereby further improving the image matching efficiency and improving the image matching accuracy.

[0072] The above description of the image matching device of the embodiment of the present invention is only an example. In the specific implementation process of the present invention, the various units in the above image matching device can be arranged in any manner, or two or more of the units can also be operated in parallel, which is not limited here. In addition, the above rough matching unit for label information acquisition and matching of the panoramic image of the subspace and the two-dimensional image of the subspace in the embodiment of the present invention can also be performed independently, or after entering the reconstruction unit and the matching unit to perform image matching through three-dimensional reconstruction and projection. For example, the image matching device of the embodiment of the present invention can also only include a rough matching unit, configured to obtain a panoramic image of at least one subspace in a three-dimensional space, and a two-dimensional image of the three-dimensional space, wherein the three-dimensional space is composed of at least one subspace, and when the three-dimensional space is composed of multiple subspaces, the subspaces do not overlap with each other; obtain the first label information corresponding to the panoramic image of the at least one subspace; obtain the second label information corresponding to the two-dimensional image of the at least one subspace in the three-dimensional space; according to the first label information and the second label information, obtain the matching relationship between the panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace. Of course, all the above-mentioned variations, modifications and optimizations of the device according to the embodiment of the present invention are included in the scope of the embodiment of the present invention.

[0073] According to the above-mentioned image matching device of the embodiment of the present invention, the projection image obtained by projecting the panoramic image of the subspace of the three-dimensional space can be matched with the two-dimensional image of the subspace, thereby obtaining the matching relationship between the panoramic image of the subspace and the two-dimensional image of the subspace, and building an association relationship accordingly. This automatic image matching device can save costs and effectively improve the image matching efficiency on the basis of ensuring the image matching accuracy.

[0074] Below, refer to Fig.12 The image matching device according to an embodiment of the present invention is described. Fig.12 FIG. 1 shows a block diagram of an image matching device 1200 according to an embodiment of the present invention. Fig.12 As shown, the device 1200 may be a computer or a server.

[0075] like Fig.12 As shown, the image matching device 1200 includes one or more processors 1210 and a memory 1220. Of course, in addition to this, the image matching device 1200 may also include an input device, an output device (not shown), etc. These components may be interconnected via a bus system and / or other forms of connection mechanisms. It should be noted that Fig.12 The components and structures of the image matching device 1200 shown are merely exemplary and non-limiting. The image matching device 1200 may also have other components and structures as required.

[0076] The processor 1210 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may utilize computer program instructions stored in the memory 1220 to perform desired functions, which may include: acquiring a panoramic image of at least one subspace in a three-dimensional space, and a two-dimensional image of the three-dimensional space, wherein the three-dimensional space is composed of at least one subspace, and when the three-dimensional space is composed of multiple subspaces, the subspaces do not overlap with each other; acquiring a two-dimensional image of at least one subspace in the three-dimensional space based on the two-dimensional image of the three-dimensional space; performing three-dimensional reconstruction on the panoramic image of the at least one subspace, and acquiring projection images corresponding to the panoramic images of the at least one subspace; acquiring a matching relationship between the panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace based on the projection image of the at least one subspace and the two-dimensional image of the at least one subspace, and establishing an association relationship between the panoramic image of the subspace and the two-dimensional image of the subspace that generate the matching relationship.

[0077] The memory 1220 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1210 may run the program instructions to implement the functions of the image matching device of the embodiment of the present invention described above and / or other desired functions, and / or may execute the image matching method according to the embodiment of the present invention. Various applications and various data may also be stored in the computer-readable storage medium.

[0078] The following describes a computer-readable storage medium according to an embodiment of the present invention, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the following steps: obtaining a panoramic image of at least one subspace in a three-dimensional space, and a two-dimensional image of the three-dimensional space; obtaining a two-dimensional image of at least one subspace in the three-dimensional space based on the two-dimensional image of the three-dimensional space, wherein the three-dimensional space is composed of at least one subspace, and when the three-dimensional space is composed of multiple subspaces, the subspaces do not overlap with each other; performing three-dimensional reconstruction on the panoramic image of the at least one subspace, and obtaining projection images corresponding to the panoramic images of the at least one subspace; obtaining a matching relationship between the panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace based on the projection image of the at least one subspace and the two-dimensional image of the at least one subspace, and establishing an association relationship between the panoramic image of the subspace and the two-dimensional image of the subspace that generate the matching relationship.

[0079] Of course, the above-mentioned specific embodiments are merely examples rather than limitations, and those skilled in the art can, based on the concept of the present invention, merge and combine some steps and devices from the various embodiments described separately above to achieve the effects of the present invention. Such merged and combined embodiments are also included in the present invention, and such merges and combinations are not described one by one herein.

[0080] Note that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details of the above invention are only for the purpose of illustration and facilitating understanding, not limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.

[0081] The block diagrams of the devices, apparatuses, equipment, and systems involved in the present invention are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The word "such as" used here refers to the phrase "such as but not limited to", and can be used interchangeably with it.

[0082] The step flow charts and the above method descriptions in the present invention are only illustrative examples and are not intended to require or imply that the steps of each embodiment must be performed in the order given. As will be appreciated by those skilled in the art, the order of the steps in the above embodiments can be performed in any order. Words such as "thereafter", "then", "next", etc. are not intended to limit the order of the steps; these words are only used to guide the reader through the description of these methods. In addition, any reference to an element in the singular, such as using "one", "one", or "the" is not to be construed as limiting the element to the singular.

[0083] In addition, the steps and devices in the various embodiments of this document are not limited to being implemented in a certain embodiment. In fact, based on the concept of the present invention, relevant partial steps and partial devices in the various embodiments of this document can be combined to conceive new embodiments, and these new embodiments are also included in the scope of the present invention.

[0084] Each operation of the method described above may be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs) or processors.

[0085] The various illustrated logic blocks, modules and circuits described may be implemented or performed using a general purpose processor, digital signal processor (DSP), ASIC, field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but as an alternative, the processor may be any commercially available processor, controller, microcontroller or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0086] The steps of the method or algorithm described in conjunction with the present invention can be directly embedded in hardware, in a software module executed by a processor, or in a combination of the two. A software module can exist in any form of tangible storage medium. Some examples of storage media that can be used include random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, etc. A storage medium can be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative manner, the storage medium can be integral with the processor. A software module can be a single instruction or many instructions, and can be distributed on several different code segments, between different programs, and across multiple storage media.

[0087] The method invented herein includes one or more actions for implementing the method described. Methods and / or actions can be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of actions is specified, the order and / or use of specific actions can be modified without departing from the scope of the claims.

[0088] The functions described can be implemented by hardware, software, firmware or any combination thereof. If implemented in software, the functions can be stored as one or more instructions on a tangible computer-readable medium. The storage medium can be any available tangible medium that can be accessed by a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device or any other tangible medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. As used herein, a disc includes a compact disc (CD), a laser disc, an optical disc, a digital versatile disc (DVD), a floppy disk and a blue disc.

[0089] Thus, a computer program product may perform the operations presented herein. For example, such a computer program product may be a computer-readable tangible medium having instructions tangibly stored (and / or encoded) thereon, which instructions may be executed by one or more processors to perform the operations described herein. The computer program product may include packaging materials.

[0090] Software or instructions may also be transmitted via a transmission medium. For example, the software may be transmitted from a website, server or other remote source using a transmission medium such as coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology such as infrared, radio or microwave.

[0091] In addition, the module and / or other appropriate means for performing the methods and techniques described herein can be downloaded and / or otherwise obtained by the user terminal and / or base station when appropriate. For example, such a device can be coupled to a server to facilitate the transmission of the means for performing the methods described herein. Alternatively, the various methods described herein can be provided via a storage component (e.g., RAM, ROM, a physical storage medium such as a CD or a floppy disk, etc.) so that the user terminal and / or base station can obtain the various methods when being coupled to the device or providing a storage component to the device. In addition, any other appropriate technology for providing the methods and techniques described herein to a device can be utilized.

[0092] Other examples and implementations are within the scope and spirit of the present invention and the appended claims. For example, due to the nature of software, the functions described above can be implemented using software executed by a processor, hardware, firmware, hard wiring, or any combination of these. Features that implement the functions can also be physically located in various locations, including being distributed so that parts of the functions are implemented at different physical locations. Moreover, as used herein, including as used in the claims, "or" used in the enumeration of items beginning with "at least one" indicates a separate enumeration, so that, for example, the enumeration of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). In addition, the wording "exemplary" does not mean that the example described is preferred or better than other examples.

[0093] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present invention is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.

[0094] The above description of the invented aspects is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features of the present invention.

[0095] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present invention to the form invented herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. An image matching method, comprising: Acquire a panoramic image of at least one subspace in a three-dimensional space, and a two-dimensional image of the three-dimensional space, wherein the three-dimensional space is composed of at least one subspace, and when the three-dimensional space is composed of a plurality of subspaces, the subspaces do not overlap with each other, wherein the two-dimensional image of the three-dimensional space describes the planar layout of the subspaces of the three-dimensional space; Acquire a two-dimensional image of at least one subspace in the three-dimensional space according to the two-dimensional image of the three-dimensional space; Perform three-dimensional reconstruction on the panoramic image of the at least one subspace, and project the three-dimensional reconstructed model of the panoramic image of the at least one subspace onto the plane where the two-dimensional image of the three-dimensional space is located or onto a parallel plane, and obtain projection images corresponding to the panoramic image of the at least one subspace; obtain a matching relationship between the panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace based on the projection image of the at least one subspace and the two-dimensional image of the at least one subspace, and establish an association relationship between the panoramic image of the subspace and the two-dimensional image of the subspace that generate the matching relationship.

2. The method of claim 1, wherein: The method further includes: acquiring first label information corresponding to the panoramic images of the at least one subspace; acquiring second label information corresponding to the two-dimensional images of the at least one subspace in the three-dimensional space; acquiring a matching relationship between the panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace according to the first label information and the second label information; and Performing three-dimensional reconstruction on the panoramic image of the at least one subspace and obtaining projection images respectively corresponding to the panoramic image of the at least one subspace includes: when a matching relationship between the panoramic image of at least one subspace and the two-dimensional image of at least one subspace cannot be obtained according to the first label information and the second label information, performing three-dimensional reconstruction based on the panoramic image of the subspace where the matching relationship cannot be generated, and obtaining the corresponding projection images.

3. The method of claim 1, wherein: Acquiring a panoramic image of at least one subspace in a three-dimensional space includes: Using a panoramic camera to acquire a panoramic image of each subspace in the at least one subspace; or A panoramic image of each subspace in the at least one subspace is acquired according to the multiple perspective images of each subspace.

4. The method of claim 1, wherein: According to the two-dimensional image of the three-dimensional space, acquiring a two-dimensional image of at least one subspace in the three-dimensional space comprises: Performing semantic recognition on each pixel of the two-dimensional image in the three-dimensional space; According to the result of semantic recognition, the two-dimensional image of the three-dimensional space is segmented to obtain a two-dimensional image of at least one subspace.

5. The method of claim 1, wherein: Performing three-dimensional reconstruction on the panoramic image of the at least one subspace includes: According to the panoramic image of the subspace, a spatial feature of at least one object forming the subspace is acquired, and three-dimensional reconstruction is performed according to the acquired spatial feature of the at least one object.

6. The method of claim 5, wherein: Acquiring projection images corresponding to the panoramic images of the at least one subspace includes: Detecting and locating targets in the panoramic image of the subspace; According to the three-dimensional reconstruction result of the panoramic image of the subspace and the detection and positioning results of the target in the panoramic image, a projection image of the subspace on the plane where the two-dimensional image of the three-dimensional space is located is obtained.

7. The method of claim 6, wherein: Acquiring a matching relationship between the panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace according to the projection image of the at least one subspace and the two-dimensional image of the at least one subspace includes: Acquire a matching relationship between the projection image of the at least one subspace and the two-dimensional image of the at least one subspace according to the projection image of the at least one subspace and the layout of the objects and / or targets in the two-dimensional image of the at least one subspace; According to the matching relationship between the projection image of the at least one subspace and the two-dimensional image of the at least one subspace, a corresponding matching relationship between the panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace is acquired.

8. The method of claim 1, wherein: Acquiring a matching relationship between the panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace according to the projection image of the at least one subspace and the two-dimensional image of the at least one subspace includes: Performing shape matching on the projection image of the at least one subspace and the two-dimensional image of the at least one subspace, and establishing a matching relationship between the projection image of the at least one subspace and the two-dimensional image of the at least one subspace according to the matching result; According to the matching relationship between the projection image of the at least one subspace and the two-dimensional image of the at least one subspace, a corresponding matching relationship between the panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace is acquired.

9. An image matching device, comprising: an image acquisition unit configured to acquire a panoramic image of at least one subspace in a three-dimensional space, and a two-dimensional image of the three-dimensional space, wherein the three-dimensional space is composed of at least one subspace, and when the three-dimensional space is composed of a plurality of subspaces, the subspaces do not overlap with each other, wherein the two-dimensional image of the three-dimensional space describes a planar layout of the subspaces of the three-dimensional space; a subspace acquisition unit, configured to acquire a two-dimensional image of at least one subspace in the three-dimensional space according to the two-dimensional image of the three-dimensional space; a reconstruction unit configured to perform three-dimensional reconstruction on the panoramic image of the at least one subspace, and project the three-dimensional reconstruction model of the panoramic image of the at least one subspace onto a plane where the two-dimensional image of the three-dimensional space is located or onto a plane parallel to the plane, and obtain projection images corresponding to the panoramic images of the at least one subspace; A matching unit is configured to obtain a matching relationship between the panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace based on the projection image of the at least one subspace and the two-dimensional image of the at least one subspace, and to establish an association relationship between the panoramic image of the subspace and the two-dimensional image of the subspace that generate the matching relationship.

10. An image matching device, comprising: processor; and a memory having computer program instructions stored therein, Wherein, when the computer program instructions are executed by the processor, the processor is caused to perform the following steps: Acquire a panoramic image of at least one subspace in a three-dimensional space, and a two-dimensional image of the three-dimensional space, wherein the three-dimensional space is composed of at least one subspace, and when the three-dimensional space is composed of a plurality of subspaces, the subspaces do not overlap with each other, wherein the two-dimensional image of the three-dimensional space describes the planar layout of the subspaces of the three-dimensional space; Acquire a two-dimensional image of at least one subspace in the three-dimensional space according to the two-dimensional image of the three-dimensional space; Performing three-dimensional reconstruction on the panoramic image of the at least one subspace, and projecting the three-dimensional reconstruction model of the panoramic image of the at least one subspace onto a plane where the two-dimensional image of the three-dimensional space is located or onto a parallel plane, and acquiring projection images corresponding to the panoramic images of the at least one subspace; Based on the projection image of the at least one subspace and the two-dimensional image of the at least one subspace, a matching relationship between the panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace is obtained, and an association relationship is established between the panoramic image of the subspace and the two-dimensional image of the subspace that generate the matching relationship.

11. A computer-readable storage medium having computer program instructions stored thereon, wherein: When the computer program instructions are executed by a processor, the following steps are implemented: Acquire a panoramic image of at least one subspace in a three-dimensional space, and a two-dimensional image of the three-dimensional space, wherein the three-dimensional space is composed of at least one subspace, and when the three-dimensional space is composed of a plurality of subspaces, the subspaces do not overlap with each other, wherein the two-dimensional image of the three-dimensional space describes the planar layout of the subspaces of the three-dimensional space; Acquire a two-dimensional image of at least one subspace in the three-dimensional space according to the two-dimensional image of the three-dimensional space; Performing three-dimensional reconstruction on the panoramic image of the at least one subspace, and projecting the three-dimensional reconstruction model of the panoramic image of the at least one subspace onto a plane where the two-dimensional image of the three-dimensional space is located or onto a parallel plane, and acquiring projection images corresponding to the panoramic images of the at least one subspace; Based on the projection image of the at least one subspace and the two-dimensional image of the at least one subspace, a matching relationship between the panoramic image of the at least one subspace and the two-dimensional image of the at least one subspace is obtained, and an association relationship is established between the panoramic image of the subspace and the two-dimensional image of the subspace that generate the matching relationship.

Images