Image processing method, device and computer-readable storage medium

Abstract

To provide an image processing method for processing a panoramic image.SOLUTION: A method includes: obtaining a panoramic image related to a room; performing door detection on the panoramic image to determine first information related to at least one door in the room; and displaying, based on the first information, a panoramic marker of the at least one door in the panoramic image, the panoramic marker indicating at least a door outline, a door type and how to open the door.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to the field of image processing, and more particularly to an image processing method, apparatus, and computer-readable storage medium for processing panoramic images. More specifically, the image processing method of the present invention relates to a method for displaying door signs in a panoramic image and identifying matching door candidates among multiple rooms, and a method for generating samples for training a deep learning-based neural network. [Background technology]

[0002] Virtual roaming is an important aspect of virtual reality (VR) technology, and it spans multiple industries, including architecture, travel, gaming, aerospace, and medicine. The combination of virtual scene construction and virtual roaming allows users to autonomously roam within three-dimensional scenes, such as buildings, cities, or game scenes, and intuitively experience the scene. In recent years, businesses such as online real estate and digital interior design have seen significant growth, leading to widespread use of virtual roaming technology in related fields. This technology enables virtual navigation of buildings and rooms, and can even generate corresponding floor plans, allowing customers to easily browse entire buildings or specific interior spaces online.

[0003] However, the information provided to customers using existing virtual roaming is still limited and may be inaccurate in some cases, often requiring customers to conduct a site survey, which also negatively impacts the generation of floor plans.

[0004] Therefore, there is a need for improved image processing methods and apparatus for processing panoramic images. Summary of the Invention [Problem to be solved by the invention]

[0005] In view of the above circumstances, the present invention provides the following image processing method, apparatus, and computer-readable storage medium to solve or improve at least the above problems in the prior art. [Means for solving the problem]

[0006] One aspect of the present invention provides an image processing method including: acquiring a panoramic image of a room; performing door detection on the panoramic image to identify first information about at least one door in the room; and displaying, based on the first information, a panoramic sign in the panoramic image for each of the at least one doors, the panoramic sign indicating at least a door outline, a door type, and an opening method.

[0007] According to an embodiment of the present invention, the method further includes: identifying second information regarding structural aspects of the room by performing structural detection on the panoramic image; generating a two-dimensional floor plan of the room based on the second information; and displaying, for at least one door, a two-dimensional door representation, a planar sign indicating at least the door type and opening method in the two-dimensional floor plan based on the first information and the second information.

[0008] According to an embodiment of the present invention, generating the two-dimensional floor plan includes generating a three-dimensional representation of the room in a three-dimensional spatial coordinate system based on the second information; and converting the three-dimensional representation to a two-dimensional planar coordinate system to generate the two-dimensional floor plan of the room.

[0009] According to an embodiment of the present invention, displaying a planar sign for at least one door in the two-dimensional plan view includes: for each door, based on the first information and the second information, projecting the two-dimensional door representation perpendicularly onto an edge corresponding to a corresponding structural surface in the two-dimensional plan view of the room to generate an alternative two-dimensional representation; and displaying the alternative two-dimensional representation as a two-dimensional door representation on the two-dimensional plan view for each door.

[0010] According to an embodiment of the present invention, the method further includes, for at least one door in the panoramic image, determining a distance between each representative point of the at least one door and a corresponding structural surface based on the first information and the second information before displaying a panoramic sign or a planar sign for each door; and removing a door whose distance does not meet a threshold range from the at least one door. According to an embodiment of the present invention, the panoramic or planar sign of each of the at least one door further indicates the types of rooms on either side of said door.

[0011] According to an embodiment of the present invention, door detection and structure detection are performed using deep learning based neural networks.

[0012] Another aspect of the present invention provides a method for training a neural network for the image processing method described above, the method including: performing a spatial transformation on a panoramic image to generate a plurality of sample panoramic images having angles different from that of the panoramic image; and training the neural network using the plurality of sample panoramic images.

[0013] Yet another aspect of the present invention provides an image processing method including: acquiring a plurality of panoramic images of a predetermined room in a building; performing door detection on each of the plurality of panoramic images to identify first information about at least one door in the predetermined room; performing structural detection on each of the plurality of panoramic images to identify second information about structural aspects of the predetermined room; generating a plurality of room floor plans associated with rooms in the building based on the first information and the second information, the plurality of room floor plans including signs indicating at least one of a door designation, a door type, a way of opening, and room types on both sides of the door, for each of the at least one door in the predetermined room; and performing matching on a plurality of inter-room doors based on each of the plurality of room floor plans to determine matching door candidates.

[0014] According to an embodiment of the present invention, the method further includes stitching the room floor plans together based on the matching door candidates to generate an architectural floor plan of the building.

[0015] Yet another aspect of the present invention provides an image processing device including: an acquisition means for acquiring a panoramic image of a room; a detection means for performing door detection on the panoramic image to determine first information about at least one door in the room; and a display means for displaying, in the panoramic image based on the first information, a panoramic sign indicating at least the outline of the door, the type of the door, and how to open it, for each door.

[0016] Yet another aspect of the present invention provides an image processing device comprising a processor and a memory in which computer program instructions are stored, wherein the computer program is executed by the processor to achieve the steps of acquiring a panoramic image of a room; performing door detection on the panoramic image to identify first information about at least one door in the room; and displaying, based on the first information, a panoramic sign indicating at least a door outline, a door type, and an opening method for each of the at least one door in the panoramic image.

[0017] Yet another aspect of the present invention provides a computer-readable storage medium having a computer program stored therein, the storage medium performing the steps of acquiring a panoramic image of a room, performing door detection on the panoramic image to identify first information about at least one door in the room, and displaying, based on the first information, a panoramic sign in the panoramic image for each of the at least one door, the panoramic sign indicating at least a door outline, a door type, and an opening method. [Effects of the Invention]

[0018] The image processing method, image processing device, and computer-readable storage medium of the present invention can provide users with richer and more accurate information by identifying each door in a room in a panoramic image based on information identified by door detection. The method of the present invention also provides information and identifies door candidates for generating a floor plan, thereby enabling a more accurate generation of a floor plan of the entire building and significantly improving the efficiency and accuracy of image processing. Furthermore, the present invention provides a method for training a neural network to execute the method, thereby further improving the accuracy of the image processing method. [Brief explanation of the drawings]

[0019] The above and other objects, features, and advantages of the present invention will become more apparent from the following detailed description of the embodiments of the present invention with reference to the accompanying drawings. These drawings are used to better understand the embodiments of the present invention and are used to explain the present invention together with the embodiments, which constitute a part of the specification, but are not intended to limit the present invention. In the drawings, the same reference numerals indicate the same parts or steps. [Figure 1] FIG. 1 is a flowchart showing an example of an image processing method according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating a panoramic image associated with a room according to an embodiment of the present invention. [Figure 3] FIG. 3 is a diagram illustrating the display of a panoramic sign on a panoramic image according to an embodiment of the present invention. [Figure 4] FIG. 4 is a flowchart showing another example of an image processing method according to an embodiment of the present invention. [Figure 5] FIG. 5 is a diagram illustrating displaying a planar sign on a two-dimensional plan view according to an embodiment of the present invention. [Figure 6] FIG. 6 is a diagram showing a door that is erroneously detected in a panoramic image according to an embodiment of the present invention. [Figure 7] FIG. 7 is a diagram illustrating the generation of a plurality of sample panoramic images according to an embodiment of the present invention. [Figure 8] FIG. 8 is a flowchart showing another example of an image processing method according to an embodiment of the present invention. [Figure 9] FIG. 9 is a diagram illustrating a plan view stitched together based on matching gate candidates according to an embodiment of the present invention. [Figure 10] FIG. 10 is a block diagram showing an example of an image processing apparatus according to an embodiment of the present invention. [Figure 11] FIG. 11 is a block diagram showing another example of an image processing apparatus according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, embodiments of an image processing method, an image processing apparatus, and a computer-readable recording medium according to the present invention will be described with reference to the accompanying drawings.

[0021] All other embodiments that a person skilled in the art can easily conceive based on the embodiments described in the present invention are included in the scope of protection of the present invention. Furthermore, the embodiments described in the present invention are only a part of the exemplary embodiments of the present invention, and are not all of them. These embodiments are merely examples and do not limit the scope of the present invention. In order to make the description concise and easy to understand, detailed descriptions of functions and structures that are well known in the technical field will be omitted, and redundant descriptions of steps and components will also be omitted.

[0022] First, a basic flow of an image processing method according to an embodiment of the present invention will be described in detail with reference to Fig. 1. As shown in Fig. 1, the image processing method includes step S101 of acquiring a panoramic image associated with a room.

[0023] According to an embodiment of the present invention, in this step, the panoramic image can be acquired from an external source such as a server or an internal source such as local storage via a wired or wireless connection, or can be captured by an image capture device such as a camera, but is not limited to these. Note that the term "panoramic image" as used herein refers to an image captured by a device such as a panoramic camera or a wide-angle camera, or an image with a wide field of view (e.g., a field of view of 120 degrees or more) obtained by synthesis, and is distinguished from a normal image captured with a standard lens or a perspective image captured using a perspective mapping method. The panoramic image acquired in this step can be, but is not limited to, a grayscale or color original image, or an image obtained by performing various image preprocessing such as cropping, scaling, calibration, or transformation on the original image.

[0024] In the embodiments of the present invention, a panoramic image of a room refers to a panoramic image showing a scene of the room, for example, a panoramic image that can display a scene of the room in a 360-degree angle of view range, captured from within the room (e.g., the center of the room).

[0025] For convenience of explanation, a residential house and its rooms are used as examples in the embodiments, but the term "room" used in the present invention has a broader meaning, and is not limited to referring to a space within a house, but also includes an indoor space or an enclosed area of ​​any building or location (e.g., an office building or a department store) depending on the specific application.

[0026] After acquiring the panoramic image of the room, the process proceeds to step S102, where door detection is performed on the panoramic image to identify first information about at least one door in the room.

[0027] According to an embodiment of the present invention, door detection is performed on the panoramic image acquired in this step. Specifically, the panoramic image of a room includes multiple doors. For example, FIG. 2 shows a floor plan of a panoramic image of a room according to an embodiment of the present invention. As shown in FIG. 2, six doors are shown in the current room. Each door has its own location, size (e.g., width, height, thickness, etc.), door type (e.g., hinged door, sliding door, accordion door, or door), and opening method (e.g., left / right to in / out rotation opening, left / right push opening, up / down / left / right folding, or simpler, left / right opening or in / out opening, etc.). In addition, each side of the door leads to a specific type of room (e.g., the room inside the door is the living room, and the room outside the door is the kitchen). In an embodiment of the present invention, an image processing method based on door detection is performed on the panoramic image to identify the above information about the doors of each room in the panoramic image, or an information set including at least the above information. Note that the above description is for illustrative purposes only and is not limiting. In another embodiment of the present invention, the door type may include other types such as the door open state (e.g., closed, slightly open, half open, almost fully open, and open), or the door material (e.g., wood or glass), or purpose (e.g., fire protection or thermal insulation), etc. In the present invention, the various pieces of information about the doors in the room are referred to as first information.

[0028] For example, in an embodiment of the present invention, door detection may include detecting corner coordinates (e.g., coordinates of corner points located at the corners of a door frame) for each door, thereby determining the door's position and dimensions; considering that doors are in different states when opened and closed, even if the door features are the same, they will exhibit certain differences depending on the door state, and these differences are usually closely related to and specific to the door type. Therefore, door detection may include identifying feature elements related to the shape, position, and dimensions of the door or specific parts of each room door, thereby determining the room type, opening method, and switch state; and further, extracting and classifying scene images inside and outside the door to determine the type of room connected to each side of each door. Furthermore, according to an embodiment of the present invention, door images are identified and classified using a neural network-based classification method, thereby determining information such as the door's opening state and opening direction. In practical applications, any known or future door detection method may be applied depending on the specific scene to determine the first information, and this is not limited thereto.

[0029] For convenience, the "gate" is described in the embodiments using a door as an example, but the "gate" described in the present invention is not limited to a room door and has a broader meaning, and may include a room window, a cabinet door, a gate, etc., as well as any room element that conforms to a predetermined size and shape depending on the actual application.

[0030] After identifying the first information associated with the door in the room, the process proceeds to step S103, where a panoramic sign is displayed on the door in the panoramic image based on the identified first information, and the displayed panoramic sign indicates at least the door profile, door type, and opening method of the door.

[0031] According to an embodiment of the present invention, in this step, a panoramic sign indicating the door outline, door type, and opening method is displayed for each door in the panoramic image based on the first information about the doors. For example, as described above, door detection can be performed on the panoramic image to identify the position and dimensions of the door. Also, for example, by detecting the corners of the door frame / door panel, the outline of the area corresponding to the door in the panoramic image, such as the outline of the door frame, the outline of the door panel, or a combination or at least a part thereof (i.e., the outline of the door).

[0032] Furthermore, according to an embodiment of the present invention, considering that a wide-angle lens causes distortion in a panoramic image, and that the degree of distortion is more severe in peripheral areas of the image (e.g., polar regions near the ceiling and floor of a panoramic image), the distortion of the panoramic image can be incorporated into the display of the door outline rather than simply identifying the area where the door is located based on corner coordinates. In other words, by introducing the distortion factor of the panoramic image, the door outline can be curved in the direction of the distortion according to the degree of distortion in the panoramic image. For example, as shown in FIG. 2, the door outline is displayed as a curved polygon in the panoramic image so that the outline (i.e., the sides of the curved polygon) representing the door fits the deformed door in the panoramic image. This method allows the user to more accurately view the shape and area of ​​each door in the panoramic image and avoids redundant display of information.

[0033] In addition, according to an embodiment of the present invention, in order to incorporate more information into the panoramic sign, the corresponding door type may be displayed using different colored outlines. Furthermore, the opening method may be displayed along a line extending perpendicular to the bottom edge of the door or the bottom edge of the door frame from a corner point on the door axis toward the door opening direction (inward or outward). Furthermore, the room types on either side of the door may be displayed using different symbols. Furthermore, the information displayed by the panoramic sign may be displayed using symbols or text in the area adjacent to the door (e.g., within the area where the door is located or along the door frame). The above description is for illustrative purposes only and is not limiting. In other embodiments, only part of the above information may be displayed, or information about more doors may be displayed. Furthermore, the door opening method may be displayed using color and the door type may be displayed using text, or a combination of these.

[0034] For example, as shown in FIG. 2, taking the leftmost door in the panoramic image as an example, the panoramic sign includes a door outline 201 indicated by a specific line type, e.g., a dashed line indicating that the door is a hinged door. The door type is indicated by a different color. An inward-extending line 202 indicates the door's opening direction. Optionally, vertical lines (e.g., line segment 203 in FIG. 2) of different gradations / colors are used on both sides of the door as a boundary to indicate the room types on either side of the door. For example, in this example, a dark line inside the door indicates that the room type inside the door is a living room, and a light line outside the door indicates that the room type outside the door is a kitchen. Alternatively, different line types may be used to indicate specific room types. In the example of FIG. 2, text (e.g., text box 204 in FIG. 2) may be displayed near the door outline to display the door's status or other information, thereby providing the user with a more specific door opening method along with the door opening direction indicated by line 202.

[0035] According to the above method, the panoramic sign displayed on the panoramic image displays various information related to the door together, thereby providing the user with more abundant and intuitive information.

[0036] 3A and 3B are diagrams showing panoramic signs in a panoramic image according to an embodiment of the present invention, illustrating multiple examples of panoramic signs. As shown in FIG. 3A, door outlines in the panoramic sign indicate the positions of the window on the left side of the panoramic image and the two doors on the right side, respectively, in the room. Curved polygonal door outlines indicate the shape of the corresponding window and door, and the color of the door outline indicates whether the corresponding window or door is a window or hinged door, respectively. Lines 301 and 302 in the panoramic sign indicate that the door opens to the inside right. Optionally, lines 303 and 304 in the panoramic sign indicate the room types on either side of the two doors. Because these two doors are in the same room, the rooms within the doors are of the same type. Note that in an embodiment of the present invention, if the door type indicated by the panoramic sign is a door type that does not connect to a room on the outside, such as a window or cabinet door, the room types on either side (or at least one side) of the door may be omitted.

[0037] In addition, in some scenes, the door may be completely closed or slightly open, so the door's open / closed state may not be determined by identifying the door's characteristic elements (such as hinges or rails). In this case, as described above, a separate type including the door type may be displayed. For example, as shown in FIG. 3(b), a panoramic sign sequentially displays the door types of each door from left to right, indicating the door type as a sliding door, a closed door, a slightly open door, and a closed door, respectively. In this example, the predicted location of the door when it is completely closed is indicated by a dotted line, providing the user with the predicted location of the door when it is closed.

[0038] The above embodiment has described the process of displaying a panoramic sign for a door in a panoramic image with reference to Figures 1 to 3. According to the above embodiment, it is possible to intuitively provide a user with abundant and accurate information in a panoramic image.

[0039] In order to provide the user with more information about the room, so that the user can switch between the panoramic image and a plan view (e.g., a bird's-eye view plan view or a floor plan), the image processing method according to the preferred embodiment of the present invention further includes a process of displaying a plan sign for the door in the two-dimensional plan view. The basic flow of this process will be described in detail with reference to FIG. 4.

[0040] As shown in FIG. 4, the image processing method includes, in step S401, processing for identifying second information related to structural aspects of the room by performing structural detection on the panoramic image.

[0041] In this step, an embodiment of the present invention performs structure detection on the panoramic image captured in step S101. Specifically, the three-dimensional space of a room is typically formed by the various structural surfaces (e.g., walls, ground, and ceiling) of the room. The spatial characteristics include the spatial coordinates of the walls, ground, and ceiling that make up the room; the perpendicular or parallel geometric relationships between walls, ground, and ceiling; and the edge positions of the boundaries. This information is identified through structure detection to generate a two-dimensional floor plan of the room. In the present invention, the various pieces of information related to the structural surfaces of the room are referred to as second information. In practical applications, the second information may be identified using any known or future structural detection method depending on the specific scene, but the present invention is not limited thereto.

[0042] For ease of explanation, in the embodiments of the present invention, "walls, ground, and ceiling" are described as examples of structural surfaces, but the term "structural surface" has a broader meaning and can also refer to any other plane or wall in a room that has divisible characteristics.

[0043] After identifying the first information about the doors in the room, the process proceeds to step S402, where a two-dimensional floor plan of the room is generated based on the identified second information.

[0044] In an embodiment of the present invention, generating a two-dimensional floor plan of a room includes generating a three-dimensional representation of the room in a three-dimensional spatial coordinate system based on second information about the structural surfaces of the room and then transforming the three-dimensional representation into a two-dimensional plane coordinate system. Thus, the two-dimensional floor plan of the room is generated. Specifically, Figures 5(a) to 5(d) illustrate two-dimensional floor plans of a room generated in an embodiment of the present invention. After identifying the second information about the structural surfaces of the room, a three-dimensional representation of the room is constructed by performing three-dimensional reconstruction using, for example, a Horizon Net deep neural network model, and obtaining the boundary lines at the side edges of each structural surface in the room, as shown in Figure 5(a). The resulting three-dimensional representation of the room is then projected onto a two-dimensional plane coordinate system parallel to the ground, thereby obtaining a floor plan of the room. As shown in Figure 5(b), the two-dimensional floor plan is a bird's-eye view of the geometric shape and dimensions of the room corresponding to the panoramic image.

[0045] After generating the two-dimensional floor plan of the room, the process proceeds to step S403. In this step, a planar marker is displayed for each door in the room on the generated two-dimensional floor plan based on the first information identified by the door detection and the second information identified by the structure detection. Here, the planar marker indicates at least the two-dimensional door representation, the door type, and how to open it.

[0046] Specifically, based on the first information about the doors in the room identified in step S102, a two-dimensional door representation of each door is identified at a corresponding position on the two-dimensional plan view of the room. For example, by using a method similar to that described in the above steps to convert the three-dimensional representation of the door detected in the panoramic image into the two-dimensional coordinate system of the plan view of the room, a two-dimensional door representation of the door is displayed on the generated two-dimensional plan view. This allows the position and dimensions of the door to be indicated on the two-dimensional plan view. In addition, based on the identified first information, further information such as the door type, opening method, and room types on both sides of each door can be displayed. As described above, two-dimensional signs for the corresponding doors may be displayed in a manner similar to or corresponding to the panoramic signs. As shown in FIG. 5(c), two-dimensional signs for each door are further displayed on the two-dimensional plan view. The two-dimensional signs may indicate the two-dimensional door representation, door type, opening method, and, optionally, the room types on both sides of the door. This allows the user to generate and provide a floor plan of the room, and by repeatedly using the information identified in the door detection process to display various information about the door on the generated two-dimensional floor plan, the user can easily switch between and compare the panorama and the two-dimensional floor plan, and the information about the room is displayed more completely.

[0047] Furthermore, as shown in Figure 5(c), the 2D door displayed on the 2D floor plan may not fit the edge of the room due to errors in corner coordinates or coordinate transformation, which may result in inaccurate information about the displayed door's position and how to open it.

[0048] In a preferred embodiment of the present invention, displaying a planar sign on a two-dimensional floor plan of a room to optimize the two-dimensional door representation of a door includes generating an alternative two-dimensional representation for each door by perpendicularly projecting the two-dimensional door representation onto a corresponding edge of a structural surface in the two-dimensional floor plan of the room based on the identified first and second information. Specifically, the coordinate values ​​in a two-dimensional coordinate system of the corner points of the door and the shape of the room (i.e., the representation of the wall surface) are determined based on the first and second information. Then, the two-dimensional door representation is projected onto the edge of the corresponding wall surface by calculating the planar coordinates, and the alternative two-dimensional representation generated by the projection is superimposed on the representation of the wall surface of the room, as shown in FIG. 5(d). Based on this, the generated alternative two-dimensional representation is displayed on the two-dimensional floor plan as a two-dimensional door representation of each door in the room, thereby correcting the position and opening manner of the door. In this embodiment, the "corresponding structural surface" in the two-dimensional floor plan refers to the wall surface on which the door is actually located. For example, by setting rules such as the angle formed between the door (door frame) and the wall surface and / or the distance between the door and the wall surface, a structural surface corresponding to each door is identified.

[0049] In addition, in an embodiment of the present invention, to ensure that the dimensions of the door are not affected by the projection process, the width of the door is determined based on the first information, and the generated alternative 2D representation is adjusted based on the determined width of the door after the projection process. For example, the distance between the corner points of the door in the horizontal direction is calculated, and the actual width of the door is determined based on the distortion of the panoramic image or the depth information of the image, or the distance from the corner points of the door to the camera, and based on a trigonometric relationship. In this way, the 2D planar representation is optimized and the dimensions of the door in the 2D planar view are not adversely affected by the projection process.

[0050] Furthermore, there are cases where doors are erroneously detected. Fig. 6 shows a door that has been erroneously detected in an embodiment of the present invention. As shown in Fig. 6, erroneous panorama signs 601 and 602 are displayed in the left and right marginal areas of the panoramic image. Since the door described in the present invention does not exist in these areas, the door has been erroneously detected.

[0051] An image processing method according to an embodiment of the present invention further includes, before displaying a panoramic sign or a planar sign for each door in the panoramic image, determining a distance from a representative point of each door to a corresponding structural surface based on the first information and the second information, and removing a door (i.e., a misdetected door) for which the determined distance is greater than a threshold value from the detected doors, for at least one door in the panoramic image. Specifically, as described above, the coordinates of each pixel related to the door and the coordinates of the corresponding structural surface of the door are determined from the first information and the second information. The distance from the representative point of each door to the corresponding structural surface is calculated based on the coordinate values. Preferably, the distance to the corresponding structural surface is calculated using the center point of the door as the representative point. The calculated distance is then compared with a preset threshold to determine whether the door is a misdetected door. If the determined distance is greater than the threshold value or outside the threshold range, the door is determined to be a misdetected door and the misdetected door is removed. This prevents the display of a sign for the misdetected door. In another embodiment, the distance from a representative point of a door to the ground or ceiling adjacent to the door can be calculated. In this case, the center point of the top or bottom edge of the door frame is used as the representative point to calculate the distance to the ground or ceiling adjacent to the door, and doors whose distance does not meet the threshold range can be eliminated. Furthermore, appropriate thresholds or range ranges can be set for different scenes according to actual applications. This further improves the accuracy of image processing and information display, and improves processing efficiency.

[0052] In addition, door detection and structure detection, or portions thereof, according to embodiments of the present invention are performed using a deep learning-based neural network. For example, as described above, a neural network-based classifier is used to identify information such as the door opening state and opening direction. The present invention also provides a method for training a neural network to perform the image processing method described herein. Specifically, this includes generating samples for training the neural network using the following method to construct the neural network.

[0053] Specifically, to implement the various methods described in the present invention, a method for training a deep learning-based neural network involves performing spatial transformation on a panoramic image (e.g., the panoramic image acquired in step S101) to generate multiple sample panoramic images with different viewpoints. By performing spatial transformation on the original panoramic image, camera movement can be simulated (i.e., simulating a camera taking images at different positions and angles). This results in the generated sample target panoramic image having a different viewpoint from the original panoramic image. In this way, the training sample collection is expanded, and, for example, samples can be trained with the expanded image features to perform multi-view comparative learning. This provides additional information to the deep learning-based neural network, enabling the machine learning model to achieve better results in subsequent processing (or downstream tasks).

[0054] Specifically, in an embodiment of the present invention, spatial transformation of a panoramic image involves transforming pixels in the original panoramic image into another spatial coordinate system using a predetermined mapping method, thereby obtaining an image composed of pixels in the other coordinate system transformed from the pixels of the original panoramic image. For example, a 3D surface mapping transformation can be used to project the original panoramic image onto the surface of a 3D solid shape to convert it into a 3D surface image, allowing subsequent processing to translate, rotate, or a combination of these to further rotate a specific region in the panoramic image. Such a 3D solid shape can be a sphere, cube, or other solid shape formed in 3D space. Correspondingly, the 3D surface mapping transformation can include a spherical mapping transformation, a cube mapping transformation, or any mapping transformation that maps a 2D panoramic image onto the surface of a 3D solid shape.

[0055] In addition, in an embodiment of the present invention, spatial transformation parameters are set to adjust the transformation for the panoramic image. For example, for an image with large distortion, the amount of translation / rotation transformation of the three-dimensional sphere can be appropriately increased to transform areas with large distortion in the original panoramic image (e.g., bipolar areas near the ceiling or floor in the panoramic image) into areas with small distortion (e.g., areas near the equator in the panoramic image), or vice versa. This generates a panoramic image with different distortion in the same area compared to the original panoramic image, which allows the results of subsequent image processing to match.

[0056] FIG. 7 illustrates the generation of multiple sample panoramic images according to an embodiment of the present invention. As shown in FIG. 7, various spatial transformations can be performed on the panoramic images to simulate panoramic images captured at different viewpoints due to camera movement. In other words, the generated sample panoramic images have different viewpoints than the original panoramic images, and the images corresponding to the same regions have different degrees of distortion. This provides rich image information. Therefore, by training a deep learning-based neural network with the generated sample panoramic images, an improved model can be constructed for implementing the method according to the present invention, thereby achieving better image processing such as detection and identification.

[0057] An embodiment in which a door sign is displayed in a panoramic image has been described above with reference to FIGS.

[0058] According to the above embodiments, doors in a room can be labeled with more information in a panoramic image, and a floor plan including the labels can be generated using the various information determined. This improves the accuracy of the information and simultaneously improves the efficiency of image processing. Furthermore, by simulating camera movement, the training samples of the neural network used in image processing can be expanded. This improves the model for improved door detection and structure detection, further improving the accuracy of the information determined.

[0059] Another example of a flow for executing an image processing method according to an embodiment of the present invention will be described below with reference to FIGS.

[0060] First, as shown in FIG. 8, the image processing method includes, in step S801, acquiring a plurality of panoramic images of a predetermined room in a building.

[0061] In this step, the embodiment of the present invention acquires multiple panoramic images using a method similar to that used in step S101. In addition, in the embodiment of the present invention, the panoramic image of a specific room in a building refers to a panoramic image of a room in the building, for example, panoramic images are taken for all or some of the rooms in the building to acquire multiple panoramic images of the specific room in the building.

[0062] After acquiring the panoramic images, the process proceeds to step S802. In step S802, door detection is performed on each of the plurality of panoramic images to identify first information related to at least one door in a predetermined room; in step S803, structure detection is performed on each of the plurality of panoramic images to identify second information related to the structural aspects of the predetermined room.

[0063] In an embodiment of the present invention, in the above two steps, door detection and structure detection are performed for each panoramic image to identify first information about at least one door in a specified room and second information about the structural aspects of the specified room. Also, in steps S802 and S803, the first information and the second information are identified by applying the same method as the door detection and structure detection method described in the above steps, but these steps may be performed in the order shown in Figure 8, or simultaneously, or S803 may be performed before S802.

[0064] After the first information and the second information are identified, the method proceeds to step S804. Based on the identified first information and second information, a room plan relating to a plurality of rooms in the building is generated, the room plan including, for each door in a given room, a sign indicating at least one of a door designation, a door type, an opening method, and the types of rooms on either side of the door.

[0065] In an embodiment of the present invention, in the above steps, multiple room floor plans are generated in a manner similar to that described in FIG. 4. Signs are displayed on the room floor plans. Here, the "room floor plan" refers to a two-dimensional plan view of one room in a building. In addition, a "sign" is displayed in a manner similar to the above-described method for displaying planar signs, to show at least one piece of information about the door (e.g., door designation, door type, opening method, and room types on both sides of the door).

[0066] After generating the multiple room floor plans and displaying the signs, the process proceeds to step S805, where matching is performed on doors between multiple rooms based on the signs on each room floor plan, and matching door candidates are determined.

[0067] In an embodiment of the present invention, this step matches doors between multiple rooms based on the corresponding door markings within the rooms. Specifically, doors between rooms of the same type are first identified based on the types of rooms on either side of the door. For example, as shown in FIG. 9(a), in the floor plan of a living room, line 901 indicates that a living room and a kitchen are located on either side of a single door. Similarly, in the floor plan of a kitchen, line 902 indicates that a living room and a kitchen are also located on either side of a single door. Therefore, it is determined that the door between these two rooms is the same, and this door is determined as a matching door candidate, as shown by the dashed line in FIG. 9(a). This initial identification of identical doors between multiple rooms as matching door candidates provides users with useful information for understanding the overall structure of a building with multiple rooms, and is also useful for subsequent processes such as identifying relationships between rooms and generating architectural floor plans.

[0068] Furthermore, according to an embodiment of the present invention, matching between rooms can be determined based on the door size, position, door type, door opening method, or a combination thereof, thereby further improving the reliability of matching. For example, if two bedrooms are adjacent to the same living room, matching door candidates can be determined based on the door opening method and position, taking into account that the rooms adjacent to both sides of the two bedroom doors are the same room type.

[0069] Furthermore, according to an embodiment of the present invention, the image processing method includes, in step S806, stitching together a plurality of room plans based on the matching door candidates to generate an architectural floor plan of the building.

[0070] In this invention, matching door candidates are the actual points connecting rooms, so multiple rooms are connected by doors to form the overall structure of the building. Therefore, after the matching door candidates between rooms are determined using the above method, the connection relationship between the rooms can be identified. In an embodiment of the present invention, in this step, floor plans of rooms with the same matching door candidate are connected to generate an architectural floor plan of the entire building. For example, as shown in Figure 9(b), the determined matching door candidate is used as the connecting point between two corresponding rooms, and the floor plans of the two rooms are connected to form a floor plan that combines the kitchen and living room. In this way, by connecting all the floor plans of multiple rooms, a floor plan of the entire room is generated.

[0071] Thus, according to the image processing method of the embodiment of the present invention, the steps can identify door candidates for a building with multiple rooms, provide information, and generate floor plans, thereby generating floor plans of the entire building in a more accurate manner and improving the efficiency and accuracy of image processing.

[0072] Next, an example of an image processing device according to the present invention will be described with reference to FIG.

[0073] 10 is a block diagram showing the configuration of an image processing device 1000 according to an embodiment of the present invention. As shown in FIG. 10, the image processing device 1000 includes an acquiring unit 1010, a detecting unit 1020, and a display unit 1030. The illustrated structure is merely exemplary and is not limiting. In addition to these units, the image processing device 1000 may also include other components, but these are not related to the embodiment of the present invention and will not be illustrated or described here.

[0074] The details of the following operations performed by the image processing device 1000 according to the embodiment of the present invention are almost the same as those explained with reference to Figures 1 to 9, so for the sake of brevity, the explanation of the same parts will be omitted. Below, each means or component of the image processing device 1000 will be explained one by one.

[0075] The acquisition unit 1010 acquires an original panoramic image. The specific processing performed by the acquisition unit 1010 corresponds to the content of step S101 described with reference to FIG.

[0076] Specifically, in an embodiment of the present invention, the acquisition unit 1010 may be an image capture device, such as a mobile phone, a camera, or a video camera, or an image capture unit included in such a device, and may also be a device for acquiring images captured from the image capture device or unit. The acquisition unit 1010 may be physically separate from other modules in the image processing device 1100, and transmit the acquired images to other modules in the image processing device 1100 via wired or wireless communication. Alternatively, the acquisition unit 1010 may be integrated with other modules in the image processing device 1100 or installed in the same system. Other modules or components of the image processing device 1100 receive the acquired images from the acquisition unit 1010 via an internal bus.

[0077] The detection unit 1020 performs door detection on the panoramic image and identifies first information related to at least one door in the room. The specific processing performed by the detection unit 1020 corresponds to the content of step S102 shown in FIG. 1. In addition, the detection unit 1020 according to the embodiment of the present invention performs structure detection on the panoramic image. Here, the door detection and structure detection are performed using a neural network based on deep learning.

[0078] The display means 1030 displays a panoramic sign for each door in the panoramic image based on the first information. The panoramic sign indicates at least one of the door outline, door type, and opening method, and may optionally indicate the types of rooms on both sides of the door. The display means 1030 according to an embodiment of the present invention also generates a two-dimensional plan view of the room and displays a plan view sign for each door on the generated two-dimensional plan view. Furthermore, the display means 1030 according to an embodiment of the present invention removes erroneously detected doors and optimizes the two-dimensional door display for the doors.

[0079] Next, another example of the image processing device of the present invention will be described with reference to FIG.

[0080] FIG. 11 is a block diagram showing the configuration of an image processing device 1100 according to an embodiment of the present invention. As shown in FIG. 11, the image processing device 1100 includes a processor 1110 and a memory 1120. The image processing device 1100 may be a computer or a server. The illustrated configuration is merely an example and is not limiting. Furthermore, the image processing device 1100 may include other components in addition to these means, but these components are not related to the embodiment of the present invention and therefore will not be illustrated or described here.

[0081] Note that the detailed operations performed by the image processing device 1100 according to the embodiment of the present invention are almost the same as those described with reference to Figures 1 to 9, and for the sake of brevity, detailed description will be omitted. Below, each module and component in the image processing device 1100 will be described.

[0082] The processor 1110 is a central processing unit (CPU) or other type of processing unit having data processing and / or instruction execution capabilities, and can implement predetermined functions by executing computer program instructions stored in the memory 1120. Execution of the computer program instructions by the processor 1110 achieves the steps of: acquiring a panoramic image of a room; performing door detection on the panoramic image to identify first information about at least one door in the room; and displaying, based on the first information, a panoramic indicator in the panoramic image for the at least one door, indicating at least a door outline, a door type, and an opening method for each door.

[0083] The memory 1120 includes one or more computer program products. The computer program products are stored in various computer-readable storage media, such as volatile and / or non-volatile memory. One or more computer program instructions are stored in the computer-readable storage media, and execution of the program instructions by the processor 1110 realizes the functionality of an image processing device according to an embodiment of the present invention and / or other desired functionality and / or realizes an image processing method according to an embodiment of the present invention. The computer-readable storage media may also store various application programs and various data.

[0084] Next, a computer-readable storage medium according to an embodiment of the present invention will be described.

[0085] The present invention also provides a computer-readable storage medium having stored therein computer program instructions, the computer program instructions being executed by a processor to perform the steps of: acquiring a panoramic image of a room; performing door detection on the panoramic image to identify first information about at least one door in the room; and displaying, based on the first information, a panoramic sign in the panoramic image for each of the at least one door, the panoramic sign indicating at least a door outline, a door type, and an opening method.

[0086] The steps executed by the processor in the embodiments of the present invention correspond to the contents of each of the above-described embodiments explained with reference to Figures 1 to 9. Note that each component and module in the image processing device described above may be realized by hardware or software, or may be realized by a combination of hardware and software.

[0087] The above embodiments are merely illustrative and not limiting. Furthermore, those skilled in the art may combine or integrate some of the steps and devices from the above-described embodiments to achieve the effects of the present invention. Such combined and integrated embodiments are also encompassed within the present invention and will not be described one by one herein. Furthermore, the advantages, benefits, and effects mentioned in the present invention are merely illustrative and not limiting, and should not be considered as essential features of the embodiments of the present invention. The details of the present invention described above are merely illustrative and are provided for ease of understanding, not limiting. These details do not limit the application of the present invention.

[0088] Block diagrams of modules, devices, facilities, and systems referred to in the present invention are merely examples for illustrative purposes. The connections, arrangements, and configurations shown in the block diagrams are not required or implied to apply. Those skilled in the art can connect, arrange, and configure these modules, devices, facilities, and systems in any manner. Furthermore, terms such as "include," "includes," and "having" are inclusive terms meaning "including but not limited to," and are interchangeable. Furthermore, the terms "or" and "and," as used herein, mean "and / or" and are interchangeable, unless the context clearly indicates otherwise. Furthermore, the term "such as," as used herein, means "such as but not limited to," and is interchangeable.

[0089] The flow diagrams of the present invention and the above method descriptions are merely illustrative examples and are not intended to require or imply that the steps of each example must be performed in the order shown. The steps shown in the above examples can be performed in any order. Furthermore, terms such as "then," "and," and "next" are not intended to limit the order of the steps, but are used to guide an understanding of the present invention. Furthermore, any reference to a singular element using, for example, the articles "a," "one," "the," or "said" is not to be construed as limiting the element to the singular.

[0090] It should be noted that the steps and devices described in each embodiment in this specification are not limited to being performed in a specific embodiment. New embodiments can be conceived by combining some of the steps or devices related to each embodiment based on the spirit of the present invention, and these new embodiments are also included within the scope of the present invention. Furthermore, methods and functions according to the present invention include one or more actions for realizing the method. Methods and / or actions may be interchanged without departing from the scope of the claims. In other words, unless otherwise specified, the order and / or use of actions may be modified without departing from the scope of the claims.

[0091] Each operation of the methods described above may be performed by any suitable means having the corresponding functionality. This means may include various hardware and / or software components and / or modules, including, but not limited to, circuits, application specific integrated circuits (ASICs), or processors. Each of the illustrative logic blocks, modules, and circuits described above may be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an ASIC, a field programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof, designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any commercially available processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing equipment, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration.

[0092] The steps of a method or algorithm according to the present invention may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may be stored on any form of tangible storage medium. Examples of storage media that may be used include random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, etc. A storage medium may be coupled to a processor such that the processor can read information from, and write information to, the storage medium. Alternatively, the storage medium and the processor may be integrated. A software module may represent either a single instruction or multiple instructions, and may be distributed over several different code segments, among different programs, and across multiple storage media.

[0093] Thus, a computer program product can perform the operations described herein. For example, such a computer program product is a tangible computer-readable medium having instructions tangibly stored (and / or encoded) thereon. The instructions are executed by one or more processors to perform the operations described above. The computer program product may also be included in a package. The software or instructions are transmitted over a transmission medium. For example, the software may be transmitted from a website, server, or other remote source using a transmission medium such as coaxial cable, fiber optic cable, twisted wire, Digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, microwave, etc.

[0094] Modules and / or other suitable means for performing the above-described methods and techniques may be downloaded and / or otherwise obtained by the user terminal and / or base station, as appropriate. For example, such equipment may be connected to a server and provided with transmission of means for performing the above-described methods. Alternatively, the above-described methods may be provided via storage means (e.g., physical storage media such as RAM, ROM, CD, or floppy disk), and the user terminal and / or base station may obtain the methods when coupled to or providing storage means to the equipment. Furthermore, any other suitable technology may be used to provide the above-described methods and techniques to the equipment.

[0095] Other examples and implementations are within the scope and spirit of the present invention and the claims. For example, due to the nature of software, the functions described above may be implemented by software executed by a processor, hardware, firmware, hardwire, or any combination thereof. Features implementing the functions may be located in various physical locations. This includes functions distributed such that portions of the functions are implemented in different physical locations. As used herein and in the claims, "or" used in a list preceding "at least one" refers to a disjunctive list; for example, the list "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, "exemplary" does not imply that the described example is preferred or preferable over other examples.

[0096] Various changes, substitutions, and alterations to the technology described above may be made without departing from the limitations of the claims. Furthermore, the scope of the claims is not limited to the particular processes, machines, manufacture, combinations of events, means, methods, and acts described above. Any now-known or later-developed processes, machines, manufacture, combinations of events, means, methods, and acts that perform substantially the same function or achieve substantially the same result as those described herein may be utilized. Accordingly, the scope of the claims includes any processes, machines, manufacture, combinations of events, means, methods, or acts that fall within their scope.

[0097] The foregoing description is provided to enable one skilled in the art to use or practice the invention, and various modifications to this description will be apparent to those skilled in the art, and the general principles defined herein may be applied elsewhere without departing from the scope of the invention.

[0098] Thus, the present invention is not limited to the above description, but is to be accorded the widest scope consistent with the inventive concept and novel features.

[0099] The foregoing description has been given for purposes of illustration and description, but not of limitation. Variations, additions, modifications and subcombinations of the embodiments described above will occur to those skilled in the art.

Claims

1. 1. A computer-implemented image processing method, comprising: Obtain a panoramic image of the room; performing door detection on the panoramic image to identify first information about at least one door in the room, the first information including at least a door profile, a door type, and an opening method of the door; and displaying a panoramic sign indicating at least a door outline, a door type, and an opening method for each of the at least one doors in the panoramic image based on the first information; An image processing method comprising:

2. Identifying second information about structural surfaces of the room by performing structural detection on the panoramic image, the second information including at least spatial position coordinates of walls, ground, and ceiling constituting the room, vertical or parallel geometric relationships between walls, ground, and ceiling, and edge positions of boundaries; generating a two-dimensional floor plan of the room based on the second information; and displaying, for each of the at least one door, at least a two-dimensional door representation, a planar sign indicating a door type and an opening method in the two-dimensional plan view based on the first information and the second information; The image processing method of claim 1 , further comprising:

3. generating a two-dimensional floor plan of the room, generating a three-dimensional representation of the room in a three-dimensional spatial coordinate system based on the second information; and transforming the three-dimensional representation into a two-dimensional planar coordinate system to generate a two-dimensional floor plan of the room; 3. The image processing method according to claim 2, further comprising:

4. Displaying a planar sign for each of the at least one doors in the two-dimensional plan view includes: based on the first information and the second information, for each door, projecting a two-dimensional door representation perpendicularly onto a side corresponding to a corresponding structural surface in the two-dimensional floor plan of the room to generate an alternative two-dimensional representation; and displaying the alternative two-dimensional representations as two-dimensional door representations on the two-dimensional plan view for each of the doors; 4. The image processing method according to claim 3, further comprising:

5. In the panoramic image, before displaying a panoramic sign or a planar sign for each of the at least one door, Identifying a distance from each representative point of the at least one door to a corresponding structural surface based on the first information and the second information; and removing from said at least one door any door for which said distance does not meet a threshold range; The image processing method according to claim 2 , further comprising:

6. the panoramic or planar sign of each of said at least one door further indicates the types of rooms on either side of said door; The image processing method according to claim 1 .

7. The door detection and the structure detection are performed using a neural network based on deep learning. The image processing method according to claim 2 .

8. 8. A neural network training method for use in training a neural network based on deep learning according to claim 7, comprising: generating a plurality of sample panoramic images having different viewpoints from the panoramic image by performing a spatial transformation on the panoramic image; and training the neural network using the plurality of sample panoramic images; A neural network training method comprising:

9. Acquire multiple panoramic images of a given room in a building; performing door detection for each of the plurality of panoramic images to identify first information about at least one door in the predetermined room, the first information including at least a door profile, a door type, and an opening method of the door; performing structural detection for each of the plurality of panoramic images to identify second information related to structural aspects of the specified room, the second information including at least spatial position coordinates of walls, a ground, and a ceiling that constitute the room, vertical or parallel geometric relationships between walls, the ground, and the ceiling, and edge positions of boundaries; generating a plurality of room plans for rooms in the building based on the first information and the second information, the room plans including indicators indicating at least one of a door designation, a door type, an opening style, and room types on either side of the door for the at least one door in a given room; and performing matching for doors between a plurality of rooms based on each of the plurality of room floor plans, and determining matching door candidates; An image processing method comprising:

10. stitching the room plans together based on the matching door candidates to generate an architectural floor plan of the building; The image processing method of claim 9 , further comprising:

11. an acquisition means for acquiring a panoramic image of the room; a detection means for performing door detection on the panoramic image to identify first information about at least one door in the room, the first information including at least a door profile, a door type, and an opening method of the door; and a display means for displaying, in the panoramic image based on the first information, a panoramic sign indicating at least a door outline, a door type, and an opening method for each of the at least one doors; An image processing device comprising:

12. a processor and a memory in which a computer program is stored; When the computer program is executed by the processor, Obtain a panoramic image of the room; performing door detection on the panoramic image to identify first information about at least one door in the room, the first information including at least a door profile, a door type, and an opening method of the door; and displaying a panoramic sign indicating at least a door outline, a door type, and an opening method for each of the at least one doors in the panoramic image based on the first information; An image processing device in which the steps are realized.

13. A computer-readable storage medium, comprising: A computer program is stored, and when the computer program is executed, Obtain a panoramic image of the room; performing door detection on the panoramic image to identify first information about at least one door in the room, the first information including at least a door profile, a door type, and an opening method of the door; and displaying a panoramic sign indicating at least a door outline, a door type, and an opening method for each of the at least one doors in the panoramic image based on the first information; The storage medium on which the steps are realized.

14. A program for causing a computer to execute the image processing method according to any one of claims 1 to 7, the neural network training method according to claim 8, or the image processing method according to claim 9 or 10.

Images