Model generation system and model construction method of architectural space

Abstract

The present invention provides a model generation system and a model construction method for a building space. A scanned image is obtained. The scanned image is generated by scanning the building space. Identify the identification information of the markers in the scanned image and the positional relationship of the markers in the building space. Construct a three-dimensional model of the building space according to the identification information and the positional relationship. The markers correspond to objects in the three-dimensional model. In this way, modeling and positioning can be quickly performed, objects can be accurately identified, and a preview function of furniture or electrical appliances can be provided.

Description

Technical Field

[0001] The present invention relates to a simulation technology, and in particular to a model generation system and a model construction method of a building space. Background Art

[0002] Nowadays, extended reality (XR) technologies for simulating sensations, perceptions and / or environments, such as virtual reality (VR), augmented reality (AR), mixed reality (MR), etc., are gaining popularity. These technologies can be applied in a variety of fields, such as gaming, military training, healthcare, remote work, etc.

[0003] For example, when residents plan to decorate a bare house, they usually cannot be there in person and can only rely on their imagination to plan the layout or arrange the furniture. Existing AR technology can help users place sofas or other furniture models in the house to simulate the furniture placement. However, if you want to further preview the wall color, floor tile style, door material, window frame style, curtain style, ceiling recessed light light, kitchenware location, air conditioning location and other fine-tuning, the existing technology still has difficulties. Summary of the invention

[0004] The present invention is directed to a building space model generation system and a model construction method, which combine modeling, positioning and object recognition technologies to provide users with more diversified building space planning functions.

[0005] According to an embodiment of the present invention, a method for constructing a model of an architectural space includes (but is not limited to) the following steps: obtaining a scanned image. The scanned image is generated by scanning the architectural space. Recognizing identification information of a marker in the scanned image and a positional relationship of the marker in the architectural space. Constructing a three-dimensional model of the architectural space based on the identification information and the positional relationship. The marker corresponds to an object in the three-dimensional model.

[0006] According to an embodiment of the present invention, a model generation system for an architectural space includes (but is not limited to). An image capture device, a memory, and a processor. The image capture device is used to capture an image. The memory is used to store a program code corresponding to a model construction method for the model generation system. The processor couples the image capture device and the memory and is configured to execute the program code. The model construction method includes the following steps: obtaining a scanned image through an image capture device. The scanned image is generated by scanning the architectural space. Recognizing identification information of a marker in the scanned image and a positional relationship of the marker in the architectural space. Constructing a three-dimensional model of the architectural space based on the identification information and the positional relationship. The marker corresponds to an object in the three-dimensional model.

[0007] Based on the above, the model generation system and model construction method of the building space of the embodiment of the present invention, based on the real mark placed in the actual building space, obtains the identification information of the object corresponding to the mark (for example, wall, door, window, furniture, etc.) and its position in the space through image recognition, and constructs the three-dimensional model of the building space accordingly. In this way, modeling and positioning can be quickly achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The accompanying drawings are included to provide a further understanding of the present invention and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the present invention and together with the description serve to explain the principles of the present invention.

[0009] Figure 1 A component block diagram of a system for generating a model of an architectural space according to an embodiment of the present invention;

[0010] Figure 2 is a flow chart of a method for constructing a model of an architectural space according to an embodiment of the present invention;

[0011] Figure 3 For example, architectural space;

[0012] Figure 4 is a marked instance;

[0013] Figure 5 A and Figure 5 B is an example of a directional mark.

[0014] Description of Figure Numbers

[0015] 100: Model generation system;

[0016] 110: image capture device;

[0017] 130: memory;

[0018] 150: Display;

[0019] 170: processor;

[0020] S210~S250: steps;

[0021] w1~w3: wall;

[0022] d1: door;

[0023] f1: floor;

[0024] m1~m8: marking;

[0025] b1, b2: bed. DETAILED DESCRIPTION

[0026] Reference will now be made in detail to exemplary embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Whenever possible, the same reference numerals are used in the drawings and the description to refer to the same or like parts.

[0027] Figure 1 FIG. 1 is a block diagram of components of a system 100 for generating a model of an architectural space according to an embodiment of the present invention. Figure 1 , the model generation system 100 includes but is not limited to: an image capture device 110, a memory 130, a display 150 and a processor 170. The model generation system 100 can be one or more electronic devices such as smart phones, tablets, VR / AR glasses or head-mounted displays (HMD).

[0028] The image capture device 110 may be a monochrome camera or a color camera, a stereo camera, a digital video camera, a depth camera, a time-of-flight (ToF) detector, or other sensors capable of capturing images. In one embodiment, the image capture device 110 is used to capture images.

[0029] The memory 130 may be any type of fixed or removable random-access memory (RAM), read-only memory (ROM), flash memory, similar devices, or a combination of the above devices. In one embodiment, the memory 130 is used to record program code, device configuration, buffer data, or permanent data (such as image data, object data, identification information, positional relationships, three-dimensional models, marking features, internal parameters of the image capture device 110, etc.), and these data will be introduced later.

[0030] The display 150 may be a liquid crystal display (LCD), a light-emitting diode (LED), an organic light-emitting diode (OLED) or other display. In one embodiment, the display 150 is used to display images. It should be noted that in some embodiments, the display 150 may be a display of an external device (such as a smartphone, a tablet, a computer, etc.), and the external device may be placed on the main body or desktop of the head-mounted display.

[0031] The processor 170 is coupled to the image capturing device 110 , the memory 130 , and the display 150 , and the processor 170 is configured to load and execute the program code stored in the memory 130 to perform the process of the exemplary embodiment of the present invention.

[0032] In some embodiments, the functions of the processor 170 may be implemented using a programmable unit such as a central processing unit (CPU), a microprocessor, a microcontroller, a digital signal processing (DSP) chip, a field programmable gate array (FPGA), etc. The functions of the processor 170 may also be implemented by an independent electronic device or an integrated circuit (IC), and the operation of the processor 170 may also be implemented by software.

[0033] It should be noted that the processor 170 may not be provided in the same device as the image capture device 110 and / or the display 150. However, the devices respectively equipped with the image capture device 110, the display 150 and the processor 170 may further include a communication transceiver with a compatible communication technology (e.g., Bluetooth, wireless fidelity (Wi-Fi), infrared (IR) or a physical transmission line) to transmit or receive data to each other. For example, the memory 130 and the processor 170 may be separated into a computing device. This computing device may be a desktop computer, a laptop computer, a server, a smart phone, or a tablet device. The image capture device 110 and the display 150 are also separated into another electronic device (e.g., a smart phone or a tablet), and the two devices can communicate wirelessly or wired.

[0034] Hereinafter, the method described in the embodiment of the present invention will be described with various components and modules in the matching model generation system 100. Each process of the method can be adjusted according to the implementation situation, and is not limited thereto.

[0035] Figure 2 Flow chart of the method for constructing a model of an architectural space according to an embodiment of the present invention. Figure 2 , the processor 170 can obtain one or more scanned images through the image capture device 110 (step S210). Specifically, it is assumed that the user wears VR / AR glasses or HMD equipped with the image capture device 110 on his / her head, or holds a scanner equipped with the image capture device 110. The user can move or rotate the image capture device 110 so that the image capture device 110 can capture in a specified direction to scan the building space (such as a room, office, etc.) and generate a scanned image accordingly. For example, Figure 3 is an example of an architectural space. Figure 3 , the user may use the image capture device 110 to photograph the walls w1 - w3 , the door d1 , and the floor f1 in the building space.

[0036] In the embodiment of the present invention, the image of the real environment within the field of view (FoV) captured by the image capture device 110 will be referred to as a scanned image, but is not limited thereto.

[0037] In one embodiment, the user may carry the image capture device 110 and walk around the building space arbitrarily or according to a preset route.

[0038] In some embodiments, if the main body of the image capture device 110 rotates a certain angle each time (for example, 10 degrees, 20 degrees, or 30 degrees along the x-axis, y-axis, or z-axis, where the x-axis, y-axis, and z-axis are perpendicular to each other), or if each preset time interval expires, the processor 170 may trigger the image capture device 110 to capture a scene image. For example, with respect to the construction of a 360-degree virtual environment, it is assumed that the viewing angle of each scene image is 30 degrees away from an adjacent scene image. The user may use the image capture device 110 to perform a 360-degree rotation horizontally, and then 12 or more scanned images may be obtained from the image capture device 110.

[0039] The processor 170 may identify the identification information of the marker in the scanned image and the positional relationship of the marker in the building space (step S230). Specifically, the marker is a physical object (e.g., paper, plastic board, etc.) formed by printing one or more words, symbols, patterns, shapes and / or colors.

[0040] Figure 4 is an example of a tag. Figure 4 In one example, the mark is an octopus pattern, and in another example, the mark is a QR code. It should be noted that the user can determine the appearance of the mark.

[0041] In addition, users can paste the markers at designated locations in the building space. It is worth noting that each marker represents an object. Objects can be walls, columns, beams, doors, windows, furniture, home appliances or other objects that may exist in the building space. Figure 3 For example, markers m1~m6 are respectively pasted on walls w1~w3, floor f1 and door d1.

[0042] In one embodiment, the identification information of the tag includes the object type and / or number. The object type is the type of object on which the tag is placed. For example, the object type is a wall, a column, a beam, a door, a window, furniture, a home appliance, etc. The processor 170 can interpret the appearance of the tag to identify the object type. The appearance of the tag may be formed by text, symbols, patterns, shapes and / or colors. The number can be used to distinguish multiple objects of the same object type. Different appearances can represent different object types and / or numbers. Figure 3For example, the appearance of the labels m1~m3 corresponding to walls w1~w3 is different. Or, different walls need to be labeled with different labels, for example, the living room wall is labeled with the label m1 of "wall 1", and the master bedroom wall is labeled with the label m2 of "wall 2". Figure 4 For example, octopus patterns or other animal patterns are used for furniture, and QR codes are used for wall decoration.

[0043] In some embodiments, the identification information of the tag may be encoded, such as a QR code, a barcode, a serial number, etc. The processor 170 may decode the tag based on the corresponding encoding mode.

[0044] In another embodiment, the identification information of the mark also includes the orientation of the mark. The orientation of the mark represents the orientation of the corresponding object in the building space. That is, the object has directionality. It is worth noting that most furniture or household appliances (e.g., refrigerators, televisions, sofas, beds, etc.) and even doors or windows have directionality. Directionality is related to the orientation of the front, back, top or other sides of the object in the building space. For example, the TV and refrigerator face away from the wall.

[0045] For example, Figure 5 A and Figure 5 B is an example of a directional mark. Figure 5 A, mark m7 indicates that the head of bed b1 faces northwest. Figure 5 B, mark m7 indicates that the head of bed b2 faces north. Figure 3 The marks m5 and m6 in the figure indicate that the top side and the bottom side of the door d1 are arranged perpendicular to the ground.

[0046] On the other hand, floors, ceilings, and walls may not have directionality, and users do not need to consider the orientation of the marker and can stick it on the object at will.

[0047] In some embodiments, the identification information of the tag needs to be decoded to obtain the orientation of the object in the building space. For example, "S" represents that the front of the refrigerator faces south.

[0048] In one embodiment, the processor 170 may determine the coordinates of the marker in the building space based on the depth information in the scanned image. For example, the depth information of a specific pixel / block in the scanned image obtains features (such as color, line, pattern, etc.). The features of these pixels / blocks will be mapped to specific three-dimensional space coordinates in the blank virtual environment based on the corresponding depth and position.

[0049] In another embodiment, the marker has a specific size and feature points. The processor 170 can determine the orientation of the marker in the building space based on the size and feature points of the marker, and the relative position and / or direction between the user and the marker when the scanned image is obtained. The processor 170 can obtain feature information in the scanned image based on image recognition technology (such as scale-invariant feature transform (SIFT), haar feature, Adaboost, support vector machine (SVM), or neural network algorithms). The feature information can be feature points and their positions, scales, rotation invariants, grayscale, color, directions, or other information set by feature extraction. The processor 170 can compare with pre-stored features corresponding to different positions and / or directions in the database, and derive the position and / or direction with higher similarity as the position and / or direction of the marker. Alternatively, the processor 170 can input the feature information extracted from the scanned image into a machine learning model trained based on the aforementioned database, and infer the position and / or direction of the marker accordingly.

[0050] In one embodiment, the processor 170 may derive the three-dimensional structure of the building space through Structure from Motion (SfM) based on the marked feature points, for example, by matching feature points between the scanned images of the sequence and estimating the three-dimensional structure accordingly.

[0051] It is worth noting that in order to ensure that any scanned image captures more than two markers, in one embodiment, the spacing of the markers can be predefined. The processor 170 can provide prompts about the spacing between multiple adjacent markers based on the specifications of the image capture device 110 (e.g., field of view, focal length, etc.) and the size of the building space. That is, the spacing of the markers is related to the range of the building space that the image capture device 110 can capture in one scanned image. During the scanning process, the processor 170 can present prompts (e.g., moving forward or backward toward the wall, or turning to a specific angle, etc.) through the display 150, so that the image capture device 110 can capture more than two markers each time, thereby obtaining enough feature points.

[0052] It should be noted that, in order to prevent the markers from being too close together, in some embodiments, a special tape may be provided to connect adjacent markers. The tape has a special pattern to assist SfM. The tape needs to be attached to the place where the image capture device 110 may scan to help positioning.

[0053] In one embodiment, the processor 170 can determine the current position of the user in the building space when the scanned image is obtained and the user's viewing angle of the mark according to the orientation of the mark. The user's position and viewing angle are used for subsequent browsing of the three-dimensional model, and will be described later.

[0054] In addition, the processor 170 may further derive a relative position relationship (or coordinate relationship) between the markers based on the coordinates of those markers.

[0055] In one embodiment, the position of the mark may represent an end point, a center, an edge, or any position on the object. Figure 3 For example, marks m5 and m6 indicate the positions of the top and bottom sides of the door d1.

[0056] In another embodiment, the processor 170 may determine the extensibility of the corresponding object in the building space according to the object type corresponding to the mark. The extensibility is related to the feature of extending outward from the location of the mark. The area of ​​object types such as floors, ceilings, walls, beams, columns, etc. may be much larger than the area of ​​the mark. The definition of the extensibility feature can assist the model generation system 100 in determining the boundary of the object. The outward extension can be any direction of the horizontal plane, any direction of the vertical plane, or a specified direction. Figure 3 For example, the mark m1 indicates that the wall w1 is extended in any direction from the horizontal plane (ie, the wall surface) at the position of the mark 1. The extensibility is used for subsequent modeling and will be explained later.

[0057] The processor 170 may construct a three-dimensional model of the building space according to the identification information and the position relationship (step S250). Specifically, the position of the marker may represent the position of the object, and some object types may also have directionality and / or extension. The processor 170 may use SfM to estimate the structure of the three-dimensional model, or generate a point cloud map of the three-dimensional model.

[0058] In addition to the structure, the processor 170 can further derive the object type at each position in the structure and its area and / or orientation. In one embodiment, for extensibility, the processor 170 can determine the boundary of the corresponding object in the dimensional model according to the extensibility. An object with extensibility means that its range extends outward from the marked position. For objects such as walls and beams, a closed loop can be formed, and the floor of the building can be retained.

[0059] In one embodiment, the marker includes a first marker and a second marker, and the object type of the first marker is a wall, a ground, or a ceiling (i.e., an extensible marker). The processor 170 can determine the termination point of the extension of the first marker (i.e., the boundary of a certain side) according to the object types of the first marker and the second marker. The first marker is adjacent to the second marker, and the termination point of the extension of the first marker is related to the intersection of the first marker and the second marker after the first marker is extended.

[0060] In one embodiment, the second mark is also a mark with extensibility (ie, its object type corresponds to an extensibility feature). Figure 3For example, two adjacent marks are walls w1 and w2. The two walls w1 and w2 extend toward each other from the positions of their marks m1 and m2 until they intersect. The intersection is the end point of the extension of the two walls w1 and w2.

[0061] In another embodiment, the second mark is a termination symbol (ie, its object type corresponds to the end of the extension), and the extension from the first mark to the position of the second mark stops.

[0062] In another embodiment, the second mark is not extensible (ie, its object type does not correspond to an extensible feature). Figure 3 For example, the mark m1 extends toward the left side of the figure to avoid the door d1 but the extension does not stop.

[0063] For the mark without extension, its position represents the position of the corresponding object (eg, upper left corner, center, lower right corner, etc.). The processor 170 may mark the positions of these objects in the structure.

[0064] In addition, for directional markers, the processor 170 may mark the orientation of these objects in the structure.

[0065] After the three-dimensional model is constructed, the processor 170 can present the three-dimensional model through the display 150. For example, the user can preview the three-dimensional model through a computer or a mobile phone. In one embodiment, since the position and viewing angle of the user taking the scanned image are known (i.e., located), the user wearing AR / VR glasses or HMD equipped with the display 150 can walk and / or turn in the actual building space and browse the three-dimensional model accordingly.

[0066] In one embodiment, the processor 170 may provide a user interface through the display 150 to receive the user's adjustment operation through an input device (e.g., a touch panel, a mouse, or a keyboard, etc.). The user interface may include options and values ​​for changing the style, material, pattern, color, and / or type of an object. The adjustment operation is related to selecting an option and changing a value. The processor 170 may change the style, color, material, and / or pattern of a specified object in the three-dimensional model according to the adjustment operation.

[0067] In some embodiments, the model generation system 100 can also provide voice-controlled adjustments. For example, Figure 3 The wall m1 is changed to yellow, and the door d1 is changed from a wooden door to a steel door. In other embodiments, the model generation system 100 can also provide an eye focus function. For example, the object that the user is looking at can be replaced (staring at the TV cabinet in front of the user, pressing a switch button, and switching to the next set of TV cabinets).

[0068] In practical applications, users can attach corresponding marks to structural objects such as ceilings, floors, walls, columns, doors, windows, etc. to help determine the structure of the three-dimensional model (i.e., the rough house). Then, the user can anticipate the location of furniture (e.g., beds, sofas, tables, etc.) or electrical appliances (e.g., refrigerators, washing machines, kitchen utensils, etc.) and attach corresponding marks to the wall or floor. Then, the user can change the paint color or veneer material of the wall, change the material or pattern of the door, or change the style of the window or curtains through the decoration software (the marked identifier is linked to the object in the software). In addition, the user can also change the style of the sofa (e.g., three-seater sofa, L-shaped sofa, etc.), change the brand or capacity of the refrigerator, and change the type of air conditioner (e.g., separate or independent).

[0069] In summary, in the model generation system and model construction method of the architectural space of the embodiment of the present invention, the markers can be placed on existing objects (e.g., walls, ceilings, etc.) or non-existent objects (e.g., expected furniture, electrical appliances, etc.) in the architectural space, and the identification information corresponding to these markers (e.g., type, directionality, extensibility, etc.) and the positional relationship in the space are obtained through image scanning, and the three-dimensional model of the architectural space is constructed accordingly. Since the objects in the three-dimensional model have specified types, the user changes the style, color or pattern of the specified object. In addition, since the relative position of the marker and the user has been recorded, the three-dimensional model can be browsed in real time through the AR / VR device after rendering. In this way, modeling and positioning can be quickly achieved, objects can be accurately identified, and a preview function for furniture or electrical appliance layout is provided.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a model of an architectural space, characterized in that: include: Obtaining a scanned image, wherein the scanned image is generated by scanning a building space; Identify identification information of a mark in the scanned image and a positional relationship of the mark in the building space, wherein the identification information includes an object type; as well as Constructing a three-dimensional model of the building space according to the identification information and the positional relationship, wherein the mark corresponds to an object in the three-dimensional model, wherein the step of constructing the three-dimensional model of the building space according to the identification information and the positional relationship comprises: Determine the extensibility of the corresponding object in the building space according to the object type, wherein the extensibility is related to a feature extending outward from the position of the mark, the mark includes a first mark and a second mark, the object type of the first mark is a wall, a ground or a ceiling, and the second mark is a termination symbol or has no extensibility; Determining a boundary of a corresponding object in the three-dimensional model according to the extensibility; and The end point of the extension of the first mark is determined according to the object types of the first mark and the second mark, wherein the first mark is adjacent to the second mark, and the end point of the extension of the first mark is related to the intersection of the first mark and the second mark after extension.

2. The architectural space model construction method according to claim 1, characterized in that: The identification information includes a number, and the step of identifying the identification information of the mark in the scanned image includes: The appearance of the mark is interpreted to identify the type of the object, wherein the appearance of the mark includes at least one of text, symbols, patterns, shapes or colors, and the number is used to distinguish multiple objects of the same object type.

3. The architectural space model construction method according to claim 2, characterized in that: The identification information also includes the orientation of the mark, and the orientation of the mark represents the orientation of the corresponding object in the building space.

4. The method for constructing a model of an architectural space according to claim 1, characterized in that: The step of identifying the positional relationship of the markers in the building space comprises: Determining the orientation of the mark in the building space according to the size and feature points of the mark; and The position of the user in the building space and the viewing angle of the user viewing the mark are determined according to the orientation of the mark, wherein the position and the viewing angle of the user are used to browse the three-dimensional model.

5. The method for constructing a model of an architectural space according to claim 1, characterized in that: After the step of constructing the three-dimensional model of the building space, the method further includes: Providing a user interface to receive user adjustment operations; and, At least one of the style, color, material, and pattern of the object in the three-dimensional model is changed according to the adjustment operation.

6. The method for constructing a model of an architectural space according to claim 1, characterized in that: Also includes: Hints are provided regarding the spacing between multiple adjacent markers based on the specifications of the image capture device and the architectural space.

7. A model generation system for architectural space, characterized in that: include: An image capture device, capturing an image; A memory storing program codes corresponding to the model building method used in the model generation system; A processor is coupled to the image capture device and the memory and is configured to execute the program code. The model construction method includes: Acquire a scanned image by the image capture device, wherein the scanned image is generated by scanning the building space; identifying identification information of a marker in the scanned image and a positional relationship of the marker in the building space, wherein the identification information includes an object type; and Constructing a three-dimensional model of the building space according to the identification information and the positional relationship, wherein the mark corresponds to an object in the three-dimensional model, wherein the step of constructing the three-dimensional model of the building space according to the identification information and the positional relationship comprises: Determine the extensibility of the corresponding object in the building space according to the object type, wherein the extensibility is related to a feature extending outward from the position of the mark, the mark includes a first mark and a second mark, the object type of the first mark is a wall, a ground or a ceiling, and the second mark is a termination symbol or has no extensibility; Determining a boundary of a corresponding object in the three-dimensional model according to the extensibility; and The end point of the extension of the first mark is determined according to the object types of the first mark and the second mark, wherein the first mark is adjacent to the second mark, and the end point of the extension of the first mark is related to the intersection of the first mark and the second mark after extension.

8. The architectural space model generation system according to claim 7, characterized in that: The identification information includes a serial number, and the model building method further includes: The appearance of the mark is interpreted to identify the type of the object, wherein the appearance of the mark includes at least one of text, symbols, patterns, shapes or colors, and the number is used to distinguish multiple objects of the same object type.

Images