Method and apparatus for recognizing space, and recording medium for recording same

Abstract

A method for recognizing a space according to the present invention comprises: a first step of generating a spatial landmark map (SLM) which is a set of first local spatial landmark maps (LSLMs) in which a plurality of sub-spaces formed by dividing a certain space are expressed using relative coordinates between a plurality of objects included in the sub-spaces; a second step of acquiring an image of a specific space of a size that corresponds to the sub-spaces, and recognizing objects belonging to each frame of the image; a third step of obtaining a second local spatial landmark map expressed using relative coordinates of the objects recognized in the second step, and comparing the second local spatial landmark map with the spatial landmark map to identify the first local spatial landmark map most similar to the spatial landmark map; and a fourth step of identifying the specific space as identified in the third step.

Description

Spatial recognition method and device, and recording medium recording the same

[0001] The present invention relates to a method for recognizing a space in which a user is located. More specifically, the present invention relates to a method for recognizing a space in which a user is located by extracting spatial object information from a video sequence that has previously captured a space, converting the information into a relational topological graph, generating a spatial landmark map (SLM), which is a data structure, and constructing a local spatial landmark map (LSLM) from images captured at an actual location and comparing it with the previously generated SLM.

[0002]

[0003] Outdoor location recognition uses the Global Positioning System (GPS), which utilizes satellites. However, indoors, GPS signals can be weak and obstructed by indoor structures, increasing location recognition errors or even causing the system to malfunction.

[0004] To address these limitations, short-range wireless communication technologies like WiFi, Bluetooth, and Ultra Wide-Band (UWB) are increasingly being used indoors instead of GPS, as well as methods utilizing magnetic fields. These wireless communication technologies can suffer from reduced location accuracy due to variations in signal strength in indoor environments. Furthermore, methods utilizing magnetic fields require precise preliminary indoor measurement and are limited to use within the measured space.

[0005] Indoor positioning technologies utilizing computer vision apply image processing and artificial intelligence to captured images. These methods don't require complex infrastructure setup and avoid the multipath effects and signal blocking issues inherent in wireless signaling methods. Indoor positioning technologies utilizing computer vision can be broadly categorized into marker-based methods, which use fixed image patterns, and markerless methods, which don't use markers.

[0006] Markers are image patterns that can be individually recognized in advance through video and are distinct from other markers. QR codes, ARTag, AprilTag, and CALTag are commonly used. Furthermore, various forms, such as three-dimensional shapes and LED elements, are also possible. Markers can be placed in a pre-recognizable space, recognized from images, and used to determine indoor locations using the information they provide. Specifically, a user recognizes a marker, photographs it, and transmits the image to a server that stores the marker's location and related information. The server then provides the user's approximate location around the marker.

[0007] Markerless image-based indoor positioning technology estimates location by comparing captured images with pre-stored images or extracting image properties. Most markerless image-based indoor positioning methods build a database (DB) of image information for indoor positioning in advance. This database contains numerous images of actual indoor spaces, indoor maps, and 3D spatial information. Then, a learning model is created that can effectively extract matching items from the DB by referencing the captured images. In the actual application stage, the learning model is used to analyze the captured indoor images and extract images or information matching the database. The indoor location is estimated based on the extracted information.

[0008] Conventional methods utilizing wireless signals or computer vision focus on identifying indoor locations with the utmost accuracy. However, these methods are inherently limited by the inherent instability of indoor conditions, with positioning errors ranging from precise to extremely large.

[0009]

[0010] Accordingly, the present invention seeks to identify the area to which a user or terminal belongs rather than specifying a specific location indoors.

[0011]

[0012] In order to solve the above technical problem, the spatial recognition method of the present invention includes a first step of generating a spatial landmark map (SLM) which is a collection of first local spatial landmark maps (LSLMs) in which the detailed space is expressed as relative coordinates between a plurality of objects included in the detailed space, for each detailed space created by dividing a certain space into a plurality of parts, a second step of obtaining an image of a specific space having a size corresponding to the detailed space and recognizing objects belonging to each frame of the image, a third step of obtaining a second local spatial landmark map expressed as relative coordinates of objects recognized in the second step and identifying a first local spatial landmark map that is most similar to the first local spatial landmark map by comparing it with the spatial landmark map, and a fourth step of identifying the specific space according to what is identified in the third step.

[0013] The above first step includes a step of acquiring an image captured by a monocular camera while rotating 360 degrees at one point in the detailed space, and a step of extracting objects from each frame of the image, wherein the objects are expressed as a rectangular bounding box (RBB) formed to fit the size of the object while including the object.

[0014] The spatial recognition method of the present invention further includes a step of removing an error caused by a photograph taken by the monocular camera, wherein the error includes at least one of: an object appearing intermittently in non-consecutive frames; an object being detected multiple times in the same frame; and an object being only partially visible before and after the shape of the object is fully revealed.

[0015] The above relative coordinates are, the above relative coordinates are, respectively and Relative coordinates between two objects based on , If so, their relative coordinates are, It is expressed as follows, and is an object and In each of the frames that appear simultaneously, and is the average RBB coordinate.

[0016] In another embodiment of the present invention, a computing device and a recording medium are also disclosed.

[0017]

[0018] According to the present invention, since it targets space rather than indoor location recognition, which focuses on finding a specific point in a very narrow area, it can provide more stable performance. Depending on the size of the space, the accuracy of indoor recognition can be increased from a specific point to an area.

[0019]

[0020] Figure 1 is a flowchart illustrating a spatial recognition method of the present invention.

[0021] Figure 2 is a flowchart showing the specific process of the offline stage.

[0022] Figure 3 shows the change in RBB width according to the frame of the object by 360-degree rotation shooting.

[0023] Figure 4 shows the results of error correction according to Type 1, Type 2, and Type 3 in the data correction process.

[0024] Figures 5 to 7 are drawings illustrating examples of operations for constructing a spatial landmark map.

[0025] Figure 8 is a block diagram showing the configuration of the computing device of the present invention.

[0026]

[0027] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, detailed descriptions of well-known functions or components that may obscure the gist of the present invention will be omitted in the following description and the attached drawings. Additionally, throughout the specification, the term "including" a component does not exclude other components, unless specifically stated otherwise, but rather implies the inclusion of other components.

[0028] Additionally, while terms such as "first" and "second" may be used to describe various components, these components should not be limited by these terms. These terms may be used to distinguish one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

[0029] The terminology used herein is merely used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprise" or "have" indicate the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0030] Unless specifically defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by those of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning within the context of the relevant technology, and shall not be construed in an idealized or overly formal sense unless explicitly defined herein.

[0031]

[0032] The present invention can be divided into two stages: offline and online. The offline stage builds a spatial landmark map (SLM) containing objects for indoor spatial recognition. The online stage recognizes objects based on video or images captured by a camera in the space, generates a spatial landmark map corresponding to the image, and then identifies matching areas in the spatial landmark map built in the offline stage to recognize the corresponding space.

[0033] The spatial recognition method of the present invention includes a first step performed offline, and second and third steps performed online.

[0034] Figure 1 is a flowchart illustrating a spatial recognition method of the present invention.

[0035] In the first step (S10), for each detailed space created by dividing a space into multiple parts, a spatial landmark map (SLM), which is a collection of first local spatial landmark maps (LSLMs) in which the detailed space is expressed as relative coordinates between multiple objects included in the detailed space, is generated.

[0036] In the second step (S20), an image of a specific space of a size corresponding to the detailed space is acquired and objects belonging to each frame of the image are recognized.

[0037] In the third step (S30), a second local spatial landmark map expressed in the relative coordinates of the objects recognized in the second step is obtained, and the first local spatial landmark map that is most similar to the spatial landmark map is identified by comparison.

[0038] Finally, in the fourth step (S40), the specific space is identified based on what was identified in the third step.

[0039] In the present invention, the first step (S10) includes a step of acquiring an image captured by a monocular camera while rotating 360 degrees at one point in the detailed space, and a step of extracting objects from each frame of the image, wherein the objects can be expressed as a rectangular bounding box (RBB) formed to fit the size of the object while including the object.

[0040] The above first step (S10) further includes a step of removing an error caused by the shooting of the monocular camera, wherein the error includes at least one of: an object appearing intermittently in non-consecutive frames; an object being detected multiple times in the same frame; and only a part of the object being visible before and after the shape of the object is fully revealed.

[0041] In addition, the above relative coordinates are,

[0042] The above relative coordinates are, respectively and Relative coordinates between two objects based on , If so, their relative coordinates are,

[0043] It is expressed as follows, and is an object and In each of the frames that appear simultaneously, and is the average RBB coordinate.

[0044]

[0045] Hereinafter, the present invention will be described in more detail.

[0046] [Offline Stage (S10)]

[0047] The offline stage is the stage of constructing a spatial landmark map (SLM) aimed at recognizing the spatial area in which the user is located indoors. Depending on the size of the recognized space, the accuracy of indoor positioning can be provided. As illustrated in Fig. 2, the offline stage includes a data collection stage (S11), a correction stage (S13) for correcting the collected data, a calculation stage (S15) for calculating relative coordinates between objects, and a map construction stage (S17) for constructing an SLM based on the obtained relative coordinates.

[0048] In the present invention, the data collection step (S11) is a process of collecting data necessary to build an SLM for a specific space. This collection step (S11) further includes the following first through fifth steps.

[0049] The first step is to divide a space into at least two subspace regions. Each subspace must contain at least two objects, but there are no specific restrictions.

[0050] The second step is to install a monocular camera at a point in one of the detailed spaces defined in the first step, and rotate it 360 degrees to capture an image of one of the detailed spaces where the monocular camera is installed.

[0051] The third step extracts objects for spatial recognition from each frame of the captured video. Multiple objects can be recognized in a single frame, and the same object can be recognized across multiple frames. Each object recognized in a frame is represented by a rectangular bounding box (RBB) that is formed to fit the object's size and contains the object. An RBB is expressed as coordinates that indicate the pixel positions horizontally and vertically from the pixel at one end of the frame as coordinates (0,0). For example, the RBB coordinates of an object are RBB = {( , ), ( , )}, where {( , )class ( , ) are the coordinates of the pixels at the upper left and lower right corners of the RBB.

[0052] The fourth step separates and organizes identical objects in consecutive image frames. Among the objects recognized across multiple frames, the k-th object is selected. When this is said, it is expressed as the following mathematical expression 1.

[0053]

[0054]

[0055] In mathematical expression 1 is the total number of frames captured, is the total number of objects recognized in this subdomain. And is the RBB coordinate of the object expressed in mathematical expression 2 as information of this object in the n-th frame ( ) and RBB area ( ) as an element.

[0056]

[0057] Step 5 repeats steps 2 to 4 described above for each of the detailed spaces partitioned in step 1.

[0058] In the present invention, the data correction step (S13) is a process of removing noise from the data acquired in the previous step (S11).

[0059] Object by 360 degree rotation shooting ( ) An example of the change in RBB width according to the frame is shown in Fig. 3. In areas where there is no object, no value is displayed, but as part of the object is captured, the width value gradually increases. In frames where the entire object is visible, the RBB width has the shape of a downward convex arc, which is due to the characteristics of the monocular camera. Afterwards, the width decreases as the object is partially visible and then disappears.

[0060] As shown in Figure 3, incorrect recognition results can occur in Type 1, Type 2, and Type 3. Type 1 is when a non-existent object is incorrectly recognized. Type 2 is when both normal and incorrect objects are recognized in a single frame. Type 3 is when only a portion of an object appears, making accurate recognition uncertain. These errors are eliminated during the data correction process.

[0061] Type 1 error correction is performed as follows:

[0062] Since the camera rotates 360 degrees and continuously captures images, objects appear in successive frames. Intermittently appearing objects in non-consecutive frames, such as Type 1, are considered misrecognitions, and errors are corrected by removing them. An example of this application is shown below.

[0063] : Number of the j-th frame in which the object is recognized ( )

[0064] For all j,

[0065] If it is discontinuous, Remove information.

[0066] Here, λ is an integer greater than or equal to 1, and can be arbitrarily set during implementation.

[0067] Type 2 error correction is performed as follows:

[0068] This step corrects for Type 2 errors, where objects are detected multiple times in the same frame. In these cases, the RBB area of ​​incorrectly recognized objects differs from that of normal objects. Therefore, objects that exhibit significant differences in RBB area changes compared to objects recognized correctly in adjacent frames should be removed.

[0069] Type 3 error correction is performed as follows:

[0070] When a camera rotates 360 degrees while capturing an image, only parts of the object are visible before and after the object's full shape is revealed. These incompletely recognized objects can cause errors in real-world spatial recognition applications and therefore need to be removed. After removing Type 1 and Type 2 errors, peaks in the RBB area are formed around the boundaries of the correctly recognized frames, as shown in Figure 3. After obtaining the frame indices of these peaks, information about the objects in the frames before and after them is removed.

[0071] Figure 4 shows the results of error correction according to Type 1, Type 2, and Type 3 in the data correction process.

[0072] Error correction according to types 1 to 3 above is performed for all objects.

[0073]

[0074] At step S15, the relative distance between objects is calculated.

[0075] When the error correction process is performed, normally recognized results are produced for all recognized objects in the frames where all objects appear. -th object The width of the RBB is expressed as in mathematical formula 3.

[0076]

[0077] object A set of frames in which objects are normally recognized Then, it can be written as in mathematical equation 4.

[0078]

[0079] Here is an arbitrary constant, and is determined according to the implementation environment through experiments, etc. Two different objects and This set of frames recognized together Then, it can be written as in mathematical equation 5.

[0080]

[0081] If the number of frames within is greater than a certain number (ζ), i.e., The following process is performed. Here, ζ is a constant and is determined according to the implementation environment through experiments, etc.

[0082] to object and In each of the frames that appear simultaneously, and If we define the average RBB coordinates, they can be written as Equation 6 below from Equations 1 and 2.

[0083]

[0084] Here, q is and This is the total number of frames that appeared simultaneously.

[0085] From this, and In this situation where they appear simultaneously, each and Relative coordinates between two objects based on class If so, they are expressed together as in mathematical expression 7.

[0086]

[0087] For all simultaneously appearing objects, repeat the above process to obtain the relative coordinates between them.

[0088] In step S17, the SLM is configured based on the obtained relative coordinates.

[0089] A set of objects recognized in a detailed space S Then, it can be written as in mathematical equation 8.

[0090]

[0091] K here s is the total number of objects recognized in the subspace s, and s is the total number of subspaces.

[0092] In this detailed space s, a data structure like mathematical expression 9 is created using the relative coordinates between objects recognized.

[0093]

[0094] D S can be structured in various forms. Any form useful for spatial matching in the online stage is possible. Finally, the spatial landmark map for the detailed space s is defined as in Equation 10.

[0095]

[0096] As described above, a space can be composed of multiple sub-spaces, and a spatial landmark map for the entire space that synthesizes the local spatial landmark maps composed for each sub-space is composed as in mathematical expression 11.

[0097]

[0098] An application example of actually implementing the spatial landmark map described above is described below.

[0099] [Application Example]

[0100] In order to show an example of the offline step of the present invention, i.e., the operation of constructing a spatial landmark map, we will consider a case where the error-corrected RBB area of ​​objects recognized from a 360-degree image taken in a detailed space composed of four objects, as shown in FIG. 5, changes.

[0101] The set of frames of objects recognized simultaneously in Fig. 5 is obtained as shown in Fig. 6.

[0102] In Fig. 6, the relative coordinates between objects in each set are obtained, and the relationship between objects recognized is as shown in Fig. 7. Fig. 7 can be expressed in various formats.

[0103]

[0104] Hereinafter, the online stage of the present invention will be described.

[0105] The online stage is the process of recognizing space using the SLM built in the offline stage.

[0106] In the online phase, the user's space is identified based on video or images captured by the camera. The basic process for this is as follows.

[0107] Step 1: Take images with a camera in a specific space and recognize objects (O′) for each captured frame.

[0108] Step 2: Apply an offline method to each object to obtain the error-corrected relative position coordinates (D′).

[0109] Step 3: Referring to the SLM constructed offline, find LSLMn of detailed regions that contain all recognized objects (O′).

[0110] Step 4: Among the found LSLMn, find the element that provides the closest match to D′, and determine the space provided by this element as the recognition target.

[0111] In addition, various methods, such as K-NN (K-Nearest Neighbor), can be used to find the most matching part for the SLM′ = {O′,D′} recognized by the camera in the SLM obtained in Equation 11. From this, the space can be recognized.

[0112]

[0113] Figure 8 is a block diagram illustrating a computing device (800) that executes the spatial recognition method of the present invention described above. It reconstructs a series of processing steps according to the above-described identification method from the perspective of hardware configuration. Therefore, to avoid redundancy in explanation, only an outline of the functions and operations of each component will be provided.

[0114] The computing device (800) is configured to include a memory (810) in which programs for implementing the above-described spatial recognition method are stored, and a processor (820) for performing a series of operations for recognizing a space based on the programs, wherein the processor (820) performs operations including a first step of generating a spatial landmark map (SLM) which is a collection of first local spatial landmark maps (LSLMs) in which the detailed space is expressed as relative coordinates between a plurality of objects included in the detailed space, for each detailed space created by dividing a certain space into a plurality of parts, a second step of acquiring an image of a specific space having a size corresponding to the detailed space and recognizing objects belonging to each frame of the image, a third step of obtaining a second local spatial landmark map expressed as relative coordinates of objects recognized in the second step, and identifying a first local spatial landmark map that is most similar to the first local spatial landmark map by comparing it with the spatial landmark map, and a fourth step of identifying the specific space based on what is identified in the third step.

[0115] The above processor (820) obtains an image captured by a monocular camera while rotating 360 degrees at one point in the detailed space in the first process, and further performs an operation process of extracting objects from each frame of the image, and the objects are expressed as a rectangular bounding box (RBB) formed to fit the size of the object while including the object.

[0116] The processor (820) performs an operation to remove an error caused by the shooting of the monocular camera, wherein the error includes at least one of: an object appearing intermittently in non-consecutive frames; an object being detected multiple times in the same frame; and an object being only partially visible before and after the shape of the object is fully revealed.

[0117] The above relative coordinates are O, respectively k Wow O l The relative coordinates between two objects based on z (k,l) , z (l,k) If so, their relative coordinates are, It can be calculated as follows, and is an object O k Wow O l In the frames where O appears simultaneously, each O k Wow O l is the average RBB coordinate.

[0118] Meanwhile, the present invention described above can be implemented as computer-readable code on a computer-readable recording medium. Computer-readable recording media include all types of recording devices that store data that can be read by a computer system.

[0119] Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disks, and optical data storage devices. Furthermore, computer-readable recording media can be distributed across network-connected computer systems, allowing computer-readable code to be stored and executed in a distributed manner. Furthermore, functional programs, codes, and code segments for implementing the present invention can be readily inferred by programmers in the technical field to which the present invention pertains.

[0120] The present invention has been described above, focusing on various embodiments thereof. Those skilled in the art will appreciate that the present invention can be implemented in modified forms without departing from its essential characteristics. Therefore, the disclosed embodiments should be considered illustrative rather than limiting. The scope of the present invention is set forth in the claims, not the foregoing description, and all differences within the scope equivalent thereto should be construed as being encompassed by the present invention.

Claims

1. A first step of generating a spatial landmark map (SLM), which is a collection of first local spatial landmark maps (LSLMs) in which the detailed space is expressed by relative coordinates between a plurality of objects included in the detailed space, for each detailed space created by dividing a space into multiple parts; A second step of acquiring an image of a specific space of a size corresponding to the above detailed space and recognizing objects belonging to each frame of the image; A third step of obtaining a second local spatial landmark map expressed in relative coordinates of objects recognized in the second step, and identifying a first local spatial landmark map that is most similar to the second local spatial landmark map by comparing it with the second local spatial landmark map; and, A fourth step of identifying the specific space according to what was identified in the third step; A spatial recognition method comprising:

2. In paragraph 1, The above first step is, A step of acquiring an image captured by a monocular camera while rotating 360 degrees at one point in the above detailed space, A step of extracting objects from each frame of the above video, Including, The above objects are expressed as a rectangular bounding box (RBB) that is formed to fit the size of the object while containing the object.

3. In paragraph 2, Further comprising a step of removing errors resulting from shooting by the above monocular camera, A method for spatial recognition, wherein the above error includes at least one of: an object appears intermittently in non-consecutive frames, an object is detected multiple times in the same frame, or only part of an object is visible before and after the entire shape of the object is revealed.

4. In paragraph 2, The above relative coordinates are, respectively, and Relative coordinates between two objects based on , If so, their relative coordinates are, It is expressed as, and is an object and In each of the frames that appear simultaneously, and A spatial recognition method, which is the average RBB coordinate.

5. A recording medium that records the spatial recognition method described in any one of clauses 1 to 4 so that a computer can execute it.

6. Memory where programs are stored, A processor that performs a series of operations to recognize space based on the above program, The above processor, A first process of generating a spatial landmark map (SLM), which is a collection of first local spatial landmark maps (LSLMs) in which the detailed space is expressed by relative coordinates between a plurality of objects included in the detailed space, for each detailed space created by dividing a space into multiple parts, A second process of acquiring an image of a specific space of a size corresponding to the above detailed space and recognizing objects belonging to each frame of the image; A third process of obtaining a second local spatial landmark map expressed in relative coordinates of objects recognized in the second process and comparing it with the first local spatial landmark map to identify the most similar first local spatial landmark map, A fourth process of identifying the specific space according to what was identified in the third process above; A computing device that performs operations involving .

7. In paragraph 6, The above processor is a computing device that acquires an image captured by a monocular camera while rotating 360 degrees at one point in the detailed space in the first process, and further performs a computing process of extracting objects from each frame of the image, and the objects are expressed as a rectangular bounding box (RBB) formed to fit the size of the object while including the object.

8. In paragraph 6, The above processor is a computing device that performs an operation to remove an error caused by a photographing operation of the monocular camera, wherein the error includes at least one of: an object intermittently appearing in non-consecutive frames, an object being detected multiple times in the same frame, or only a part of an object being visible before and after the shape of the object is fully revealed.

9. In paragraph 6, The above relative coordinates are, respectively, and Relative coordinates between two objects based on , If so, their relative coordinates are, It is expressed as, and is an object and In each of the frames that appear simultaneously, and The average RBB coordinate of the computational unit.

Images