Object restoration system using real-time location tracking technology of multi-view RGB-D camera using image markers

The system addresses the challenge of fixed camera positions in RGB-D cameras by employing real-time position tracking with image markers, enabling flexible camera adjustments for comprehensive 3D object scanning.

KR103046442B1Active Publication Date: 2026-11-11ELECTRONICS & TELECOMM RES INST
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
KR1020220013185
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2022-01-28
Publication Date
2026-11-11
Estimated Expiration
2042-01-28

AI Technical Summary

Technical Problem

Conventional RGB-D cameras struggle with precise reconstruction of 3D objects due to fixed camera positions, leading to incomplete reconstruction of object sides and backs.

Method used

An object restoration system using real-time position tracking technology with multi-view RGB-D cameras and image markers, allowing dynamic adjustment of camera positions and directions for accurate scanning.

Benefits of technology

Enables flexible camera configurations and efficient object restoration by transforming the coordinate system, facilitating complete 3D scanning of objects regardless of their orientation or surface.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112022011321770-PAT00001_ABST
    Figure 112022011321770-PAT00001_ABST
Patent Text Reader

Abstract

The present invention relates to an object restoration system utilizing real-time position tracking technology of a multi-view RGB-D camera using image markers. The present invention includes a plurality of clients that process images captured by each connected camera and transmit point cloud data calculated using an image marker reference coordinate system; and a server that stores integrated point cloud data synthesized from the point cloud data received from each of the clients.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present invention relates to a technology for scanning or restoring an object in three dimensions by placing multiple RGB-D cameras around an actual object to be restored, placing an image marker at the location of the object to be restored, calculating the position and orientation of the RGB-D cameras based on the coordinate system of the image marker, and then placing the object to be restored at the location of the image marker. Background Technology

[0002] Unlike general cameras that display only 2D image information, RGB-D cameras have the advantage of easily generating 3D objects by additionally including distance information for each point in the image. However, like general cameras, RGB-D cameras provide accurate RGB-D information for the front of the object to be reconstructed, but information for the sides or back is poor or non-existent; thus, while the front of the object can be reconstructed, the sides and back cannot. For this reason, devices that place multiple RGB-D cameras around an object to reconstruct it in 3D are being developed, but they have the disadvantage that precise object reconstruction is difficult because the camera positions are fixed. The problem to be solved

[0003] The present invention was devised to solve conventional problems and aims to provide an object restoration system utilizing real-time position tracking technology of a multi-view RGB-D camera using image markers, which enables precise object restoration by adjusting the position and direction of the camera to match the object to be scanned and then scanning the object.

[0004] The objectives of the present invention are not limited to those mentioned above, and other unmentioned objectives will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0005] An object restoration system utilizing real-time position tracking technology of a multi-view RGB-D camera utilizing image markers according to an embodiment of the present invention for achieving the above objective includes: a plurality of clients that process images captured by each connected camera and transmit point cloud data calculated using an image marker reference coordinate system; and a server that scans an object located at an image marker in three dimensions using integrated point cloud data synthesized from the point cloud data received from each of the clients. Effects of the invention

[0006] According to one embodiment of the present invention, unlike conventional fixed scanning equipment, the position or direction of the camera can be easily changed arbitrarily, and there is an effect of making it easy to add or remove a client system equipped with an RGB-D camera.

[0007] In addition, since the object is scanned by the marker coordinate system, the present invention has the effect of facilitating the transformation of the coordinate system. For example, if the image marker is rotated 90 degrees, the X-axis and Y-axis are reversed so that the object is scanned, and if the image marker is placed on a slightly inclined surface rather than a horizontal surface in marker scan mode, the object can be scanned at an angle.

[0008] In addition, the present invention allows for various camera configurations depending on the scanned object, thereby enabling efficient object restoration. Brief explanation of the drawing

[0009] FIG. 1 is a system configuration diagram for explaining an object restoration system utilizing real-time position tracking technology of a multi-view RGB-D camera utilizing image markers according to an embodiment of the present invention. Figure 2 is a flowchart of a program performed on the client shown in Figure 1. Figure 3 is a flowchart of a program executed on the server shown in Figure 1. FIG. 4 is an example of a client-side system in an image marker recognition mode in one embodiment of the present invention. FIG. 5 is an example of a client-side system in object scanning mode in one embodiment of the present invention. Specific details for implementing the invention

[0010] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Meanwhile, the terms used in this specification are for describing the embodiments and are not intended to limit the present invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text. The terms "comprises" and / or "comprising" as used in this specification do not exclude the presence or addition of one or more other components, steps, actions, and / or elements in addition to the mentioned components, steps, actions, and / or elements.

[0011] FIG. 1 is a system configuration diagram for explaining an object restoration system utilizing real-time position tracking technology of a multi-view RGB-D camera utilizing image markers according to an embodiment of the present invention.

[0012] As illustrated in FIG. 1, an object restoration system utilizing real-time position tracking technology of a multi-view RGB-D camera utilizing image markers according to one embodiment of the present invention includes a plurality of clients (200) and one server (100).

[0013] Each client (200) is connected to an RGB-D camera (201) and processes the image coming from the RGB-D camera (201) to convert point cloud data into an image marker reference coordinate system (204) and then transmits it to the server (100). The RGB-D camera (201) may be installed in multiple locations with different heights in the same direction. Here, the image marker may be a Charuco board consisting of a chessboard pattern containing a QR code, and a world coordinate system in which the Charuco board is provided on four sides may be used.

[0014] Figure 2 is a flowchart of a program performed by a client (200) of the present invention.

[0015] When the client (200) is executed, the execution of the marker recognition mode (210) is determined according to the user's selection.

[0016] If the marker recognition mode (210) is selected, an image marker (203) placed in the middle of the space to be scanned is recognized, the distance to the RGB-D camera (201) connected to the current client (200) is calculated based on the coordinate system of the image marker, and the degree of tilt is calculated.

[0017] On the other hand, if the marker recognition mode (210) is not selected, an object placed in the middle of the space to be scanned is scanned to generate three-dimensional point cloud data, and then the generated point cloud data is transmitted to the server (100).

[0018] In the marker recognition mode, the client (200) undergoes a marker selection step (S220) that allows the user to select a marker to use. Since each marker used in one embodiment of the present invention basically has information such as the size of the scan space built in, when scanning a small object or a large object, different markers can be selected to determine the size of the valid scan space.

[0019] In the marker recognition and camera pose calculation step (S230), the position of the image marker (203) in the RGB image captured by the RGB-D camera (201) is recognized by the selected marker, and the distance and tilt of the RGB-D camera from the reference coordinate system (204) of the recognized image marker is calculated.

[0020] The scan space setting step (240) is a step for setting the valid range of the space to be scanned, and is intended to remove point cloud data that is outside the valid range.

[0021] Basically, the marker selected by the user contains valid scan space information, but in the scan space setting step (S240), more precise settings can be made by the user.

[0022] The image marker space setting step (S240) sets a range of valid coordinate values ​​along the X, Y, and Z axes based on the origin of the image marker reference coordinate system (204).

[0023] Since marker recognition, marker selection, and scan space setting must be performed simultaneously by clients (200) connected to the server (100), control over the marker recognition mode (S210), marker selection (S220), and scan space setting (S240) is configured to be performed through the server (100).

[0024] When not in marker recognition mode, an RGB-D image acquisition step (S250) is performed to acquire an RGB-D image captured through an RGB-D camera (201), and then a local point cloud generation step (S260) is performed to create point cloud data from the RGB-D image. In this local point cloud data generation step (S260), a three-dimensional point cloud is generated using respective depth map information based on the RGB-D camera coordinate system.

[0025] Subsequently, in the global point cloud generation step (S270), the local point cloud data generated in the local point cloud generation step (S260) is converted into an image marker reference coordinate system (204). In this global point cloud data generation step (270), the local point cloud data is converted into global point cloud data based on the camera position and attitude information created in the camera pose calculation step (S230).

[0026] In the global point cloud filtering step (S280), point cloud data outside the valid spatial range for the X, Y, and Z axes set in the scan space setting step (S240) is removed.

[0027] In the global point cloud transmission step (290), the finally created global point cloud data is transmitted to the server (100).

[0028] In the present invention, three-dimensional point cloud data acquired from a specific camera is converted into the coordinate axes of a marker image and used. Converting the point cloud data into the coordinate axes of a marker image is possible by multiplying the matrix of the camera from which the point cloud was acquired (4x4 matrix) and the point cloud data (a vector composed of four elements: x, y, z, and 1 (three-dimensional coordinates and 1)).

[0030] The server (100) scans an object located at an image marker in three dimensions using a single point cloud data that synthesizes point cloud data received from each client (200). At this time, the server (100) shows the scanned object to the user and saves the point cloud data based on the user's selection.

[0031] FIG. 3 shows a flowchart of a program executed in a server (100) in one embodiment of the present invention.

[0032] In the global point cloud reception step (S110), three-dimensional point cloud data is received from the client (200).

[0033] The global point cloud data merging step (S120) is a step of merging global point cloud data received from multiple clients (200) into a single point cloud data. In this global point cloud data merging step (S120), the space set in the scan space setting step (S240) is represented by voxels of a size desired by the user along the X, Y, and Z axes, and then each 3D point is mapped to the voxel for processing. If two or more points are mapped to a single voxel, the color value of that voxel is used by averaging the color values ​​of all mapped 3D points.

[0034] The global point cloud data visualization step (S130) is a step of visualizing point cloud data expressed in voxel form to the user.

[0035] The global point cloud data storage step (S140) is a step performed when the user wants to store point cloud data, and stores point cloud data represented by a single voxel in a storage device.

[0036] FIG. 4 is an example of a client (200) side system in image marker recognition mode, and FIG. 5 is an example of a client (200) side system in object scanning mode.

[0037] In FIGS. 4 and FIGS. 5, a system was configured using four clients (200).

[0038] After selecting a marker recognition mode (S210), the user places an image marker where it can be seen from all RGB-D cameras (201), then performs a camera pose calculation step (S230) following a marker recognition step (S220) to find the position and orientation of each RGB-D camera on the marker reference coordinate system (203). In the marker reference coordinate system, the direction of the Z-axis is perpendicular to the image marker plane.

[0040] Although the configuration of the present invention has been described in detail with reference to the accompanying drawings, this is merely illustrative, and it is obvious that those skilled in the art can make various modifications and changes within the scope of the technical concept of the present invention. Accordingly, the scope of protection of the present invention should not be limited to the aforementioned embodiments but should be determined by the description in the following claims.

Claims

Claim 1 Multiple clients that process images captured by each connected camera and transmit point cloud data calculated using an image marker reference coordinate system; The multi-viewpoint utilizing image markers is characterized by including a server that scans an object where an image marker is located in 3D using integrated point cloud data synthesized from point cloud data received from each of the above clients, wherein the client determines whether to perform a marker recognition mode based on a user's selection, and if the marker recognition mode is selected, recognizes an image marker placed in the middle of the space to be scanned, calculates the distance from an RGB-D camera currently connected to the client based on the coordinate system of the image marker, sets the valid range of the space to be scanned, filters the global point cloud by removing point cloud data outside the valid space range for the set X, Y, and Z axes, and transmits the generated global point cloud data to the server, and if the marker recognition mode is not selected, acquires an RGB-D image through the connected RGB-D camera, generates local point cloud data from the RGB-D image, converts the generated local point cloud data into an image marker reference coordinate system, and converts the local point cloud data into global point cloud data based on the camera's position and attitude information, and since the marker basically has information such as the size of the scan space embedded, different markers are selected to determine the size of the valid scan space when scanning small or large objects. Object restoration system utilizing real-time position tracking technology of RGB-D cameras. Claim 2 An object restoration system utilizing real-time position tracking technology of a multi-view RGB-D camera using image markers, characterized in that, in claim 1, the client is connected to an RGB-D camera, processes an image coming from the RGB-D camera to convert point cloud data into an image marker reference coordinate system, and then transmits it to the server side. Claim 3 An object restoration system utilizing real-time position tracking technology of a multi-view RGB-D camera using image markers, characterized in that, in paragraph 2, the RGB-D cameras are formed in multiple numbers at different heights in the same direction and are installed in all directions. Claim 4 An object restoration system utilizing real-time position tracking technology of a multi-view RGB-D camera utilizing an image marker, characterized in that, in paragraph 2, the image marker reference coordinate system may utilize a world coordinate system in which the image marker is a charco board consisting of a chessboard pattern including a QR code, and the charco board is provided on four sides. Claim 5 The method comprises: a step in which a client determines whether to perform a marker recognition mode based on a user's selection; if the marker recognition mode is selected in the determination step, a step of recognizing an image marker placed in the middle of the space to be scanned; a step of calculating the distance to an RGB-D camera currently connected to the client based on the coordinate system of the image marker; a step of setting an effective range of the space to be scanned; a step of filtering a global point cloud by removing point cloud data outside the effective space range for the set X, Y, and Z axes; and a step of transmitting the generated global point cloud data to a server; wherein, if the marker recognition mode is not selected in the determination step, a step of acquiring an RGB-D image through a connected RGB-D camera; and a step of generating local point cloud data from the RGB-D image. An object restoration method utilizing real-time position tracking technology of a multi-view RGB-D camera utilizing image markers, further comprising the step of converting generated local point cloud data into an image marker reference coordinate system and converting the local point cloud data into global point cloud data based on the camera's position and attitude information; wherein the markers are characterized by having information such as the size of the scan space built-in by default, so that different markers are selected to determine the size of the valid scan space when scanning small or large objects. Claim 6 In claim 5, the step of generating the point cloud data is characterized by generating a three-dimensional point cloud using respective depth map information based on the RGB-D camera coordinate system, in an object restoration method utilizing real-time position tracking technology of a multi-view RGB-D camera utilizing image markers. Claim 7 delete Claim 8 delete Claim 9 An object restoration method utilizing real-time position tracking technology of a multi-view RGB-D camera utilizing image markers, comprising: receiving three-dimensional point cloud data from clients by the server; merging global point cloud data received from multiple clients into one point cloud data; visualizing point cloud data expressed in voxel form to a user; and storing point cloud data expressed in one voxel in a storage device if the user wishes to save the point cloud data. Claim 10 In claim 9, the step of combining with the point cloud data is characterized by representing the space set in the scan space setting step as voxels of a size desired by the user along the X, Y, and Z axes, and then processing each 3D point by mapping it to the voxel, wherein if two or more points are mapped to a single voxel, the color value of that voxel is the average of the color values ​​of all mapped 3D points, thereby providing an object restoration method utilizing real-time position tracking technology of a multi-view RGB-D camera utilizing image markers.

Citation Information

Patent Citations

  • Apparatus and method for generating point cloud

    KR1020200071565A

  • Method and system for generating target 3D point cloud in medical imaging system

    JP2021007739A

  • Apparatus and method for point cloud compression

    KR1020190089115A

  • A point cloud registration method based on RGB-D camera for shooting volumetric objects

    KR102206108B1