Method and apparatus for mapping mirror-based real-time dynamic projection

Abstract

A mirror-type real-time dynamic projection mapping method and apparatus are presented. A mirror-type real-time dynamic projection mapping method performed by a computer device according to an embodiment of the present invention comprises the steps of: irradiating an IR light source through an IR camera and providing a visible light projector light source using a projector; passing or reflecting the IR light source and the visible light projector light source using a dichroic mirror; and irradiating a light source at a desired location by controlling the light source passed or reflected by the dichroic mirror in at least one of the X-axis and Y-axis using the mirror, wherein the dichroic mirror can pass the IR light source and reflect the visible light projector light source.

Description

Technology Field

[0001] The embodiments of the present invention below relate to a mirror-type real-time dynamic projection mapping method and apparatus, and more specifically, to a mirror-type real-time dynamic projection mapping method and apparatus that tracks and maps a moving subject. Background Technology

[0002] Projection mapping is a technology that makes real-world objects appear to have different characteristics by projecting light-based images onto the surface of an object to alter it. Graphics implemented on objects through projection mapping overlay augmented reality images of the space, evoking a more expanded sense of space and realism.

[0003] Projection mapping specialized for large screens has given birth to a new form of expression technique. Projection mapping utilizes various types of surfaces as screens and is recently being widely used as stage equipment for performances. However, unlike this, projection mapping techniques used in performances have spatial limitations. By developing real-time object projection mapping that can compensate for these constraints on stage equipment and moving objects, a new type of performance exhibition system is proposed, enabling a response to diverse markets.

[0004] Recently, real-time vision machine-based sensing technology has diverse applications in the cultural technology industry, as it is a technology-intensive industry where cutting-edge science and technology are converged. In particular, the exhibition and performance industry is increasingly integrating advanced technologies to meet customer needs. Among these, there is a need for technological development to provide mapping functions to props, actors, and audiences by applying real-time object mapping solutions to stage sets, exhibition spaces, and theme parks utilizing projection mapping. Furthermore, there is an urgent need to develop new exhibition and stage performance devices utilizing immersive media that allow audience participation in live stage performances or experiential theme spaces.

[0005] As such, existing projection technology used for fixed objects is reaching its limits in terms of utility, requiring the development of technology capable of tracking and mapping moving subjects.

[0006] Korean Published Patent No. 10-2024-0103264 relates to a projection mapping system using such a digital art twin, and describes a technology regarding a projection mapping system using a digital twin based on a 3D object converted from a 2D image and the creator's intention. Prior art literature

[0007] Korean Patent Publication No. 10-2024-0103264 The problem to be solved

[0008] Embodiments of the present invention describe a mirror-type real-time dynamic projection mapping method and apparatus, and more specifically, provide a technology that enables response to various markets such as performance exhibitions and theme parks by developing a real-time dynamic projection mapping system based on dynamically moving objects.

[0009] Embodiments of the present invention provide a mirror-type real-time dynamic projection mapping method and apparatus that eliminates delay by developing hardware capable of tracking elements corresponding to real-time objects and performing ultra-high-speed image processing. Specifically, the invention involves the development of a real-time object image analysis and projection focusing control unit, an integrated operating system for a real-time object tracking system, and an image processing library for projection mapping, and the final result can implement a server for application in an indoor theme park. means of solving the problem

[0010] A mirror-type real-time dynamic projection mapping method performed by a computer device according to an embodiment of the present invention comprises: a step of irradiating an IR light source through an IR camera and providing a visible light projector light source using a projector; a step of passing or reflecting the IR light source and the visible light projector light source using a dichroic mirror; and a step of irradiating a light source at a desired location by controlling the light source passed or reflected by the dichroic mirror in at least one of the X-axis and Y-axis using a mirror, wherein the dichroic mirror can pass the IR light source and reflect the visible light projector light source.

[0011] The above dichroic mirror may have a front surface coated with a surface dielectric multilayer film and a back surface polished.

[0012] The step of illuminating a light source at the desired location can be performed by using a single mirror that can rotate along the X-axis and Y-axis to illuminate the light source at the desired location.

[0013] The step of illuminating the light source at the desired location can be performed by moving the mirror through servo control to shift the coordinates to the center in order to correct distortion of the projector image through the mirror.

[0014] The above servo control can perform servo control processing after calculating the relative coordinates to move to the extracted coordinates.

[0015] The step of irradiating a light source at the desired location above can provide an image distortion device integrated with a UV map during center point coordinate correction and up, down, left, and right coordinate movement using an absolute coordinate extraction camera capable of correcting the difference between absolute coordinates and measured coordinates.

[0016] The step of illuminating a light source at the desired location above can be achieved by projecting the deformation of the UVMap through absolute coordinate extraction onto a projector screen via Unity's real-time renderer, and controlling the mirror through servo control using the coordinate values ​​of a high-speed camera for real-time coordinate extraction of the object to enable real-time mapping.

[0017] The step of illuminating a light source at the desired location can be performed by inserting a virtual frame into the frame difference between the high-speed IR camera and the projector to predict the direction of the coordinates and calculate the latency, thereby enabling servo control in advance.

[0018] A mirror-type real-time dynamic projection mapping device according to another embodiment of the present invention comprises: a light source providing unit that irradiates an IR light source through an IR camera and provides a visible light projector light source using a projector; a light source transmission and reflection unit that passes or reflects the IR light source and the visible light projector light source using a dichroic mirror; and a servo control unit that irradiates a light source at a desired location by controlling the light source passed or reflected by the dichroic mirror in at least one of the X-axis and Y-axis using the mirror, wherein the dichroic mirror can pass the IR light source and reflect the visible light projector light source.

[0019] The above servo control unit can irradiate a light source at a desired position using a single mirror that can rotate along the X and Y axes.

[0020] The above servo control unit can move the mirror to the center by moving the mirror through servo control to correct distortion of the projector image through the mirror.

[0021] The above servo control can perform servo control processing after calculating the relative coordinates to move to the extracted coordinates.

[0022] The above servo control unit can provide an image distortion device integrated with a UV map during center point coordinate correction and up, down, left, and right coordinate movement by using an absolute coordinate extraction camera capable of correcting the difference between absolute coordinates and measured coordinates.

[0023] The above servo control unit can project the deformation of the UVMap through absolute coordinate extraction onto the projector screen via Unity's real-time renderer, and control the mirror through servo control using the coordinate values ​​of the high-speed camera for real-time coordinate extraction of the object to enable real-time mapping.

[0024] The above servo control unit can predict the direction of the coordinates and calculate the latency by inserting a virtual frame into the frame difference between the high-speed IR camera and the projector, and perform servo control in advance. Effects of the invention

[0025] Embodiments of the present invention can provide a mirror-type real-time dynamic projection mapping method and apparatus that is free from delay by developing hardware capable of tracking elements corresponding to real-time objects and performing ultra-high-speed image processing. Brief explanation of the drawing

[0026] FIG. 1 is a diagram illustrating an image processing method of a camera controller module and an image processing module according to an embodiment of the present invention. FIG. 2 is a diagram showing the image processing result according to one embodiment of the present invention. FIG. 3 is a diagram illustrating camera image coordinate processing through a dichroic mirror according to an embodiment of the present invention. FIG. 4 is a flowchart illustrating a mirror-type real-time dynamic projection mapping method according to an embodiment of the present invention. FIG. 5 is a block diagram showing a mirror-type real-time dynamic projection mapping device according to an embodiment of the present invention. FIG. 6a is a drawing for explaining a mirror-type real-time dynamic projection mapping device according to an embodiment of the present invention. FIG. 6b is a diagram showing that the center direction of an IR camera and a projector coincides according to one embodiment of the present invention. FIG. 7 is a drawing showing a dichroic mirror according to one embodiment of the present invention. FIG. 8 is a drawing for explaining a dynamic mapping projector utilizing a mirror according to an embodiment of the present invention. FIG. 9 is a drawing showing a mirror-type real-time dynamic projection mapping device according to an embodiment of the present invention. FIG. 10 is a drawing illustrating the use of a single mirror according to one embodiment of the present invention. FIG. 11 is a drawing for explaining image distortion of a projector screen according to one embodiment of the present invention. FIG. 12 is a drawing for explaining image distortion of a camera screen according to one embodiment of the present invention. FIG. 13 is a diagram illustrating coordinate movement to the center according to one embodiment of the present invention. FIG. 14 is a diagram showing a camera and a UV Map for coordinate extraction according to an embodiment of the present invention. FIG. 15 is a diagram illustrating the image analysis and UV map processing process of Unity according to an embodiment of the present invention. FIG. 16 is a drawing showing a real-time renderer of the Unity engine according to one embodiment of the present invention. FIG. 17 is a diagram illustrating mirror control and UV map operation according to one embodiment of the present invention. FIG. 18 is a drawing for explaining an image according to absolute coordinates and relative coordinates according to an embodiment of the present invention. FIGS. 19 and FIGS. 20 are drawings for explaining a servo control processing method according to an embodiment of the present invention. Specific details for implementing the invention

[0027] Embodiments of the present invention will be described below with reference to the attached drawings. However, the described embodiments may be modified in various different forms, and the scope of the present invention is not limited by the embodiments described below. Furthermore, various embodiments are provided to more fully explain the present invention to those skilled in the art. In the drawings, the shapes and sizes of elements may be exaggerated for clearer explanation.

[0029] For real-time object video tracking, a vision camera with a high FPS capable of capturing images to detect fast-moving objects is required; therefore, a camera with 800 FPS or higher (e.g., an IR camera) is selected, and a USB 3.0 interface or higher is required to secure bandwidth for real-time processing of high-FPS camera images. For instance, in the case of Golfzon's instant capture camera products, the initial model was 850 FPS, and based on instant capture processing speed, it is currently used at 2,000 FPS.

[0030] The IR band is used to resolve light source noise from the projector, and the provided lens and 820nm IR filter can be added. The normal operation of the IR lens can be verified, for example, by taking a picture indoors to check if only the IR light source is displayed in the image.

[0032] The image processing module can be classified into a camera controller module, an image processing module, and a servo controller module.

[0033] FIG. 1 is a diagram illustrating an image processing method of a camera controller module and an image processing module according to an embodiment of the present invention.

[0034] Referring to FIG. 1, the camera controller module may include a camera connection module (110) and a camera image grab module (120). The camera image grab module (120) performs parallel processing and SIMD operations for high-speed computation, and the data values ​​obtained from the camera can be converted into a matrix form for high-speed processing. Here, the camera may be an IR camera, measured at an average speed of 1.00 to 1.25 ms, and the image read time is approximately 1.25 ms based on 800 FPS.

[0035] The camera controller module can acquire an image in response to an image request from the camera (hardware) on the stream and measure the time taken until image conversion.

[0036] The image processing module may comprise an image preprocessing module (130), an image processing module (140), an object parameter extraction module (150), and an image postprocessing module (160).

[0037] The order of the image processing module read from the hardware (camera) can be as follows: image preprocessing, image processing, and parameter extraction followed by image postprocessing. Here, image preprocessing may utilize threshold (Adaptive_ThresHold) processing and / or Gaussian processing methods.

[0038] FIG. 2 is a diagram showing the image processing result according to one embodiment of the present invention.

[0039] Referring to Fig. 2a, the result of image preprocessing is shown, and referring to Fig. 2b, the result of image preprocessing using a Gaussian processing method is shown. In particular, the changes in the preprocessed image according to the Gaussian block size are, in order: original, 3, 5, 7, 9, and 11. At this time, if the block size becomes too large, the ratio of averaging the values ​​in the periphery increases, which has the disadvantage that the classification of the boundary line is not clear, so it can be confirmed that a value between 7 and 15 is appropriate.

[0040] The servo controller module may include a servo connection module, a servo control module, a servo status check module, and a parameter value preprocessing module. For example, the servo controller module can control the X-axis and Y-axis movement of the mirror described below.

[0041] The following describes parameter extraction for a servo controller.

[0042] FIG. 3 is a diagram illustrating camera image coordinate processing through a dichroic mirror according to an embodiment of the present invention.

[0043] Referring to FIG. 3, camera image coordinate processing can be performed by extracting intrinsic parameters based on images detected through each camera, and boundary detection can be performed by separating pixel values ​​that are distinguished from the surrounding background through contours. An IR light source can be irradiated using a dichroic mirror, and an IR image can be processed through the dichroic mirror.

[0044] The image processing method involves performing image preprocessing, object detection and tracking on the preprocessed image, extracting relevant parameters from the detected objects, performing image post-processing, and utilizing these as reference data for subsequent object tracking. Here, the detected object parameters can be processed as Global Events so that they can be used in other systems.

[0046] FIG. 4 is a flowchart illustrating a mirror-type real-time dynamic projection mapping method according to an embodiment of the present invention.

[0047] Referring to FIG. 4, a mirror-type real-time dynamic projection mapping method performed by a computer device according to an embodiment of the present invention may include the steps of: irradiating an IR light source through an IR camera and providing a visible light projector light source using a projector (S410); passing or reflecting the IR light source and the visible light projector light source using a dichroic mirror (S420); and irradiating a light source at a desired location by controlling the light source passed or reflected by the dichroic mirror in at least one of the X-axis and Y-axis using the mirror (S430). Here, the dichroic mirror may refer to a mirror capable of passing an IR light source and reflecting a visible light projector light source.

[0048] A mirror-type real-time dynamic projection mapping method according to one embodiment of the present invention can be explained in more detail by exemplifying a mirror-type real-time dynamic projection mapping device according to one embodiment of the present invention.

[0050] FIG. 5 is a block diagram showing a mirror-type real-time dynamic projection mapping device according to an embodiment of the present invention.

[0051] Referring to FIG. 5, a mirror-type real-time dynamic projection mapping device (500) according to one embodiment of the present invention may comprise a light source providing unit (510), a light source transmission and reflection unit (520), and a servo control unit (530).

[0052] In step (S410), the light source providing unit (510) can irradiate an IR light source through an IR camera and provide a visible light projector light source using a projector.

[0053] In step (S420), the light source transmission and reflection section (520) can use a dichroic mirror to pass through or reflect the IR light source and the visible light projector light source.

[0054] A dichroic mirror is a reflector composed of many thin layers of materials with different refractive indices, possessing the property of reflecting light of certain colors while transmitting all other colors. Compared to ordinary color filters, dichroic mirrors have very low loss due to absorption and feature the ability to adjust the wavelength range of selectively reflected light based on the thickness or structure of the material.

[0055] Here, a dichroic mirror may refer to a mirror capable of passing an IR light source through and reflecting a visible light projector light source, and may include a dichroic film, a filter, a mirror including a filter, a bandpass filter, etc. For example, the front surface of the dichroic mirror may be coated with a surface dielectric multilayer film, and the back surface may be a polished surface.

[0056] In other words, an IR light source can be irradiated using a dichroic mirror that passes the IR light source through while reflecting visible light, and an IR image can be processed through the dichroic mirror.

[0057] In step (S430), the servo control unit (530) can direct the light source that has passed or been reflected by the dichroic mirror to a desired position by controlling the light source in at least one of the X-axis and Y-axis using the mirror.

[0058] The servo control unit (530) can irradiate a light source at a desired location using a single mirror that can rotate along the X-axis and Y-axis. Meanwhile, when using separate X-axis and Y-axis mirrors, the projector light source is diffused by the mirror that can rotate along the X-axis and then diffused by the mirror that rotates along the Y-axis; in this case, the size of the mirror that rotates along the Y-axis must be enlarged due to the diffusion.

[0059] The servo control unit (530) may use, for example, a 2-axis gimbal structure and a 3-axis gimbal structure. Here, a ball bearing servo may be used considering the load of the rotation axis.

[0060] In this mirror-type real-time dynamic projection mapping device, image distortion occurs as objects move because the light emitted by the projector spreads, and awkwardness arises due to time delay. To resolve this, the issue of image distortion can be addressed by predicting future positions while considering the distortion and performing image correction in advance. Additionally, the awkwardness caused by time delay can be mitigated by performing video analysis and image correction (servo control) at the time interval between two adjacent frames of the projector. In this case, image correction can be performed two or more times.

[0061] The servo control unit (530) can move the mirror to the center by moving the mirror through servo control to correct distortion of the projector image through the mirror. The servo control can perform servo control processing after calculating the relative coordinates to be moved to the extracted coordinates.

[0062] The servo control unit (530) can provide an image distortion device integrated with a UV map when correcting center point coordinates and moving up, down, left, and right coordinates using an absolute coordinate extraction camera capable of correcting the difference between absolute coordinates and measured coordinates.

[0063] The servo control unit (530) can project the deformation of the UVMap through absolute coordinate extraction onto the projector screen via Unity's real-time renderer, and control the mirror through servo control using the coordinate values ​​of the high-speed camera for real-time coordinate extraction of the object to enable real-time mapping.

[0064] The servo control unit (530) can predict the direction of the coordinates and calculate the latency by inserting a virtual frame into the frame difference between the high-speed IR camera and the projector, and can perform servo control in advance.

[0065] A mirror-type real-time dynamic projection mapping method and apparatus according to one embodiment of the present invention will be described in more detail below.

[0067] FIG. 6a is a drawing for explaining a mirror-type real-time dynamic projection mapping device according to an embodiment of the present invention, and FIG. 6b is a drawing showing that the center direction of an IR camera and a projector coincides according to an embodiment of the present invention.

[0068] Referring to FIG. 6a, to configure a mirror-type real-time dynamic projection mapping device, a projector (610) that emits a visible light projector source and an IR camera (620) that emits an IR light source are configured, and an IR light source is emitted using a dichroic mirror (630), and an IR image can be processed through the dichroic mirror (630). Then, the light source passed or reflected by the dichroic mirror (630) can be controlled in at least one of the X-axis and Y-axis using a mirror (640) to emit the light source at a desired location.

[0069] In this way, by using a dichroic mirror (630) that matches the projection direction of the projector (610), the side IR camera (620) can act like a mirror while reflecting.

[0070] This is a sensing method that includes structural content, where the projection mapping viewpoint and the vision sensor viewpoint are identical.

[0071] Referring to Fig. 6b, it shows that the center direction of the IR camera and the projector are aligned, and this can be utilized for real-time stage equipment mapping, etc. In addition, mapping through the movement of actors is smooth and has little error, and there is little error when using live object mapping images or videos.

[0072] By analyzing the outline of an object, it is possible to set 2D and 3D image mappings that match the outline and mapping images by selection, and accordingly, an automatic conversion device such as shading and shadows can be provided.

[0073] FIG. 7 is a drawing showing a dichroic mirror according to one embodiment of the present invention.

[0074] Referring to FIG. 7, the dichroic mirror (700) can reflect (mirror) visible light and allow (transmit) IR light sources to pass through. At this time, the front surface (710) of the dichroic mirror (700) may be coated with a surface dielectric multilayer film, and the back surface (720) may be made of a polished surface.

[0075] Here, the dichroic mirror (700) may refer to a bandpass filter, a dichroic filter, a mirror including a filter, etc., and may reflect certain parts and absorb other specific parts.

[0077] The following describes the application of a dynamic mapping projector utilizing a mirror. Here, dynamic projection refers to the application of SKULL mapping equipment or a real-time dynamic mapping system according to the present embodiment.

[0078] FIG. 8 is a drawing for explaining a dynamic mapping projector utilizing a mirror according to an embodiment of the present invention.

[0079] Referring to Fig. 8a, the case where the X-axis mirror and Y-axis mirror are used separately is shown. In this case, the projector beam is diffused, so the X-axis image and the Y-axis image are different, and it is difficult to match the size of the mirrors.

[0080] Referring to Fig. 8b, in the case where the X-axis mirror and the Y-axis mirror are used separately, the rotation of the mirror by the axis of rotation is either the X-axis or the Y-axis. Here, the projector light source is diffused by the mirror rotatable along the X-axis and then diffused by the mirror rotatable along the Y-axis. In this case, the mirror rotatable along the Y-axis must have its size enlarged due to the diffusion.

[0081] Referring to FIG. 8c, when using a single mirror capable of XY-axis rotation, two axes of the same rotation axis can be used. In this case, the XY axes can be operated at the same point, and, for example, as shown in FIG. 8d, a 2-axis gimbal structure and a 3-axis gimbal structure can be used.

[0082] Accordingly, in this embodiment, a device capable of real-time mapping through projection via a mirror or coordinate transformation of a projector image can be used, and an object tracking device through reflection of a 2-axis mirror can be used.

[0084] FIG. 9 is a drawing showing a mirror-type real-time dynamic projection mapping device according to an embodiment of the present invention.

[0085] Referring to FIG. 9, a focusing section through a dichroic mirror (930) is provided, and a test of the lens distance and aperture with respect to the focal length of the projector (910) is performed to provide a servo control section of the optical mirror (940).

[0086] At this time, considering that the beam projection angle of the projector (910) is diffuse, the size of the mirror (940) can be expanded and adjusted. As explained above, if a single mirror (940) capable of rotating along the XY axis is used instead of using separate XY mirrors, the size of the mirror (940) may not need to be expanded even if the beam projection angle of the projector (910) is diffuse. Here, 920 is an IR camera.

[0087] In such mirror-type real-time dynamic projection mapping devices, image distortion occurs as objects move because the light emitted by the projector spreads, and awkwardness occurs due to time delay.

[0088] To address this, image distortion can be resolved by predicting future positions while considering image distortion and performing image correction in advance. Additionally, awkwardness caused by time delay can be addressed by performing video analysis and image correction (servo control) at the time interval between two adjacent frames of the projector. In this case, image correction can be performed two or more times.

[0089] FIG. 10 is a drawing illustrating the use of a single mirror according to one embodiment of the present invention.

[0090] Referring to FIG. 10, the X and Y axis servo design can be changed with a single mirror by taking into account the reflectivity of the mirror. By switching the design from the X and Y axis mirrors to a gimbal type that takes direction into account, it can be made to match the projection direction of the projector. For example, a 2-axis gimbal structure and a 3-axis gimbal structure can be used. Here, a ball bearing servo can be used to account for the load on the rotation axis.

[0092] FIG. 11 is a drawing for explaining image distortion of a projector screen according to one embodiment of the present invention.

[0093] Referring to Fig. 11a, an actual projector (screen) image is shown. This indicates that projector image distortion (UVMap) through a mirror occurs as illustrated in Fig. 11b.

[0094] FIG. 12 is a drawing for explaining image distortion of a camera screen according to one embodiment of the present invention.

[0095] Referring to FIG. 12a, which shows the actual camera screen, image distortion occurs as illustrated in FIG. 12b. Consequently, since the image is not centered, the image can be moved to the center by shifting the XY coordinates through servo control. In other words, the UVMap can be modified by looking at the absolute XY coordinates.

[0096] FIG. 13 is a diagram illustrating coordinate movement to the center according to one embodiment of the present invention.

[0097] Referring to FIG. 13a, the screen coordinates viewed from the IR camera are shown, and as illustrated in FIG. 13b, the coordinates can be moved to the center through mirror movement via servo control.

[0098] In this way, distortion (UVMap) of the projector image through the mirror occurs, and an automatic correction program using the coordinate values ​​of the space where the original image is to be displayed can be used by utilizing an image correction program to correct this.

[0100] By developing GPU-based embedded image processing technology for high-speed image processing, and providing a general-purpose image port for the camera interface, an image analysis algorithm for object tracking is applied and metadata for servo control is converted to generate control data for projector lens control by calculating distance using TOF.

[0101] By applying AI deep learning algorithms, a Kalman filter can be applied to improve processing speed and minimize latency. To enable X / Y axis control within 1ms for real-time control, a dichroic filter can be used to align the visible light view of the projector image with the camera view to minimize the error range.

[0102] The development of a TOF distance sensor linkage and projector focusing motor linkage for object distance measurement involves designing a control unit for motor control tailored to projector focusing based on the calculated distance and a TOF sensor linkage for object distance measurement; this enables synchronization for detection through ultra-high-speed control.

[0103] A moving yoke for moving subjects, utilizing tracking-based path control technology, can move along the X-axis at a speed of 0.66 cm / sec to expand the object tracking area and minimize vibration, thereby enabling a moving rail control device for tracking object movement in a stage space. By incorporating a detachable enclosure design that maintains structural compatibility with the projector's fixed mount, the accuracy of object coordinate analysis has been enhanced through improved processing speed of the vision machine camera. This has reduced the latency of the XY mirror, enabling real-time control.

[0105] The development of a monitoring UI screen for identifying the movement of objects was carried out simultaneously with the module development, and the development of a video synchronization processing module for synchronizing the performance between the mapping video and the show control system was implemented by utilizing a development tool called Touch Designer and adopting the MDX interface and timeline synchronization for synchronization.

[0106] We implemented scene direction guides and authoring software for realistic live performances and exhibitions, and implemented UI / UX scene direction tools.

[0107] It was developed as a UI program to efficiently manage operations by generating 2D object modeling data for object interaction suitable for characteristics and converting 3D image data for an image processing library for projection mapping.

[0108] Below, image processing for projection mapping is described.

[0109] Since the development of modules for TOF distance sensor integration and projector focusing motor integration for measuring the distance of an object must be prioritized, a TOF distance sensor and an interface part are provided, and a TOF camera module with higher resolution than a simple distance sensor can be applied to apply 3D images of the object.

[0110] By analyzing object types and converting situational image data according to viewing angles, a method of creating an optical illusion by changing the viewpoint in real time based on the mapping video image attribute data according to projector resolution is mastered, thereby providing the actual viewer with the feeling of being mapped in 3D.

[0111] By applying image extraction information obtained through 3D mapping data conversion and library creation, situational object image mapping data for image output can be generated and connected to a real-time projector to be applied to the target.

[0112] At this time, as a result of rotating the mirror, the UVMap was distorted by the rotation of the mirror.

[0113] FIG. 14 is a diagram showing a camera and a UV Map for coordinate extraction according to an embodiment of the present invention.

[0114] Referring to FIG. 14, it can be seen that the shape of the UVMap (1430) resulting from rotation and diffusion is shown, and that absolute coordinate values ​​of the screen are required. For this reason, at least two cameras (1410, 1420) are required, and two cameras can be used: a camera (1420) for extracting the coordinates of a high-speed object and rotating the mirror, and a low-speed camera (1410) for extracting the absolute coordinates of the object on the projector (screen).

[0115] The UVMap (1430) can be determined by using the first absolute coordinate camera (1410) to distort the UVMap (1430) in the opposite direction. At this time, the absolute coordinate point correction is required, and the vector transformation can be distorted in Unity.

[0116] FIG. 15 is a diagram illustrating the image analysis and UV map processing process of Unity according to an embodiment of the present invention.

[0117] Referring to Fig. 15, for real-time object coordinate analysis, the object coordinate calculation can be processed using the time difference between a high-speed camera, i.e., 815 FPS, and a projector refresh frequency of 60 Hz.

[0118] As shown in the formula 60Hz * 13 Frames = 780 FPS, the movement and direction of movement of an object can be predicted by analyzing 6 frames of images using the gap of 13 frames, and projection mapping to the object can be completed through servo control by considering the mirror movement time during the remaining 7 frames.

[0119] FIG. 16 is a drawing showing a real-time renderer of the Unity engine according to one embodiment of the present invention.

[0120] Referring to Fig. 16, an algorithm can be provided that distorts the image image through a real-time mesh renderer and transmits it to a projector using the UV map reference values ​​of the image image through object coordinate analysis data and the absolute coordinate camera coordinate values.

[0121] It can operate using two cameras for absolute coordinates and object tracking, a method using a camera for high-speed object tracking and a camera for absolute coordinates to correct distortion, or a method of projecting an image onto an object through mirror rotation control and tracking it at high speed. In this case, distortion occurs due to the mirror rotation method.

[0122] FIG. 17 is a drawing for explaining mirror control and UV map operation according to an embodiment of the present invention, and FIG. 18 is a drawing for explaining an image according to absolute coordinates and relative coordinates according to an embodiment of the present invention.

[0123] Referring to Fig. 17, the deformation of the UVMap through absolute coordinate extraction is projected onto a projector screen via Unity's real-time renderer, and real-time mapping can be achieved by controlling the servo mirror with the coordinate values ​​of a high-speed camera for real-time coordinate extraction of the object.

[0124] In this way, by using an absolute coordinate extraction camera capable of correcting the difference between absolute coordinates and measured coordinates, an image distortion device integrated with a UV map can be provided for center point coordinate correction and vertical and horizontal coordinate movement.

[0126] FIGS. 19 and FIGS. 20 are drawings for explaining a servo control processing method according to an embodiment of the present invention.

[0127] The servo control processing method can be explained through the mechanical analysis of the projector mount and focal length testing.

[0128] It is necessary to enlarge the mirror size, and to drive this, a 12-bit servo motor is selected. A servo with a resolution of 0.087 degrees and acceleration / deceleration functions for mirror load can be used. An ARM Cortex-based servo motor is used with a basic data transmission speed (Baud Rate): 9,600 [bps] ~ 4.5 [Mbps], and the resolution can be 0.0878 degrees per Tick (360 degrees / 4096). The measurement and control unit is coded by dividing the X and Y coordinates, and the control unit processes the center point parameter coordinate values ​​received from the image processing module. The received parameter values ​​are converted into the servo motor's resolution values ​​(0~4096) to calculate the Tick value, and a module for processing servo control and real-time position verification of the coordinates moved according to the servo control can be implemented.

[0130] Referring to Fig. 19, a 1 / 2 speed reduction is expected considering camera exposure compensation, and servo position correction per pixel considering resolution is required. At this time, the error range is predicted to be 6.25 mm, and the object center point can be measured.

[0131] Referring to Fig. 20, with a servo speed of 70 rpm - 1.66 rotations per second, 1 degree: 4.6ms, 0.0878 degrees: 0.4ms, it can be estimated to be about 8ms considering the communication speed (considering 100 bytes 115200bps).

[0132] Servo control has a resolution of 360 degrees / 4096 = 0.0878 degrees, 0.0878 degrees - 3mm, and 6mm. The conclusion depends on the camera resolution.

[0133] The servo motor control method calculates the relative coordinates to move to based on the extracted coordinates, performs servo control processing, and verifies the moved servo coordinates.

[0134] An algorithm can be provided that predicts the direction of coordinates by inserting a virtual frame into the frame difference between a high-speed camera of 800 FPS and a standard projector (60 Hz) and overcomes the latency (delay) through this.

[0135] For example, the direction of movement of a ball can be captured at high speed to create three point coordinates, the direction of these coordinates can be predicted, and latency can be calculated to perform servo control in advance. At this time, a program algorithm can be provided that incorporates feedback by correcting the error between the predicted direction and the mapped image obtained through the absolute camera.

[0137] Dynamic projection mapping technology used in theme parks, exhibitions, performances, and mega-events still relies on foreign technology. The development of domestic technology is urgent, and price must be considered to facilitate mass adoption. The embodiments of this invention enable import substitution by localizing expensive overseas equipment and allow for application in various industries by utilizing the underlying technology. By applying machine learning-based technology and artificial intelligence algorithms, video analysis speed is improved and latency is minimized. Furthermore, by aligning the camera viewpoint of a dynamically moving subject with the projection focusing point, object mapping analysis time is reduced, enabling effective content projection.

[0138] By producing a universal modular product that enables technology implementation simply by adding modules to existing projectors, it is applicable to various manufacturers and models without being limited by lens sizes specific to each brand, thereby allowing for cost savings. Through real-time mapping solutions and functions utilizing projection mapping, it provides immersive media that allows for audience participation. Furthermore, the introduction of a system that is reusable in exhibition, stage, and performance venues and allows for the configuration and expansion of effects makes it easier to adapt to improvisational acting or unexpected situations, rather than relying on fixed scenarios like traditional lighting.

[0140] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0141] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0142] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0143] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0144] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 A real-time dynamic projection mapping method of a mirror type performed by a computer device, comprising: a step of irradiating an IR light source through an IR camera and providing a visible light projector light source using a projector; a step of passing or reflecting the IR light source and the visible light projector light source using a dichroic mirror; and a step of irradiating a light source at a desired location by controlling the light source passed or reflected by the dichroic mirror in at least one of the X-axis and Y-axis using the mirror, wherein the dichroic mirror is characterized by passing the IR light source and reflecting the visible light projector light source, and the step of irradiating a light source at a desired location is characterized by providing an image distortion device integrated with a UV map when correcting center point coordinates and moving up, down, left, and right coordinates using an absolute coordinate extraction camera capable of correcting the difference between absolute coordinates and measured coordinates. Claim 2 A dynamic projection mapping method according to claim 1, wherein the dichroic mirror is characterized in that the front surface is coated with a surface dielectric multilayer film and the back surface is polished. Claim 3 A dynamic projection mapping method according to claim 1, wherein the step of illuminating a light source at a desired location is characterized by illuminating a light source at a desired location using a single mirror rotatable in the X-axis and Y-axis. Claim 4 A dynamic projection mapping method according to claim 1, wherein the step of irradiating a light source at the desired location is characterized by moving the mirror through servo control to move the coordinates to the center in order to correct distortion of the projector image through the mirror. Claim 5 A dynamic projection mapping method according to claim 4, wherein the servo control is characterized by calculating relative coordinates to be moved to the extracted coordinates and then performing servo control processing. Claim 6 delete Claim 7 A dynamic projection mapping method according to claim 1, wherein the step of irradiating a light source at the desired location is characterized by projecting the deformation of the UVMap through absolute coordinate extraction onto a projector screen via Unity's real-time renderer, and controlling the mirror through servo control using the coordinate values ​​of a high-speed camera for real-time coordinate extraction of the object to enable real-time mapping. Claim 8 A dynamic projection mapping method according to claim 1, wherein the step of irradiating a light source at the desired location is characterized by predicting the direction of the coordinates and calculating the latency by inserting a virtual frame into the frame difference between the high-speed IR camera and the projector, and performing servo control in advance. Claim 9 A dynamic projection mapping device of a mirror type real-time dynamic projection mapping device comprising: a light source providing unit that irradiates an IR light source through an IR camera and provides a visible light projector light source using a projector; a light source transmission and reflection unit that passes or reflects the IR light source and the visible light projector light source using a dichroic mirror; and a servo control unit that controls the light source passed or reflected by the dichroic mirror in at least one of the X-axis and Y-axis using the mirror to irradiate the light source at a desired location, wherein the dichroic mirror is characterized by passing the IR light source and reflecting the visible light projector light source, and the servo control unit is characterized by providing an image distortion device integrated with a UV map during center point coordinate correction and up, down, left, and right coordinate movement using an absolute coordinate extraction camera capable of correcting the difference between absolute coordinates and measured coordinates. Claim 10 In claim 9, the dynamic projection mapping device is characterized in that the servo control unit irradiates a light source at a desired position using a single mirror rotatable in the X-axis and Y-axis. Claim 11 A dynamic projection mapping device according to claim 9, wherein the servo control unit moves the mirror through servo control to move the coordinates to the center in order to correct distortion of the projector image through the mirror. Claim 12 A dynamic projection mapping device according to claim 11, wherein the servo control is characterized by calculating relative coordinates to be moved to extracted coordinates and then performing servo control processing. Claim 13 delete Claim 14 A dynamic projection mapping device according to claim 9, wherein the servo control unit projects the deformation of the UVMap through absolute coordinate extraction onto a projector screen via Unity's real-time renderer, and controls the mirror through servo control using the coordinate values ​​of a high-speed camera for real-time coordinate extraction of an object to enable real-time mapping. Claim 15 A dynamic projection mapping device according to claim 9, wherein the servo control unit predicts the direction of coordinates and calculates latency by inserting a virtual frame into the frame difference between the high-speed IR camera and the projector, and performs servo control in advance.

Images