[0029] Below in conjunction with the accompanying drawings, the embodiments of the present invention are described in detail: the present embodiment is implemented on the premise of the technical solution of the present invention, and provides detailed embodiments and specific operation processes, but the protection scope of the present invention is not limited to the following described embodiment.
[0030] As shown in FIG. 2 , the system of this embodiment includes a personal computer, a projector, and a common camera. The interactive identification module and the rendering module run in the computer, wherein the interactive identification module will control the camera in the picture to obtain the image, then analyze and then use the rendering module to control the projector to output the picture. The camera is plugged into the USB port of the computer, and the projector is connected to the DVI interface on the graphics card of the same machine. The shooting range of the camera covers the projection range of the projector, and obtains the interactive images of the projector and system users.
[0031] As shown in Figure 1, the rendering module is a rendering engine, which provides users with relatively powerful and convenient rendering functions. High-quality rendering can be performed with various rendering technologies for various input scenes. With the acceleration of GPU hardware, users can easily implement their own lighting and shadow algorithms by writing their own shading programs. In addition to the general rendering process, the rendering module also supports the rendering of some special effects, such as the rendering of physical water, clouds, fire and terrain. This module includes: geometric transformation module, resource manager module, special effect library module, lighting and texture module, and rendering abstraction layer. These sub-modules work together as nodes of the entire rendering pipeline.
[0032] Resource manager module: User data is generally model data of some 3D scenes, including Maya (the famous 3D modeling and rendering software developed by Autodesk) model data, 3ds MAX (the famous 3D modeling and rendering software developed by Autodesk) model data, these data There are geometry data, texture data, material data. When these data come in, they are firstly loaded into the resource manager module, so that they can be managed uniformly throughout the rendering process. For large resources, the algorithm strategies such as loading, releasing, and preloading are performed in the resource manager module. Some The longest unused resources will be released. On the other hand, the lighting and texture module algorithms can calculate the resources that will be used. At this time, the lighting and texture modules will tell the resource manager module this information. Resource management The server module is preloaded, which improves performance. In short, the resources used in the rendering process, including geometric data, texture data, material data, etc., will be loaded and released by the resource manager uniformly. The geometric transformation module and the lighting and texture module will provide the resource manager module with some resource requirements information. , then the resource manager module manages according to the least recently used and prefetch policies.
[0033] Geometry transformation module: After the resource is loaded into the resource manager module, the geometry transformation module will first go to the resource manager module to apply for model geometry data including triangle slices, quadrilateral slices and other data. And organize the applied data according to its own format into custom model data that the renderer can recognize, including vertex position, vertex normal vector, vertex texture coordinates, etc. After the format is converted, three coordinate system transformations are performed. The first is to convert the model coordinates to the user-defined world coordinate system, including translation, rotation, and scaling. These data can be input through input devices such as mouse and keyboard, which is convenient for users to control. Then the next step is to convert the geometric data to the camera space according to the camera position input by the user, and finally convert the geometric data to the standard clipping space by perspective transformation.
[0034] Lighting and texture module: After the resource manager module is loaded and the geometry transformation module data is preprocessed, the processed data will be sent to the lighting and texture module for rendering. According to the requirements set by the user, this module can perform global illumination calculations of various techniques and various precisions, and supports some of the following rendering techniques: the simplest ray tracing, photon tracing using photon trees to solve Monte Carlo integration ( Support direct light, indirect light, mirror effect, defocus effect, you can set the sampling accuracy, the light source can be surface light source, point light source, ambient light using elevation dynamic light), according to the image error requirements proposed by the user, you can adjust the rendered video Perform bilateral video filtering (a filtering method that considers the influence of multiple factors) to achieve motion blur, depth of field, anti-aliasing and other effects. If the error is too large, some pixels can be resampled. This sub-module is the core and most important module of the system.
[0035] Special effects library module: The technical implementation of lighting and texture modules requires accelerated implementation of the special effects library module. The lighting algorithm requires a lot of calculations, and a special effects library module was specially designed during the design. This module contains some packaged technologies, and each technology contains some rendering processes rendered by the graphics processing unit, so that the Some rendering pipeline techniques are encapsulated and accelerated with graphics processing unit hardware. This module generally uses various hardware features to implement the algorithms used in the lighting and texture modules. Make full use of the parallel features of the latest DirectX10 (a high-performance 3D graphics development kit developed by Microsoft) graphics card's multi-processing unit, as well as the multi-threading, sse2/sse3 (extended multimedia instruction set) multimedia extension instructions of Intel's multi-core processors Technology.
[0036] Rendering abstraction layer: It is the implementation layer of the most specific details. The rendering module is designed considering the possibility of cross-platform and cross-rendering libraries, so this sub-module is to implement an abstract interface, and the underlying implementation uses OpenGL (a high-performance 3D graphics development kit) or Direct3D ( A high-performance 3D graphics development kit developed by Microsoft) implementation, this sub-module provides a large number of rendering and system interfaces for the above lighting rendering layer and special effects library module to call.
[0037] For general rendering, data processing and flow is performed from top to bottom in the order of the block diagram. When a specific effect needs to be displayed, the shader program that needs special effects will be obtained from the special effect library module after observing the projection change steps. Set, and then the shader assembly will be rendered directly on the GPU according to the rendering process of the predetermined special effect. Each special effect has its own dedicated rendering process, so that the special effect shader assembly can flexibly add algorithms for new special effects. The resource manager is a relatively independent module used to organize the loading, release and use of system rendering resources, such as textures, vertex buffers, shader buffers, etc. The advantage of this is that the calculation and data are better separated, which is convenient for management and increases scalability. The rendering abstraction layer is the data processing before submitting various input rendering data to the hardware at the bottom layer. The design of the module interface is abstract, either Direct3D or OpenGL can be used. The specific API is independent and has a certain cross-platform. ability.
[0038] This embodiment is an interactive display work integrating digital technology, digital art and traditional Chinese painting art. In an open space, it provides a natural and flexible interaction between viewers and ink works. An ink lotus pond is projected on the ground, and the fish are no longer still on the paper, but swim freely among the lotuses. When visitors walk into the picture, they will find that the water waves are rippling under their feet, the schools of fish are startled, and the lotus leaves are floating; the sleeping lotus is awakened by the visitors and slowly opens; when the visitor leaves, everything gradually subsides and returns to its original state; Drops of water on the rocks, splashing microwaves on the water. An agile ink painting, a freehand nature, a tranquility in the hustle and bustle, a traditional and fashionable interest. The projector installed above projects the ink painting vertically onto the ground. In this way, when people visit, they can interact with the ground with their hands, feet, the whole body, or even any other object. The background program tracks and locates the visitor, and stimulates the drawing program to respond accordingly, such as fish escaping, water waves rippling, lotus leaf fluttering, etc.
[0039] The interactive identification module is installed on an HP personal computer, and by analyzing the content captured by the camera, the message event of the analysis result is sent to the rendering module through inter-process communication, and then the rendering module will draw a new response screen in real time. .
[0040] The interaction recognition module uses a camera to detect moving objects, and mainly uses background frame difference to recognize the interaction input mode. This technology is also used in video tracking and traffic monitoring.
[0041] Since the motion of the dynamic graphic object in the picture will interfere with the recognition, the interactive recognition module first selects the rendered picture without the dynamic graphic object as the background frame. The camera will compare the difference between the screen captured by each monitoring and the background image captured before. If it exceeds a certain threshold, it is considered that there may be objects interacting. Because of the color characteristics of different backgrounds, in this embodiment, when the difference between the two images is performed, the difference is performed on the three RGB channels of the color image respectively, and the results are weighted. The grayscale histograms of the three channels of R, G, and B are shown in Figure 6, Figure 7 and Figure 8. Figure 9 is the weighted average result of the difference images of the background and camera images in three RGB channels. If the weight of the channel with a larger gray value is set larger, the distinction between the background and the foreground can be larger, and the recognition effect will be better.
[0042] After the difference image is binarized, due to various noises, changes in the light environment, and the influence of small objects, the interactive recognition module needs to expand the difference image, so that the original connected areas with holes can be removed from the hollow, and then eroded to eliminate small plaques . We end up with some large connected regions, which are the added interactions on the background. After identifying these large areas, calculate several points near the centroid in these connected blocks as input messages and send them to the rendering module. When the rendering module receives the interactive input, it will generate animation responses in the area around the input points. .
[0043] The rendering module is also installed on the same computer, the graphics card is Geforce7900, and the final display is projected onto the ground from top to bottom through a MITSUBISHI LX390 projector installed on the ceiling. A Logitech QuickCam camera is also installed next to the projector, which also shoots the entire projected image area from top to bottom.
[0044] The rendering of a scene by the rendering module involves the rendering organization of the entire scene in addition to the individual rendering technology of each part. In the real-time rendering of the entire scene, this embodiment makes full use of the characteristics of modern graphics hardware to optimize rendering, and combines modern graphics engine technology to organize rendering.
[0045] The following is an example of the animation and interaction of the ink-style "water, fish, lotus" and other content to illustrate the interaction process of the entire system. The system uses C++, OpenGL, and CG languages to implement on Microsoft WindowsXP system.
[0046]The rendering module provides interactive water surface rendering, which can realize real-time dynamic rendering of a realistic water surface triggered by interaction. The water surface modeling is represented by a plane grid, the grid size is 64×64, and the length and width are both 75. The water surface nodes on each grid are described by four quantities: space position, space normal vector, phase, and velocity. As shown in Figure 3, the center of the grid is placed at the coordinate origin (0, 0, 0), and the grid faces are on the XOZ plane. During initialization, the Vertex value of each node is (n/GridSizeX*75-38.5, 0, (1-m/GridSizeY)*75-38.5), and the initial value of Normal is (0, 1, 0) pointing to the positive y direction , the initial value of Position is 0, and the value of Velocity is 0. The initial mesh is static. The rippling of water waves is reflected in the movement of grid nodes. In order to achieve real-time effects, each grid node in this embodiment only moves up and down in the y direction, not horizontally, and overlaps and breaks with water waves. The experimental results are as follows The visual effect is also very good. Water waves have phenomena such as transmission and interference. When the position of the water wave is relatively high, it must become lower next, and then spread the energy to the surrounding area. The change of each node is affected by its own state of the previous frame and the state of the surrounding nodes. , the same is accompanied by the decay of energy. According to this idea, the water node grid calculation adopts a recursive formula to calculate (refer to Figure 4). The above steps are repeated for all grid nodes for each frame of the video. With the coordinate position and normal vector of each node on the grid, rendering can be performed.
[0047] The rendering in this embodiment is performed in the vertex renderer and the pixel renderer of nVidia's Geforce7900GS graphics card. In the pixel renderer, each fragment in the triangle is rendered, and the final color of each point on the water surface is mixed from the refracted light of the water bottom, the reflected light of the environment, the ambient light, and the reflected light of the light source. For the calculation of the refracted light at the bottom of the water, this embodiment can easily find the intersection of the refracted ray and the bottom of the water according to the coordinate position of the vertex, the normal vector, the depth of the pool and the position of the viewpoint, and then use the coordinates of this intersection to convert to the texture space at the bottom of the pool, The color value of the ink painting at the bottom of the sampling pool is the best result.
[0048] The rendering module uses the cube environment mapping technology to calculate the reflected light of the ambient light.
[0049] The rendering module provides the implementation of texture animation of graphics objects. FIG. 5 is an animation of a series of actions of a certain graphic object used in the application embodiment to decompose a picture to perform a single action animation, and the results are mixed into a rendering scene to realize animation drawing and interactive response.
[0050] The rendering module provides state management of graphic objects and realizes complex animations and interactive responses. In the application embodiment, the motion law of the graphic object and the switching between actions are very complicated. The system provides a variety of state management for the actions of each graphic object in the screen. The system will determine the state of the next state of the graphic object and the switching sequence between the two states according to the interactive input parameters and the current state of the graphic object (including speed, direction, attitude, etc.), to ensure the smoothness of the motion sequence between the two states. stitching.
[0051] This embodiment uses computer vision and graphics technology to realize a more natural human-computer interaction mode, which greatly improves the scope of application of the system while saving costs.
