A three-dimensional interactive method and system
By collecting images of physical props in a three-dimensional interaction system and displaying corresponding three-dimensional virtual objects and scenes, the problem of lack of three-dimensional interaction in the existing technology is solved, and the display of three-dimensional virtual scenes and objects is realized, improving user experience and teaching effect .
Patent Information
- Application Number
- CN201810458559.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2018-05-14
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2038-05-14
AI Technical Summary
The existing technology lacks human-computer interaction in three-dimensional teaching and/or game entertainment scenes, and has a low user experience. It mainly adopts a planar display method, and it is impossible to perform three-dimensional interaction in a three-dimensional three-dimensional space.
By placing physical props on a two-dimensional plane and using the camera of the three-dimensional interactive system to collect their images, displaying three-dimensional virtual objects and/or virtual scenes corresponding to the props, converting them in real time according to the movement trajectory of the props, using holographic projection technology or enhancement Realistic technology displays virtual objects and scenes in physical three-dimensional space.
It realizes the display of three-dimensional virtual scenes and objects in three-dimensional space, improves user experience and teaching effects, and enhances interactivity.
Smart Images

Figure CN110478892B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of information technology, and in particular, relates to a three-dimensional interaction method and system. Background Art
[0002] In the current teaching and game entertainment process, physical objects such as books, toys, and physical teaching aids are commonly used. With the continuous advancement of educational informatization, electronic devices such as televisions (electronic display screens), electronic whiteboards, projectors, and interactive all-in-one machines have entered the classroom in large numbers, and more and more display content has begun to be presented in a digital way. At present, classroom interaction is not only the interaction between teachers and students, but also a large number of interactive games and interactive questions and answers have entered the classroom, providing students with richer display content. At the same time, in terms of game entertainment, a large number of electronic devices with novel touch operation functions are slowly replacing traditional key-operated personal computers and game terminals. This makes the user experience richer.
[0003] The current digital teaching methods and game entertainment methods all use a flat display method, for example, allowing students to operate the teaching system, teaching application software or games through keyboard and mouse, so as to achieve the purpose of teaching or game entertainment.
[0004] However, the prior art does not involve the application of three-dimensional props. Even if there are sporadic three-dimensional prop applications on the market, they are only based on three-dimensional display on a two-dimensional plane, and cannot perform human-computer interaction in three-dimensional teaching and / or game entertainment scenes in a three-dimensional three-dimensional space, resulting in a low user experience. Summary of the invention
[0005] The present invention provides a three-dimensional interaction method and system, which solves the problem that the prior art lacks human-computer interaction in three-dimensional teaching and / or game entertainment scenes.
[0006] In order to achieve the above object, the present invention provides a three-dimensional interaction method, comprising:
[0007] Placing a physical prop on a two-dimensional plane, wherein the prop is within the acquisition range of a camera of the three-dimensional interactive system;
[0008] The three-dimensional interactive system displays a three-dimensional virtual object and / or a virtual scene corresponding to the prop;
[0009] When the prop moves within the acquisition range of the three-dimensional interactive system, the three-dimensional virtual scene and / or the virtual object are converted accordingly according to the moving trajectory of the prop.
[0010] Optionally, the three-dimensional interactive system displays a three-dimensional virtual object and / or a virtual scene corresponding to the prop, including:
[0011] The three-dimensional interactive system displays three-dimensional virtual objects and / or virtual scenes corresponding to the props in a physical three-dimensional space through holographic projection technology.
[0012] as well as,
[0013] The three-dimensional interactive system displays virtual scenes and / or virtual objects corresponding to the props on a two-dimensional display screen, and the virtual scenes and virtual objects displayed on the two-dimensional display screen and / or the three-dimensional virtual scenes and virtual objects displayed in the physical three-dimensional space correspond to each other.
[0014] Optionally, the three-dimensional interactive system displays a three-dimensional virtual object and / or a virtual scene corresponding to the prop, including:
[0015] The three-dimensional interactive system uses augmented reality (AR) technology to display three-dimensional virtual objects and / or virtual scenes through a helmet-mounted display (HMD) or a mobile terminal.
[0016] Optionally, the method further comprises:
[0017] The three-dimensional interactive system displays a virtual scene and / or virtual object corresponding to the prop on a two-dimensional display screen, and the virtual scene and / or virtual object displayed on the two-dimensional display screen corresponds to the three-dimensional virtual scene and / or virtual object;
[0018] The three-dimensional virtual scene and / or the virtual object is converted accordingly according to the moving trajectory of the prop, including:
[0019] The virtual scene and / or virtual object displayed on the two-dimensional display screen is synchronously or asynchronously transformed with the three-dimensional virtual scene and / or virtual object displayed by the HMD or the mobile terminal according to the moving trajectory of the prop.
[0020] Optionally, the three-dimensional interactive system uses augmented reality (AR) technology to display three-dimensional virtual objects and / or virtual scenes through a helmet-mounted display (HMD) or a mobile terminal, including:
[0021] Using computer graphics (CG) technology to construct the three-dimensional virtual object and / or three-dimensional scene model;
[0022] Using three-dimensional tracking and registration technology to perform virtual-to-real registration processing on the three-dimensional virtual object and / or three-dimensional scene model and the image captured by the HMD or mobile terminal;
[0023] The three-dimensional virtual object and / or three-dimensional scene model is drawn and rendered in real time, and according to the mapping relationship between the three-dimensional virtual object and / or virtual scene after the virtual-to-real registration processing in the real scene, the three-dimensional virtual object and / or three-dimensional scene model is superimposed and fused with the image of the material culture carrier, and the superimposed and fused image is displayed through the HMD or mobile terminal.
[0024] Optionally, the using of the three-dimensional tracking and registration technology to perform virtual-to-real registration processing on the three-dimensional virtual object and / or the three-dimensional scene model and the image captured by the HMD or the mobile terminal includes:
[0025] Acquire identification information of the prop through the HMD or the mobile terminal; the identification information is a pre-set identification mark;
[0026] The obtained identification mark is sent to the cloud server, and the cloud server determines the three-dimensional virtual object or three-dimensional scene model corresponding to the identification information according to the identification information, and then calculates the prop image collected by the HMD or mobile terminal, performs three-dimensional coordinate transformation on the image, and loads the three-dimensional virtual object or three-dimensional scene model at a specific position in the image based on the transformed coordinates.
[0027] Optionally, the three-dimensional virtual scene and / or the virtual object are converted accordingly according to the moving trajectory of the prop, including:
[0028] Moving the virtual object and / or the virtual scene according to the moving trajectory of the prop;
[0029] The virtual objects and / or virtual scenes are processed with special effects, and / or the virtual objects and / or the areas where the virtual objects are located in the scenes are processed with special effects, and / or the virtual objects and / or virtual scenes are made to interact according to a specific storyline.
[0030] Optionally, the three-dimensional virtual scene and / or the virtual object are converted accordingly according to the moving trajectory of the prop, including:
[0031] Moving the virtual object in the virtual scene according to the moving trajectory of the prop;
[0032] When the virtual object moves to a preset position in the virtual scene, a three-dimensional scene interaction is displayed.
[0033] An embodiment of the present invention further provides a three-dimensional interactive system, the system comprising: a processor and a memory for storing a computer program that can be run on the processor; wherein the processor is used to execute the above-mentioned three-dimensional interactive method when running the computer program.
[0034] An embodiment of the present invention further provides a computer-readable storage medium on which computer-executable instructions are stored. The computer-executable instructions are used to execute the above-mentioned three-dimensional interaction method.
[0035] The method and system of the embodiment of the present invention have the following advantages:
[0036] Through the method and system provided by the embodiment of the present invention, the image of the physical prop is collected by the camera, and the three-dimensional virtual object and / or three-dimensional virtual scene corresponding to the physical prop is displayed in the three-dimensional space, and when the physical prop moves, the corresponding three-dimensional virtual object and / or three-dimensional virtual scene is interactively displayed. The three-dimensional virtual scene and / or virtual object display in the three-dimensional space is realized, which improves the user experience and the teaching effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 is a flow chart of a three-dimensional interaction method in an embodiment of the present invention;
[0038] Figure 2a is a schematic diagram of a prop frame and a display content area in an embodiment of the present invention;
[0039] Figure 2b Schematic diagram of a border with a direction mark and a display content area of a prop in an embodiment of the present invention;
[0040] Figure 2c is another schematic diagram of a prop frame and a display content area in an embodiment of the present invention;
[0041] Figure 3 This is a schematic diagram of an application example in an embodiment of the present invention;
[0042] Figure 4 is another schematic diagram of an application example in an embodiment of the present invention;
[0043] Figure 5 It is a schematic diagram of the structure of the three-dimensional interactive system in an embodiment of the present invention. DETAILED DESCRIPTION
[0044] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in each embodiment of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0045] To achieve the above objectives, Figure 1 As shown, the present invention provides a three-dimensional interaction method, the method comprising:
[0046] S101, placing a physical prop on a two-dimensional plane, wherein the prop is placed within the acquisition range of a camera of the three-dimensional interactive system;
[0047] The physical props can be a physical object with a radio frequency identification code, such as an intelligent toy with a built-in radio frequency chip or a sensor device, or a physical object with a two-dimensional identification code, such as a table printed with a two-dimensional code, a card printed with a two-dimensional code, etc., or a display card of a separate pinyin, tone, Chinese character or number (the display card material can be a paper sheet, a plastic sheet, a wood sheet, a rubber sheet or a metal sheet, etc.), or a small toy and a small physical object, such as a small fish, an apple, a small table, a building block, a jigsaw puzzle, etc. The embodiment of the present invention can identify the different colors and different shapes (such as polygons, circles, ellipses, sectors, etc.) of the small toy / small physical object, not only can accurately locate the specific position of the physical prop in space, but also can provide the rotation angle, whether it needs to be straightened and other suggestive information, so as to facilitate the teaching interaction with students. The display card can also be some teaching aids printed with pinyin, tone, Chinese character or number, such as conventional teaching aids such as books, small sticks, rulers, triangles, teaching rulers / whips, compasses, etc. Taking the display card as an example, in the embodiment of the present invention, the display card has a frame, which can be closed (or semi-closed) and have a certain width, and the frame color is obviously different from the background color of the display content area. For example, the frame color can be black, and the background color of the display content area can be white, or the frame color is white and the color of the display content area is black (such as Figure 2a The frame is a polygon, such as a triangle, rectangle, rhombus, trapezoid, etc. The frame can also be an ellipse or a perfect circle. For the convenience of explanation, the embodiment of the present invention takes a quadrilateral display card as an example. Figure 2a As shown, the quadrilateral can be a square or an approximately square shape, such as a square, a rectangle, a rounded rectangle, etc., and the border is separated from the display content area by a certain width. The feature of the quadrilateral display card having a border has the following technical effects: the border can be quickly located and identified through an image recognition algorithm, thereby quickly identifying the display content of the teaching area within the border. Compared with existing image recognition technologies, the solution of first identifying the border and then identifying the display content in the embodiment of the present invention ensures that the time for identifying the display card is shorter and the recognition accuracy is higher. In addition, in order to facilitate the acquisition of the display content in the props from any angle, for some content that appears ambiguous in different directions (such as 6 and 9, u and n), the display card also includes a direction mark, and the direction mark is located in the content area or placed on the border. As shown Figure 2bAs shown, the direction mark can be a thicker border on one side (i.e. one side is thick and the other three sides are narrow), or it can be a dot, two rectangular corners, a horizontal line, etc. If the border is circular, the direction mark can be added to one section of the circular border, such as thickening a section directly above the circular border, or adding one or more dots, or adding some irregular shapes. The direction mark has the following technical effects: it can quickly locate the direction of the displayed content, thereby quickly locating and identifying the information of the displayed content, and improving the accuracy and speed of recognition. Figure 2c is a composition example of a display card in an embodiment of the present invention, Figure 2c It is a display card, and the card has a closed frame of a rounded rectangle, the frame has a certain width (can be divided into an inner frame and an outer frame), and there is a certain distance between the frame and the display content area. In addition, the frame meets the characteristics of a thick bottom edge and thin other three sides, and the bottom edge is the direction mark of the display card. The embodiment of the present invention can quickly locate and identify the frame, and identify the rotation angle of the display card according to the direction mark, so as to quickly identify the display content information in the display content area in the frame.
[0048] The three-dimensional interactive system can be an image processing and display system, which has a display screen and a camera, such as a smart TV, a PDA, a tablet computer, etc. If the physical prop is on a certain horizontal plane, in order to facilitate the acquisition of the physical prop, the camera horizontal line can be at an angle to the plane. In addition, the camera can be a single camera or a dual camera, and its acquisition range includes not only the plane but also the physical three-dimensional space (such as the entire classroom), so the physical prop can be in the physical three-dimensional space, for example, the user holds the physical prop in the physical three-dimensional space and is within the acquisition range of the camera.
[0049] S102, the three-dimensional interactive system displays a three-dimensional virtual object and / or a virtual scene corresponding to the prop;
[0050] After detecting the prop, the system displays a three-dimensional virtual object and / or a virtual scene corresponding to the prop.
[0051] The specific detection method is as follows (taking the prop as a display card as an example):
[0052] S1021, image binarization processing: using an adaptive threshold algorithm to convert the image into a black and white binary image, so that the dotted line points appear as independent black areas on the binary image;
[0053] S1022, extracting the contour of the black area: using a contour following algorithm to process the black area in the binary image to obtain contour lines of all black areas, and adding them to the contour set;
[0054] S1023, extracting a contour: extracting and deleting a contour line from the contour set for analysis;
[0055] S1024, rough screening of contours: Calculate the contour length, the area of the enclosed area, the center of gravity, the perimeter-to-area ratio and other parameters, and use a specific threshold for preliminary screening to remove candidates whose shapes are obviously inconsistent with the dotted line points;
[0056] S1025, dotted line point tangent analysis: perform ellipse fitting on the contour, and take the major axis of the ellipse as the tangent;
[0057] S1026, adding a dotted line point set: adding the filtered contour to the dotted line point set for subsequent algorithm analysis;
[0058] S1027, point set analysis: performing subsequent analysis on the dotted line points that pass the screening to determine the scope of the game area;
[0059] S1028, dotted line point neighbor analysis: search for dotted line points with adjacent positions and similar tangent directions in the dotted line point set. Perform ellipse fitting on the contour, and take the major axis of the ellipse as the tangent direction;
[0060] S1029, isolated point screening: delete the dashed line points whose number of neighbors is less than 5 (because there will be other dashed line points in the same direction around the real dashed line points);
[0061] S10210, Hough cluster analysis: Perform Hough voting analysis on the remaining points to obtain a series of clusters;
[0062] S10211, cluster screening: first remove the clusters with fewer dotted points, then try to select two clusters with the largest number of dotted points in the horizontal direction (with an angle less than 45 degrees to the horizontal line) from the remaining clusters, and then select two clusters with the largest number of dotted points in the vertical direction (with an angle less than 45 degrees to the vertical line). If the selection fails (the number of valid clusters is insufficient), the current frame image will not be processed;
[0063] S10212, straight line fitting: performing straight line fitting on the selected clusters in the horizontal and vertical directions respectively to determine the boundary equation of the game area;
[0064] S10213, calculate correction parameters: calculate the intersection points of the four corners of the game area according to the fitted straight line equation, and then use the regression analysis algorithm to calculate the correction parameters of the game area. This set of parameters can be used to correct subsequent image frames.
[0065] Optionally, when the prop is a smart toy equipped with radio frequency identification, the detection area may also be a wireless sensing area, that is, when the smart toy appears in the detection area, the virtual scene / character / object corresponding to the smart toy is automatically sensed and displayed.
[0066] In addition, when the prop is a display card with a border, the method for identifying the prop is as follows:
[0067] S10214, image binarization. Use an adaptive threshold algorithm to convert the image captured by the camera into a black and white binary image to highlight the border of the prop. The adaptive threshold algorithm is based on:
[0068] in
[0069] Among them, v is the gray value of the pixel, N(v) is the set of pixels near v, C is the preset threshold, and v' represents the pixel point in the area N(v).
[0070] S10215, performing binary contour extraction on the result, scanning the connected regions in the binary image, and obtaining data such as the contour lines, area, and perimeter of the region.
[0071] S10216, perform contour screening, that is, perform a quick geometric morphological analysis on the contour extracted in the previous step, and only retain contours that resemble quadrilaterals (the use scenarios of the embodiment of the present invention include but are not limited to quadrilaterals, and only quadrilaterals are used as an example for explanation) to reduce the processing time of subsequent steps. Specifically, first use the local average method to smooth the contour line, then use the nearest difference method to calculate the tangent vector of each point on the contour line, and finally use the hierarchical clustering method to analyze all the tangent vector coordinates. If there are exactly 4 significant clusters formed, it is considered that the contour is similar to a quadrilateral.
[0072] S10217, split the processing result of S110213 into four sides. Based on the cluster analysis result of the previous step, extract the coordinates of the contour points corresponding to the four significant clusters into four sets, which correspond to the fitting data of the four sides of the quadrilateral.
[0073] S10218, perform least squares fitting on the fitting data of S1104. The data of the four edges generated in the previous step can be respectively fitted with straight lines to obtain equations of the four edges. The present invention uses the least squares algorithm to perform straight line fitting, and the optimization goal is:
[0074]
[0075] After completing the fitting of the four sides, the specific position of the border of the display card in the image can be determined.
[0076] S10219, correct the image of the adjacent area. Due to the various shooting angles, the card will be deformed in the image. The deformation of the card can be corrected using the frame obtained in the previous step. At this time, the card content may have four different orientations of 0 degrees, 90 degrees, 180 degrees, and 270 degrees.
[0077] S10220, detect the direction mark. The embodiment of the present invention uses a machine learning method to detect and identify the direction mark in the card. By collecting thousands of card images with different orientations and annotating them (for example, 5 categories can be used, namely 0 degrees, 90 degrees, 180 degrees, 270 degrees, and no direction mark), a deep neural network is used for training to obtain a direction mark classifier, which can detect and distinguish the direction mark of the result obtained in the previous step. The classifier trained by this scheme can achieve an accuracy rate of more than 99.6% for the recognition of direction marks.
[0078] S10221a, rotating the image according to the mark. For a card with a direction mark detected, the card can be rotated according to the direction mark. Rotating means rotating the card to a horizontal position.
[0079] S10221a, acquiring images in all directions. For cards without detected direction marks, the embodiment of the present invention directly generates images in four different directions for subsequent content recognition algorithm analysis.
[0080] S10222. Identify the content of the card. The present invention uses a machine learning method to identify the content of the card. First, there are hundreds to thousands of defined cards in the sample database. After extracting the directional gradient histogram features (HOG) of these samples, the SVM multi-category classifier is trained; if there are a lot of samples in the database (more than one thousand), a deep neural network can also be used directly for training. Use the classifier to distinguish the image obtained in the previous step (if no direction mark is detected before, the images in the four directions are distinguished, as long as one of them is valid). After the distinction, the result is compared (verified) with the standard sample in the database. After the verification, it is considered that the result is successfully detected.
[0081] After the props are detected, the corresponding three-dimensional virtual scene and / or virtual object is displayed, and the method may specifically be:
[0082] Displaying a three-dimensional virtual object and / or a virtual scene corresponding to the prop in a physical three-dimensional space by means of holographic projection technology,
[0083] as well as,
[0084] The three-dimensional interactive system displays virtual scenes and / or virtual objects corresponding to the props on a two-dimensional display screen, and the virtual scenes and virtual objects displayed on the two-dimensional display screen and / or the three-dimensional virtual scenes and virtual objects displayed in the physical three-dimensional space correspond to each other.
[0085] For example, a two-dimensional virtual scene and / or virtual object is displayed on the two-dimensional display screen of the system, and a three-dimensional virtual scene and virtual object are displayed in three-dimensional space at the same time. The scene and object can be of the same type or of different types. For example, the prop is a physical fish toy. After the fish toy is collected, the system displays a virtual kitten and several small fish (two-dimensional virtual objects) on the two-dimensional display screen, as well as virtual scenes, such as a river, a fishing rod held by a kitten, and a beach, etc. In the two-dimensional picture presented to the user, a kitten holds a fishing rod and is ready to fish, and the fish swims in the river as the physical prop (fish toy) moves, and the movement trajectory is one-to-one corresponding, and when the swimming small fish moves to the hook of the fishing rod, the kitten hangs up the small fish and puts it in a fish tank or bucket. At this time, through holographic projection technology, the fishing rod is not only displayed on a two-dimensional plane, but part of the fishing rod can "stretch out" from the display screen, and the image of the fish can also form a three-dimensional virtual fish, swimming freely in the physical space. At this time, the three-dimensional fish image and the two-dimensional fish image can appear at the same time. In addition, the three-dimensional fish image can move according to the movement of the physical fish toy, while the three-dimensional fish image moves randomly and freely.
[0086] Holography is a photographic technology that records all the information (amplitude, phase) in the light waves reflected (or transmitted) by the photographed object, and the light reflected or transmitted by the object can be completely reconstructed through the recording film, as if the object is there. By observing the photos from different directions and angles, different angles of the photographed object can be seen, so the recorded image can give people a three-dimensional vision. Holography is a technology that uses the principles of interference and diffraction to record and reproduce the real three-dimensional image of an object.
[0087] The first step is to use the interference principle to record the object's light wave information, which is the shooting process: the object forms a diffuse object beam under laser irradiation; another part of the laser is used as a reference beam to shoot onto the holographic film, which is superimposed with the object beam to produce interference, converting the phase and amplitude of each point on the object's light wave into a spatially varying intensity, thereby using the contrast and interval between the interference fringes to record all the information of the object's light wave. After the film recording the interference fringes undergoes processing procedures such as development and fixing, it becomes a Nolide hologram, or holographic photo;
[0088] The second step is to use the diffraction principle to reproduce the object's light wave information, which is the imaging process: the hologram is like a complex grating. Under the irradiation of coherent laser, the diffracted light waves of a linearly recorded sinusoidal hologram can generally give two images, namely the original image (also called the initial image) and the conjugate image. The reproduced image has a strong three-dimensional sense and a real visual effect. Each part of the hologram records the light information of each point on the object, so in principle each part of it can reproduce the entire image of the original object. Through multiple exposures, multiple different images can be recorded on the same film, and they can be displayed separately without interfering with each other.
[0089] In addition, in addition to the realization of three-dimensional virtual objects and / or virtual scenes by holographic projection technology, embodiments of the present invention can also use augmented reality (AR) technology to realize the construction of three-dimensional virtual objects / scenes. The method is specifically: using augmented reality AR technology to display three-dimensional virtual objects and / or virtual scenes through a helmet-mounted display (Head Mounted Display, HMD) (such as AR glasses, etc.) or a mobile terminal. In addition, the system displays the virtual scene and / or virtual object corresponding to the prop on a two-dimensional display screen, and the virtual scene and / or virtual object displayed on the two-dimensional display screen corresponds to the three-dimensional virtual scene and / or virtual object;
[0090] The three-dimensional virtual object and / or virtual scene is displayed by the HMD or the mobile terminal, specifically:
[0091] Using computer graphics (CG) technology to construct the three-dimensional virtual object and / or three-dimensional scene model;
[0092] Using three-dimensional tracking and registration technology to perform virtual-to-real registration processing on the three-dimensional virtual object and / or three-dimensional scene model and the image captured by the HMD or mobile terminal;
[0093] The three-dimensional virtual object and / or three-dimensional scene model is drawn and rendered in real time, and according to the mapping relationship between the three-dimensional virtual object and / or virtual scene after the virtual-to-real registration processing in the real scene, the three-dimensional virtual object and / or three-dimensional scene model is superimposed and fused with the image of the material culture carrier, and the superimposed and fused image is displayed through the HMD or mobile terminal.
[0094] Three-dimensional tracking registration means that during the operation of the system, it needs to detect and identify the image features of the real environment in real time, and determine the position and direction of the camera in the three-dimensional space relative to the real environment based on the features of the acquired real-world video image frames, so as to accurately place the virtual object in the correct position in the real scene. To achieve seamless integration of virtual objects and real scenes, three-dimensional tracking registration technology is the core key technology of the AR system.
[0095] The implementation of 3D registration technology can generally be divided into three approaches: computer vision-based 3D registration technology, hardware-based 3D registration technology, and hybrid 3D registration technology.
[0096] Three-dimensional tracking and registration technology based on computer vision: It mainly determines the position and posture of the camera by identifying the features of the image in the real scene. This registration method can be divided into two types: the method based on artificial markers (Marker) and the method based on natural image features. The system obtains the internal and external parameters of the camera by real-time identification of parallel lines, vertical lines, plane objects, corner points, texture features and other methods of markers in the real scene. The advantage of the marker-based method is that the real scene features are obvious, it has high robustness and fast computing speed, but it requires artificially setting black and white markers in the scene in advance, which will affect the beauty of the scene. The method based on natural image features can overcome the above-mentioned shortcomings of artificial markers.
[0097] Three-dimensional tracking and registration technology based on hardware equipment: mainly uses signal transmitting devices and sensing devices to obtain relevant data and calculate the position and posture information required for three-dimensional registration. Commonly used hardware devices include electromagnetic tracking devices, optical tracking devices, ultrasonic tracking devices, GPS positioning devices, inertial navigation, electronic compass, etc. The electromagnetic tracking device uses a coil signal transmitter to determine the relative position and posture information of the target object in the real scene based on the coupling interaction between the magnetic transmission signal and the magnetic induction signal. The optical tracking device uses a camera device or a photosensitive component to determine the six-degree-of-freedom information of the target object through the image, light source information and the three-dimensional spatial position information of the sensor based on the received light source information or the light emitted by the reflection ball. The ultrasonic tracking device performs three-dimensional registration based on the time difference, phase difference and sound pressure generated by the ultrasonic waves emitted by different sound sources reaching the specific target object in the real scene. The relative position information of the target object in the real scene determined by hardware devices such as GPS positioning, inertial navigation, and electronic compass is often used in augmented reality information service systems based on location services.
[0098] Hybrid 3D tracking and registration technology: In some specific applications, the above-mentioned multiple 3D tracking and registration methods are combined to determine the position information of the virtual object relative to the real scene, thereby realizing the 3D tracking and registration function of the system.
[0099] Specifically, in the embodiment of the present invention, the system captures a sequence of frames of video images of the real environment through a camera on an HMD or a mobile terminal, identifies a target object in the real environment, and further determines the posture relationship of the smart terminal camera relative to the target object in the real environment. The system determines the exact position of the virtual object (i.e., virtual object and / or virtual scene) in the real environment based on the acquired camera posture data information and the positioning data information obtained from the real scene, and finally completes the real-time drawing and generation of the virtual object in the real environment, integrates the virtual object with the real environment and displays it on the mobile terminal, forming a new scene of virtual-real integration, and completing the three-dimensional tracking and registration process, thereby achieving the purpose of enhancing the cultural information of the real environment.
[0100] During the operation of the system, on the one hand, the position information of the mobile terminal camera is in a state of dynamic change. The system must obtain the position information of the camera relative to the target object in the real environment in real time and accurately in order to complete the subsequent accurate three-dimensional registration task. On the other hand, the system must be able to detect, identify and track the label information in the real environment in real time to quickly obtain the position data information of the camera, and detect, identify and track the target object (label, such as a physical prop can be used as a label) in real time in the three-dimensional space environment and complete the virtual and real scene registration in real time. The three-dimensional tracking and registration technology based on computer vision is that the system uses the image frame containing the target object as the positioning reference in the video image sequence frame of the real scene, and obtains the position relationship between the camera and the real environment in real time by detecting, identifying and matching the feature points of the target object image frame and tracking it in real time, and updates the coordinate transformation relationship from the virtual object to the target object in the real environment in real time, and then updates the virtual and real fusion scene in real time to complete the three-dimensional registration task. The three-dimensional tracking and registration process based on computer vision mainly includes four links: image feature point detection, identification and matching, real-time tracking of the target object and virtual and real fusion.
[0101] In the system, 3D tracking registration is a key technology to ensure that virtual objects are accurately superimposed and integrated into the real environment. In this process, the system obtains data information of the real scene through the video image of the real scene captured by the camera. The data of 3D tracking registration comes directly from the image information captured by the camera. If the internal parameters of the camera are inaccurate, it will seriously affect the accuracy of 3D registration and even cause registration failure. Therefore, the augmented reality system needs to perform camera calibration to determine the relevant parameters of the camera. The camera parameters include internal parameters and external parameters. The internal parameters refer to the parameters related to the geometric and optical characteristics of the camera itself. The external parameters refer to the three-dimensional spatial position and direction of the camera relative to a certain world coordinate system. The process of camera calibration in the system is the process of determining the internal and external parameters of the camera. In addition, the coordinate transformation in 3D tracking registration The essence of the 3D tracking registration process involves the conversion relationship between the world coordinate system, the camera coordinate system, the imaging plane coordinate system and the pixel coordinate system. By determining the conversion matrix between these four coordinate systems, the system can accurately superimpose the virtual object in the scene of the real environment. The 3D tracking registration process mainly includes three main aspects: camera calibration, real-time tracking and 3D scene drawing fusion. The purpose of camera calibration is to help the system determine the internal and external parameters of the camera, obtain the conversion relationship between the real-world coordinate system, camera coordinate system, imaging plane coordinate system and pixel coordinate system, and provide data information for the subsequent accurate superposition of virtual objects in the real scene. The real-time tracking process includes real-time detection and tracking of target objects in the real scene. Through real-time detection, identification, matching and tracking of target objects in the real scene, the position and direction of the camera relative to the real-world coordinate system are obtained to determine the conversion matrix between the real-world coordinate system and the camera coordinate system. The three-dimensional scene rendering fusion mainly obtains the conversion relationship between the world coordinate system, camera coordinate system, imaging plane coordinate system and pixel coordinate system, and uses computer graphics image rendering generation technology to superimpose virtual objects in the target object area of the real scene to form a new scene of virtual-real fusion.
[0102] Therefore, the three-dimensional tracking and registration technology is used to perform virtual-real registration processing on the three-dimensional virtual object and / or three-dimensional scene model and the image collected by the HMD or mobile terminal, which can be specifically:
[0103] Acquire identification information of the prop through the HMD or the mobile terminal; the identification information is a pre-set identification mark;
[0104] The obtained identification mark is sent to the cloud server, and the cloud server determines the three-dimensional virtual object or three-dimensional scene model corresponding to the identification information according to the identification information, and then calculates the prop image collected by the HMD or mobile terminal, performs three-dimensional coordinate transformation on the image, and loads the three-dimensional virtual object or three-dimensional scene model at a specific position in the image based on the transformed coordinates.
[0105] S103: When the prop moves within the acquisition range of the three-dimensional interactive system, the three-dimensional virtual scene and / or the virtual object are converted accordingly according to the moving trajectory of the prop.
[0106] The three-dimensional virtual scene and / or the virtual object are converted accordingly according to the moving trajectory of the prop, specifically: the virtual scene and / or virtual object displayed on the two-dimensional display screen and the three-dimensional virtual scene and / or virtual object displayed by the HMD or mobile terminal are converted synchronously or asynchronously according to the moving trajectory of the prop.
[0107] The three-dimensional virtual scene and / or the virtual object are converted accordingly according to the moving trajectory of the prop, which may be:
[0108] Moving the virtual object and / or the virtual scene according to the moving trajectory of the prop;
[0109] The virtual objects and / or virtual scenes are processed with special effects, and / or the virtual objects and / or the areas where the virtual objects are located in the scenes are processed with special effects, and / or the virtual objects and / or virtual scenes are made to interact according to a specific storyline.
[0110] Optionally, the three-dimensional virtual scene and / or the virtual object are transformed accordingly according to the moving trajectory of the prop, which may be:
[0111] Moving the virtual object in the virtual scene according to the moving trajectory of the prop;
[0112] When the virtual object moves to a preset position in the virtual scene, a three-dimensional scene interaction is displayed.
[0113] Figure 3 and Figure 4 is one of the specific application examples in the embodiment of the present invention, wherein, Figure 3 It is the aforementioned teaching game of the kitten fishing. It can be seen that the prop is a fish toy placed on the table and within the camera's acquisition range. The user next to it watches it through AR glasses. AR glasses convert the two-dimensional plane image into a three-dimensional virtual object for display. For another user who does not wear AR glasses, all he sees is the two-dimensional kitten fishing game. Figure 4 In the example, the user replaces the AR glasses with a mobile terminal, and can also display the three-dimensional cat fishing game. When the user moves the physical fish toy, the three-dimensional fish also moves with it, and when the fish moves to the three-dimensional fishing rod, the cat lifts the fish and puts it in the bucket.
[0114] Through the method and system provided by the embodiment of the present invention, the image of the physical prop is collected by the camera, and the three-dimensional virtual object and / or three-dimensional virtual scene corresponding to the physical prop is displayed in the three-dimensional space, and when the physical prop moves, the corresponding three-dimensional virtual object and / or three-dimensional virtual scene is interactively displayed. The three-dimensional virtual scene and / or virtual object display in the three-dimensional space is realized, which improves the user experience and the teaching effect.
[0115] An embodiment of the present invention further provides a device, which includes: a processor and a memory for storing a computer program that can be run on the processor, wherein the processor is configured to execute the above-mentioned method for realizing three-dimensional interaction when running the computer program.
[0116] An embodiment of the present invention further provides a storage medium on which computer instructions are stored. When the instructions are executed by a processor, the above-mentioned method for realizing three-dimensional interaction is implemented.
[0117] Figure 5 : is a schematic diagram of a system structure provided by an embodiment of the present invention. The system 500 may include one or more central processing units (CPU) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 532 or data 534. Among them, the memory 520 and the storage medium 530 may be short-term storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the device. Furthermore, the central processing unit 510 may be configured to communicate with the storage medium 530 and execute a series of instruction operations in the storage medium 530 on the system 500. The system 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, and one or more input and output interfaces 560. The steps performed by the above method embodiments may be based on the Figure 5 The system structure shown.
[0118] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0119] Those of ordinary skill in the art will appreciate that the modules and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0120] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0121] Each part of this specification is described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment part.
[0122] Finally, it should be noted that the above is only a preferred embodiment of the technical solution of the present application and is not intended to limit the scope of protection of the present application. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. If these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, any modifications, equivalent substitutions, improvements, etc. made shall be included in the scope of protection of the present application.
Claims
1. A three-dimensional interactive method, It is characterized in that include: Placing a physical prop on a two-dimensional plane, wherein the prop is within the acquisition range of a camera of a three-dimensional interactive system; The three-dimensional interactive system displays a three-dimensional virtual object and / or a virtual scene corresponding to the prop; When the prop moves within the acquisition range of the three-dimensional interactive system, the three-dimensional virtual object and / or virtual scene is converted accordingly according to the movement trajectory of the prop; Obtaining identification information of the prop through a helmet-mounted display HMD or a mobile terminal, wherein the identification information is a pre-set identification mark; The three-dimensional interactive system displays a three-dimensional virtual object and / or a virtual scene corresponding to the prop, including: The three-dimensional interactive system displays three-dimensional virtual objects and / or virtual scenes corresponding to the props in a physical three-dimensional space through holographic projection technology. as well as, The three-dimensional interactive system displays a virtual scene and / or virtual object corresponding to the prop on a two-dimensional display screen, and the virtual object and / or virtual scene displayed on the two-dimensional display screen corresponds to the three-dimensional virtual object and / or virtual scene displayed in the physical three-dimensional space; The three-dimensional virtual object and / or virtual scene is converted accordingly according to the moving trajectory of the prop, including: The virtual object is moved in the virtual scene according to the moving track of the prop; when the virtual object moves to a preset position in the virtual scene, a three-dimensional scene interaction is displayed.
2. The method according to claim 1, It is characterized in that The three-dimensional interactive system displays a three-dimensional virtual object and / or a virtual scene corresponding to the prop, including: The three-dimensional interactive system uses augmented reality (AR) technology to display the three-dimensional virtual objects and / or virtual scenes through the HMD or mobile terminal.
3. The method according to claim 2, It is characterized in that The three-dimensional virtual object and / or virtual scene is converted accordingly according to the moving trajectory of the prop, including: The virtual objects and / or virtual scenes displayed on the two-dimensional display screen are synchronously or asynchronously transformed with the three-dimensional virtual objects and / or virtual scenes displayed by the HMD or the mobile terminal according to the moving trajectory of the prop.
4. The method according to claim 2, It is characterized in that The three-dimensional interactive system uses augmented reality (AR) technology to display three-dimensional virtual objects and / or virtual scenes through a helmet-mounted display (HMD) or a mobile terminal, including: Using computer graphics (CG) technology to construct the three-dimensional virtual object and / or three-dimensional scene model; Using three-dimensional tracking and registration technology to perform virtual-to-real registration processing on the three-dimensional virtual object and / or three-dimensional scene model and the image captured by the HMD or mobile terminal; The three-dimensional virtual object and / or three-dimensional scene model is drawn and rendered in real time, and according to the mapping relationship between the three-dimensional virtual object and / or virtual scene after the virtual-to-real registration processing in the real scene, the three-dimensional virtual object and / or three-dimensional scene model is superimposed and fused with the image of the material culture carrier, and the superimposed and fused image is displayed through the HMD or mobile terminal.
5. The method according to claim 4, It is characterized in that The three-dimensional tracking and registration technology is used to perform virtual-real registration processing on the three-dimensional virtual object and / or three-dimensional scene model and the image collected by the HMD or the mobile terminal, including: Acquire identification information of the prop through the HMD or the mobile terminal; the identification information is a pre-set identification mark; The obtained identification mark is sent to the cloud server, and the cloud server determines the three-dimensional virtual object or three-dimensional scene model corresponding to the identification information according to the identification information, and then calculates the prop image collected by the HMD or mobile terminal, performs three-dimensional coordinate transformation on the image, and loads the three-dimensional virtual object or three-dimensional scene model at a specific position in the image based on the transformed coordinates.
6. The method according to claim 1, It is characterized in that The three-dimensional virtual object and / or virtual scene is converted accordingly according to the moving trajectory of the prop, including: Moving the virtual object and / or the virtual scene according to the moving trajectory of the prop; The virtual object and / or the virtual scene are processed with special effects, and / or the virtual object and / or the location area where the virtual object is located in the virtual scene are processed with special effects, and / or the virtual object and / or the virtual scene are made to interact according to a specific storyline.
7. A three-dimensional interactive system, It is characterized in that The system comprises: a processor and a memory for storing a computer program that can be run on the processor; wherein the processor is used to execute the three-dimensional interaction method according to any one of claims 1 to 6 when running the computer program.
8. A computer-readable storage medium having computer-executable instructions stored thereon, It is characterized in that The computer executable instructions are used to execute the three-dimensional interaction method described in any one of claims 1 to 6.
Citation Information
Patent Citations
Augmented-reality-based three-dimensional interactive learning system and method
CN103366610A
Augmented reality technology-based handheld reading equipment and reading method thereof
CN104102412A
Civil air defense interactive experience method and system
CN106710348A
