Memory scene reproduction and interaction method

Abstract

The invention discloses a memory scene reproduction and interaction method, which comprises the following steps: S1, scene acquisition: a mobile terminal scans a scene based on a depth camera module, uploads acquired information to a cloud server, and builds a three-dimensional scene model; s2, scene reproduction: aligning the mobile terminal to the anchor position of the three-dimensional scene model, and completing dynamic superposition of the three-dimensional scene model according to the current angle of the mobile terminal; and S3, scene interaction: a depth camera module of the mobile terminal is aligned with a user, and the user photographs in the three-dimensional scene model. According to the method, the scene is automatic by the user, and the experience feeling is good.

Description

Technical Field

[0001] The present invention relates to the technical field of augmented reality, and in particular to a memory scene reproduction and interaction method. Background Art

[0002] Augmented Reality (AR) is a technology that "seamlessly" integrates real-world information and virtual-world information. It uses specific technical means to simulate and superimpose physical information that is difficult to experience in a certain time and space range in the real world, and applies virtual information to the real world, which is perceived by human senses, thereby achieving a sensory experience beyond reality. The real environment and virtual objects are superimposed on the same screen or space in real time and exist simultaneously.

[0003] In current AR applications, developers usually store designed images in smart devices. The experience scenes cannot be customized by users, are not personalized enough, and cannot meet user experience needs. Summary of the invention

[0004] The purpose of the present invention is to provide users with memory scene reproduction and interaction functions based on smart devices to enhance the user's scene experience.

[0005] To achieve the above object, the present invention provides the following technical solutions: A memory scene reproduction and interaction method comprises the following steps: S1, scene acquisition: the mobile terminal scans the scene based on the depth camera module, uploads the collected information to the cloud server, and builds a three-dimensional scene model; S2, scene reproduction, aligning the mobile terminal with the anchor point position of the 3D scene model, and completing the dynamic superposition of the 3D scene model according to the current angle of the mobile terminal; S3, scene interaction, the depth camera module of the mobile terminal is aimed at the user, and the user takes a picture in the three-dimensional scene model.

[0006] Preferably, the step S1 specifically comprises: S101, scene scanning, calibrating anchor points and slowly scanning the scene; S102, data upload, uploading the scene video, depth information parameters, and user-defined text information and voice information generated by the scan to the cloud server; S103, image frame extraction: the cloud program performs frame processing on the video, performs similarity comparison on each frame image, finds the key frame image and forms a key frame image collection; S104, 3D modeling, performing boundary stitching on the key frame images according to the depth information of the key frame images and their corresponding frames, to obtain a 3D scene model fused with the key frame images; S105, data storage, binding the key frame image collection, text information, voice information, and labels of the three-dimensional scene model, and completing classified storage.

[0007] Preferably, the step S2 is specifically as follows: S201, terminal positioning, moving the mobile terminal to a designated location based on GPS positioning; S202, model superposition, the mobile terminal checks the current position, finds the scene position corresponding to the anchor point of the 3D scene model, and then superimposes and reproduces the scene and the 3D scene model; S203, motion tracking, performing angle conversion on the three-dimensional scene model according to the current angle information of the mobile terminal, so as to achieve dynamic superposition of the scene and the three-dimensional scene model; S204, picture adjustment, identifying and filtering the current scene object type according to the universal recognition algorithm, and completing the masking of the three-dimensional scene model picture.

[0008] Preferably, the step S3 includes two modes: self-portrait and remote group photo taking; Selfie: Users download the 3D scene model and use the depth camera module to shoot themselves into the model screen to complete the selfie action; Remote group photo taking: Users can send 3D scene models to other users through the cloud sharing function. Two users can use remote communication technology to shoot themselves into the model and avoid overlapping of characters by adjusting the model angle.

[0009] Preferably, in step S101, the anchor point is selected by using an AR card or a screen indicator box, and the screen indicator box selects scene elements by using the User Defined Targets technology.

[0010] Preferably, in step S202, the mobile terminal performs an image similarity check on the screen display screen based on the first key frame image, finds the corresponding reference object, and then finds the anchor point of the three-dimensional scene model based on the angle parameter of the first key frame image.

[0011] The present invention adopts the memory scene reproduction and interaction method of the above structure, and the scene is made by the user according to his own needs, with high degree of freedom and higher user immersion. It is automatically generated in the cloud, with low operation difficulty and short waiting time, which can meet the user's function of shooting and using at any time, and supports remote sharing, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1It is a module schematic diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0013] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0014] A memory scene reproduction and interaction method as shown in the figure includes the following steps: S1. Scene Collection The mobile terminal scans the scene based on the depth camera module, uploads the collected information to the cloud server, and builds a three-dimensional scene model.

[0015] S101, scene scanning, calibrating anchor points and slowly scanning the scene. Anchor points are selected by AR cards or screen indicator boxes, and the screen indicator box selects scene elements by using User Defined Targets technology.

[0016] S102, data upload, upload the scene video, depth information parameters, and user-defined text information and voice information generated by scanning to the cloud server. Among them, the depth information parameter is a continuous recording process, which includes time, three-dimensional point cloud data, angle and other parameters. Each time point is encapsulated in a dictionary format, and the dictionary format is stored in a list format.

[0017] S103, image frame extraction: the cloud program processes the video in frames, performs similarity comparison on each frame image, finds the key frame image and forms a key frame image collection.

[0018] Similarity comparison, using image recognition algorithm to extract image features. Taking the first two key frame images as an example, analyze the similarity between the first frame image and the subsequent images respectively. When the similarity between the first frame image and the Nth frame image is the smallest and there are still similar parts, the N+1th frame image is judged to be the second key frame image.

[0019] S104, three-dimensional modeling, based on the depth information of the key frame images and their corresponding frames, the key frame images are spliced ​​at the edges, the overlapping parts are removed, and the spliced ​​parts are smoothed using an iterative or convolution method to obtain a three-dimensional scene model fused with the key frame images.

[0020] S105, data storage, binding the key frame image collection, text information, voice information, and labels of the three-dimensional scene model, and completing classified storage.

[0021] S2. Scene Reproduction Aim the mobile terminal at the anchor point of the 3D scene model and dynamically overlay the 3D scene model based on the current angle of the mobile terminal.

[0022] S201: Terminal positioning: moving the mobile terminal to a designated location based on GPS positioning.

[0023] S202, model superposition, the mobile terminal checks the current position, finds the scene position corresponding to the anchor point of the 3D scene model, and then superimposes and reproduces the current scene and the 3D scene model. In the process of position checking, the mobile terminal performs an image similarity check on the screen display screen according to the first key frame image, finds the corresponding reference object, and then finds the anchor point of the 3D scene model according to the angle parameter of the first key frame image.

[0024] S203 , motion tracking, performing angle conversion on the three-dimensional scene model according to the current angle information of the mobile terminal, so as to achieve dynamic superposition of the scene and the three-dimensional scene model.

[0025] S204, picture adjustment, identifying and filtering the current scene object type according to the universal recognition algorithm, and completing the masking of the three-dimensional scene model picture.

[0026] S3. Scene Interaction The depth camera module of the mobile terminal is aimed at the user, and the user takes photos in the three-dimensional scene model, including two modes: selfie and remote group photo.

[0027] Selfie: Users download a 3D scene model and use the depth camera module to capture themselves within the model image, thus taking a selfie.

[0028] Remote group photo taking: Users can send 3D scene models to other users through the cloud sharing function. Two users can use remote communication technology to shoot themselves into the model and avoid overlapping of characters by adjusting the model angle.

[0029] In this method, scene interaction can be used separately from scene reproduction, that is, during the recording process, it is only necessary to connect multiple users to one scene, and the background environment can be processed by green screen matting.

[0030] In this embodiment, the mobile terminal can be a smart phone. Taking the iPhone X as an example, the phone comes with a depth camera and a variety of sensors, and has the latitude and longitude positioning function of the phone. In terms of self-positioning and map reconstruction navigation of the depth camera, the three-dimensional visual SLAM application is significantly better than the two-dimensional visual SLAM, so that more complete image information can be obtained, which is convenient for generating three-dimensional models in the cloud. During remote interaction. With the help of 5G high-quality channels, the delay in the process of adjusting the model angle can be significantly reduced, thereby improving the efficiency of remote group shooting.

[0031] The above are specific embodiments of the present invention, but the protection scope of the present invention should not be limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention, so the protection scope of the present invention should be based on the protection scope defined in the claims.

Claims

1. A memory scene reproduction and interaction method, characterized in that: The steps include: S1, scene acquisition: the mobile terminal scans the scene based on the depth camera module, uploads the collected information to the cloud server, and builds a three-dimensional scene model; S2, scene reproduction, aligning the mobile terminal with the anchor point position of the 3D scene model, and completing the dynamic superposition of the 3D scene model according to the current angle of the mobile terminal; S3, scene interaction, the depth camera module of the mobile terminal is aimed at the user, and the user takes a picture in the three-dimensional scene model.

2. The memory scene reproduction and interaction method according to claim 1, characterized in that: The S1 step is specifically as follows: S101, scene scanning, calibrating anchor points and slowly scanning the scene; S102, data upload, uploading the scene video, depth information parameters, and user-defined text information and voice information generated by the scan to the cloud server; S103, image frame extraction: the cloud program performs frame processing on the video, performs similarity comparison on each frame image, finds the key frame image and forms a key frame image collection; S104, 3D modeling, performing boundary stitching on the key frame images according to the depth information of the key frame images and their corresponding frames, to obtain a 3D scene model fused with the key frame images; S105, data storage, binding the key frame image collection, text information, voice information, and labels of the three-dimensional scene model, and completing classified storage.

3. The memory scene reproduction and interaction method according to claim 1, characterized in that: The S2 step is specifically as follows: S201, terminal positioning, moving the mobile terminal to a designated location based on GPS positioning; S202, model superposition, the mobile terminal checks the current position, finds the scene position corresponding to the anchor point of the 3D scene model, and then superimposes and reproduces the scene and the 3D scene model; S203, motion tracking, performing angle conversion on the three-dimensional scene model according to the current angle information of the mobile terminal, so as to achieve dynamic superposition of the scene and the three-dimensional scene model; S204, picture adjustment, identifying and filtering the current scene object type according to the universal recognition algorithm, and completing the masking of the three-dimensional scene model picture.

4. The memory scene reproduction and interaction method according to claim 1, characterized in that: The step S3 includes two modes: self-portrait and remote group photo taking; Selfie: Users download the 3D scene model and use the depth camera module to shoot themselves into the model screen to complete the selfie action; Remote group photo taking: Users can send 3D scene models to other users through the cloud sharing function. Two users can use remote communication technology to shoot themselves into the model and avoid overlapping of characters by adjusting the model angle.

5. The memory scene reproduction and interaction method according to claim 2, characterized in that: In the step S101, the anchor point is selected by using an AR card or a screen indicator box, and the screen indicator box selects scene elements by using the User Defined Targets technology.

6. The memory scene reproduction and interaction method according to claim 3, characterized in that: In the step S202, the mobile terminal performs an image similarity check on the screen display screen according to the first key frame image, finds the corresponding reference object, and then finds the anchor point of the three-dimensional scene model according to the angle parameter of the first key frame image.

Images