A reconstruction system for user experience AR product scenes for developers

Abstract

The present invention discloses a reconstruction system for a user experience AR product scene for developers, including: a user database module for storing user personal information and data generated by experiencing AR products; a resource library module for storing encoded products and their three-dimensional models, and user character models; an environmental image module for storing image data of the user's location; a data processing module for constructing a feature data set for each user based on the user's personal information; a scene component module for storing action components and animation components; a scene reconstruction module for receiving the user's feature data set and establishing a complete virtual scene; and a scene display module for displaying the entire scene on the developer's device. The present invention can perform three-dimensional reconstruction of the virtual scene of the user's experience of AR products, and display it to the developer in a visual manner, thereby accelerating the trial and error and improvement process of augmented reality technology in application, and facilitating providing users with a better AR experience.

Description

Technical Field

[0001] The present invention belongs to the field of augmented reality (AR), and specifically is a reconstruction system for user experience AR product scenes for developers. Background Art

[0002] AR (Augmented Reality) is a technology that calculates the position and angle of camera images in real time and adds corresponding images and 3D models. This technology combines virtual scenes and real scenes by applying virtual information to the real world. That is, real scenes and virtual scenes are superimposed on the same screen or space in real time and exist at the same time, so that virtual information can be perceived by human senses, thereby achieving a more realistic sensory experience.

[0003] Today, augmented reality (AR) technology is developing rapidly and is being applied in various fields such as medicine and games. More and more AR applications are entering people's daily lives. Especially in online sales, AR displays of goods are beginning to appear to help users experience products more effectively. However, the relevant data collected by developers through AR systems are usually not visualized, and it is impossible to receive users' suggestions and feedback on AR display methods in a short period of time, which leads to developers spending a long time on program deployment and trial and error, and slow technology updates. Summary of the invention

[0004] The present invention aims to solve the deficiencies of the above-mentioned prior art and proposes a reconstruction system for user experience AR product scenes for developers, in order to make up for the deficiencies of the existing AR application system and provide user data to developers in a timely manner in a visualized manner, thereby accelerating the trial and error and improvement process of augmented reality (AR) technology in the application, and facilitating providing users with a better AR experience.

[0005] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme:

[0006] The invention provides a reconstruction system for a user experience AR product scene for developers, which includes: a user database module, a resource library module, an environment image module, a data processing module, a scene component module, a scene reconstruction module and a scene display module;

[0007] The user database module is used to store user personal information and data generated by experiencing AR products, and transmit them to the data processing module; the user personal information is the ID of the user identity; the data generated by experiencing AR products include the AR products selected by the user to experience, and the user's interactive behavior data on the AR products;

[0008] The resource library module is used to store the encoded goods and their three-dimensional models, the user's character model, and transmit them to the data processing module;

[0009] The environment image module is used to store image data of the location where the user is when experiencing the AR product, and transmit it to the data processing module; the image data is the environment image obtained by the user using the camera of the device held by the user;

[0010] The data processing module associates other data of the received related users according to the user personal information, thereby constructing a feature data set for each user and transmitting it to the scene reconstruction module;

[0011] The scene component module is used to store action components and animation components; the action components are used to be added to the character model to generate corresponding actions, and the animation components are used to be added to the three-dimensional model of the AR product, and add corresponding animation effects to the three-dimensional model according to the corresponding actions of the character model;

[0012] The scene reconstruction module receives the user's feature data set, and uses the camera in the virtual scene as the origin of the coordinate system, the direction of the camera as the Y axis, and the side direction of the camera as the X axis, so as to determine the Z axis using the spiral rule and establish a virtual scene coordinate system; then, the depth data of the space where the user is located is extracted according to the image data to determine the size of the scene, and reconstruct the scene where the user experiences the AR product, then, the three-dimensional model of the AR product to be experienced and the character model of the corresponding user are added to the scene, and the size and position of the model are adjusted according to the depth data; then, action components are added to the character model according to the interactive behavior data to generate corresponding actions; finally, animation components are added to the three-dimensional model of the AR product, so as to generate corresponding animation effects on the three-dimensional model of the AR product according to the corresponding actions of the character model;

[0013] The scene display module displays the entire scene on the developer's device.

[0014] Compared with the prior art, the present invention has the following beneficial effects:

[0015] The present invention applies virtual reality technology to online application scenarios. Different from the previous AR applications that only provide display effects for users, the system cooperates with the user database module, resource library module, environment image module, scene component module, data processing module, scene reconstruction module and scene display module. On the basis of obtaining the data generated by the user's experience of AR products, the real environment of the user is modeled through 3D reconstruction technology, and character models and 3D models of AR products are added to the reconstructed scene. According to the interactive behavior data and the scene component module, corresponding action effects are generated for the model, thereby realizing the visualization of user data. Developers can obtain user feedback on AR applications more timely and effectively based on the visualized scenes, helping developers to accelerate the trial and error and improvement process, so as to provide users with a better AR experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of an embodiment of a system for reconstructing a user experience AR product scene for developers according to the present invention;

[0017] Figure 2 This is a system schematic diagram of a system for reconstructing a user experience AR product scene for developers according to the present invention. DETAILED DESCRIPTION

[0018] In this embodiment, a reconstruction system for a user experience AR product scene for developers can save the scene in which the user experiences the AR product and the user's experience data, and through the cooperation between the user database module, the resource library module, the environment image module, the scene component module, the data processing module, the scene reconstruction module and the scene display module, the virtual scene of the user experience AR product is three-dimensionally reconstructed and displayed to the developer in a visual manner; Figure 1 As shown, in a specific implementation, the data generated after the user experiences the AR product in the real scene 1 is displayed to the developer through the scene reconstruction system 3 and displayed in the form of scene 2. In the real scene 1, the user holds a mobile device 102 to experience the AR product in front of the real building 101, or uses the building 101 to perform related AR displays. The entire or partial image of the building 101 should be displayed in the mobile device. The relevant data when the user experiences the AR product in the real scene 1 is saved by the scene reconstruction system 3 and displayed to the developer in the form of scene 2. Scene 2 mainly includes 3 sub-scenes. Module 201 mainly includes the user's personal information, such as an ID representing the user's identity; scene 202 displays the specific environment when the user experiences the AR product, and a virtual scene is built through three-dimensional modeling technology; scene 203 displays the animation effects used by AR on the user's mobile device;

[0019] In this embodiment, Figure 2As shown, a reconstruction system for user experience AR product scene for developers includes: a user database module 301, a resource library module 302, an environment image module 303, a data processing module 304, a scene component module 305, a scene reconstruction module 306 and a scene display module 307;

[0020] The user database module 301 is used to store user personal information and data generated by experiencing AR products, and transmit them to the data processing module;

[0021] Furthermore, the data generated after the user experiences the AR product in the real scene 1 will be stored in the user database module 302 and obtained through the permissions provided by the device; the user's personal information is the ID of the user's identity; the data generated by experiencing the AR product includes the AR product selected by the user to experience, and the user's interactive behavior data on the AR product;

[0022] The resource library module 302 is used to store the dimensional data of the encoded commodity and its three-dimensional model, and the user's character model, and transmit it to the data processing module;

[0023] Furthermore, the product code is used to distinguish different types of products and associate the three-dimensional model of the product; the user's character model is used to represent the user himself in the virtual scene; the product three-dimensional model and the character model are designed models, the product model is consistent with the physical appearance of the product, and the character model is represented by two universal male and female models; the size data of the product three-dimensional model and the user's character model are set to a certain normal value;

[0024] The environmental image module 303 is used to store image data of the location where the user experiences the AR product and transmit it to the data processing module;

[0025] Furthermore, the image data is scanned and recorded by calling the camera of the device 102 held by the user to scan and record the surrounding environment, and the recorded data is saved in the form of an image;

[0026] The data processing module 304 associates the received other data of the relevant users according to the user personal information, thereby constructing a feature data set for each user and transmitting it to the scene reconstruction module;

[0027] Furthermore, user personal information is an ID representing the user's identity. Based on each user's unique ID, other data about the user can be searched, such as which products the user has experienced, which interactive behaviors the user has implemented on these products, and the scene images the user is in. All the user's data is collected and a feature data set representing the user is constructed;

[0028] The scene component module 305 is used to store the action components and animation components used in the process of reconstructing the virtual scene, and transmit them to the scene reconstruction module;

[0029] Furthermore, the action component is used to be added to the character model so that the character model can make human-like movements. According to the interactive behavior data, corresponding actions can be added to the character model, such as user clicks, slides, and two-finger slides on the screen. The animation component is used to be added to the three-dimensional model of the AR product so that the three-dimensional model of the AR product has a smooth animation display effect. According to the corresponding actions of the character model, corresponding animation effects are added to the three-dimensional model, such as product rotation, zooming in and out, etc.

[0030] The scene reconstruction module 306 receives the user's feature data set, and uses the camera in the virtual scene as the origin of the coordinate system, the direction of the camera as the Y axis, and the side direction of the camera as the X axis, so as to determine the Z axis using the spiral rule and establish a virtual scene coordinate system; then, the depth data of the user's space is extracted according to the image data to determine the size of the scene, and reconstruct the scene of the user experiencing the AR product; then, the three-dimensional model of the AR product to be experienced and the character model of the corresponding user are added to the scene, and the size and position of the model are adjusted according to the depth data; then, action components are added to the character model according to the interactive behavior data to generate corresponding actions; finally, animation components are added to the three-dimensional model of the AR product, so as to generate corresponding animation effects on the three-dimensional model of the AR product according to the corresponding actions of the character model;

[0031] Furthermore, the camera in the virtual scene is an AR camera. A coordinate system is established in the virtual scene, with the AR camera as the origin of the coordinate system, the direction of the camera as the coordinate axis Y, the side of the camera (usually 90 degrees to the right) as the coordinate axis X, and the coordinate axis Z is determined according to the screw rule to obtain the virtual scene coordinate system;

[0032] Furthermore, the image data is an environmental image of the scene where the user is located, and the depth data in the image is extracted using the SIFT algorithm; the depth data represents the distance between a feature point in the virtual scene and the origin of the coordinate system, and can be used to determine the size of the entire virtual scene; the depth data of each feature point in the scene is different, and can be used to process the occlusion relationship and the distance relationship between objects in the scene, and can be used to determine the position between objects in the scene, and reconstruct the scene where the user experiences the AR product; Figure 1 , reconstructing the real scene not including the user in scene 1 into a virtual scene not including the user in scene 202;

[0033] Furthermore, a 3D model of the AR product experienced by the user and a character model corresponding to the user are added to the virtual scene. The model includes dimension data, and the size and position of the model in the scene can be determined based on the scene depth data.

[0034] Furthermore, the interactive behavior data is the interaction between the user and the AR product. Generally, the product model is placed by clicking the screen, the product model is rotated by sliding the screen, and the product model is zoomed in and out by sliding the screen with two fingers. The interactive behavior data is the collection of these action data.

[0035] Furthermore, after adding an action component to the character model, the action in the interaction process is identified using the action component according to the interaction data, and the action is added to the character model so that the character model can perform the same actions as the user, such as clicking the screen, sliding the screen, and sliding the screen with two fingers.

[0036] Furthermore, after adding an animation component to the 3D model of the AR product, the animation component will add corresponding animation effects to the 3D model according to the action data saved in the action component, such as the movement, rotation, and scaling of the model, so that a smooth display effect can be achieved between the 3D model of the product and the 3D model of the character;

[0037] The scene display module 307 displays the entire scene on the developer's device 308;

[0038] Furthermore, the entire scene is the entire virtual scene established by the scene reconstruction module 306, which includes the environment of the user's location, the model representing the user and the three-dimensional model of the AR product, and the smooth display effect between the models;

[0039] Furthermore, the device 308 is mainly for the developer side, and is connected to the scene display module 307, and is used to display the entire scene to the developer; the information displayed in the device 308 can be as follows: Figure 1 In the scene 2, in addition to displaying the entire virtual scene 202, specific data 201 of the user's personal information and specific AR animation effects 203 during the user's experience in the handheld device 102 are also displayed.

Claims

1. A reconstruction system for user experience AR product scenes for developers, which features include: User database module, resource library module, environment image module, data processing module, scene component module, scene reconstruction module and scene display module; The user database module is used to store user personal information and data generated by experiencing AR products, and transmit them to the data processing module; the user personal information is the ID of the user identity; the data generated by experiencing AR products include the AR products selected by the user to experience, and the user's interactive behavior data on the AR products; The resource library module is used to store the encoded goods and their three-dimensional models, the user's character model, and transmit them to the data processing module; The environment image module is used to store image data of the location where the user is when experiencing the AR product, and transmit it to the data processing module; the image data is the environment image obtained by the user using the camera of the device held by the user; The data processing module associates other data of the received related users according to the user personal information, thereby constructing a feature data set for each user and transmitting it to the scene reconstruction module; The scene component module is used to store action components and animation components; the action components are used to be added to the character model to generate corresponding actions, and the animation components are used to be added to the three-dimensional model of the AR product, and add corresponding animation effects to the three-dimensional model according to the corresponding actions of the character model; The scene reconstruction module receives the user's feature data set, and takes the camera in the virtual scene as the origin of the coordinate system, the direction of the camera as the Y axis, and the direction of the side of the camera as the X axis, so as to determine the Z axis using the screw rule and establish a virtual scene coordinate system; then, the depth data of the space where the user is located is extracted according to the image data to determine the size of the scene, and reconstruct the scene in which the user experiences the AR product, then, the three-dimensional model of the AR product to be experienced and the character model of the corresponding user are added to the scene, and the size and position of the model are adjusted according to the depth data; then, action components are added to the character model according to the interactive behavior data to generate corresponding actions; finally, animation components are added to the three-dimensional model of the AR product, so as to generate corresponding animation effects on the three-dimensional model of the AR product according to the corresponding actions of the character model; The scene display module displays the entire scene on the developer's device.

Images