File browsing system based on AR (Augmented Reality) equipment
By designing a file browsing system on an AR device, using high-precision sensors and cameras to obtain user gesture information, realizing three-dimensional visualization and interactive operation of files, solving the problem that files can only be viewed in two-dimensionally in the existing technology, and improving operation efficiency and Security, suitable for applications in multiple fields.
Patent Information
- Application Number
- CN202411136023.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-12-06
AI Technical Summary
In the prior art, files can only be viewed on two-dimensional planes, which are troublesome and inconvenient in harsh working scenarios, especially when you need to view drawings, instructions, videos and other media, especially if you need to manually read paper materials or click on your mobile phone. inconvenient.
A file browsing system based on AR devices is designed, including file classification unit, file browsing unit, user gesture acquisition unit and file operation unit, which supports three-dimensional operation of files in the AR environment, and obtain user gestures through high-precision sensors and cameras. Information, combined with image processing and spatial positioning algorithms, realizes the visualization and interactive operation of files.
It realizes the three-dimensional operation of files by users in the AR environment, with simple interaction operation, supports browsing and operation of multiple types of files, improves operation efficiency and security, eliminates security risks in some jobs, and is suitable for multiple fields. application.
Smart Images

Figure CN119088217A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of augmented reality, and in particular, the present invention relates to a file browsing system based on an AR device. Background Art
[0002] Augmented Reality (AR) is an innovative technology that superimposes virtual information on the real world. It enhances the user's perception and interactive experience of the real environment through computer-generated images, videos, audio or other sensory inputs. The core of AR technology is "enhancement", that is, adding additional layers and functions to the real world without replacing it.
[0003] Human-Computer Interaction Techniques refers to the technology that enables people to communicate with computers in an effective way through computer input and output devices. Human-Computer Interaction Techniques include machines providing people with a large amount of relevant information and prompts through output or display devices, and people inputting relevant information to machines through input devices, answering questions and prompts, etc.
[0004] The existing documents can only be viewed in two dimensions, which has certain limitations. For example, if frontline workers need to view work details or related documents based on drawings, instructions, videos and other media on site, most of them require paper materials or electronic devices such as mobile phones. At this time, they need to manually flip through paper materials or click on mobile phones to view relevant information, which is troublesome, especially in some harsh working scenarios. Summary of the invention
[0005] The present invention aims to overcome the shortcomings of the prior art and proposes a file browsing system based on an AR device to achieve the following objectives: enabling users to perform three-dimensional operations on files in an AR environment, with simple interactive operations and support for multiple types of files.
[0006] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a file browsing system based on an AR device, the system includes a file classification unit and a file browsing unit, the file browsing unit further includes a file display unit, a user gesture acquisition unit, a user gesture position judgment unit, and a file operation unit; the file classification unit is used to obtain files in an AR scene and put files of the same type therein into the same folder; the file browsing unit is used to realize file browsing and interaction of users in an AR scene, wherein the file display unit is used to realize visualization of files of different types in an AR scene; the user gesture acquisition unit is used to obtain gesture information of users in an AR scene; the user gesture position judgment unit is used to analyze the gesture information and determine the position and action of the user gesture relative to the file in the AR scene; the file operation unit is used to combine the gesture information and the position and action of the user gesture relative to the file in the AR scene to realize interaction with the file.
[0007] Preferably, the file classification unit identifies the file type by identifying the file extension, and creates a corresponding folder for each file type to store files of the corresponding type, and the file types include pictures, videos, PDFs, and models.
[0008] Preferably, the file display unit supports the identification and display of different types of files; the file display unit supports three-dimensional visualization of model files; the file display unit supports static thumbnail display and video content playback for video files; the file display unit supports simultaneous display of multiple files, that is, supports users to view multiple file contents side by side in an AR scene; the file display unit supports augmented reality overlay, that is, superimposing virtual file contents in an AR scene on a real-world view; the file display unit supports interactive browsing, that is, users interact with files in the AR scene through gesture input.
[0009] Preferably, the gestures supported for recognition by the user gesture acquisition unit include but are not limited to single-finger click, two-finger zoom, slide, rotate, and grab.
[0010] Preferably, the gesture information acquired by the user gesture acquisition unit includes the user operation intention obtained by analyzing and identifying the gesture.
[0011] Preferably, the user gesture acquisition unit includes a high-precision sensor and a high-precision camera, and the high-precision sensor and the high-precision camera are used to acquire the position information and movement trajectory of the user's hand in real time.
[0012] Preferably, the user gesture position judgment unit determines the position and action of the user gesture relative to the file in the AR scene based on image processing and spatial positioning algorithms and in combination with the gesture information of the user gesture acquisition unit to determine whether the user gesture interacts with the file in the AR scene.
[0013] Preferably, for pictures and PDF files, the file operation unit combines the gesture information, and the position and action of the user gesture relative to the file in the AR scene, to support the user to zoom in, zoom out, rotate and turn pages of the file through gestures.
[0014] Preferably, for model-type files, the file operation unit supports transformation of the model in combination with the gesture information, and the position and action of the user gesture relative to the file in the AR scene, and specifically includes a model expansion judgment unit, a model closure judgment unit, a model transformation control unit and a model state storage unit, wherein the model expansion judgment unit is used to judge the opening and closing state of the user's left hand gesture, and when it is detected that the left thumb and the left index finger are opened and closed to reach a preset expansion threshold, the model expansion is triggered; the model closure judgment unit is used to judge the closing state of the user's left hand gesture, and when it is detected that the left thumb and the left index finger are closed to reach a preset closing threshold, the model closure is triggered; the model transformation control unit is used to control the expansion and closing actions of the model according to the judgment results of the model expansion judgment unit and the model closure judgment unit; the model state storage unit is used to store the current state of the model, including the expansion state and the closing state.
[0015] Preferably, for video files, the file operation unit combines the gesture information, and the position and action of the user gesture relative to the file in the AR scene, to support: controlling the playback and pause of the video by clicking the video playback area with a single finger when playing the video file; adjusting the playback progress of the video on the video playback bar through left and right sliding gestures; and adjusting the volume of the video through up and down sliding gestures.
[0016] The technical effect of the present invention is: the present invention enables users to wear AR devices to view local pictures, PDFs, videos and model files, and sets different operation methods for different files, which not only enriches the interactive methods of users to operate files, but also abandons the original two-dimensional viewing of files, allowing users to perform three-dimensional operations on files in an AR environment, and the interactive operation is simple, with certain practical use effects. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 The present invention is a structural block diagram of a file browsing system based on an AR device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0018] The following is a further detailed description of the specific implementation of the present invention through the description of the embodiments with reference to the accompanying drawings, the purpose of which is to help those skilled in the art have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention, and to facilitate its implementation. It should be noted that the words "first", "second" and the like described in this application are only for the convenience of describing the technical solution to distinguish different components, and are not used to limit this application. In order to make the technical solution of the present invention clearer, the present invention is explained through the following embodiments.
[0019] like Figure 1 A file browsing system based on an AR device is shown, the system includes a file classification unit and a file browsing unit, the file browsing unit further includes a file display unit, a user gesture acquisition unit, a user gesture position judgment unit, and a file operation unit; the file classification unit is used to acquire files in an AR scene and place files of the same type in the same folder; the file browsing unit is used to enable users to browse and interact with files in an AR scene, wherein the file display unit is used to visualize files of different types in an AR scene; the user gesture acquisition unit is used to acquire gesture information of the user in the AR scene; the user gesture position judgment unit is used to analyze the gesture information and determine the position and action of the user gesture relative to the file in the AR scene; the file operation unit is used to combine the gesture information and the position and action of the user gesture relative to the file in the AR scene to enable interaction with the file.
[0020] Specifically, in this embodiment, the file classification unit identifies the file type by identifying the file extension, and creates a corresponding folder for each file type to store files of the corresponding type, including pictures, videos, PDF, and models. For example, a picture folder is created to automatically store picture files such as JPG and PNG; a PDF folder is created to automatically store PDF files; a model folder is created to automatically store model files such as FBX and OBJ; a video folder is created to automatically store video files such as MP4 and AVI. After all files are classified into corresponding folders, click the required folder button in the system to pop up files of the corresponding type.
[0021] In this embodiment, the file browsing unit is used to adopt different operation modes for viewing different files in an AR scene. The file browsing unit includes a file display unit, a user gesture acquisition unit, a user gesture position determination unit, and a file operation unit.
[0022] The file display unit of this embodiment is mainly used to display pictures, PDF, models and video files in AR scenes. Specifically, the file display unit supports the identification and display of different types of files, and the file type can also be identified by identifying the file extension; the file display unit supports three-dimensional visualization of model files, that is, the file display unit can display model files in three dimensions in the AR environment, so that users can observe the details of the model from different angles and dimensions; the file display unit supports dynamic content display. For video files, it can not only display static thumbnails of videos, but also play video file contents, providing a richer visual experience; the file display unit supports multi-file synchronous display, that is, it supports users to view multiple file contents side by side in AR scenes; the file display unit supports augmented reality overlay, that is, overlaying virtual file contents in AR scenes on the real world view to achieve the fusion display of file contents and the real world; the file display unit supports interactive browsing, that is, users interact with files in AR scenes through gesture input, such as zooming in to view the details of pictures or PDF files, or operating model files to view different sides. In summary, the file display unit is suitable for most commonly used file formats and has a certain adaptability.
[0023] The user gesture acquisition unit of this embodiment includes a high-precision sensor and a high-precision camera, which are used to obtain the position information and movement trajectory of the user's hand in real time. Among them, the gestures supported by the user gesture acquisition unit include but are not limited to common gestures such as single-finger click, double-finger zoom, slide, rotate, grab, etc. At the same time, this embodiment also obtains the user's operation intention by analyzing and processing the gesture data, and sends the user's operation intention as part of the gesture information to the user gesture position judgment unit for further analysis. Based on the above, the user gesture acquisition unit can provide users with an intuitive, flexible and efficient operation interface, making the file browsing system based on AR devices more intelligent and user-friendly.
[0024] In this embodiment, the user gesture position judgment unit is used to judge whether the gesture information is in contact with the operated file. Specifically, the user gesture position judgment unit determines the position and action of the user gesture relative to the file in the AR scene based on the existing image processing and spatial positioning algorithms and combined with the gesture information of the user gesture acquisition unit to judge whether the user gesture interacts with the file in the AR scene, such as clicking, grabbing, dragging and other operations. Specifically, in the user gesture position judgment unit, the image processing algorithm first recognizes and tracks the gesture, and then the spatial positioning algorithm calculates the precise position and direction of the hand in the AR environment based on the results of the image processing. In this way, the system can determine whether the user's gesture is in contact with the virtual file and determine the intention of the gesture, such as clicking, dragging or zooming.
[0025] The image processing algorithm plays a key role in the user gesture position determination unit. It is mainly responsible for extracting and analyzing gesture features from the visual data received by the user gesture acquisition unit. The image processing algorithm includes but is not limited to the following algorithms:
[0026] Skin color detection: By identifying the skin color of the user's hand, the hand area is separated from the background.
[0027] Contour Tracking: Identify and track the contours of your hands to determine the shape and motion of your gestures.
[0028] Feature point recognition: Identify specific feature points in hand images, such as fingertips, palm centers, etc. The movement of these points can be used to recognize specific gestures.
[0029] Gesture recognition: Identify the user's gesture type by analyzing the changes in feature points and the shape of the hand contour.
[0030] The spatial positioning algorithm is responsible for determining the position and direction of the user's gesture in three-dimensional space. The spatial positioning algorithm includes but is not limited to the following algorithms:
[0031] Stereo vision: Use multiple cameras to capture images from different angles and determine the spatial position of gestures through triangulation.
[0032] Structured light or time of flight (ToF): This technique uses light of a known pattern to emit and measure the time or intensity of the reflected light to obtain the 3D shape and position of the hand.
[0033] Inertial Measurement Unit (IMU): Combines accelerometer and gyroscope data to estimate the position and orientation of the hand in space.
[0034] The present invention can ensure that users can interact with virtual files accurately and naturally in AR scenes through the user gesture position judgment unit, providing reliable support for file browsing and operation. The accurate judgment of the user gesture position judgment unit also ensures the stability and security of the system, avoiding misoperation and recognition errors.
[0035] In this embodiment, the file operation unit is used to combine the gesture information. After the file is judged by the user gesture position judgment unit, the file changes according to the different operations of the gesture. The file changes include moving, rotating, scaling, etc. At the same time, the file operation unit of the present invention sets different operation methods for different files.
[0036] For images and PDF files, the file operation unit combines gesture information, as well as the position and movement of the user's gesture relative to the file in the AR scene, to support users to zoom in, zoom out, rotate, and turn pages of files through gestures, thereby greatly improving the user's operating experience and interaction efficiency.
[0037] For model files, the file operation unit supports transformation of the model by combining gesture information, as well as the position and movement of the user's gesture relative to the file in the AR scene. When operating a model file, the user can move the position of the model in the AR environment through a single-finger drag gesture, and adjust the position and angle of the model in the view, making it easier to view and operate; the user can rotate the model freely in three-dimensional space through a single-finger or multi-finger rotation gesture to view all sides and angles of the model; the user can scale the model through a two-finger pinch gesture to enlarge or reduce the size of the model in order to observe the details or overall effect of the model in more detail. For complex model files, users can dynamically split and combine the model, split the complex model into separate components through gesture operations for independent viewing, or recombine the components to view the overall structure. For example, the model can expand and close as the left-hand gesture expands and closes.
[0038] For model type files, the file operation unit of this embodiment specifically includes a model expansion judgment unit, a model closure judgment unit, a model transformation control unit and a model state storage unit, wherein the model expansion judgment unit is used to judge the opening and closing state of the user's left hand gesture, and when it is detected that the left thumb and the left index finger are opened and closed to reach a preset expansion threshold (which can be flexibly set according to actual conditions), the model expansion is triggered; the model closure judgment unit is used to judge the closing state of the user's left hand gesture, and when it is detected that the left thumb and the left index finger are closed to reach a preset closing threshold (which can be flexibly set according to actual conditions), the model closure is triggered; the model transformation control unit is used to control the expansion and closing actions of the model according to the judgment results of the model expansion judgment unit and the model closure judgment unit; the model state storage unit is used to store the current state of the model, including the expansion state and the closing state.
[0039] For video files, the file operation unit combines gesture information, as well as the position and action of the user gesture relative to the file in the AR scene, and supports the following when playing video files: controlling the playback and pause of the video by clicking the video playback area with one finger. A single click can start or pause the playback of the video, providing a simple and intuitive control method; adjusting the playback progress of the video on the video playback bar through left and right sliding gestures, that is, sliding the gesture left or right can forward or rewind the video, allowing users to quickly find content of interest; adjusting the volume of the video through up and down sliding gestures, where swiping up increases the volume and swiping down decreases the volume, providing users with convenient volume control.
[0040] The present invention provides a file browsing system based on an AR device, which enables users to perform three-dimensional operations on files in an AR environment, and the interactive operation is simple, with certain practical effects. In view of the inconvenient operation in the prior art, the present invention realizes the browsing and viewing of drawings, instructions, 3D models, videos and other files through AR devices. The operation is not only simple and convenient, but also can eliminate safety hazards during operations under certain circumstances. At the same time, the present invention has a wide range of applications and has convenient and reliable applications in multiple fields. For example:
[0041] Engineering field: When engineers are performing on-site construction or maintenance, they can use AR devices to view engineering drawings and 3D models with their hands free, intuitively understand complex engineering structures, and play relevant installation or maintenance video tutorials to guide on-site operations. This approach not only improves work efficiency, but also reduces engineering problems caused by misunderstandings.
[0042] Education: Teachers and students can use the system for interactive learning in the classroom. For example, in biology class, teachers can display 3D biological models and disassemble them in real time to help students better understand complex biological structures; in history class, historical figures or events can be vividly presented in the form of videos through AR technology, making learning more interesting and intuitive.
[0043] Design field: In the product design and architectural design industries, designers can use the system to intuitively view and modify 3D models in an AR environment. With simple gestures, designers can rotate, scale, and split models to view details, or overlay and compare models with objects in the real world to improve design accuracy and efficiency. In addition, designers can quickly browse related PDF instructions or watch design demonstration videos without switching between different applications or devices.
[0044] The present invention is described above by way of example in conjunction with the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-mentioned methods. As long as various non-substantial improvements are made using the method concept and technical solution of the present invention; or the above-mentioned concept and technical solution of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.
Claims
1. A file browsing system based on an AR device, characterized in that: The system includes a file classification unit and a file browsing unit, and the file browsing unit includes a file display unit, a user gesture acquisition unit, a user gesture position judgment unit, and a file operation unit; the file classification unit is used to acquire files in an AR scene and place files of the same type in the same folder; the file browsing unit is used to enable users to browse and interact with files in an AR scene, wherein the file display unit is used to visualize files of different types in an AR scene; the user gesture acquisition unit is used to acquire gesture information of the user in the AR scene; the user gesture position judgment unit is used to analyze the gesture information and determine the position and action of the user gesture relative to the file in the AR scene; the file operation unit is used to combine the gesture information and the position and action of the user gesture relative to the file in the AR scene to achieve interaction with the file.
2. The file browsing system based on AR device according to claim 1, characterized in that: The file classification unit identifies the file type by identifying the file extension, and creates a corresponding folder for each file type to store files of the corresponding type, wherein the file types include pictures, videos, PDFs, and models.
3. A file browsing system based on an AR device according to claim 1 or 2, characterized in that: The file display unit supports the identification and display of different types of files; the file display unit supports three-dimensional visualization of model files; the file display unit supports static thumbnail display and video content playback for video files; the file display unit supports simultaneous display of multiple files, that is, supports users to view multiple file contents side by side in an AR scene; the file display unit supports augmented reality overlay, that is, overlaying virtual file contents in an AR scene on a real-world view; The file display unit supports interactive browsing, that is, the user interacts with the file in the AR scene through gesture input.
4. The file browsing system based on AR device according to claim 1, characterized in that: The gestures supported by the user gesture acquisition unit include but are not limited to single-finger click, double-finger zoom, slide, rotate, and grab.
5. The file browsing system based on AR device according to claim 4, characterized in that: The gesture information acquired by the user gesture acquisition unit includes the user operation intention obtained by analyzing and identifying the gesture.
6. A file browsing system based on an AR device according to any one of claims 1, 4 or 5, characterized in that: The user gesture acquisition unit includes a high-precision sensor and a high-precision camera, and the high-precision sensor and the high-precision camera are used to acquire the position information and movement trajectory of the user's hand in real time.
7. The file browsing system based on AR device according to claim 1, characterized in that: The user gesture position determination unit determines the position and action of the user gesture relative to the file in the AR scene based on image processing and spatial positioning algorithms and in combination with the gesture information of the user gesture acquisition unit to determine whether the user gesture interacts with the file in the AR scene.
8. The file browsing system based on AR device according to claim 1, characterized in that: For pictures and PDF files, the file operation unit combines the gesture information and the position and action of the user gesture relative to the file in the AR scene to support the user to zoom in, zoom out, rotate and turn pages of the file through gestures.
9. The file browsing system based on AR device according to claim 1, characterized in that: For model-type files, the file operation unit supports transformation of the model in combination with the gesture information, and the position and action of the user gesture relative to the file in the AR scene, and specifically includes a model expansion judgment unit, a model closure judgment unit, a model transformation control unit and a model state storage unit, wherein the model expansion judgment unit is used to judge the opening and closing state of the user's left hand gesture, and when it is detected that the left thumb and the left index finger are opened and closed to reach a preset expansion threshold, the model expansion is triggered; the model closure judgment unit is used to judge the closing state of the user's left hand gesture, and when it is detected that the left thumb and the left index finger are closed to reach a preset closing threshold, the model closure is triggered; the model transformation control unit is used to control the expansion and closing actions of the model according to the judgment results of the model expansion judgment unit and the model closure judgment unit; the model state storage unit is used to store the current state of the model, including the expansion state and the closing state.
10. The file browsing system based on AR device according to claim 1, characterized in that: For video files, the file operation unit combines the gesture information, and the position and action of the user gesture relative to the file in the AR scene, to support: controlling the playback and pause of the video by clicking the video playback area with one finger; adjusting the playback progress of the video on the video playback bar through left and right sliding gestures; and adjusting the volume of the video through up and down sliding gestures.
