Method for recording user behavior using an AR device and AR device

Abstract

The present invention relates to a method for recording user behavior using an AR device, the method comprising: constructing a recording space coordinate system in response to a recording start command on a first AR device; recording first user behavior data of the first AR device to generate a first user behavior data packet, the first user behavior data including pose data of the first AR device and first user operation data, the first user behavior data being configured to be used to change AR content displayed by the first AR device; associating the first user behavior data packet with the AR content to generate a first AR content scene data packet; and storing the first AR content scene data packet in response to a recording end command. The recorded first AR content scene data packet can be shared with others for viewing or editing, facilitating dissemination and derivative creation.

Description

Technical Field

[0001] (Cross-Reference to Related Applications) This application claims priority to a Chinese patent application filed with the China National Intellectual Property Administration on December 13, 2022, with application number 202211606710.X and title of invention "User Behavior Recording Method and AR Device Based on AR Device", the entire content of which is incorporated herein by reference.

[0002] This disclosure relates to the field of Augmented Reality (AR), and particularly to a user behavior recording method and an AR device using an AR device.

Background Art

[0003] With the popularization of AR devices, more and more users have started to use AR devices, and further view and / or create AR content using AR devices, and further interact with AR content through various interaction methods provided by AR devices. Users also have a need to record their interaction actions with the AR content they create as videos and spread and share them.

[0004] However, in the existing user behavior recording methods of AR devices, the recording of user behavior is performed by recording the rendering screen of the current AR device of the recorder, and the recorded content is saved as a video to facilitate dissemination on the Internet. In such an existing video recording mode, viewers can only play the recorded video on an AR device or other smart devices. Viewers lack a sense of immersion and participation and cannot perform secondary creation. In addition, videos recorded at high resolution require a large storage capacity (memory capacity) for the device, so they are not suitable for storage and dissemination.

Disclosure of the Invention

[0005] The purpose of this disclosure is to provide a new method for recording user behavior using an AR device and an AR device itself. By establishing a unified coordinate system for the recorder and the viewer, and recording user operations and pose data (pose data, position and orientation data) under the same content, viewers can see the recorder's past operations within the same content space, thereby enhancing immersion and participation. At the same time, it reduces storage costs and facilitates dissemination and derivative works.

[0006] This disclosure provides, in a first embodiment, a method for recording user behavior using an AR device. The method includes constructing a recording spatial coordinate system in response to a recording start command on a first AR device; recording first user behavior data of the first AR device and generating a first user behavior data packet, wherein the first user behavior data includes pose data and first user operation data of the first AR device, and the first user behavior data is configured to be used to change AR content displayed by the first AR device; associating the first user behavior data packet with the AR content and generating a first AR content scene data packet; and saving the first AR content scene data packet in response to a recording end command.

[0007] In a second embodiment, this disclosure provides an AR device, the AR device comprising a camera, a processor, memory, and an inertial sensor IMU, wherein the memory stores a computer program, and when the computer program is executed by the processor, the AR device performs the following steps: in response to a recording start command on the first AR device, construct a recording spatial coordinate system; record first user behavior data of the first AR device and generate a first user behavior data packet, the first user behavior data comprising pose data and first user operation data of the first AR device, the first user behavior data configured to be used to change AR content displayed by the first AR device; associate the first user behavior data packet with the AR content and generate a first AR content scene data packet; and in response to a recording end command, save the first AR content scene data packet.

[0008] One or more embodiments of this disclosure allow users to record user behavior with relatively small memory capacity by recording only pose information (position and orientation information) and user operation information of the AR device associated with the AR content, and furthermore, such user behavior can be edited or extended by other users, making it easy to disseminate and create derivative works. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 shows an exemplary system architecture 100 of an embodiment used in the user behavior recording method of this disclosure. [Figure 2] Figure 2 is a flowchart illustrating a method for recording user behavior using an AR device in one or more embodiments of the present disclosure. [Figure 3] Figure 3 is a schematic diagram of the recorder's viewpoint in one or more embodiments of the present disclosure. [Figure 4] Figure 4 is a schematic diagram of the viewpoints of the recorder and the viewer in one or more embodiments of the present disclosure. [Figure 5] Figure 5 is a schematic block diagram of a user behavior recording method using an AR device in one or more embodiments of the present disclosure. [Figure 6] Figure 6 is a schematic diagram of an AR device 600 in one or more embodiments of the present disclosure. [Modes for carrying out the invention]

[0010] To further illustrate the technical means and effects used by this disclosure to achieve the objectives of the invention, specific embodiments, configurations, features, and effects of the user behavior recording method using an AR device and the AR device proposed by this disclosure will be described in detail below, in conjunction with the drawings and preferred embodiments.

[0011] Figure 1 shows an exemplary system architecture 100 of an embodiment used in the user behavior recording method of this disclosure.

[0012] As shown in Figure 1, the exemplary system architecture 100 includes an AR device 11 and a target device 12.

[0013] The AR device 11 may include one or two display screens 111. These display screens are used to display AR content. Furthermore, the AR device 11 may further include a glasses frame 112. In some embodiments, the sensors, processing unit, memory, and battery of the AR device 11 may be located within the glasses frame 112. In some optional implementations of some embodiments, one or more of the sensors, processing unit, memory, and battery may be integrated into a separate, independent accessory (not shown) that can be connected to the glasses frame 112 via a data cable. In some optional implementations of some embodiments, the AR device 11 may have only display capabilities and some sensors, with the target device 12 providing functions such as data processing, data storage, and power supply capabilities. In one or more embodiments of the present disclosure, the AR device 11 includes an augmented reality or mixed reality device capable of displaying AR content, such as AR glasses, mixed reality (MR) glasses, an AR headband, or an MR headband.

[0014] The target device 12 may further include a touch-sensitive display screen 121 (touchscreen). In some embodiments, the AR device 11 and the target device 12 can communicate via a wireless connection. In one or more embodiments of the present disclosure, the target device 12 may include a computing device such as a smartphone, mobile data terminal, smart tablet, or laptop. In some optional implementations of some embodiments, the AR device 11 and the target device 12 may be connected via a data cable (not shown), and the AR device 11 can be controlled by interaction functions (e.g., a touchpad) provided by the target device 12.

[0015] The number of AR devices and target devices shown in Figure 1 should be understood as illustrative. Any appropriate number of AR devices and target devices can be used depending on the implementation needs.

[0016] For example, in the process of a user using the AR device 11 shown in Figure 1, the AR device 11 can be used to view, edit, and create AR content. The AR content includes one or more combinations of three-dimensional (3D) models, images, videos, and text, and there is no limit to the quantity of each type. In one embodiment, the AR content can be a combination of two 3D models, for example, a teacup and a teapot. In another embodiment, the AR content can be a combination of a 3D model and text, for example, the 3D model may be a virtual character and the text may be a brief introduction to the virtual character. In yet another embodiment, the AR content can be dynamic or static. Dynamic AR content is a display mode in which the AR content changes continuously over time when there is no human interaction, while static AR content is a display mode in which the AR content remains fixed when there is no human interaction. Users can perform various operations on AR content using the interaction functions provided by the AR device 11. For example, they can perform operations such as scaling, rotating, zooming in and out, modifying and editing, adding and deleting elements, and attaching and detaching them. For example, in one embodiment, the AR content is a white clothing model, and the user can draw a related pattern on the white clothing model. In this case, it may be necessary to rotate and scale the AR content, and it may be necessary to draw on the white clothing using a drawing tool provided by the AR content system. At this time, the user may have a need to record the production process in order to share it externally. Normally, the user's production process can be recorded by recording a video, but such video recording methods may require a relatively large amount of storage capacity and are not suitable for dissemination. According to one or more embodiments of the present disclosure, the user's production process can be recorded using a user activity recording method and can be viewed by other users on other devices.As an example, in the following, a method for recording user actions by an AR device will be described in detail based on FIGS. 2 to 5.

[0017] FIG. 2 is a flowchart diagram of a method for recording user actions by an AR device in one or more embodiments of the present disclosure, and includes the following.

[0018] S1. In response to a recording start command on the first AR device, construct a recording space coordinate system.

[0019] S2. Record the first user action data of the first AR device to generate a first user action data packet, where the first user action data includes the pose data of the first AR device and the first user operation data, and the first user action data is configured to be used to change the AR content displayed by the first AR device.

[0020] S3. Associate the first user action data packet with the AR content to generate a first AR content scene data packet.

[0021] S4. In response to a recording end command, save the first AR content scene data packet.

[0022] Specifically, the first user wears the first AR device, and the first user can view and edit one or more AR contents through the first AR device.

[0023] In step S1, according to one or more embodiments of this disclosure, if a first user wants to start recording their actions, they can start recording by pressing the start recording button on the first AR device. At this time, the first AR device is initialized and a recording space coordinate system is constructed. The purpose of constructing the recording space coordinate system is to ensure that all user actions recorded thereafter are recorded based on a unified coordinate system, thereby enabling accurate reconstruction of user actions. After the AR device starts recording, the user can be notified that recording has started by displaying it on the user interface (UI) on the first AR device.

[0024] In one of the embodiments, the initial position of the three Degrees of Freedom (3DOF) sensor of the first AR device can be set as the coordinate origin of the recording space coordinate system. In one of the embodiments, the initial position of the 3DOF sensor is default set to the physical position where the 3DOF sensor is located. The 3DOF sensor can include one or a combination of an accelerometer, a gyroscope, and a magnetometer. The 3DOF sensor is configured to calculate the 3DOF information of the first AR device. The 3DOF information includes Pitch, Roll, and Yaw. Through the 3DOF information of the AR device, the user can place AR content in the 3DOF space, thereby realizing the role of the extended display. The initial position of the 3DOF sensor is the origin position automatically defined when the 3DOF sensor is activated, and is usually in the physical position area of the 3DOF sensor. In another embodiment, in addition to the 3DOF information, one or more cameras on the first AR device are used to perform Simultaneous Localization and Mapping (SLAM) for the current space, and form a SLAM space through SLAM modeling. In one embodiment of the present disclosure, the SLAM space refers to including, in addition to the 3DOF information, further translational information of the X-axis, Y-axis, and Z-axis in the space. The coordinate origin of the SLAM space is used as the coordinate origin of the recording space coordinate system.

[0025] In step S2, after the first AR device starts recording, the first AR device records user behavior data of the first user on the AR device and further summarizes the user behavior data during the recording process to generate a data packet. The data packet may include all of the first user's actions related to AR content interaction. In one or more embodiments, the user behavior data includes pose data collected by sensors on the AR device and other user interaction data with the AR content recorded on the AR device. Depending on the type of sensor compatible with the AR device, the pose data typically includes 3DOF or 6DOF data. For example, if the AR device is equipped only with a 6-axis or 9-axis inertial measurement unit (IMU), it can typically only detect 3DOF data. If the AR device is further equipped with one or more cameras with spatial awareness capabilities, the AR device can detect the user's translation along the X, Y, and Z axes, thereby obtaining 6DOF data. Specifically, the pose data of the AR device includes X, Y, and Z translation data, and roll, pitch, and yaw rotation data. In one or more embodiments, user operation data may include one or more types of user gesture commands, voice commands, and key commands. For example, gesture commands may include dynamic and static gestures, of which dynamic gestures may include actions such as "spreading the palm," "making a fist," "moving fingers," and "moving the palm." Static gestures may include gestures such as "fist," "palm," "one finger," "two fingers," "three fingers," and "four fingers." Voice commands may be offline or online voice commands, for example, the user may input operations such as "zoom in," "zoom out," and "rotate" by voice. Key commands may be user commands input via various operation keys (operation buttons) on the AR device or via other interactive devices (e.g., mouse, Feishu, keyboard).Through these user commands, users can manipulate the AR content displayed by the AR device, for example, by zooming in, zooming out, rotating, moving, and editing, thereby changing the presentation effect of the AR content.

[0026] In step S3, the first AR content scene data packet is generated by associating the first user behavior data packet with the AR content. Specifically, the AR content scene data packet is generated by associating the user behavior data packet generated in step S2 with the AR content. In one embodiment, associating the user behavior data packet with the AR content means linking the AR content and the user behavior data packet, and by adding a link statement of the AR content's file address to the user behavior data packet, the AR content opens simultaneously when the user opens the user behavior data packet. In another embodiment, associating the user behavior data packet with the AR content means merging (combining) the user behavior data packet with the AR content. When associating the user behavior data packet with the AR content, only the initial state at the start of recording of the AR content is associated, and the subsequent state of the AR content can be reproduced according to the data in the user behavior data packet. If the AR content contains one or more types of material, all the material of the AR content can be associated with the user behavior data packet.

[0027] In step S4, the first AR content scene data packet is saved (stored) in response to a recording termination command. Specifically, when a user completes recording and needs to terminate recording, they can give a recording termination or recording save command using a key, gesture, or the like. The first AR content scene data of the first AR content scene data packet is saved as a separate file so that it can be read or edited later by the same user or other users. The specific format of the saved file is not limited to one or more embodiments of this disclosure. It is sufficient that the format can be pre-configured between AR devices and mutually supported (interchangeable).

[0028] Figure 3 is a schematic diagram of the recorder's viewpoint in one or more embodiments of the present disclosure. In Figure 3, the recorder is the first user. When the recorder begins recording user behavior, the sensor generates a recording space coordinate system including a fixed spatial origin. All user behavior by the recorder in this coordinate system is recorded through steps S1 to S4 described above.

[0029] The activity log file, which includes AR content scene data and is recorded based on steps S1 to S4 described above, contains only user activity data and basic AR content information, and does not include rendered images. The file size is relatively small and can be easily transmitted and shared. For example, the recorder can share it with other users through file transfer or the built-in sharing function, and other users can read it via other AR devices, thereby allowing them to reconstruct the recorder's interaction process with the AR content. The method for reading the recorder's user activity file is, for example, as described in steps S5A, S6A, and S7A of Figure 2.

[0030] S5A: In response to a reading operation of the second AR device, the first AR content scene data packet is read.

[0031] S6A: Open the AR content associated with the first user behavior data packet and render the AR content in the recording space coordinate system.

[0032] S7A: The AR content is modified based on the pause data of the first AR device and the first user operation data in the first user behavior data packet.

[0033] Specifically, the second user wears a second AR device. The first and second AR devices may be of the same type or different types, as long as they can support reading the saved file generated in step S4. In step S5A, if the second user wants to view the user behavior recorded by the first user, they can use a file reading method and read the first AR content scene data packet using the second AR device. In step S6A, when the second AR device reads the first AR content scene data packet, it detects that there is AR content associated with the first AR content scene data packet, so the second AR device can open the associated AR content and render the AR content in the recording space coordinate system. In step S7A, the second AR device can adjust the presentation (display) method of the AR content based on the user operation information for the first AR device and the pose data of the first AR device, which are included in the first user behavior data packet. The second user can observe the recording space themselves, for example, by adjusting the observation angle to observe the first user's creative process, and by moving closer to or further away from the AR content to see the details or the whole of the AR content.

[0034] Figure 4 shows schematic diagrams of the viewpoints of the recorder and viewer in one or more embodiments of the present disclosure. In Figure 4, the recorder is the first user, and the viewer is the second user. In Figure 4, the second user, who is the viewer, and the first user, who is the recorder, use a unified recording space coordinate system. The second user can move and change orientation within the space to observe the first user's actions within the space from various angles and distances, and to adjust the AR content.

[0035] In addition to recording the actions of the first user toward the AR content as described in steps S1 to S4, one or more embodiments of this disclosure further disclose that multiple users can create AR content simultaneously and record the actions of multiple users. Specifically, this is as described in steps S5B, S6B, S7B, and S8B of Figure 2.

[0036] S5B. The recording space coordinate system and the AR content are shared with a third AR device via a communication connection, and the third AR device and the first AR device display the same AR content.

[0037] S6B. The third AR device is configured to record third user behavior data of the third AR device and generate third user behavior data packets based on the recording space coordinate system, wherein the third user behavior data includes pose data and third user operation data of the third AR device, and the third user behavior data is used to change the AR content displayed by the first AR device.

[0038] S7B. The third user behavior data packet is associated with the AR content to generate a third AR content scene data packet.

[0039] S8B. In response to the recording termination command, the third AR content scene data packet and the first AR content scene data packet are packaged and saved.

[0040] Specifically, a third user wears a third AR device. Through steps S5B to S8B, the third user can collaboratively edit the same AR content with the first user, and the actions of both the first and third users can be recorded together. Specifically, in one embodiment, the third AR device communicates with the first AR device via a wired or wireless method, thereby acquiring the AR content from the first AR device and displaying it to the third user. The third AR device records the third user's action data based on the same recording space coordinate system constructed by the first AR device, and further generates a third user action data packet. Considering that the first and third users may be creating AR content simultaneously, the first user action data packet and the third user action data packet are arranged to modify the AR content together. Therefore, the third AR content scene data packet and the first AR content scene data packet are packaged and stored so that other users can easily observe the operations of the first and third users simultaneously.

[0041] Furthermore, one or more embodiments of this disclosure disclose: a method for creating and recording AR content using a second AR device after steps S5A to S7A. Specifically, the method involves, in response to a recording start command on the second AR device, recording second user behavior data of the second AR device based on the recording spatial coordinate system to generate a second user behavior data packet, the second user behavior data including pose data and second user operation data of the second AR device, and the second user behavior data being configured to be used to modify the AR content displayed by the second AR device. A second content scene data packet is generated by associating the second user behavior data packet with the AR content. In response to a recording end command, the second AR content scene data packet and the first AR content scene data packet are packaged and stored.

[0042] Figure 5 shows a schematic block diagram of a method for recording user behavior using an AR device in one or more embodiments of the present disclosure. Figure 5 includes a first user 501, a second user 502, a third user 503, and AR content 510 located in space 500. Although not shown, each user has their own AR device, namely the first AR device, the second AR device, and the third AR device. Each of these AR devices is capable of displaying the AR content 510. However, because each user has a different field of view and position in the coordinate system, different users may see different parts of the AR content 510. The first user can perform steps S1 to S4, thereby recording their behavior toward the AR content 510 and obtaining the first AR content scene data packet. The second user opens the first AR content scene data packet according to steps S5A to S7A, thereby allowing the second user to observe the first user's AR content 510 creation process from their perspective. A third user can participate in the first user's production process, record their own production actions toward the AR content 510 as a third-party AR content scene data packet, and save it together with the first-party AR content scene data packet before sharing it.

[0043] In some extended embodiments, the first AR content scene data packet can be encrypted while it is being stored, so that when reading the first AR content scene data packet, a corresponding decryption operation is required to reproduce the user's actions. In one embodiment, the user recording the actions can add a payment module to the AR content scene data package, and viewers can only decrypt it after paying a certain fee, making it easier for creators to generate revenue. In another embodiment, the user recording the actions can configure viewers to only decrypt the data after performing a specific action, thereby increasing the enjoyment of the sharing process.

[0044] The following schematic diagram of the structure of an AR device 600 (e.g., the smart terminal in Figure 1) suitable for implementing some embodiments of the present disclosure is shown with reference to Figure 6. The AR device shown in Figure 6 is merely an example and does not limit the functionality and scope of use of the embodiments of the present disclosure.

[0045] As shown in Figure 6, the AR device 600 may include a processing unit 601 (e.g., a central processing unit, graphics processor, etc.), which can perform various appropriate operations and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data necessary for the operation of the AR device 600. The processing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0046] Typically, the following devices can be connected to the I / O interface 605: input devices 606 such as touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, and gyroscopes; output devices 607 such as liquid crystal displays (LCDs), speakers, and vibrators; and communication devices 609. The communication device 609 enables the AR device 600 to communicate with other devices wirelessly or via wired connections to exchange data. Figure 6 shows an AR device 600 with various devices, but it should be understood that it is not necessary to implement or have all of the illustrated devices. Alternatively, more or fewer devices may be implemented or have been included. Each block shown in Figure 6 may represent one device, or multiple devices as needed.

[0047] In particular, according to some embodiments of this disclosure, the processes described in the preamble with reference to flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a type of computer program product, which includes a computer program recorded on a computer-readable medium, and which includes program code for performing the methods shown in the flowcharts. In some embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or from a ROM 602. When the computer program is executed by the processing unit 601, the above-described functions, limited to the methods of some embodiments of this disclosure, are performed.

[0048] It should be noted that the computer-readable media described in some embodiments of this disclosure may be computer-readable signal media, computer-readable storage media, or any combination thereof. Computer-readable storage media may be, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In some embodiments of this disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which can be used with or in conjunction with an instruction execution system, device, or apparatus. In some embodiments of this disclosure, the computer-readable signal medium may include data signals propagated in the baseband or as part of a carrier wave, and the computer-readable signal medium may contain computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transfer programs used by or in combination with instruction execution systems, apparatus, or devices.Program code contained in a computer-readable medium can be transmitted by any suitable medium, including but not limited to electrical cables, optical fibers, radio frequency (RF), or any suitable combination thereof.

[0049] In some embodiments, clients and servers can communicate using any currently known or future-developed network protocol, such as the HyperText Transfer Protocol (HTTP), and can interconnect via digital data communication (e.g., communication networks) in any format or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), the internet (e.g., the Internet), peer-to-peer networks (e.g., ad hoc peer-to-peer networks), and any currently known or future-developed networks.

[0050] The computer-readable medium may be included in the AR device or may exist independently without being incorporated into the AR device. The computer-readable medium contains one or more programs. When the one or more programs are executed by the AR device, the AR device, in response to a recording start command on the first AR device, constructs a recording spatial coordinate system, records first user action data of the first AR device and generates a first user action data packet, the first user action data of which includes pose data and first user operation data of the first AR device, the first user action data is configured to be used to change the AR content displayed by the first AR device, associates the first user action data packet with the AR content and generates a first AR content scene data packet, and saves the first AR content scene data packet in response to a recording end command.

[0051] Computer program code that performs operations of some of the embodiments described herein can be written in one or more programming languages, or a combination thereof, and such programming languages ​​include, for example, object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may run entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it may be connected to an external computer (for example, via the Internet using an Internet service provider).

[0052] The flowcharts and block diagrams in the drawings illustrate the implementable architectures, functions, and operations of the systems, methods, and computer program products of various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code, which includes executable instructions for implementing one or more defined logical functions. It should also be noted that in some alternative implementations, the functions described within a block may be executed in an order different from that shown in the drawings. For example, two consecutively shown blocks may actually be executed in parallel or in reverse order, depending on the function. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, may be implemented by a system based on dedicated hardware for performing the defined functions or operations, or by a combination of dedicated hardware and computer instructions.

[0053] The units described in some embodiments of this disclosure may be implemented in software or in hardware.

[0054] The functions described in the preamble of this text can be performed, at least partially, by one or more hardware logic components. Examples of hardware logic components that can be used include, but are not limited to, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems on chips (SOCs), and complex programmable logic devices (CPLDs).

[0055] The above description merely illustrates some relatively preferred embodiments and operational technical principles of the present disclosure. Those skilled in the art should understand that the scope of the inventions included in the embodiments of the present disclosure is not limited to technical configurations obtained by specific combinations of the aforementioned technical features, but also includes other technical configurations obtained by any combination of the aforementioned technical features or their equivalents, without departing from the inventive concept described above. For example, the aforementioned features may be replaced with technical configurations obtained by similarly functioning technical features disclosed in the embodiments of the present disclosure, but are not limited thereto.

Claims

1. A method for recording user behavior using an augmented reality (AR) device, In response to a recording start command on the first AR device, construct a recording spatial coordinate system. The system is configured to record first user behavior data of the first AR device and generate first user behavior data packets, wherein the first user behavior data includes pause data and first user operation data of the first AR device, and the first user behavior data is used to change the AR content displayed by the first AR device. The first user behavior data packet is associated with the AR content to generate the first AR content scene data packet. A method for recording user behavior by an augmented reality (AR) device, comprising saving the first AR content scene data packet in response to a recording termination command.

2. The user behavior recording method according to claim 1, characterized in that the initial position of the 3DOF sensor having 3 degrees of freedom of the first AR device is set as the coordinate origin of the recording space coordinate system.

3. Simultaneous self-localization and mapping are performed on the current space, and a SLAM space is formed using SLAM modeling. The user behavior recording method according to claim 1, characterized in that the coordinate origin of the SLAM space is the coordinate origin of the recording space coordinate system.

4. The user behavior recording method according to claims 1 to 3, characterized in that the pose data of the AR device includes movement data in X, Y, and Z directions, and rotation data in roll, pitch, and yaw directions.

5. The user operation data is characterized in that it includes user gesture commands, voice commands, and key commands, as described in claims 1 to 3.

6. The user behavior recording method according to claims 1 to 3, characterized in that the AR content includes one or more types of combinations of three-dimensional 3D models, images, videos, and text.

7. In response to a reading operation of the second AR device, read the first AR content scene data packet, The AR content associated with the first user behavior data packet is opened, and the AR content is rendered in the recording space coordinate system. The user behavior recording method according to claim 1, further comprising modifying the AR content based on the pause data of the first AR device and the first user operation data in the first user behavior data packet.

8. In response to the recording start command on the second AR device, Based on the recording spatial coordinate system, the system records the second user behavior data of the second AR device and generates a second user behavior data packet, the second user behavior data of which includes the pause data and second user operation data of the second AR device, and the second user behavior data is configured to be used to change the AR content displayed by the second AR device. The second user behavior data packet is associated with the AR content to generate a second AR content scene data packet. In response to the recording termination command, the second AR content scene data packet and the first AR content scene data packet are packaged and saved. The user behavior recording method according to claim 7, further comprising:

9. The aforementioned recording spatial coordinate system and the aforementioned AR content are shared with a third AR device via a communication connection, and the third AR device and the first AR device display the same AR content. The third AR device is configured to record third user behavior data of the third AR device based on the recording space coordinate system and to generate third user behavior data packets, the third user behavior data of which includes pose data and third user operation data of the third AR device, and the third user behavior data is used to change the AR content displayed by the first AR device. The third user behavior data packet is associated with the AR content to generate a third AR content scene data packet. In response to the recording termination command, the third AR content scene data packet and the first AR content scene data packet are packaged and saved. The user behavior recording method according to claim 1, further comprising the following:

10. It is an augmented reality (AR) device. Includes camera, processor, memory, and inertial sensor IMU, The augmented reality (AR) device is characterized in that a computer program is stored in the memory, and when the computer program is executed by the processor, the method according to any one of claims 1, 2, 3, 7, 8, or 9 is executed.

11. An augmented reality (AR) device, Includes camera, processor, memory, and inertial sensor IMU, The augmented reality (AR) device is characterized in that a computer program is stored in the memory, and when the computer program is executed by the processor, the method according to claim 4 is executed.

12. An augmented reality (AR) device, Includes camera, processor, memory, and inertial sensor IMU, The augmented reality (AR) device is characterized in that a computer program is stored in the memory, and when the computer program is executed by the processor, the method according to claim 5 is executed.

13. An augmented reality (AR) device, Includes camera, processor, memory, and inertial sensor IMU, The augmented reality (AR) device is characterized in that a computer program is stored in the memory, and when the computer program is executed by the processor, the method according to claim 6 is executed.

Images