AR device and method for recording user behavior

Abstract

The present invention relates to a method for recording user behavior using an AR device, the method comprising: constructing a recording space coordinate system in response to a recording start command on a first AR device; recording first user behavior data of the first AR device to generate a first user behavior data packet, the first user behavior data including pose data of the first AR device and first user operation data, the first user behavior data being configured to be used to change AR content displayed by the first AR device; associating the first user behavior data packet with the AR content to generate a first AR content scene data packet; and storing the first AR content scene data packet in response to a recording end command. The recorded first AR content scene data packet can be shared with others for viewing or editing, facilitating dissemination and derivative creation.

Description

[Technical Field]

[0001] (Cross-Incorporation of Related Applications) This application claims priority to a Chinese patent application, application number 202211606710.X, filed with the China Intellectual Property Office on December 13, 2022, for the invention entitled "User behavior recording method based on AR device and AR device," the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to the field of Augmented Reality (AR), and more particularly to a method for recording user behavior using an AR device and an AR device. [Background technology]

[0003] As AR devices become more popular, more and more users are using them, viewing and / or creating AR content, and interacting with the AR content through various interaction methods provided by AR devices. Users also have a need to record, distribute, and share their interactions with the AR content they have created as videos.

[0004] However, in existing methods for recording user behavior in AR devices, user behavior is recorded by recording the user's current AR device rendering screen, and the recorded content is saved as a video for easy distribution over the Internet. In this existing video recording mode, viewers can only play back the recorded video on the AR device or other smart devices, which lacks a sense of immersion and participation and prevents viewers from creating derivative works. Furthermore, videos recorded in high resolution require a large storage capacity (memory capacity) on the device, making them unsuitable for storage and distribution. DISCLOSURE OF THE INVENTION

[0005] The purpose of this disclosure is to provide a new AR device and a method for recording user behavior. By establishing a unified coordinate system for the recorder and the viewer and recording the user's operations and pose data (pose data, position and orientation data) under the same content, the viewer can view the recorder's past operations within the same content space, thereby improving the sense of immersion and participation. At the same time, this reduces storage costs and facilitates dissemination and derivative creation.

[0006] In a first aspect, the present disclosure provides a method for recording user behavior using an AR device, the method including: constructing a recording space coordinate system in response to a recording start command on a first AR device; recording first user behavior data of the first AR device to generate a first user behavior data packet, where the first user behavior data includes pose data of the first AR device and the first user operation data, the first user behavior data being configured to be used to modify AR content displayed by the first AR device; associating the first user behavior data packet with the AR content to generate a first AR content scene data packet; and saving the first AR content scene data packet in response to a recording stop command.

[0007] In a second aspect, the present disclosure provides an AR device including a camera, a processor, a memory, and an inertial sensor (IMU), wherein the memory stores a computer program, and when the computer program is executed by the processor, the AR device performs the following steps: constructing a recording space coordinate system in response to a recording start command on a first AR device; recording first user behavior data of the first AR device to generate a first user behavior data packet, the first user behavior data including pose data of the first AR device and the first user operation data, the first user behavior data being configured to be used to change AR content displayed by the first AR device; associating the first user behavior data packet with the AR content to generate a first AR content scene data packet; and saving the first AR content scene data packet in response to a recording stop command.

[0008] One or more embodiments of the present disclosure allow a user to record user behavior with a relatively small storage capacity by recording only the pose information (pose information, position and orientation information) of the AR device associated with the AR content and user operation information, and further, the user behavior can be edited or extended by other users, facilitating dissemination and derivative creation. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an exemplary system architecture 100 for an embodiment of the user behavior recording method of the present disclosure. [Figure 2] FIG. 2 is a flowchart diagram of a method for recording user behavior by an AR device in accordance with one or more embodiments of the present disclosure. [Figure 3] FIG. 3 is a schematic diagram of a recorder's perspective in one or more embodiments of the present disclosure. [Figure 4] FIG. 4 is a schematic diagram of the recorder and viewer perspectives in one or more embodiments of the present disclosure. [Figure 5] FIG. 5 is a schematic block diagram of a method for recording user behavior by an AR device in one or more embodiments of the present disclosure. [Figure 6] FIG. 6 is a schematic diagram of an AR device 600 in accordance with one or more embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0010] In order to further explain the technical means and effects used by the present disclosure to achieve the specified inventive objectives, the following provides a detailed description of specific embodiments, configurations, features and effects of the user behavior recording method using an AR device and the AR device proposed by the present disclosure, in combination with drawings and preferred embodiments.

[0011] FIG. 1 illustrates an exemplary system architecture 100 for an embodiment of the user behavior recording method of the present disclosure.

[0012] As shown in FIG. 1, the exemplary system architecture 100 includes an AR device 11 and a target device 12.

[0013] The AR device 11 may include one or two display screens 111. The display screens are used to display AR content. The AR device 11 further includes an eyeglass frame 112. In some embodiments, a sensor, a processing unit, a memory, and a battery of the AR device 11 may be disposed within the eyeglass frame 112. In some optional implementations of some embodiments, one or more of the sensor, the processing unit, the memory, and the battery may be integrated into another independent accessory (not shown) and connected to the eyeglass frame 112 via a data cable. In some optional implementations of some embodiments, the AR device 11 may only have a display function and some sensors, with the target device 12 providing functions such as data processing, data storage, and power supply capabilities. In one or more embodiments of the present disclosure, the AR device 11 includes an augmented reality or mixed reality device capable of displaying AR content, such as AR glasses, mixed reality (MR) glasses, an AR headband, or an MR headband.

[0014] The target device 12 may further include a touch-sensitive display screen 121 (touch screen). In some embodiments, the AR device 11 and the target device 12 may communicate via a wireless connection. In one or more embodiments of the present disclosure, the target device 12 may include a device with computing capabilities, such as a smartphone, a mobile data terminal, a smart tablet, or a laptop. In some optional implementations of some embodiments, the AR device 11 and the target device 12 may be connected via a data cable (not shown), and the AR device 11 may be controlled by an interaction function (e.g., a touchpad) provided by the target device 12.

[0015] It should be understood that the number of AR devices and target devices in Figure 1 is exemplary, and any suitable number of AR devices and target devices can be included depending on the needs of the implementation.

[0016] For example, when a user uses the AR device 11 shown in FIG. 1 , the AR device 11 can be used to view, edit, and create AR content. The AR content may include one or a combination of three-dimensional (3D) models, images, videos, and text, with no limit on the number of each type. In one embodiment, the AR content may be a combination of two 3D models, such as a teacup and a teapot. In another embodiment, the AR content may be a combination of a 3D model and text, such as a 3D model representing a virtual character and a text providing a brief description of the virtual character. In other embodiments, the AR content may be dynamic or static. Dynamic AR content refers to a display mode in which the AR content changes continuously over time without human interaction, while static AR content refers to a display mode in which the AR content remains fixed without human interaction. The interaction functions provided by the AR device 11 allow the user to perform various operations on the AR content, such as scaling, rotating, zooming in and out, modifying and editing, adding and deleting elements, and attaching and detaching the AR content. For example, in one embodiment, the AR content is a white clothing model, and the user can draw a related pattern on the white clothing model. The AR content may need to be rotated and scaled, and the user may need to draw on the white clothing using a drawing tool provided by the AR content system. At this time, the user may need to record the creation process for external sharing. While the user's creation process can usually be recorded by recording a video, this method may require a relatively large storage capacity and is not suitable for distribution. According to one or more embodiments of the present disclosure, a user behavior recording method can be used to record the user's creation process, which can then be viewed by other users on other devices.As an example, a method for recording user behavior using an AR device will be described in detail below with reference to FIGS.

[0017] FIG. 2 is a flowchart diagram of a method for recording user behavior by an AR device in one or more embodiments of the present disclosure, including:

[0018] S1. In response to a recording start command on the first AR device, a recording space coordinate system is constructed.

[0019] S2. Record first user behavior data of the first AR device to generate a first user behavior data packet, where the first user behavior data includes pose data of the first AR device and the first user operation data, and the first user behavior data is configured to be used to modify AR content displayed by the first AR device.

[0020] S3. Associate the first user action data packet with the AR content to generate a first AR content scene data packet.

[0021] S4. In response to a recording end command, the first AR content scene data packet is stored.

[0022] Specifically, a first user wears a first AR device, and the first user can view and edit one or more AR contents via the first AR device.

[0023] In step S1, according to one or more embodiments of the present disclosure, when a first user wants to start behavior recording, the first user can start behavior recording by pressing a start recording button on the first AR device. At this time, the first AR device is initialized and a recording space coordinate system is constructed. The purpose of constructing the recording space coordinate system is to ensure that all subsequently recorded user behaviors are recorded based on a unified coordinate system, thereby enabling accurate restoration of the user behaviors. After the AR device starts recording, the first AR device can notify the user that recording has started by displaying a user interface (UI) on the first AR device.

[0024] In one embodiment, the initial position of a three degrees of freedom (3DOF) sensor of the first AR device can be the coordinate origin of the recording space coordinate system. In one embodiment, the initial position of the 3DOF sensor is defaulted to the physical location of the 3DOF sensor. The 3DOF sensor can include one or more combinations of an accelerometer, a gyroscope, and a magnetometer. The 3DOF sensor is configured to calculate 3DOF information of the first AR device. The 3DOF information includes pitch, roll, and yaw. Through the 3DOF information of the AR device, a user can position AR content within the 3DOF space, thereby realizing the role of an augmented display. The initial position of the 3DOF sensor is an origin position automatically defined when the 3DOF sensor is activated, and is typically located within the physical location of the 3DOF sensor. In another embodiment, a SLAM space is formed by SLAM modeling, in which simultaneous localization and mapping (SLAM) is performed on the current space using one or more cameras on the first AR device, superimposed on the 3DOF information. In one embodiment of the present disclosure, the SLAM space refers to a space that includes not only the 3DOF information but also translation information about the X-, Y-, and Z-axes in the space. The coordinate origin of the SLAM space is used as the coordinate origin of the recording space coordinate system.

[0025] In step S2, after the first AR device starts recording, the first AR device records user behavior data of the first user on the AR device and further summarizes the user behavior data during the recording process to generate a data packet. The data packet may include all of the first user's AR content manipulation behavior. In one or more embodiments, the user behavior data includes pose data collected by sensors on the AR device and other user manipulation data for the AR content recorded on the AR device. Depending on the type of sensor compatible with the AR device, the pose data typically includes 3DOF data or 6DOF data. For example, if the AR device only has a 6-axis or 9-axis inertial sensor (Inertial Measurement Unit, IMU), it can typically only detect 3DOF data. If the AR device also has one or more cameras with spatial recognition capabilities, it can detect the user's translation in three axes: X, Y, and Z, thereby obtaining 6DOF data. Specifically, the pose data of the AR device includes X, Y, and Z movement data and roll, pitch, and yaw rotation data. In one or more embodiments, the user operation data may include one or more of a user's gesture command, a voice command, and a key command. For example, the gesture command may include a dynamic gesture and a static gesture, where dynamic gestures include actions such as "open palm," "clenched fist," "moving fingers," and "moving palm." Static gestures may include gestures such as "fist," "palm," "one finger," "two fingers," "three fingers," and "four fingers." The voice command may be an offline voice command or an online voice command. For example, the user may input operations such as "zoom in," "zoom out," and "rotate" by voice. The key command may be a user command input via various operation keys (operation buttons) on the AR device or other interactive devices (e.g., a mouse, a keyboard, or a keyboard).Through these user commands, the AR content displayed by the AR device can be manipulated, for example, by zooming in, zooming out, rotating, moving, editing, etc., thereby changing the presentation effect of the AR content.

[0026] In step S3, a first AR content scene data packet is generated by associating the first user behavior data packet with the AR content. Specifically, an AR content scene data packet is generated by associating the user behavior data packet generated in step S2 with the AR content. In one embodiment, associating the user behavior data packet with the AR content means linking the AR content with the user behavior data packet, and adding a link statement of the AR content's file address to the user behavior data packet so that the AR content opens simultaneously when the user opens the user behavior data packet. In another embodiment, associating the user behavior data packet with the AR content can merge the user behavior data packet with the AR content. When associating the user behavior data packet with the AR content, only the initial state at the start of recording of the AR content is associated, and the subsequent state of the AR content can be reproduced according to the data in the user behavior data packet. If the AR content includes one or more types of materials, the user behavior data packet can be associated with all of the materials of the AR content.

[0027] In step S4, the first AR content scene data packet is saved (stored) in response to a command to end recording. Specifically, when the user has completed recording and needs to end the recording, the user can issue a command to end recording or save the recording by using a key, gesture, or other means. The first AR content scene data in the first AR content scene data packet is saved as an independent file so that it can be read or edited later by the same user or another user. The specific format of the saved file is not limited in one or more embodiments of the present disclosure. It is sufficient that the format be preset between AR devices and mutually supported (interchangeable).

[0028] 3 is a schematic diagram of a recorder's viewpoint in one or more embodiments of the present disclosure. In FIG. 3, the recorder is a first user. When the recorder starts recording user actions, a sensor generates a recording space coordinate system with a fixed space origin. All user actions by the recorder in this coordinate system are recorded through steps S1 to S4 described above.

[0029] The behavior record file containing AR content scene data recorded according to steps S1 to S4 includes only user behavior data and basic AR content information, but does not include rendering images. The file size is relatively small, making it very convenient to distribute and share. For example, the recorder can share the file with other users through file transfer or a built-in sharing function, and other users can read it through other AR devices, thereby allowing other users to restore the recorder's operation process for the AR content. The method for reading the recorder's user behavior file is, for example, as described in steps S5A, S6A, and S7A of FIG. 2.

[0030] S5A: In response to a read operation of the second AR device, read the first AR content scene data packet.

[0031] S6A: Open the AR content associated with the first user behavior data packet and render the AR content in the record space coordinate system.

[0032] S7A: Modify the AR content based on the pose data of the first AR device and the first user operation data in the first user behavior data packet.

[0033] Specifically, a second user wears a second AR device. The first and second AR devices may be the same or different types of AR devices, as long as they support reading the saved file generated in step S4. In step S5A, if the second user wants to view the user behavior recorded by the first user, they can use a file reading method to read the first AR content scene data packet using the second AR device. In step S6A, when the second AR device reads the first AR content scene data packet, it detects that there is AR content associated with the first AR content scene data packet. The second AR device can open the associated AR content and render the AR content in the recording space coordinate system. In step S7A, the second AR device can adjust the presentation (display) method for the AR content based on the user's operation information on the first AR device and the pose data of the first AR device contained in the first user behavior data packet. The second user can observe the recording space by themselves, for example, by adjusting the observation angle by themselves to observe the first user's creation process, and by moving closer to or farther away from the AR content, they can view the AR content in detail or in its entirety.

[0034] FIG. 4 illustrates a schematic diagram of the viewpoints of a recorder and a viewer in one or more embodiments of the present disclosure. The recorder in FIG. 4 is a first user, and the viewer in FIG. 4 is a second user. In FIG. 4, the second user (viewer) and the first user (recorder) use a unified recording space coordinate system. The second user can move and turn around within the space to observe the first user's actions within the space from various angles and distances, and adjust the AR content.

[0035] In addition to recording the first user's behavior with respect to AR content as described in steps S1-S4, one or more embodiments of the present disclosure further disclose that multiple users can simultaneously create AR content and record the behavior of multiple users, as described in steps S5B, S6B, S7B, and S8B of FIG.

[0036] S5B. Sharing the storage space coordinate system and the AR content to a third AR device via a communication connection, wherein the third AR device and the first AR device display the same AR content.

[0037] S6B. The third AR device records third user behavior data of the third AR device based on the recording space coordinate system and generates a third user behavior data packet, in which the third user behavior data includes pose data of the third AR device and the third user operation data, and the third user behavior data is configured to be used to modify the AR content displayed by the first AR device.

[0038] S7B. Associate the third user action data packet with the AR content to generate a third AR content scene data packet.

[0039] S8B. In response to a recording end command, the third AR content scene data packet and the first AR content scene data packet are packaged and stored.

[0040] Specifically, a third user wears a third AR device. Through steps S5B to S8B, the third user can collaborate with the first user to edit the same AR content, and the actions of the first and third users can be recorded together. Specifically, in one embodiment, the third AR device communicates with the first AR device via a wired or wireless method, thereby acquiring the AR content of the first AR device and displaying it to the third user. The third AR device records the third user's action data based on the same recording space coordinate system established by the first AR device and further generates a third user action data packet. Considering that the first and third users may be creating AR content simultaneously, the first user action data packet and the third user action data packet are arranged to modify the AR content together. Therefore, the third AR content scene data packet and the first AR content scene data packet are packaged and stored to facilitate other users' simultaneous observation of the operations of the first and third users.

[0041] One or more embodiments of the present disclosure further disclose the following: A method for creating and recording AR content by a second AR device after steps S5A to S7A. The method is specifically configured to: in response to a recording start command on the second AR device, record second user behavior data of the second AR device based on the recording space coordinate system to generate a second user behavior data packet, where the second user behavior data includes pose data of the second AR device and the second user operation data, and the second user behavior data is used to change the AR content displayed by the second AR device; associate the second user behavior data packet with the AR content to generate a second content scene data packet; and in response to a recording stop command, package and save the second AR content scene data packet and the first AR content scene data packet.

[0042] FIG. 5 illustrates a schematic block diagram of a method for recording user behavior using AR devices, according to one or more embodiments of the present disclosure. FIG. 5 includes a first user 501, a second user 502, a third user 503, and AR content 510 located within a space 500. Although not shown, each user is equipped with an AR device, i.e., a first AR device, a second AR device, and a third AR device, respectively. Each of these AR devices can display the AR content 510. However, due to the different viewing angles and positions in the coordinate system of each user, different users may view different parts of the AR content 510. The first user can perform steps S1 to S4, thereby recording their own behavior with respect to the AR content 510 and obtaining a first AR content scene data packet. The second user can open the first AR content scene data packet according to steps S5A to S7A, thereby observing the creation process of the first user's AR content 510 from the second user's perspective. The third user can participate in the first user's production during the first user's production process, and record his or her production actions on the AR content 510 as a third AR content scene data packet, which can then be saved together with the first AR content scene data packet and shared secondarily.

[0043] In some advanced embodiments, the first AR content scene data packet can be encrypted during storage, so that when the first AR content scene data packet is read, a corresponding decryption operation is required to play back the user's actions. In one embodiment, the user who records the action can add a payment module to the AR content scene data package, so that viewers can only decrypt it after paying a certain fee, thereby making it easier for the creator to earn revenue. In another embodiment, the user who records the action can set it so that viewers can only decrypt the data after performing a specific action, thereby increasing the enjoyment of the sharing process.

[0044] 6, a structural schematic diagram of an AR device 600 (e.g., the smart terminal of FIG. 1) suitable for implementing some embodiments of the present disclosure is shown. The AR device shown in FIG. 6 is merely an example and does not limit the functionality and scope of use of the embodiments of the present disclosure.

[0045] 6, the AR device 600 may include a processing unit 601 (e.g., a central processing unit, a graphics processor, etc.), which may perform various appropriate operations and processes based on programs stored in a read-only memory (ROM) 602 or loaded from a storage device 608 into a random access memory (RAM) 603. The RAM 603 also stores various programs and data necessary for the operation of the AR device 600. The processing unit 601, the ROM 602, and the RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0046] Typically, the following devices can be connected to the I / O interface 605: input devices 606, such as a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, and gyroscope; output devices 607, such as a liquid crystal display (LCD), speaker, and vibrator; and a communication device 609. The communication device 609 enables the AR device 600 to communicate with other devices wirelessly or via a wired connection to exchange data. While FIG. 6 illustrates the AR device 600 with various devices, it should be understood that it is not necessary for the device 600 to implement or include all of the illustrated devices. Alternatively, the device 600 may implement or include more or fewer devices. Each block illustrated in FIG. 6 may represent one device or multiple devices as needed.

[0047] In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program recorded on a computer-readable medium, the computer program including program code for performing the methods shown in the flowcharts. In some embodiments, the computer program can be downloaded and installed from a network via the communication device 609, or can be installed from the storage device 608 or from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the methods of some embodiments of the present disclosure are performed.

[0048] It should be noted that the computer-readable medium described in some embodiments of the present disclosure can be a computer-readable signal medium, a computer-readable storage medium, or any combination of both. The computer-readable storage medium can be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, the computer-readable signal medium carrying computer-readable program code. Such a propagated data signal may take various forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.The program code contained in the computer readable medium can be transmitted by any suitable medium, including, but not limited to, electrical cable, optical fiber, radio frequency (RF), or the like, or any suitable combination thereof.

[0049] In some embodiments, clients and servers may communicate using any now known or later developed network protocol, such as HyperText Transfer Protocol (HTTP), and may be interconnected by any format or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, a peer-to-peer network (e.g., an ad hoc peer-to-peer network), and any now known or later developed network.

[0050] The computer-readable medium may be included in the AR device or may exist independently without being incorporated into the AR device. The computer-readable medium has one or more programs stored thereon. When the one or more programs are executed by the AR device, the AR device: in response to a start recording command on a first AR device, construct a recording space coordinate system; record first user behavior data of the first AR device to generate a first user behavior data packet, where the first user behavior data includes pose data of the first AR device and the first user operation data, the first user behavior data being configured to be used to modify AR content displayed by the first AR device; associate the first user behavior data packet with the AR content to generate a first AR content scene data packet; and save the first AR content scene data packet in response to a stop recording command.

[0051] Computer program code for carrying out operations of some embodiments of the present disclosure can be written in one or more programming languages, or a combination thereof, including, for example, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code can run entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0052] The flowcharts and block diagrams in the figures illustrate possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code, which includes executable instructions for implementing one or more specified logical functions. It should also be noted that in some alternative implementations, the functions described in the blocks may be executed in a different order than that depicted in the figures. For example, two successively shown blocks may actually be executed in parallel or in the reverse order, depending on the functionality. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or by a combination of dedicated hardware and computer instructions.

[0053] The units described in some embodiments of the present disclosure may be implemented in a software manner or in a hardware manner.

[0054] The functionality described in the preceding paragraphs herein may be performed, at least in part, by one or more hardware logic components, such as, but not limited to, Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Specific Standard Parts (ASSPs), Systems on Chips (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.

[0055] The above description merely describes some relatively preferred embodiments of the present disclosure and the technical principles employed therein. Those skilled in the art should understand that the scope of the invention included in the embodiments of the present disclosure is not limited to a technical configuration obtained by a specific combination of the above technical features, but also includes other technical configurations obtained by any combination of the above technical features or equivalent features without departing from the above inventive concept. For example, the above features include, but are not limited to, technical configurations obtained by replacing the above features with technical features having similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A method for recording user behavior using an augmented reality (AR) device, comprising: constructing a recording space coordinate system in response to a recording start command on the first AR device; Recording first user behavior data of the first AR device to generate a first user behavior data packet, wherein the first user behavior data includes pose data of the first AR device and the first user operation data, and the first user behavior data is configured to be used to modify AR content displayed by the first AR device; Associating the first user behavior data packet with the AR content to generate a first AR content scene data packet; A method for recording user behavior by the augmented reality (AR) device, comprising storing the first AR content scene data packet in response to a recording end command.

2. The user behavior recording method according to claim 1 , wherein the initial position of a 3DOF sensor having three degrees of freedom of the first AR device is set as the coordinate origin of the recording space coordinate system.

3. For the current space, self-location estimation and mapping are simultaneously performed, and a SLAM space is formed by SLAM modeling.

2. The user behavior recording method according to claim 1, wherein the coordinate origin of the SLAM space is set as the coordinate origin of the recording space coordinate system.

4. The user behavior recording method according to any one of claims 1 to 3, wherein the pose data of the AR device includes X, Y, and Z movement data and roll, pitch, and yaw rotation data.

5. 4. The user behavior recording method according to claim 1, wherein the user operation data includes a user's gesture command, a voice command, and a key command.

6. The user behavior recording method according to any one of claims 1 to 3, wherein the AR content includes one or a combination of a three-dimensional 3D model, an image, a video, and text.

7. reading the first AR content scene data packet in response to a read operation of the second AR device; Opening AR content associated with the first user behavior data packet and rendering the AR content into the record space coordinate system; The user behavior recording method of claim 1 , further comprising: modifying the AR content based on the pose data of the first AR device and the first user operation data in the first user behavior data packet.

8. in response to a start recording command on the second AR device; Recording second user behavior data of the second AR device based on a recording spatial coordinate system to generate a second user behavior data packet, wherein the second user behavior data includes pose data and the second user operation data of the second AR device, and the second user behavior data is configured to be used to modify AR content displayed by the second AR device; Associating the second user behavior data packet with the AR content to generate a second AR content scene data packet; in response to a recording end command, packaging the second AR content scene data packet and the first AR content scene data packet and storing the packaged data; The user behavior recording method according to claim 7, further comprising:

9. Sharing the storage space coordinate system and the AR content to a third AR device via a communication connection, wherein the third AR device and the first AR device display the same AR content; the third AR device is configured to record third user behavior data of the third AR device based on the recording spatial coordinate system and generate a third user behavior data packet, wherein the third user behavior data includes pose data of the third AR device and the third user operation data, and the third user behavior data is configured to be used to change the AR content displayed by the first AR device; associating the third user behavior data packet with the AR content to generate a third AR content scene data packet; and in response to a recording end command, packaging and storing the third AR content scene data packet and the first AR content scene data packet. The user behavior recording method according to claim 1 , further comprising:

10. An augmented reality (AR) device, It includes a camera, a processor, a memory, and an inertial sensor IMU, The augmented reality (AR) device, characterized in that the memory stores a computer program, which, when executed by the processor, performs the method according to claims 1 to 9.

Images