A method for high-frequency collection and playback of user behavior data

Abstract

The present invention discloses a method for high-frequency collection and playback of user behavior data. By monitoring a method for triggering events based on the Document Object Model (DOM), user behavior Event information is obtained from the browser cache. The data in the Event information is intercepted and processed through a JavaScript asynchronous programming mechanism, and uploaded to a server to realize process data collection. The collected process data is stored in different data tables in different databases after transmission. When the process data needs to be exported, different process data are taken out from different databases and integrated and exported to facilitate researchers to consult and use. This method has high collection accuracy and real-time performance, and is comprehensive and flexible. Different collection strategies and collection information can be customized for different behavior event types and trigger conditions to meet the needs of multimodal data collection of process data.

Description

Technical Field

[0001] The present invention relates to the field of network communication technology, and in particular to a method for high-frequency collection and playback of user behavior data. Background Art

[0002] Most current front-end systems are developed using single-page applications (SPAs). However, as single-page applications grow in size and functionality, they become bulky and difficult to maintain. These applications are often referred to as monolithic applications. To address this issue, the concept of micro frontends has been proposed. Micro frontends are not simply front-end frameworks or tools, but rather an architectural system, similar to the microservices architecture, that breaks down front-end applications into smaller, simpler applications and then combines them into a larger whole.

[0003] CLASS is a web application that uses a browser as a client. The collection of procedural data on user operations must be based on web front-end technology, mainly including the Document Object Model (DOM) and event mechanisms based on the DOM model, JavaScript asynchronous programming, browser caching technology, etc. The procedural data collected by the system is divided into three categories: chat data, which is generated in the IM module and records the content of user chats; behavioral data, which is generated from the internal operations of the task component and records the actions and sequence of users solving problems in the process of collaborative course tasks; playback data, which is generated from the internal operations of the task component and records the order and attribute changes of HTML elements in the task component. Both behavioral data and playback data are derived from the user's behavior in the course unit component interface and rely on mouse behavior and text typing; chat data is generated from the text typing of users communicating using IM tools. Currently, there are the following problems in the collection of procedural data:

[0004] First, process data is generated during user collaboration, and each task instance corresponds to thousands of pieces of information, placing high demands on the efficiency and real-time nature of data collection. Second, the actions recorded must be extremely detailed, resulting in very small granularity in process data, thus placing high demands on the accuracy of the data collected by the system. Finally, the multimodality of process data leads to significant structural differences between different data, posing significant challenges to the implementation of unstructured data storage and integration. Existing CSCL (Computer Supported Collaborative Learning) systems rarely focus on the collection and evaluation of process data, and the accuracy and granularity of the collected data naturally cannot meet the requirements of CPS capability research. Summary of the Invention

[0005] The purpose of the present invention is to provide a high-frequency collection and playback method for user behavior data. This method has high collection accuracy and real-time performance, and is comprehensive and flexible. Different collection strategies and collection information can be customized for different behavior event types and trigger conditions to meet the needs of multimodal data collection of process data.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] A method for high-frequency collection and playback of user behavior data, comprising:

[0008] Step 1: Obtain user behavior event information from the browser cache by monitoring the triggering event based on the Document Object Model (DOM). Use JavaScript asynchronous programming to intercept and process the data in the event information and upload it to the server to achieve process data collection.

[0009] Step 2: The collected process data is transmitted and stored in different data tables of different databases;

[0010] Step 3: When process data needs to be exported, different process data are taken out from different databases and integrated and exported to facilitate researchers' access and use.

[0011] It can be seen from the technical solution provided by the above invention that the above method has high acquisition accuracy and real-time performance, and the acquisition is comprehensive and flexible. Different acquisition strategies and acquisition information can be customized for different behavioral event types and trigger conditions to meet the needs of multimodal data acquisition of process data. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 A flowchart of a method for high-frequency collection and playback of user behavior data provided by an embodiment of the present invention;

[0014] Figure 2 A schematic diagram of the composition structure of the procedural data field according to an embodiment of the present invention;

[0015] Figure 3 This is a schematic diagram of the composition of the unified identification field according to an embodiment of the present invention;

[0016] Figure 4This is a schematic diagram of the composition of the unstructured field according to an embodiment of the present invention;

[0017] Figure 5 This is a schematic diagram of the chat data collection process according to an embodiment of the present invention;

[0018] Figure 6 Schematic diagram of the behavior data and playback data collection process according to an embodiment of the present invention;

[0019] Figure 7 This is a schematic diagram of an example of exporting procedural data according to an embodiment of the present invention;

[0020] Figure 8 This is a schematic diagram of a data model after the behavior data and playback data are merged according to an embodiment of the present invention;

[0021] Figure 9 This is a schematic diagram of optimizing the export data format according to an embodiment of the present invention;

[0022] Figure 10 This is a schematic diagram of the results of the data collection correctness test of the example given in the present invention;

[0023] Figure 11 This is a schematic diagram of the results of the integrity comparison of some collected data in the examples given in the present invention. DETAILED DESCRIPTION

[0024] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments, and do not constitute a limitation of the present invention. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0025] like Figure 1 FIG2 is a flow chart of a method for high-frequency collection and playback of user behavior data provided by an embodiment of the present invention, wherein the method includes:

[0026] Step 1: Obtain user behavior event information from the browser cache by monitoring the triggering event based on the Document Object Model (DOM). Use JavaScript asynchronous programming to intercept and process the data in the event information and upload it to the server to achieve process data collection.

[0027] In this step, the process data has multimodal characteristics and is composed of a unified identification field and unstructured fields, such as Figure 2 FIG. 1 is a schematic diagram showing the composition structure of the procedural data field according to an embodiment of the present invention, wherein:

[0028] The unified identification field is a common component of multimodal data and consists of id, unitId, classroomId, curriculumId, groupId, userId and createTime, such as Figure 3 FIG2 is a schematic diagram showing the composition of the unified identification field according to an embodiment of the present invention, where id represents the unique identifier of the data, unitId represents the course unit ID, classroomId represents the classroom ID, curriculumId represents the course ID, groupId represents the group ID, userId represents the user ID, and createTime represents the time when the data is uploaded to the server;

[0029] Unstructured fields include chat data, behavioral data, and playback data, such as Figure 4 FIG. 1 is a schematic diagram showing the composition of the unstructured field according to an embodiment of the present invention, wherein:

[0030] Chat data consists of the chatGroupId, content, msgRead, msgType, fromUserid, and recordId fields; chatGroupId indicates the current group ID, content indicates the chat content, msgRead indicates whether the message has been read, msgType indicates the message type, fromUserid indicates the source user ID of the message, and recordId indicates the original message ID.

[0031] Behavior data consists of count, classKey, and behaviorData fields; count represents count, classKey represents page ID, and behaviorData represents custom fields for different course units, which consists of an array.

[0032] The playback data consists of the moduleKey, classKey, eventTime, orderNo, moduleValue, location, and baseData fields. ModuleKey, classKey, baseData, and location are standard fields passed by the course unit component to the system, indicating custom operation information within the course unit component.

[0033] In the specific implementation, due to the huge amount of behavioral data and playback data, in order to alleviate bandwidth pressure, the system will pre-store the data in the browser cache and upload it to the server after reaching the specified data volume. Therefore, the unified identification field createTime in the same batch of data will record the same time node. The introduced eventTime and orderNo fields indicate the occurrence time and order of each data.

[0034] During the process of procedural data collection, the objects of procedural data collection include chat data, behavior data, and playback data, among which:

[0035] Chat data is generated from the native input box event under the Document Object Model (DOM) element. By binding the native input box and the keyboard Enter event and monitoring the changes in the input box value, when the content inside the input box is not empty and the user presses the Enter key, the content in the input box is intercepted and converted into a JSON string and sent directly to the server. Figure 5 The following diagram illustrates the chat data collection process described in an embodiment of the present invention. First, the HTML Input native box is bound to the keyboard's Enter event. The sendMessage() function is called via the keydownHandle() handler function to pass parameters to the server. The keydownHandle() function specifies that it prevents line breaks on Enter to ensure that the Enter message event is correctly monitored. Compared to behavioral data, chat data only focuses on the final text input result, which is relatively small. Therefore, it does not require temporary storage in the browser cache for batch transmission. Instead, the system collects each item individually and transmits it directly to the server.

[0036] Behavioral data and playback data are generated inside the task component. Since the course unit component itself does not have the function of data collection, the system actively monitors the occurrence of DOM element events inside the course unit component and completes the data collection operation. The system actively retrieves the preset standard parameter information from the course unit component, processes these parameters in a unified manner and puts them into the browser cache. When the data of a certain item is stored, it is uploaded to the server to realize data collection. Among them, unlike behavioral data, the collection of playback data is based on the changes in HTML elements inside the task component. By reproducing the changes and sequence of elements, it can achieve the effect of visual playback of user behavior.

[0037] like Figure 6The figure shows a schematic diagram of the behavior data and playback data collection process described in an embodiment of the present invention. Taking behavior data collection as an example, the Widget course component unit will preset the embedded data, that is, define the DOM event elements that need to be collected, and the system will declare the customized behavior field data. When an event occurs, the task component sends it to the system in a standard format; the system micro-front-end framework will listen to the message events sent by the task component to the system through the addEventListener() function, and call the SavaBehaviorDataMsgOption method through the system storage class to save the data to the browser cache, and then transmit it to the server. Unlike behavior data, playback data has more stringent requirements on data real-time performance. Therefore, in addition to the server time createTime, an eventTime field is added to record the local behavior occurrence time. Since the system uses the sessionStorage temporary session storage mechanism, the system determines whether it is data after the page is refreshed and recalculates eventTime.

[0038] Step 2: The collected process data is transmitted and stored in different data tables of different databases;

[0039] In this step, during the transmission of the collected process data, behavioral data and playback data are derived from micro-operations of user behavior. They have a large cardinality and require high real-time performance and system data management capabilities. Therefore, the behavior data and playback data are cached in the client browser through the HTML5 native LocalStorage object. Combined with the JSON data exchange format, they are sent to the server in batches and in fixed quantities to ensure data bandwidth stability and collection integrity. Compared with behavioral data and playback data, chat data is collected less frequently and has a smaller cardinality. Therefore, browser caching technology is not required as a transit technology to reduce transmission bandwidth pressure.

[0040] When storing procedural data in different data tables in different databases, the unstructured field structure of behavioral data and playback data is flexible and contains a large number of custom fields. Therefore, after the behavioral data and playback data are submitted to the server, they are stored in the unstructured database MongoDB. Chat data only has two types: private chat and one-on-one chat. The unstructured field information is fixed, so it is stored as structured data in the database MySql.

[0041] In its specific implementation, the system uses the localStorage method to implement browser caching of data. When behavioral events are monitored, these unstructured data fields are encapsulated in messages according to the data model of the preset behavioral data standard. The Micro-App sub-application sends data to the CLASS base framework. The micro-frontend framework monitors the messages sent by the sub-application and submits the data to the cache through the localStorage.setItem() method. When the number of entries reaches the preset limit or the page is closed or the course ends, the data is actively uploaded to the server and the cache is cleared. The system uses JavaScript asynchronous mechanism for data transmission. Without refreshing the page, the HttpPost interface call is implemented to upload the body to the server. Prior to this, the system will convert the value information in its field from the default string type to JSON format using the JSON.stringify() method for storage.

[0042] Step 3: When process data needs to be exported, different process data are taken out from different databases and integrated and exported to facilitate researchers' access and use.

[0043] In this step, the standard for process data export fields consists of a unified identification field and an unstructured field. The unified identification field represents the basic information corresponding to the process data.

[0044] In the unstructured field, type is the data type; 1 is behavior data; 2 is chat data; operationModuleValue is the template content; location is the level name, which is a custom data field for the component; and data value represents the variable value in the template content.

[0045] Table 1 below shows an example of a process data field export standard according to an embodiment of the present invention:

[0046]

[0047]

[0048] like Figure 7 FIG2 is a schematic diagram of an example of process data export according to an embodiment of the present invention, wherein some of the process data information items exported are as follows: Figure 7 As shown, the template content is [A]send[B], which means that user A sent a message with content B to the group, and data1 and data2 are the values ​​of A and B respectively.

[0049] In addition, in the process of collecting process data, since both behavior data and playback data are collected from student behaviors in the task component, they are composed of unified identification fields and unstructured fields. There are overlapping fields in the unstructured fields. Collecting them separately will not only increase the pressure on the transmission bandwidth and thus affect the system response efficiency, but also cause a waste of storage resources. Therefore, the behavior data and playback data can be merged and collected in a unified manner. Specifically:

[0050] like Figure 8 Figure 2 shows a schematic diagram of the data model after the behavior data and playback data are merged according to an embodiment of the present invention. The behavior data fields are merged with the playback data fields, and a type field is added to the unstructured fields: 0 represents playback data, 1 represents behavior data, and 2 represents behavior plus playback data. The count field, which is unique to the behavior data, is deleted and replaced with the eventTime and orderNo fields to avoid duplication of field identifiers.

[0051] During the process of exporting procedural data, the type in the process data export field is used to distinguish chat data from behavioral data. Considering that the behavioral data in the template content of the exported data is defined by a third party, in order to reduce unnecessary fields, the two data types are distinguished in the template content field, such as Figure 9 The figure shows a schematic diagram of the optimized export data format according to an embodiment of the present invention, which defines the template content of the chat data: the group chat template content is [A]send[B], where [A] and [B] are the contents of the data fields data1 and data2, respectively indicating the person sending the message and the content; the single chat template content is [A]send[B]to[C], where [A] and [B] are the contents of the data fields data1, data2 and data3, respectively indicating the person sending the message, the content and the person receiving the message.

[0052] In addition, the embodiment of the present application is also equipped with a playback engine, which reproduces user behavior by collecting and reproducing playback data. For the reproduction of playback data and chat data, the implementation of data reproduction is divided into two steps: system preparation of data and distribution of data; playback data is retrieved and distributed in the database, and the distributed playback data is obtained in the system page. By comparing the classKey value, the data is left for processing. The data in the course unit component is sent out using the microApp.setData() method. When each page receives the playback data, it only needs to assign the value of the playback data to the current data. Specifically:

[0053] (1) The system prepares to replay data

[0054] The system has two ways to prepare playback data: First, when the initial playback request is made or the playback object is switched, the system requests the server to obtain fixed-point or full playback data; the fixed-point data is used to locate the playback time node, and the full playback data is used to reproduce the user operation at the current time node; second, when the playback data is updated or the playback data is requested for a second drag and drop, the system obtains 5 seconds of playback data through the playback data interface. After the system has prepared the playback data, data distribution is enabled.

[0055] (2) System distribution data to achieve data reproduction

[0056] After the system prepares the playback data, it opens and loads the playback page containing the course unit component. After the page is ready, it will notify the playback page to distribute the data. The system listens for the distribution message, and the base application receives the playback data. The data of the base application will be kept for processing, and the data of the course unit component sub-application will be distributed using the Micro-App sub-page. After the Micro-App sub-page receives the playback data, it compares the classKey value and assigns the compared data to the corresponding attributes of the current page to achieve data reproduction.

[0057] It should be noted that the contents not described in detail in the embodiments of the present invention belong to the prior art known to those skilled in the art.

[0058] The following is a specific example to verify the effectiveness of the method described in the embodiment of the present invention. This example verifies the process data management function from the two dimensions of the correctness and completeness of the collected process data. Taking the basic course writing task as an example, this example designs a set of data collection automation test scripts to verify whether the process data management function meets the requirements.

[0059] The scope of verification of data collection accuracy is to conduct a longitudinal comparison of the accuracy of data collected from individual students.

[0060] The test script is specifically designed to verify data collection accuracy. In the input function student_input1(driver), send_keys("ABC DEF") and send_keys(Keys.BACK_SPACE) are used to input the text "ABC+space+DEF" and delete the letter F. A for a in range(0,400) loop is used to repeat the input. This ensures that the data in the CSV file displayed a specific graphical pattern, making it easier to verify data collection accuracy.

[0061] like Figure 10The figure shows a result diagram of the correctness test of the example data collection given in the present invention. The column where the value of the CSV file is located shows an increasing trend of character strings. Due to the operations of typing a space and deleting F, there will be repeated rows after the last letter C is displayed, and there will be a rollback phenomenon when displaying the last letter F. The collected data meets the design expectations and the test passes.

[0062] Verification of data test integrity. Due to the large amount of data collected and the maximum capacity limit of the character box, a longitudinal comparison is adopted during the test process. That is, the system collects process data of different devices performing the same operation at the same time for consistency comparison to verify the integrity of data collection. By repeatedly executing the student_input1() script function, the data collection when multiple students perform the same operation in collaborative learning is simulated. Figure 11 The figure shows the results of the integrity comparison of some collected data in the example of the present invention. In the figure, the exported Excel files are all 20.4MB in size and contain the same number of data cells, proving that the data collected by the automated test script is complete and has passed the verification.

[0063] To sum up, the method described in the embodiment of the present invention has the following advantages compared with traditional data collection methods (manual collection, web crawler technology, log file analysis, etc.): First, the collection accuracy is high, and the collected data comes from the corresponding behavioral event information captured when the user interacts with the web page, which meets the basic needs of process data collection for user collaborative learning behavior; second, the real-time performance is high, and data collection is performed when the behavioral event is triggered, and the corresponding data processing or operation is triggered by monitoring the DOM event; third, the collection is comprehensive and flexible, and different collection strategies and collection information can be customized for different behavioral event types and trigger conditions to meet the needs of multimodal data collection of process data.

[0064] In addition, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiment method can be implemented by instructing the relevant hardware through a program, and the corresponding program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.

[0065] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or any form of implication that the information constitutes prior art already known to those skilled in the art.

Claims

1. A method for high-frequency collection and playback of user behavior data, characterized in that: The method comprises: Step 1: Obtain user behavior event information from the browser cache by monitoring the triggering event based on the Document Object Model (DOM). Use JavaScript asynchronous programming to intercept and process the data in the event information and upload it to the server to achieve process data collection. The process data has multimodal characteristics and is composed of a unified identification field and unstructured fields, where: The unified identification field is a common component of multimodal data and consists of id, unitId, classroomId, curriculumId, groupId, userId, and createTime. id represents the unique identifier of the data, unitId represents the course unit ID, classroomId represents the classroom ID, curriculumId represents the course ID, groupId represents the group ID, userId represents the user ID, and createTime represents the time when the data is uploaded to the server. Unstructured fields include chat data, behavior data, and playback data, where: Chat data consists of the chatGroupId, content, msgRead, msgType, fromUserid, and recordId fields; chatGroupId indicates the current group ID, content indicates the chat content, msgRead indicates whether the message has been read, msgType indicates the message type, fromUserid indicates the source user ID of the message, and recordId indicates the original message ID. Behavior data consists of count, classKey, and behaviorData fields; count represents count, classKey represents page ID, and behaviorData represents custom fields for different course units, which consists of an array. The playback data consists of the moduleKey, classKey, eventTime, orderNo, moduleValue, location, and baseData fields. ModuleKey, classKey, baseData, and location are standard fields passed by the course unit component to the system, representing custom operation information within the course unit component. The introduced eventTime and orderNo fields represent the occurrence time and order of each piece of data. Step 2: The collected process data is transmitted and stored in different data tables of different databases; In step 2, during the transmission of the collected process data, the behavior data and playback data are derived from micro-operations of user behavior. Therefore, the behavior data and playback data are cached in the client browser through the HTML5 native LocalStorage object. Combined with the JSON data exchange format, they are sent to the server in batches and in fixed quantities to ensure the stability of data bandwidth and the integrity of the collection. While procedural data is stored in different tables across different databases, behavioral data and playback data contain a large number of custom fields. Therefore, these data are submitted to the server and stored in the unstructured MongoDB database. Chat data, on the other hand, consists of only two types: private chat and one-on-one chat. Since the unstructured field information is fixed, it is stored as structured data in the MySQL database. Step 3: When process data needs to be exported, different process data are taken out from different databases and integrated and exported to facilitate researchers’ access and use; The method also includes the reproduction of playback data and chat data. The data reproduction is implemented in two steps: (1) The system prepares to replay data The system has two ways to prepare playback data: First, when the initial playback request is made or the playback object is switched, the system requests the server to obtain fixed-point or full playback data; the fixed-point data is used to locate the playback time node, and the full playback data is used to reproduce the user operation at the current time node; second, when the playback data is updated or the playback data is requested for a second drag and drop, the system obtains 5 seconds of playback data through the playback data interface. After the system has prepared the playback data, data distribution is enabled. (2) System distribution data to achieve data reproduction After the system prepares the playback data, it opens and loads the playback page containing the course unit component. After the page is ready, it will notify the playback page to distribute the data. The system listens for the distribution message, and the base application receives the playback data. The data of the base application will be kept for processing, and the data of the course unit component sub-application will be distributed using the Micro-App sub-page. After the Micro-App sub-page receives the playback data, it compares the classKey value and assigns the compared data to the corresponding attributes of the current page to achieve data reproduction.

2. The method for high-frequency collection and playback of user behavior data according to claim 1, characterized in that: In step 1, the objects of process data collection include chat data, behavior data, and playback data, among which: Chat data is generated from the native input box event under the Document Object Model (DOM) element. By binding the native input box and the keyboard Enter event and monitoring the changes in the input box value, when the content inside the input box is not empty and the user presses Enter, the content in the input box is intercepted and converted into a JSON string and sent directly to the server. Behavior data and playback data are generated within the task component. The system actively monitors the occurrence of DOM element events within the course unit component and completes the data collection operation. The system actively retrieves the pre-set standard parameter information from the course unit component, processes these parameters uniformly and stores them in the browser cache. When the data of a certain item is stored, it is uploaded to the server to realize data collection. Unlike behavioral data, playback data is collected based on changes in HTML elements within the task component. By reproducing the changes and sequence of elements, it can achieve the effect of visual playback of user behavior.

3. The method for high-frequency collection and playback of user behavior data according to claim 1, characterized in that: In step 3, the standard for the process data export field consists of a unified identification field and an unstructured field. The unified identification field represents the basic information corresponding to the process data. In the unstructured field, type is the data type; 1 is behavior data; 2 is chat data; operationModuleValue is the template content; location is the level name, which is a custom data field for the component; and data value represents the variable value in the template content.

4. The method for high-frequency collection and playback of user behavior data according to claim 1, characterized in that: When collecting process data, the behavior data and playback data are combined and collected in a unified manner. Specifically: Merge the fields of behavior data and playback data, and add a type field to the unstructured field: 0 for playback data, 1 for behavior data, and 2 for behavior plus playback data; Delete the count field, which is unique to behavioral data, and replace it with the eventTime and orderNo fields to avoid duplication of field identifiers. During the process of exporting procedural data, the type in the procedural data export field is used to distinguish chat data from behavioral data. Therefore, the template content of the chat data is defined: the group chat template content is [A]send[B], where [A] and [B] are the contents of the data fields data1 and data2, indicating the sender and content of the message respectively; the single chat template content is [A]send[B]to[C], where [A] and [B] are the contents of the data fields data1, data2, and data3, indicating the sender, content, and recipient of the message respectively.

Images