AI-based method and system for crawling dynamic multimodal psychological test questions

Abstract

This invention relates to the field of digital psychology and provides an AI-based method and system for crawling dynamic multi-modal psychological test questions. The method includes the following steps: obtaining the URL of the target psychological test page and selecting a set of device configuration parameters that includes at least desktop and mobile versions according to a preset device type library; loading and rendering the target psychological test page according to each configuration parameter in the device configuration parameter set, and crawling the complete document object model tree presented in each simulation environment; using AI to compare multiple document object model trees to obtain unique content nodes between versions, and selecting effective nodes to merge into a unified content tree structure; generating standardized test question data based on the unified content tree structure. This invention achieves efficient, accurate, and automated crawling of dynamic multi-modal psychological test questions by simulating multiple client access environments and introducing an AI-driven intelligent comparison, filtering, and fusion mechanism.

Description

Technical Field

[0001] This invention relates to the field of digital psychology, specifically to an AI-based method and system for crawling dynamic multi-modal psychological test questions. Background Technology

[0002] With the digitization of psychological research and the widespread adoption of online assessments, a vast amount of psychological testing resources have emerged on the internet, covering various forms from classic standardized scales to fun self-tests. Systematically collecting and aggregating these scattered and heterogeneous test items is of great value for constructing standardized question banks, conducting cross-cultural research, or developing adaptive assessment systems.

[0003] Currently, the collection of publicly available information online primarily relies on web crawling technology. Traditional crawling methods typically involve parsing webpage source code, identifying fixed HTML tags or document structures to locate and extract target content. However, modern webpages generally employ responsive web design, dynamically adjusting page layout, content, and even the visibility of functional modules based on the screen size, resolution, and other characteristics of the accessing device to provide the best user experience.

[0004] This design paradigm introduces new challenges to the automated crawling of test questions. For the same psychological test page, there may be differences in structure, interaction methods, and even some content between desktop and mobile browsers. For example, options clearly displayed in a table format on desktop may be collapsed or converted into card-style sliding components on mobile; some touch-interactive test questions optimized for mobile devices may not be triggered or displayed correctly on desktop. If crawling is performed using only a single-device mode, it is highly likely to result in incomplete or distorted test information. Therefore, this paper proposes an AI-based dynamic multimodal psychological test question crawling method and system. This method utilizes artificial intelligence (AI) technology to achieve intelligent perception, understanding, and reconstruction of dynamic content, multimodal expressions, and cross-device differences, fundamentally solving the content integrity and consistency issues brought about by responsive design. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the purpose of this invention is to provide an AI-based method and system for crawling dynamic multimodal psychological test questions, so as to solve the problems existing in the above-mentioned background technology.

[0006] This invention is implemented as follows: an AI-based method for crawling dynamic multi-modal psychological test questions, the method comprising the following steps:

[0007] Obtain the URL of the target psychological test page, and select a set of device configuration parameters that includes at least two specifications, desktop and mobile, based on a preset device type library to simulate different client access environments;

[0008] The target psychological test page is loaded and rendered according to each configuration in the device configuration parameter set, and the complete document object model tree presented in each simulation environment is captured.

[0009] AI is used to compare multiple document object model trees to obtain unique content nodes between versions, and effective nodes are selected and merged into a unified content tree structure.

[0010] Based on a unified content tree structure, the stems, options, and instructions of the test questions are extracted to generate standardized test question data.

[0011] As a further aspect of the present invention: the step of obtaining the URL of the target psychological test page and selecting a set of device configuration parameters that includes at least desktop and mobile versions according to a preset device type library specifically includes:

[0012] The URL is subjected to syntax validation, and a verification network request is initiated to confirm its accessibility.

[0013] After the request is approved, access the URL, parse the viewport control meta tags and link style sheets in the HTML document header, and extract the declarative rules related to device adaptation.

[0014] Based on the extracted adaptation rules, the device types are selected from the preset device type library, and a set of device configuration parameters containing a defined viewport size and user agent identifier is generated.

[0015] The generated set of device configuration parameters is deredundant and the priority order is determined based on the coverage of the adaptation rules.

[0016] As a further aspect of the present invention: the step of loading and rendering the target psychological test page according to each configuration item in the device configuration parameter set, and capturing the complete document object model tree presented in each simulation environment, specifically includes:

[0017] For each configuration item in the device configuration parameter set, initialize a corresponding browser emulation instance and apply all its parameters;

[0018] Control each browser emulation instance to navigate to the resource pointed to by the URL, and monitor the page loading process;

[0019] After the page loads, a set of predefined simulated user interaction operations are executed, and the current document object model tree is captured when the document structure reaches a stable state after each interaction.

[0020] By comparing and merging multiple Document Object Model (DOM) trees captured under the same browser emulation instance, a complete DOM tree that can contain all possible rendered content is obtained.

[0021] As a further aspect of the present invention: the step of using AI to compare multiple document object model trees to obtain unique content nodes between versions, and then selecting effective nodes to merge into a unified content tree structure, specifically includes:

[0022] Choose one version of the Document Object Model Tree as the baseline, and establish the mapping relationship between nodes in other versions of the Document Object Model Tree and the baseline tree by calculating the semantic and structural similarity of nodes;

[0023] Based on the mapping relationship, nodes that do not have a corresponding mapping in the baseline tree are defined as unique content nodes, and they are classified according to their attributes and context features.

[0024] For the unique content nodes after classification, a filtering rule based on the semantic features of psychological test questions is applied to identify and filter out the unique and valid content nodes that belong to the components of the test questions.

[0025] The selected unique and valid content nodes are integrated into the baseline tree based on their classification characteristics and structural context in the original tree to form the unified content tree structure.

[0026] As a further aspect of the present invention: the step of defining nodes without corresponding mappings in the baseline tree as unique content nodes based on mapping relationships, and classifying them according to their attributes and contextual features, specifically includes:

[0027] Analyze the expected layout positions of unique content nodes and their neighboring nodes on different devices to determine whether their content belongs to the same logical unit and is a visually unexpected break.

[0028] Analyze whether unique content nodes have event listener attributes or whether their display style is controlled by scripts to infer whether their presentation depends on specific user interactions;

[0029] Check the semantic tags or accessibility role attributes of the container element containing the unique content node to determine whether the container is suitable for holding such node content.

[0030] The local subtree structure changes containing the unique content node are matched with known responsive layout transition patterns and assigned descriptive category labels.

[0031] As a further aspect of the present invention: the step of extracting the question stem, options, and instructions of test questions based on a unified content tree structure to generate standardized test question data specifically includes:

[0032] Based on the unified content tree structure, the content is divided into independent question candidate blocks according to the title level, list structure and visual separation semantics;

[0033] Based on each candidate question block, the question stem text, option set text, and instruction text are identified and extracted;

[0034] In the content tree structure, the extracted text elements are located and associated with their corresponding image and audio non-text resource identifiers;

[0035] The extracted and associated text elements are encapsulated according to a predefined structured data pattern to obtain the standardized test question data.

[0036] Another objective of this invention is to provide an AI-based dynamic multimodal psychological test item crawling system, the system comprising:

[0037] The target configuration module is used to obtain the URL of the target psychological test page and select a set of device configuration parameters that includes at least two specifications, desktop and mobile, based on a preset device type library to simulate different client access environments.

[0038] The multi-modal crawling module is used to load and render the target psychological test page according to each configuration in the device configuration parameter set, and to crawl the complete document object model tree presented in each simulation environment;

[0039] The content fusion module is used to compare multiple document object model trees using AI to obtain unique content nodes between versions, and then select effective nodes to merge into a unified content tree structure.

[0040] The question generation module is used to extract the question stem, options, and instructions from the questions based on a unified content tree structure to generate standardized question data.

[0041] As a further aspect of the present invention: the target configuration module includes:

[0042] The syntax verification unit is used to perform syntax verification on the URL and initiate a verification network request to confirm its accessibility.

[0043] The rule extraction unit is used to access the URL after the request is approved, parse the viewport control meta tags and link style sheets in the HTML document header, and extract declarative rules related to device adaptation.

[0044] The parameter generation unit is used to filter out device types from a preset device type library based on the extracted adaptation rules, and generate a set of device configuration parameters containing a defined viewport size and user agent identifier.

[0045] The set optimization unit is used to perform redundancy removal on the generated set of device configuration parameters and determine the priority order based on the coverage of the adaptation rules.

[0046] As a further aspect of the present invention: the multi-mode grasping module includes:

[0047] An instance initialization unit is used to initialize a corresponding browser emulation instance for each configuration item in the device configuration parameter set and apply all its parameters.

[0048] The controlled loading unit is used to control each browser emulation instance to navigate to the resource pointed to by the URL and to monitor the page loading process;

[0049] The interactive capture unit is used to execute a set of predefined simulated user interaction operations after the page is loaded, and to capture the current document object model tree when the document structure reaches a stable state after each interaction.

[0050] The State Fusion Unit is used to compare and merge multiple Document Object Model Trees captured under the same browser emulation instance to obtain a complete Document Object Model Tree that can contain all possible rendered content.

[0051] As a further aspect of the present invention: the content fusion module includes:

[0052] The tree alignment unit is used to select a version of the Document Object Model tree as a baseline and establish the mapping relationship between nodes in other versions of the Document Object Model tree and the baseline tree through node semantic and structural similarity calculation.

[0053] The difference classification unit is used to define nodes that have no corresponding mapping in the baseline tree as unique content nodes based on the mapping relationship, and classify them according to their attributes and context features.

[0054] The validity screening unit is used to judge the unique content nodes after classification by applying filtering rules based on the semantic features of psychological test questions, and to screen out the unique valid content nodes that belong to the constituent parts of the test questions.

[0055] The structural fusion unit is used to integrate the selected unique and valid content nodes into the base tree based on their classification characteristics and structural context in the original tree, to form the unified content tree structure.

[0056] Compared with the prior art, the beneficial effects of the present invention are:

[0057] This invention, by simulating parallel crawling across multiple client access environments such as desktop and mobile devices, can automatically acquire the complete document object model of the same test page under different device specifications. This fundamentally avoids information omissions caused by differences in layout adaptation, content visibility, or interaction methods, ensuring the comprehensiveness of test data collection. Furthermore, it can intelligently identify and filter valid unique content nodes across versions, reconstructing fragmented and heterogeneous multi-version data into a complete and structurally unified test question tree, effectively solving the data inconsistency problem caused by dynamic content changes in responsive design. Through validity filtering and structured extraction, irrelevant information can be accurately removed, improving data quality and the efficiency of subsequent processing. In summary, this invention, by simulating multiple client access environments and introducing an AI-driven intelligent comparison, filtering, and fusion mechanism, achieves efficient, accurate, and automated crawling of dynamic multi-modal psychological test questions, providing a reliable data source for building large-scale standardized question banks and precise psychological research. Attached Figure Description

[0058] Figure 1 This is a flowchart of an AI-based method for crawling dynamic multimodal psychological test questions.

[0059] Figure 2 This is a flowchart illustrating the process of selecting a set of device configuration parameters based on a pre-defined device type library in an AI-based dynamic multimodal psychological test question crawling method.

[0060] Figure 3 This is a flowchart illustrating the process of retrieving a complete document object model tree in an AI-based dynamic multimodal psychological test question crawling method.

[0061] Figure 4 This is a flowchart illustrating the process of obtaining unique content nodes and integrating them into a content tree structure in an AI-based dynamic multimodal psychological test question crawling method.

[0062] Figure 5 This is a flowchart illustrating the process of defining unique content nodes (bi) and classifying them in an AI-based dynamic multimodal psychological test question crawling method.

[0063] Figure 6 This is a flowchart for generating standardized test data in an AI-based dynamic multimodal psychological test item crawling method.

[0064] Figure 7 This is a schematic diagram of the structure of an AI-based dynamic multimodal psychological test question crawling system.

[0065] Figure 8 This is a schematic diagram of the target configuration module in an AI-based dynamic multimodal psychological test question crawling system.

[0066] Figure 9This is a schematic diagram of the multi-modal crawling module in an AI-based dynamic multi-modal psychological test question crawling system.

[0067] Figure 10 This is a schematic diagram of the content fusion module in an AI-based dynamic multimodal psychological test question crawling system. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0069] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0070] like Figure 1 As shown in the figure, this embodiment of the invention provides an AI-based method for crawling dynamic multimodal psychological test questions, the method comprising the following steps:

[0071] S100: Obtain the URL of the target psychological test page and select a set of device configuration parameters that includes at least two specifications, desktop and mobile, from the preset device type library to simulate different client access environments.

[0072] S200 loads and renders the target psychological test page according to each configuration in the device configuration parameter set, and captures the complete document object model tree presented in each simulation environment;

[0073] S300 uses AI to compare multiple document object model trees to obtain unique content nodes between versions, and selects effective nodes to merge into a unified content tree structure.

[0074] S400, based on a unified content tree structure, extracts the question stem, options, and instructions of test questions to generate standardized test question data.

[0075] In this embodiment of the invention, by simulating parallel crawling of multiple client access environments such as desktop and mobile devices, the invention can automatically acquire the complete document object model of the same test page under different device specifications. This fundamentally avoids information omissions caused by differences in layout adaptation, content visibility, or interaction methods, ensuring the comprehensiveness of test data collection. Furthermore, it can intelligently identify and filter valid unique content nodes between versions, reconstructing fragmented and heterogeneous multi-version data into a complete and structurally unified test question tree, effectively solving the data inconsistency problem caused by dynamic content changes in responsive design. Through validity filtering and structured extraction, irrelevant information can be accurately removed, improving data quality and the efficiency of subsequent processing. In summary, this invention, by simulating multiple client access environments and introducing an AI-driven intelligent comparison, filtering, and fusion mechanism, achieves efficient, accurate, and automated crawling of dynamic multi-modal psychological test questions.

[0076] like Figure 2 As shown, in a preferred embodiment of the present invention, the step of obtaining the URL of the target psychological test page and selecting a set of device configuration parameters that includes at least desktop and mobile versions according to a preset device type library specifically includes:

[0077] S101, Perform syntax validation on the URL and initiate a verification network request to confirm its accessibility.

[0078] S102, after the request is approved, access the URL, parse the viewport control meta tags and link style sheets in the HTML document header, and extract the declarative rules related to device adaptation.

[0079] S103, based on the extracted adaptation rules, select the device type from the preset device type library and generate a set of device configuration parameters containing the determined viewport size and user agent identifier;

[0080] S104, perform redundancy removal on the generated set of device configuration parameters, and determine the priority order based on the coverage of the adaptation rules.

[0081] In this embodiment of the invention, the system first receives the URL (a standard address identifier for internet resources) of a target psychological test page input or specified by the user. To ensure the effectiveness of subsequent processing, the system proactively performs basic syntax validation on the URL (e.g., confirming the standardization of its protocol, domain name, and path format) and attempts to initiate a lightweight network request to verify the actual accessibility of the link, preventing the entire process from being interrupted due to link failure or network obstacles. After successful verification, the system formally accesses the page and obtains its HTML source code. Subsequently, it automatically parses the key viewport control meta tags in the document header and all external cascading style sheet files. By scanning these style codes, the system can intelligently extract declarative rules related to device adaptation, such as media query statements (e.g., identifying breakpoint rules that indicate mobile adaptation). Then, based on these extracted specific adaptation rules, the system makes dynamic decisions: it selects the key device types from the library that best trigger and override the identified rules. For example, if a breakpoint rule targeting a width of 768px is detected, the system may simultaneously select a "1920×1080" configuration representing a typical desktop window and a "375×667" configuration representing a typical mobile device, and generate a complete parameter set for each configuration, including the precise viewport size, device pixel ratio, and corresponding user agent string. Finally, the initially generated parameter set will be optimized, for example, by merging redundant configurations with overly similar viewport sizes, and determining their processing priority based on the range of adaptation rules each configuration can cover. This ensures optimal resource allocation and improved processing efficiency while maintaining comprehensive content capture.

[0082] like Figure 3 As shown, in a preferred embodiment of the present invention, the steps of loading and rendering the target psychological test page according to each configuration item in the device configuration parameter set, and retrieving the complete document object model tree presented in each simulation environment, specifically include:

[0083] S201, initialize a corresponding browser emulation instance for each configuration item in the device configuration parameter set and apply all its parameters;

[0084] S202, control each browser emulation instance to navigate to the resource pointed to by the URL, and monitor the page loading process;

[0085] S203, after the page is loaded, execute a set of predefined simulated user interaction operations, and capture the current document object model tree when the document structure reaches a stable state after each interaction;

[0086] S204 compares and merges multiple Document Object Model Trees captured under the same browser emulation instance to obtain a complete Document Object Model Tree that can contain all possible rendered content.

[0087] In this embodiment of the invention, based on the set of device configuration parameters mentioned above, a browser simulation instance (such as a headless browser instance based on Headless Chrome) is independently initialized for each configuration (e.g., "1920×1080" for desktop and "375×667" for mobile). All parameters, including the corresponding viewport size and user agent string, are precisely loaded into this instance, thereby creating a highly realistic access environment for a specific device. Each simulation instance is then driven to navigate to the target URL, and its network requests and page rendering status are continuously monitored until all critical resources are loaded and initial rendering is complete. To ensure the capture of test content that dynamically appears only through user interaction (e.g., standard answers that require clicking the "expand explanation" button to display, or hidden options that need to be swiped to view), a series of predefined simulated interactive operations, such as simulated clicks, scrolling, or touch events, are automatically executed after the page is ready. After each interaction, a snapshot of the Document Object Model (DOM) tree is captured only when the page document structure is stable. This method of multiple interactions and snapshot capture effectively covers the content presentation of the page under various interactive states. Finally, for multiple DOM tree snapshots obtained under the same simulation instance, intelligent comparison and node merging will be performed. For example, features such as node path and content hash will be used to identify and merge newly added or different nodes in different snapshots, thereby integrating and generating a complete DOM tree that can fully reflect all potentially visible content in the device environment.

[0088] like Figure 4 As shown, in a preferred embodiment of the present invention, the step of using AI to compare multiple document object model trees to obtain unique content nodes between versions, and then selecting effective nodes to merge into a unified content tree structure, specifically includes:

[0089] S301. Select a version of the Document Object Model tree as the baseline, and establish the mapping relationship between nodes in other versions of the Document Object Model tree and the baseline tree through node semantic and structural similarity calculation.

[0090] S302, Based on the mapping relationship, nodes that do not have a corresponding mapping in the baseline tree are defined as unique content nodes, and they are classified according to their attributes and context features;

[0091] S303, For the unique content nodes after classification, apply the filtering rules based on the semantic features of psychological test questions to identify and select the unique and valid content nodes that belong to the constituent parts of the test questions.

[0092] S304, the selected unique and valid content nodes are integrated into the baseline tree based on their classification characteristics and structural context in the original tree to form the unified content tree structure.

[0093] In this embodiment of the invention, the core lies in applying AI technology to intelligently analyze and fuse the captured multi-version Document Object Model (DOM) trees. After obtaining multiple DOM trees, one version of the DOM tree is first selected as a baseline, for example, the desktop DOM tree is used as the basic framework. Then, by calculating the semantic similarity (such as comparing text content and tag types) and structural similarity (such as comparing the depth of nodes in the tree and the structure of child nodes) between nodes, the most likely corresponding node in the baseline tree is found for each node in other versions of the DOM tree, such as mobile devices, thereby establishing a precise mapping relationship network for cross-version nodes. Based on this mapping relationship network, nodes that cannot be found in the baseline tree, i.e., unique content nodes, can be clearly identified. These nodes may be a prompt text displayed separately on mobile devices, a collapsible panel that only appears on tablet devices, or an extra line break node generated due to the layout changing from multiple columns on desktop to a single column on mobile devices. Next, based on the HTML attributes, CSS class names, and surrounding context of these nodes, they are initially classified, for example, they can be divided into "layout fill type", "interactive extension type", or "conditional annotation type". For the specific domain of psychological testing, the system applies a set of pre-defined semantic filtering rules rich in domain knowledge for deep discrimination. For example, a unique node classified as "interactive and extended" is considered a unique and valid content node that contributes to the construction of a complete test question if its text content contains typical question phrases (such as "What do you think…"), option markers (such as "A", "①"), or scale keywords (such as "strongly agree", "never"). Conversely, if its content is only a "back to top" button or an advertising link, it will be filtered out. Finally, these selected valid nodes are stitched to the corresponding logical positions in the baseline DOM tree according to their classification and their original structural position in the tree (such as determining whether their parent node is a question container or an option list). This merges and reconstructs multiple perspectives of potentially different content fragments into a comprehensive and structurally unified content tree structure.

[0094] like Figure 5 As shown, in a preferred embodiment of the present invention, the step of defining nodes without corresponding mappings in the baseline tree as unique content nodes based on mapping relationships, and classifying them according to their attributes and contextual features, specifically includes:

[0095] S312, Analyze the expected layout positions of unique content nodes and their neighboring nodes on different devices to determine whether their content belongs to the same logical unit and is a visually unexpected break.

[0096] S322, Analyze whether unique content nodes have event listener attributes or their display style is controlled by scripts, in order to infer whether their presentation depends on specific user interactions;

[0097] S332, check the semantic tag or accessibility role attribute of the container element where the unique content node is located to determine whether the container is suitable to hold such node content.

[0098] S342, match the local subtree structure changes containing the unique content node with known responsive layout transition patterns, and assign them descriptive category labels.

[0099] In this embodiment of the invention, the analysis first examines visual continuity. By calculating the expected rendering coordinates of a node and its neighboring text nodes under different device viewports, it determines whether the content belongs to the same logical semantic unit (such as a complete sentence or the stem of a question) that has been accidentally fragmented due to responsive adjustments such as line breaks and column splits. For example, a question stem displayed as a single line on a desktop may be broken into two independent text nodes on a narrow mobile screen. The system analyzes and identifies the continuity of these two nodes in the visual flow and associates them. Secondly, the interaction dependency attributes of the node are analyzed. It checks whether the node or its parent element is bound to event listeners such as onclick and onmouseover, or whether its display style is dynamically controlled by JavaScript, thereby inferring whether the node's content (such as a detailed explanation of a question or a hidden option) requires specific user interactions such as clicking or hovering to be displayed. Then, container semantic adaptability is checked, mainly examining the container element that carries the unique node and determining whether it uses HTML5 semantic tags (such as...). <section> 、 <aside>Whether the WAI-ARIA accessibility role attributes match the intended function of the node content itself. For example, a node containing option text, if placed in... <footer>Within the tags, it may be judged as semantically mismatched noise. Finally, the structural change features of this local subtree (such as from the desktop) are analyzed. The list is matched against a pre-built knowledge base of responsive layout transition patterns. The matching process assigns a category label with a clear design intent description to the unique node (such as "layout break compensation node", "interactive trigger hidden content", or "container semantic migration content"). This category result will directly guide the subsequent accurate selection of node validity and reasonable integration into the baseline tree.

[0100] like Figure 6 As shown, in a preferred embodiment of the present invention, the step of extracting the question stem, options, and instructions of test questions based on a unified content tree structure to generate standardized test question data specifically includes:

[0101] S401, on the unified content tree structure, the content is divided into independent test question candidate blocks according to the title level, list structure and visual separation semantics;

[0102] S402, based on each candidate question block, identifies and extracts the question stem text, option set text, and instruction text;

[0103] S403, in the content tree structure, the extracted text elements are located and associated with their corresponding image and audio non-text resource identifiers;

[0104] S404, the extracted and associated text elements are encapsulated according to a predefined structured data pattern to obtain the standardized test question data.

[0105] In this embodiment of the invention, the content tree is first intelligently segmented. Based on the hierarchical relationship of title tags, the nested structure of list elements, and HTML elements with visual separation semantics, the continuous content flow is automatically divided into independent candidate question blocks, thus defining the boundaries for subsequent fine extraction. For each candidate block, a hybrid method based on natural language processing and rule matching is used to identify and extract the core question stem text, the sequentially arranged set of option text, and any possible answer guidance text. To construct a complete question object, the nodes corresponding to these text elements are located back in the original content tree, and non-text resource identifiers closely related to the question are associated and obtained, such as the URL of the explanatory image referenced in the question stem, or the address of an audio file related to the auditory test. Finally, these extracted text elements and associated multimodal resource identifiers are encapsulated into a standard question data object with a unified format according to a predefined structured data pattern, such as JSON format, which explicitly includes standardized fields such as question type, question stem, option list, answer, explanation, and resource links, thereby outputting structured data that can be directly used by question bank systems or research tools.

[0106] like Figure 7 As shown, this embodiment of the invention also provides an AI-based dynamic multimodal psychological test question crawling system, the system comprising:

[0107] The target configuration module 100 is used to obtain the URL of the target psychological test page and select a set of device configuration parameters that includes at least two specifications, desktop and mobile, according to a preset device type library, in order to simulate different client access environments.

[0108] The multi-modal crawling module 200 is used to load and render the target psychological test page according to each configuration in the device configuration parameter set, and to crawl the complete document object model tree presented in each simulation environment;

[0109] The content fusion module 300 is used to compare multiple document object model trees using AI to obtain unique content nodes between versions, and then select effective nodes to merge into a unified content tree structure.

[0110] The question generation module 400 is used to extract the question stem, options, and instructions of the questions based on a unified content tree structure to generate standardized question data.

[0111] In this embodiment of the invention, the target configuration module 100 first receives the URL of the target psychological test page, performs normative and accessibility verification on it, then parses the page's viewport meta tags and style sheets to extract device adaptation rules, and dynamically generates a parameter set covering key configurations such as desktop and mobile devices from a preset device library. The multi-modal crawling module 200 initializes an independent browser simulation instance for each configuration based on this parameter set, loads the target page and monitors its rendering process, triggers dynamic content by simulating predefined user interactions, then captures the document object model tree under each stable interaction state, and merges multiple tree structures under the same instance into a complete DOM tree. The content fusion module 300 selects a baseline DOM tree, establishes cross-version node mapping through node semantic and structural similarity, identifies and classifies unique content nodes, applies rules based on psychological test semantic features to filter out effective nodes, and finally intelligently integrates them into the baseline tree to form a unified content tree structure. The question generation module 400 is based on this unified content tree. It divides the content into question candidate blocks according to the title, list and visual separation semantics. Within each block, it identifies and extracts the question stem, options and instruction text, associates the corresponding non-text resource identifiers such as images or audio, and finally encapsulates and generates standardized question data according to the predefined structured data pattern.

[0112] like Figure 8 As shown, in a preferred embodiment of the present invention, the target configuration module 100 includes:

[0113] The syntax verification unit 101 is used to perform syntax verification on the URL and initiate a verification network request to confirm its accessibility.

[0114] The rule extraction unit 102 is used to access the URL after the request is approved, parse the viewport control meta tags and link style sheets in the HTML document header, and extract declarative rules related to device adaptation.

[0115] The parameter generation unit 103 is used to filter out the device type from the preset device type library based on the extracted adaptation rules, and generate a set of device configuration parameters containing a determined viewport size and user agent identifier.

[0116] The set optimization unit 104 is used to perform redundancy removal on the generated set of device configuration parameters and determine the priority order based on the coverage of the adaptation rules.

[0117] like Figure 9 As shown, in a preferred embodiment of the present invention, the multi-mode grasping module 200 includes:

[0118] The instance initialization unit 201 is used to initialize a corresponding browser emulation instance for each item in the device configuration parameter set and apply all its parameters.

[0119] The controlled loading unit 202 is used to control each browser emulation instance to navigate to the resource pointed to by the URL and to monitor the page loading process;

[0120] The interactive capture unit 203 is used to execute a set of predefined simulated user interaction operations after the page is loaded, and to capture the current document object model tree when the document structure reaches a stable state after each interaction.

[0121] The state fusion unit 204 is used to compare and merge multiple document object model trees captured under the same browser emulation instance to obtain a complete document object model tree that can contain all possible rendered content.

[0122] like Figure 10 As shown, in a preferred embodiment of the present invention, the content fusion module 300 includes:

[0123] Tree alignment unit 301 is used to select a version of the document object model tree as a baseline, and establish the mapping relationship between nodes in other versions of the document object model tree and the baseline tree through node semantic and structural similarity calculation.

[0124] The difference classification unit 302 is used to define nodes that have no corresponding mapping in the baseline tree as unique content nodes based on the mapping relationship, and classify them according to their attributes and context features.

[0125] The validity screening unit 303 is used to judge the unique content nodes after classification by applying filtering rules based on the semantic features of psychological test questions, and to screen out the unique valid content nodes that belong to the constituent parts of the test questions.

[0126] The structural fusion unit 304 is used to integrate the selected unique and valid content nodes into the base tree based on their classification characteristics and structural context in the original tree, to form the unified content tree structure.

[0127] The above description only details the preferred embodiments of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0128] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0129] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0130] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the disclosure in the specification and embodiments. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims. The layout has been changed to mobile. < / footer> < / aside> < / section>

Claims

1. An AI-based method for crawling dynamic multi-modal psychological test questions, characterized in that, The method includes the following steps: Obtain the URL of the target psychological test page, and select a set of device configuration parameters that includes at least two specifications, desktop and mobile, based on a preset device type library to simulate different client access environments; The target psychological test page is loaded and rendered according to each configuration in the device configuration parameter set, and the complete document object model tree presented in each simulation environment is captured. AI is used to compare multiple document object model trees to obtain unique content nodes between versions, and effective nodes are selected and merged into a unified content tree structure. Based on a unified content tree structure, the question stem, options, and instructions are extracted to generate standardized question data; The step of using AI to compare multiple document object model trees to obtain unique content nodes between versions, and then selecting effective nodes to merge into a unified content tree structure, specifically includes: Choose one version of the Document Object Model Tree as the baseline, and establish the mapping relationship between nodes in other versions of the Document Object Model Tree and the baseline tree by calculating the semantic and structural similarity of nodes; Based on the mapping relationship, nodes that do not have a corresponding mapping in the baseline tree are defined as unique content nodes, and they are classified according to their attributes and context features. For the unique content nodes after classification, a filtering rule based on the semantic features of psychological test questions is applied to identify and filter out the unique and valid content nodes that belong to the components of the test questions. The selected unique and valid content nodes are integrated into the baseline tree based on their classification characteristics and structural context in the original tree to form the unified content tree structure.

2. The AI-based dynamic multi-modal psychological test item crawling method according to claim 1, characterized in that, The step of obtaining the URL of the target psychological test page and selecting a set of device configuration parameters that includes at least desktop and mobile versions according to a preset device type library specifically includes: The URL is subjected to syntax validation, and a verification network request is initiated to confirm its accessibility. After the request is approved, access the URL, parse the viewport control meta tags and link style sheets in the HTML document header, and extract the declarative rules related to device adaptation. Based on the extracted adaptation rules, the device types are selected from the preset device type library, and a set of device configuration parameters containing a defined viewport size and user agent identifier is generated. The generated set of device configuration parameters is deredundant and the priority order is determined based on the coverage of the adaptation rules.

3. The AI-based dynamic multi-modal psychological test item crawling method according to claim 1, characterized in that, The steps of loading and rendering the target psychological test page according to each configuration item in the device configuration parameter set, and retrieving the complete document object model tree presented in each simulation environment, specifically include: For each configuration item in the device configuration parameter set, initialize a corresponding browser emulation instance and apply all its parameters; Control each browser emulation instance to navigate to the resource pointed to by the URL, and monitor the page loading process; After the page loads, a set of predefined simulated user interaction operations are executed, and the current document object model tree is captured when the document structure reaches a stable state after each interaction. By comparing and merging multiple Document Object Model (DOM) trees captured under the same browser emulation instance, a complete DOM tree that can contain all possible rendered content is obtained.

4. The AI-based dynamic multi-modal psychological test item crawling method according to claim 1, characterized in that, The step of defining nodes without corresponding mappings in the baseline tree as unique content nodes based on mapping relationships, and classifying them according to their attributes and contextual features, specifically includes: Analyze the expected layout positions of unique content nodes and their neighboring nodes on different devices to determine whether their content belongs to the same logical unit and is a visually unexpected break. Analyze whether unique content nodes have event listener attributes or whether their display style is controlled by scripts to infer whether their presentation depends on specific user interactions; Check the semantic tags or accessibility role attributes of the container element containing the unique content node to determine whether the container is suitable for holding such node content. The local subtree structure changes containing the unique content node are matched with known responsive layout transition patterns and assigned descriptive category labels.

5. The AI-based dynamic multi-modal psychological test item crawling method according to claim 1, characterized in that, The steps for extracting the question stem, options, and instructions from test questions based on a unified content tree structure to generate standardized test question data specifically include: Based on the unified content tree structure, the content is divided into independent question candidate blocks according to the title level, list structure and visual separation semantics; Based on each candidate question block, the question stem text, option set text, and instruction text are identified and extracted; In the content tree structure, the extracted text elements are located and associated with their corresponding image and audio non-text resource identifiers; The extracted and associated text elements are encapsulated according to a predefined structured data pattern to obtain the standardized test question data.

6. An AI-based dynamic multi-modal psychological test item crawling system, characterized in that, The system includes: The target configuration module is used to obtain the URL of the target psychological test page and select a set of device configuration parameters that includes at least two specifications, desktop and mobile, based on a preset device type library to simulate different client access environments. The multi-modal crawling module is used to load and render the target psychological test page according to each configuration in the device configuration parameter set, and to crawl the complete document object model tree presented in each simulation environment; The content fusion module is used to compare multiple document object model trees using AI to obtain unique content nodes between versions, and then select effective nodes to merge into a unified content tree structure. The question generation module is used to extract the question stem, options, and instructions of questions based on a unified content tree structure to generate standardized question data; The content fusion module includes: The tree alignment unit is used to select a version of the Document Object Model tree as a baseline and establish the mapping relationship between nodes in other versions of the Document Object Model tree and the baseline tree through node semantic and structural similarity calculation. The difference classification unit is used to define nodes that have no corresponding mapping in the baseline tree as unique content nodes based on the mapping relationship, and classify them according to their attributes and context features. The validity screening unit is used to judge the unique content nodes after classification by applying filtering rules based on the semantic features of psychological test questions, and to screen out the unique valid content nodes that belong to the constituent parts of the test questions. The structural fusion unit is used to integrate the selected unique and valid content nodes into the base tree based on their classification characteristics and structural context in the original tree, to form the unified content tree structure.

7. The AI-based dynamic multi-modal psychological test item crawling system according to claim 6, characterized in that, The target configuration module includes: The syntax verification unit is used to perform syntax verification on the URL and initiate a verification network request to confirm its accessibility. The rule extraction unit is used to access the URL after the request is approved, parse the viewport control meta tags and link style sheets in the HTML document header, and extract declarative rules related to device adaptation. The parameter generation unit is used to filter out device types from a preset device type library based on the extracted adaptation rules, and generate a set of device configuration parameters containing a defined viewport size and user agent identifier. The set optimization unit is used to perform redundancy removal on the generated set of device configuration parameters and determine the priority order based on the coverage of the adaptation rules.

8. The AI-based dynamic multi-modal psychological test item crawling system according to claim 6, characterized in that, The multi-mode capture module includes: An instance initialization unit is used to initialize a corresponding browser emulation instance for each configuration item in the device configuration parameter set and apply all its parameters. The controlled loading unit is used to control each browser emulation instance to navigate to the resource pointed to by the URL and to monitor the page loading process; The interactive capture unit is used to execute a set of predefined simulated user interaction operations after the page is loaded, and to capture the current document object model tree when the document structure reaches a stable state after each interaction. The State Fusion Unit is used to compare and merge multiple Document Object Model Trees captured under the same browser emulation instance to obtain a complete Document Object Model Tree that can contain all possible rendered content.

Images