A system and method based on screen level perception and human behavior learning

Abstract

The application discloses a system and method based on screen-level perception and human behavior learning, and the system comprises a multi-modal data acquisition unit, a time sequence segmentation unit, a semantic analysis and intention extraction unit, a visual object recognition and interface state analysis unit, a multi-modal consistency judgment and event alignment unit, a knowledge graph node relationship construction unit, a matching error inquiry unit and a data storage unit. Through multi-modal perception of the operation behavior in front of the screen, the application realizes dynamic adjustment of the automatic learning workflow according to the business rules and the page structure change.

Description

Technical Field

[0001] This invention belongs to the field of intelligent processing systems for multi-source data, specifically, it relates to a system and method based on screen-level perception and human behavior learning. Background Technology

[0002] In many real-world work scenarios, especially within enterprise information systems, web systems, and desktop application systems, there are numerous repetitive or semi-structured operational processes that rely on manual completion. These processes typically have the following characteristics:

[0003] The operation process relies heavily on human understanding and judgment of the interface content; The system itself does not provide a stable and complete programming interface (API) to the outside world, or the interface permissions are restricted; The workflow is dynamically adjusted according to changes in business rules and page structure, making it difficult to maintain in the long term using a fixed script; Existing automation tools mainly rely on predefined rules or static processes, which are prone to failure once the interface or logic changes. Manual training and execution are costly, and the consistency and replicability of the results are poor.

[0004] The shortcoming of existing technologies is that existing application software systems lack the ability to perceive on-screen user behavior and to dynamically adjust automated learning workflows as business rules and page structures change. Summary of the Invention

[0005] To address the problem that existing application software systems lack perception based on screen-level user behavior and thus fail to achieve automated learning, this invention provides a system and method based on screen-level perception and human behavior learning.

[0006] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows: A system based on screen-level perception and human behavior learning includes: a multimodal data acquisition unit, a temporal segmentation unit, a semantic parsing and intent extraction unit, a visual object recognition and interface state parsing unit, a multimodal consistency judgment and event alignment unit, a knowledge graph node relationship construction unit, an error query matching unit, and a data storage unit. The multimodal data acquisition unit is used to collect multi-source data and the corresponding unified timestamps of the multi-source data during the task execution process of the currently running system. The time-series segmentation unit is used to segment continuous multi-source data streams using user-generated operation events as key anchor points. The semantic parsing and intent extraction unit is used to perform natural language processing on speech data from multi-source data to achieve intent recognition and keyword extraction. The visual object recognition and interface state parsing unit is used to perform image recognition and interface parsing on screenshots or video frames associated with operation events in multi-source data. The multimodal consistency judgment and event alignment unit integrates and analyzes the speech parsing results and visual recognition results to determine whether the multi-source data are consistent in the three dimensions of event object, action and result. Construct knowledge graph node relationship units. When the event objects, actions and results of multi-source data are consistent in three dimensions, the knowledge graph is automatically generated or updated and stored in the data storage unit. If a matching error occurs, and the data from multiple sources is inconsistent across the dimensions of event object, action, and result, the user is asked to provide supplementary explanations or confirm the relationship. Data storage unit, used to store multi-source data and knowledge graphs generated from multi-source data.

[0007] Furthermore, the multi-source data includes: voice data, screen image data, screen video data, and operation event data; Voice data: including continuous voice input generated by the user during operation; Screen image data: Automatic screenshots are taken at a fixed frequency or triggered by events; Screen video data: Continuous screen recording of the operation process; Operation event data: including mouse clicks, drags, keyboard inputs, and window switching.

[0008] Furthermore, the detailed steps for segmenting continuous multi-source data streams in the time-series segmentation unit are as follows: When an operation event is detected, audio and visual data within a preset time window are captured forward and backward, respectively. Form behavioral segments centered around a single operational event; Each action segment includes: the intention expressed before the operation, the operation itself, and supplementary explanations or feedback after the operation. This segmentation method avoids the semantic drift problem caused by relying solely on continuous audio or visual streams.

[0009] Furthermore, the semantic parsing and intent extraction unit performs natural language processing on the speech data from the multi-source data to achieve detailed steps for intent recognition and keyword extraction: An audio-to-text data conversion model is used to convert speech into text. Intent recognition and keyword extraction; Extraction of semantic elements of objects, actions, conditions, and results; Determine whether the voice message is a directive, an explanation, or a result feedback.

[0010] The audio data to text data conversion model includes a front-end processing module, an acoustic model, a language model, and a decoder; The front-end processing module is used to convert the raw audio signal into an acoustic feature sequence suitable for model processing, including noise reduction, gain control, frame windowing (segmenting the continuous signal into short time frames), and feature extraction (such as extracting Mel frequency cepstral coefficients (MFCC) or filter bank features). Acoustic models employ end-to-end deep learning-based architectures, such as convolutional neural networks (CNN), recurrent neural networks (RNN / LSTM), or Transformers and their variants (such as Conformer), to establish probabilistic mappings from acoustic features to phonemes or characters. They directly learn the mapping from acoustic features to text sequences through a connection-time classification (CTC) loss function. The language model provides syntactic and semantic constraints for the decoding process. It corrects the output errors of the acoustic model by statistically analyzing the prior probabilities of word sequences. During decoding, the scores of the language model and the acoustic model are fused to improve the fluency and accuracy of the recognition results.

[0011] Furthermore, the detailed steps for image recognition and interface parsing of screenshots or video frames associated with operation events in the visual object recognition and interface state parsing unit include: Identify interactive objects in the current interface, including buttons, input boxes, and menu items. Identify the visual characteristics of the object being manipulated and its state changes; Extract the differences between the interface before and after the operation to determine the operation result.

[0012] Furthermore, the detailed steps for determining whether multi-source data are consistent across the three dimensions of event object, action, and result in the multimodal consistency judgment and event alignment unit are as follows: Determine whether the object mentioned in the voice matches the object being manipulated on the interface; Determine whether the actions described in the voice are consistent with the actual actions performed. Determine whether the expected result mentioned in the voice message matches the changes in the interface.

[0013] Furthermore, the detailed steps for automatically generating or updating the knowledge graph in the knowledge graph node relationship unit are as follows: The interface element or business entity to be manipulated is treated as an object node. Treat the user's actions as action relationships; Treat the interface changes or business results triggered by the operation as result nodes; Record the context and constraints of the operation event.

[0014] This knowledge graph serves as a structured foundation for subsequent process learning, reproduction, and reasoning.

[0015] Furthermore, in cases where multi-source data in the matching error query unit is inconsistent across the three dimensions of event object, action, and result, the following situations exist: Insufficient matching between voice and the target being operated on; The voice description is ambiguous; The operation result was inconsistent with the expected voice prompt.

[0016] The system will automatically generate targeted queries to request users to provide supplementary explanations or confirm relationships, in order to further improve the understanding of the event and update the knowledge structure.

[0017] A method based on screen-level perception and human behavior learning includes the following steps: S1. Collect multi-source data and the corresponding unified timestamps of the multi-source data during the task execution process of the currently running system; S2. Use user-generated operation events as key anchor points to segment continuous multi-source data streams; S3. Perform natural language processing on speech data from multi-source data to achieve intent recognition and keyword extraction; S4. Perform image recognition and interface parsing on screenshots or video frames associated with operation events in multi-source data; S5. Combine and analyze the speech parsing results with the visual recognition results to determine whether the multi-source data are consistent in terms of event object, action and result. S6. When the event objects, actions and results of multi-source data are consistent in three dimensions, a knowledge graph is automatically generated or updated and stored in the data storage unit. S7. If there are inconsistencies in the three dimensions of multi-source data event objects, actions, and results, request the user to provide supplementary explanations or confirm the relationship.

[0018] Compared with the prior art, the present invention has the following advantages: This system employs multimodal data fusion and temporal consistency analysis to uniformly model operational behaviors and expressive information during task execution. The system synchronously collects voice data, automatic screenshots, and continuous screen recordings, aligning the multi-source data based on timestamps. It then semantically segments the continuous data stream by combining user-generated operation events, forming behavioral fragments centered around these events. Within each behavioral fragment, the algorithm correlates the user's voice description within a certain time window before and after the operation with corresponding interface changes and operational behaviors. Natural language processing and image recognition technologies are used to determine whether the voice content, visual objects, and operational behaviors point to the same event and object. Based on this, the system automatically constructs a knowledge graph with objects, actions, conditions, and results as core elements, achieving a structured expression of the task flow. When a definite matching relationship cannot be formed between multimodal information, the system proactively generates targeted queries to supplement contextual information, thereby confirming event correlation and continuously optimizing understanding accuracy. Attached Figure Description

[0019] Figure 1 This is an overall structural block diagram of a system based on screen-level perception and human behavior learning in an embodiment of the present invention; Figure 2 This is an overall flowchart of a method based on screen-level perception and human behavior learning in an embodiment of the present invention. Detailed Implementation

[0020] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and accompanying drawings. The content mentioned in the embodiments is not intended to limit the present invention.

[0021] like Figure 1 As shown, this embodiment provides a system based on screen-level perception and human behavior learning, including: a multimodal data acquisition unit, a temporal segmentation unit, a semantic parsing and intent extraction unit, a visual object recognition and interface state parsing unit, a multimodal consistency judgment and event alignment unit, a knowledge graph node relationship construction unit, an error query matching unit, and a data storage unit; The multimodal data acquisition unit is used to collect multi-source data and the corresponding unified timestamps of the multi-source data during the task execution process of the currently running system. The time-series segmentation unit is used to segment continuous multi-source data streams using user-generated operation events as key anchor points. The semantic parsing and intent extraction unit is used to perform natural language processing on speech data from multi-source data to achieve intent recognition and keyword extraction. The visual object recognition and interface state parsing unit is used to perform image recognition and interface parsing on screenshots or video frames associated with operation events in multi-source data. The multimodal consistency judgment and event alignment unit integrates and analyzes the speech parsing results and visual recognition results to determine whether the multi-source data are consistent in the three dimensions of event object, action and result. Construct knowledge graph node relationship units. When the event objects, actions and results of multi-source data are consistent in three dimensions, the knowledge graph is automatically generated or updated and stored in the data storage unit. If a matching error occurs, and the data from multiple sources is inconsistent across the dimensions of event object, action, and result, the user is asked to provide supplementary explanations or confirm the relationship. Data storage unit, used to store multi-source data and knowledge graphs generated from multi-source data.

[0022] Multi-source data includes: voice data, screen image data, screen video data, and operation event data; Voice data: including continuous voice input generated by the user during operation; Screen image data: Automatic screenshots are taken at a fixed frequency or triggered by events; Screen video data: Continuous screen recording of the operation process; Operation event data: including mouse clicks, drags, keyboard inputs, and window switching.

[0023] Detailed steps for segmenting continuous multi-source data streams in the time-series segmentation unit: When an operation event is detected, audio and visual data within a preset time window are captured forward and backward, respectively. Form behavioral segments centered around a single operational event; Each action segment includes: the intention expressed before the operation, the operation itself, and supplementary explanations or feedback after the operation. This segmentation method avoids the semantic drift problem caused by relying solely on continuous audio or visual streams.

[0024] The semantic parsing and intent extraction unit performs natural language processing on speech data from multi-source datasets to achieve intent recognition and keyword extraction. The detailed steps are as follows: The audio data to text conversion model is used to convert speech to text. Intent recognition and keyword extraction; Extraction of semantic elements of objects, actions, conditions, and results; Determine whether the voice message is a directive, an explanation, or a result feedback.

[0025] The audio data to text data conversion model includes a front-end processing module, an acoustic model, a language model, and a decoder; The front-end processing module is used to convert the raw audio signal into an acoustic feature sequence suitable for model processing, including noise reduction, gain control, frame windowing (segmenting the continuous signal into short time frames), and feature extraction (such as extracting Mel frequency cepstral coefficients (MFCC) or filter bank features). Acoustic models employ end-to-end deep learning-based architectures, such as convolutional neural networks (CNN), recurrent neural networks (RNN / LSTM), or Transformers and their variants (such as Conformer), to establish probabilistic mappings from acoustic features to phonemes or characters. They directly learn the mapping from acoustic features to text sequences through a connection-time classification (CTC) loss function. The language model provides syntactic and semantic constraints for the decoding process and corrects the output errors of the acoustic model by statistically analyzing the prior probabilities of word sequences. The decoder fuses the scores from the language model and the acoustic model to improve the fluency and accuracy of the recognition results.

[0026] The core of a decoder is finding the optimal path between the acoustic model and the language model. Commonly used algorithms include: Viterbi Decoding: Implemented using dynamic programming, with a time complexity of O(TN²). Suitable for small vocabulary lists (<10k words).

[0027] WFST Decoding: The probabilities of HMM states, words, and language models are compiled into finite state transformers. Supports real-time decoding (latency <300ms); Memory usage optimization techniques: state compression and arc merging.

[0028] Beam search optimization: Beam Size is dynamically adjusted (initially 32, gradually reduced to 8); Historical path pruning (retaining the top 90% of paths by probability); Length normalization (add path length penalty).

[0029] The detailed steps for image recognition and interface parsing of screenshots or video frames associated with operation events in the visual object recognition and interface state parsing unit include: Identify interactive objects in the current interface, including buttons, input boxes, and menu items. Identify the visual characteristics of the object being manipulated and its state changes; Extract the differences between the interface before and after the operation to determine the operation result.

[0030] Detailed steps for determining whether multi-source data is consistent across the three dimensions of event object, action, and result in the multimodal consistency judgment and event alignment unit: Determine whether the object mentioned in the voice matches the object being manipulated on the interface; Determine whether the actions described in the voice are consistent with the actual actions performed. Determine whether the expected result mentioned in the voice message matches the changes in the interface.

[0031] Detailed steps for automatically generating or updating the knowledge graph in the node relationship unit of the knowledge graph construction: The interface element or business entity to be manipulated is treated as an object node. Treat the user's actions as action relationships; Treat the interface changes or business results triggered by the operation as result nodes; Record the context and constraints of the operation event.

[0032] This knowledge graph serves as a structured foundation for subsequent process learning, reproduction, and reasoning.

[0033] Inconsistencies in multi-source data across the event object, action, and result dimensions within the match error query unit include: Insufficient matching between voice and the target being operated on; The voice description is ambiguous; The operation result was inconsistent with the expected voice prompt.

[0034] The system will automatically generate targeted queries to request users to provide supplementary explanations or confirm relationships, in order to further improve the understanding of the event and update the knowledge structure.

[0035] like Figure 2 As shown, a method based on screen-level perception and human behavior learning includes the following steps: Collect multi-source data and the corresponding unified timestamps of the multi-source data during the task execution process of the currently running system; Using user-generated operation events as key anchor points, continuous multi-source data streams are segmented; Natural language processing is performed on speech data from multi-source datasets to achieve intent recognition and keyword extraction; Perform image recognition and interface parsing on screenshots or video frames associated with operation events from multi-source data; By fusing and analyzing the speech analysis results with the visual recognition results, it can be determined whether the multi-source data are consistent in terms of event object, action and result. When the event objects, actions, and results of multi-source data are consistent in three dimensions, a knowledge graph is automatically generated or updated and stored in the data storage unit. If there are inconsistencies in the event objects, actions, and results across multiple data sources, request the user to provide supplementary explanations or confirm the relationships.

[0036] Compared with the prior art, the present invention has the following advantages: This system employs multimodal data fusion and temporal consistency analysis to uniformly model operational behaviors and expressive information during task execution. The system synchronously collects voice data, automatic screenshots, and continuous screen recordings, aligning the multi-source data based on timestamps. It then semantically segments the continuous data stream by combining user-generated operation events, forming behavioral fragments centered around these events. Within each behavioral fragment, the algorithm correlates the user's voice description within a certain time window before and after the operation with corresponding interface changes and operational behaviors. Natural language processing and image recognition technologies are used to determine whether the voice content, visual objects, and operational behaviors point to the same event and object. Based on this, the system automatically constructs a knowledge graph with objects, actions, conditions, and results as core elements, achieving a structured expression of the task flow. When a definite matching relationship cannot be formed between multimodal information, the system proactively generates targeted queries to supplement contextual information, thereby confirming event correlation and continuously optimizing understanding accuracy.

[0037] The above provides a detailed description of a system and method based on screen-level perception and human behavior learning provided in this application. The specific embodiments are described only to aid in understanding the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A system based on screen-level perception and human behavior learning, characterized in that, include: The system includes a multimodal data acquisition unit, a temporal segmentation unit, a semantic parsing and intent extraction unit, a visual object recognition and interface state parsing unit, a multimodal consistency judgment and event alignment unit, a knowledge graph node relationship construction unit, an error query matching unit, and a data storage unit. The multimodal data acquisition unit is used to collect multi-source data and the corresponding unified timestamps of the multi-source data during the task execution process of the currently running system. The time-series segmentation unit is used to segment continuous multi-source data streams using user-generated operation events as key anchor points. The semantic parsing and intent extraction unit is used to perform natural language processing on speech data from multi-source data to achieve intent recognition and keyword extraction. The visual object recognition and interface state parsing unit is used to perform image recognition and interface parsing on screenshots or video frames associated with operation events in multi-source data. The multimodal consistency judgment and event alignment unit integrates and analyzes the speech parsing results and visual recognition results to determine whether the multi-source data are consistent in the three dimensions of event object, action and result. Construct knowledge graph node relationship units. When the event objects, actions and results of multi-source data are consistent in three dimensions, the knowledge graph is automatically generated or updated and stored in the data storage unit. If a matching error occurs, and the data from multiple sources is inconsistent across the dimensions of event object, action, and result, the user is asked to provide supplementary explanations or confirm the relationship. Data storage unit, used to store multi-source data and knowledge graphs generated from multi-source data.

2. The system based on screen-level perception and human behavior learning according to claim 1, characterized in that, Multi-source data includes: voice data, screen image data, screen video data, and operation event data.

3. The system based on screen-level perception and human behavior learning according to claim 2, characterized in that, Detailed steps for segmenting continuous multi-source data streams in the time-series segmentation unit: When an operation event is detected, audio and visual data within a preset time window are captured forward and backward, respectively. Form behavioral segments centered around a single operational event; Each action paragraph includes: the expression of intent before the operation, the operation itself, and supplementary explanations or feedback after the operation.

4. The system based on screen-level perception and human behavior learning according to claim 3, characterized in that, The semantic parsing and intent extraction unit performs natural language processing on speech data from multi-source datasets to achieve intent recognition and keyword extraction. The detailed steps are as follows: An audio-to-text data conversion model is used to convert speech into text. Intent recognition and keyword extraction; Extraction of semantic elements of objects, actions, conditions, and results; Determine whether the voice message is a directive, an explanation, or a result feedback.

5. A system based on screen-level perception and human behavior learning according to claim 4, characterized in that, The audio data to text data conversion model includes a front-end processing module, an acoustic model, a language model, and a decoder; The front-end processing module is used to convert the raw audio signal into an acoustic feature sequence suitable for model processing. The acoustic model adopts an end-to-end architecture based on deep learning to establish a probabilistic mapping relationship between acoustic features and phonemes or characters. It directly learns the mapping from acoustic features to text sequences by connecting a temporal classification loss function. The language model provides syntactic and semantic constraints for the decoding process and corrects the output errors of the acoustic model by statistically analyzing the prior probabilities of word sequences. The decoder fuses the scores from the language model and the acoustic model.

6. A system based on screen-level perception and human behavior learning according to claim 5, characterized in that, The detailed steps for image recognition and interface parsing of screenshots or video frames associated with operation events in the visual object recognition and interface state parsing unit include: Identify interactive objects in the current interface, including buttons, input boxes, and menu items. Identify the visual characteristics of the object being manipulated and its state changes; Extract the differences between the interface before and after the operation to determine the operation result.

7. A system based on screen-level perception and human behavior learning according to claim 6, characterized in that, Detailed steps for determining whether multi-source data is consistent across the three dimensions of event object, action, and result in the multimodal consistency judgment and event alignment unit: Determine whether the object mentioned in the voice matches the object being manipulated on the interface; Determine whether the actions described in the voice are consistent with the actual actions performed. Determine whether the expected result mentioned in the voice message matches the changes in the interface.

8. A system based on screen-level perception and human behavior learning according to claim 7, characterized in that, Detailed steps for automatically generating or updating the knowledge graph in the node relationship unit of the knowledge graph construction: The interface element or business entity to be manipulated is treated as an object node. Treat the user's actions as action relationships; Treat the interface changes or business results triggered by the operation as result nodes; Record the context and constraints of the operation event.

9. A system based on screen-level perception and human behavior learning according to claim 8, characterized in that, Inconsistencies in multi-source data across the event object, action, and result dimensions within the match error query unit include: Insufficient matching between voice and the target being operated on; The voice description is ambiguous; The operation result was inconsistent with the expected voice prompt.

10. A method based on screen-level perception and human behavior learning, characterized in that, Including the following steps: Collect multi-source data and the corresponding unified timestamps of the multi-source data during the task execution process of the currently running system; Using user-generated operation events as key anchor points, continuous multi-source data streams are segmented; Natural language processing is performed on speech data from multi-source datasets to achieve intent recognition and keyword extraction; Perform image recognition and interface parsing on screenshots or video frames associated with operation events from multi-source data; By fusing and analyzing the speech analysis results with the visual recognition results, it can be determined whether the multi-source data are consistent in terms of event object, action and result. When the event objects, actions, and results of multi-source data are consistent in three dimensions, a knowledge graph is automatically generated or updated and stored in the data storage unit. If there are inconsistencies in the event objects, actions, and results across multiple data sources, request the user to provide supplementary explanations or confirm the relationships.

Images