A method and system for human-computer interaction control of an LCD screen

Abstract

This application provides a human-machine interaction control method and system for a liquid crystal display screen. The method includes: acquiring the original capacitance signals generated by all sensing areas on a touch panel to form a signal distribution reflecting touch events; identifying one or more touch areas from the signal distribution, and extracting the geometric features of each touch area and the spatial relationship between multiple touch areas to construct touch structure information; comparing the touch structure information and its changes over time with preset multiple gesture patterns, and determining the operator's intention gesture within a preset difference tolerance range; responding to the intention gesture, executing the corresponding industrial operation command, and providing a visual presentation of the operation result. This application can achieve accurate identification and response to complex industrial operations, effectively improving the efficiency and accuracy of human-machine interaction.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction control technology, and more specifically, to a human-computer interaction control method and system for a liquid crystal display screen. Background Technology

[0002] In precision industrial production such as semiconductor wafer manufacturing, traditional human-machine interfaces are often inefficient and prone to errors when processing complex parameters. Especially in special environments such as cleanrooms, complex and variable lighting conditions can cause glare and reflection on the display screen, further affecting the operator's accurate judgment and gesture recognition, making it difficult for advanced multi-touch and complex gesture recognition systems to fully realize their advantages. Summary of the Invention

[0003] This application discloses a human-computer interaction control method and system for a liquid crystal display screen, so as to improve the efficiency and accuracy of human-computer interaction.

[0004] The technical solution of this application is as follows: In a first aspect, this application discloses a human-computer interaction control method for a liquid crystal display screen, comprising: Acquire the raw capacitance signals generated by all sensing areas on the touch panel to form a signal distribution that reflects touch events; From the signal distribution, one or more touch areas are identified, and the geometric features of each touch area and the spatial relationship between multiple touch areas are extracted to construct touch structure information; The touch structure information and its changes over time are compared with multiple preset gesture patterns, and the operator's intention gesture is determined within a preset tolerance range of differences. In response to the intended gesture, the corresponding industrial operation instructions are executed, and a visual presentation of the operation results is provided.

[0005] Secondly, this application also discloses a liquid crystal display human-computer interaction control system, the system comprising: The signal acquisition module is used to acquire the raw capacitance signals generated by all sensing areas on the touch panel to form a signal distribution that reflects touch events. The structure recognition module is used to identify one or more touch areas from the signal distribution, and extract the geometric features of each touch area and the spatial relationship between multiple touch areas to construct touch structure information; The gesture comparison module is used to compare the touch structure information and its changes over time with a variety of preset gesture patterns, and determine the operator's intention gesture within a preset difference tolerance range. The instruction execution module is used to respond to the intentional gesture, execute the corresponding industrial operation instruction, and provide a visual presentation of the operation result.

[0006] Compared with the prior art, this application has at least the following beneficial effects: This application enables accurate identification and response to complex industrial operations, effectively improving the efficiency and accuracy of human-computer interaction. It overcomes the problems of low efficiency and error-proneness of traditional single-point touch systems when processing a large number of complex parameters, and solves the challenge of decreased recognition accuracy of new multi-point touch systems in actual industrial application environments. Attached Figure Description

[0007] Figure 1 This is a flowchart illustrating a human-computer interaction control method for a liquid crystal display screen provided in this application.

[0008] Figure 2 This application provides a schematic diagram of the structure of a human-computer interaction control system for a liquid crystal display screen. Detailed Implementation

[0009] The technical solutions in this application will now be clearly and completely described in conjunction with the accompanying drawings.

[0010] This application proposes a human-computer interaction control method for a liquid crystal display screen, such as... Figure 1 As shown, it includes the following steps: Acquire the raw capacitance signals generated by all sensing areas on the touch panel to form a signal distribution that reflects touch events; From the signal distribution, one or more touch areas are identified, and the geometric features of each touch area and the spatial relationship between multiple touch areas are extracted to construct touch structure information; The touch structure information and its changes over time are compared with multiple preset gesture patterns, and the operator's intention gesture is determined within the preset difference tolerance range. Responding to intentional gestures, it executes corresponding industrial operation commands and provides a visual representation of the operation results.

[0011] To better understand the above-mentioned technical solutions proposed in this application, some key terms involved will be explained first.

[0012] Raw capacitance signals refer to the capacitance values ​​directly collected from the sensing area on the touch panel before any processing. These signals can directly reflect the changes in the electric field when a finger or other touch object comes into contact with the panel.

[0013] Signal distribution refers to a two-dimensional or three-dimensional data set formed by spatially arranging the original capacitance signals of all sensing areas. It intuitively shows the position, size, and intensity of touch events on the panel.

[0014] The touch area refers to a continuous area in the signal distribution where the capacitive signal strength exceeds a preset threshold, which usually corresponds to the actual contact point of the operator's finger or tool.

[0015] Geometric features include the center coordinates, area, shape (such as circle, ellipse or irregular shape), and aspect ratio of the touch area. These features help to distinguish different touch actions.

[0016] Spatial relationships refer to the relative positions, distances, angles, etc., between multiple touch areas, such as the change in distance between two touch areas when pinching with two fingers, or the relative movement trajectories of each touch area when sliding with multiple fingers.

[0017] Touch structure information is a data structure formed by integrating the geometric features and spatial relationships of the touch area. It comprehensively describes the static and dynamic characteristics of the current touch event.

[0018] Gesture patterns refer to a predefined series of touch structure information and their sequence of changes over time, such as single-finger swipe, two-finger pinch, three-finger tap, etc. Each pattern corresponds to a specific operation intention.

[0019] The tolerance range for differences refers to the range of deviations allowed between the actual touch structure information and the preset gesture pattern when comparing gesture patterns, in order to accommodate individual differences and inaccuracies in operator gestures.

[0020] Intention gestures refer to the specific operations that the system determines the operator wants to perform based on the comparison results between touch structure information and gesture patterns.

[0021] Industrial operation instructions are commands executed in response to intentional gestures to control industrial equipment or systems, such as adjusting parameters, switching modes, starting or stopping equipment, etc.

[0022] Visual presentation refers to the system providing feedback to the operator on the operation results or system status through graphics, text, animation, etc. on the display screen.

[0023] The core of the human-machine interaction control method for LCD screens proposed in this application lies in achieving efficient and accurate industrial operation through refined touch signal processing and intelligent gesture intent recognition.

[0024] Specifically, acquiring the raw capacitance signals generated by all sensing areas on the touch panel to form a signal distribution reflecting touch events can be achieved in various ways. For example, the touch panel can be configured as a capacitive touch sensor array, where each sensor periodically collects the capacitance value of its sensing area and transmits these raw capacitance signals directly to the processing unit. Another approach is to integrate optical sensors into the touch panel, generating signals by detecting the obstruction or reflection of light by a finger or stylus. These signals are then converted into analog or digital capacitance signals to form a signal distribution. Furthermore, resistive touch technology can be used, acquiring signals by measuring the voltage change at the touch point on the resistive layer and converting it into an equivalent representation of a capacitance signal.

[0025] Identifying one or more touch regions from a signal distribution and extracting the geometric features of each touch region, as well as the spatial relationships between multiple touch regions, to construct touch structure information can be achieved in various ways. For example, the system can perform threshold segmentation on the acquired signal distribution, aggregating consecutive pixels or sensor areas with signal strength above a specific threshold into a single touch region. For each identified touch region, its centroid can be calculated as its geometric center, its shape (such as a rectangle, circle, or irregular polygon) can be determined by analyzing its boundary pixels, and its area can be calculated. When multiple touch regions exist, the system can calculate their Euclidean distance, relative angle, or degree of overlap to quantify their spatial relationships. These geometric features and spatial relationship data are then integrated to form touch structure information containing detailed information on all current touch events.

[0026] The system compares touch structure information and its changes over time with multiple preset gesture patterns, and determines the operator's intended gesture within a preset difference tolerance range. This can be achieved in various ways. For example, the system can pre-store a gesture pattern library containing touch structure information sequences of standard gestures corresponding to various industrial operations. When receiving a real-time touch structure information stream, the system can use pattern recognition techniques such as Dynamic Time Warping (DTW) or Hidden Markov Models (HMMs) to compare the real-time touch structure information sequence with each standard gesture in the gesture pattern library. During the comparison, the system calculates a similarity score between the real-time gesture and each standard gesture, considering a preset difference tolerance range. For example, if a real-time gesture has the highest similarity score to the zoom gesture pattern, and this score is within the preset difference tolerance range, the system will determine the operator's intended gesture as zoom.

[0027] Responding to an operator's gesture, executing the corresponding industrial operation command, and providing a visual representation of the operation result can be implemented in various ways. For example, once the system determines the operator's gesture, such as adjusting parameter A, it immediately looks up the industrial operation command associated with that gesture and sends it to the corresponding industrial control system. Simultaneously, to provide intuitive feedback to the operator, the human-machine interface updates the value of parameter A on the display screen or animates the parameter adjustment process, such as the movement of a slider or a real-time change in a value. If the gesture is a view switch, the system loads and displays a new interface layout, possibly accompanied by a smooth transition animation to enhance the user experience.

[0028] The human-machine interaction control method for LCD displays proposed in this application works by constructing a complete chain from low-level signal acquisition to high-level intent recognition, thereby addressing the complexity and uncertainty of human-machine interaction in industrial environments. First, by acquiring the raw capacitive signals generated by all sensing areas on the touch panel, the system can comprehensively and in real-time capture every touch action of the operator, forming a fine signal distribution. This step is the foundation for all subsequent processing, ensuring the integrity and accuracy of the information.

[0029] Next, from these signal distributions, the system intelligently identifies one or more touch areas and accurately extracts the geometric features (such as position, size, and shape) of each touch area, as well as the spatial relationships (such as distance and relative position) between multiple touch areas. These features and relationships are integrated to construct comprehensive touch structure information. This step overcomes the limitations of traditional touch systems that only focus on single-point touch or simple multi-point touch, enabling the handling of more complex and refined touch modes.

[0030] Subsequently, the system compares this dynamically changing touch structure information with multiple preset gesture patterns. By introducing a difference tolerance range, the system can accommodate subtle differences and inaccuracies in operator gestures, thereby more robustly determining the operator's intended gestures. This comparison process is one of the core innovations of this application, as it closely links physical touch actions with the operator's semantic intent, effectively solving the problem of inaccurate gesture recognition in complex industrial scenarios due to environmental interference or individual operator differences.

[0031] Finally, once the operator's intention gesture is confirmed, the system immediately responds and executes the corresponding industrial operation command, such as adjusting equipment parameters, switching operating modes, or starting a specific program. Simultaneously, the system provides a visual representation of the operation result, such as updating parameter values ​​on the display screen, showing operation progress, or providing feedback on whether the operation was successful. This feedback mechanism not only improves the transparency of the operation but also enhances the operator's trust in and sense of control over the system.

[0032] Through the aforementioned interconnected steps, this application effectively addresses the problems mentioned in the background art, such as the low efficiency, error-proneness, and difficulty in leveraging the advantages of advanced systems in complex industrial environments. For example, in cleanrooms, even with glare and reflections, this application, through meticulous processing of the original capacitive signals and comprehensive analysis of touch structure information, can accurately identify the operator's intentional gestures, avoiding misjudgments and operational errors. This method not only improves operational accuracy and efficiency but also significantly reduces the operator's cognitive load, thereby enhancing the overall stability of the production line and product yield.

[0033] This application goes beyond simple multi-touch recognition. It constructs sophisticated touch structure information and combines this information with its changes over time, intelligently comparing it with multiple preset gesture patterns to determine the operator's intended gesture within a preset tolerance range. This method offers the following significant advantages compared to the closest existing technology: First, at the signal acquisition and processing level, this application acquires the raw capacitance signals generated by all sensing areas on the touch panel to form a signal distribution reflecting touch events, which lays the foundation for subsequent accurate recognition. In contrast, existing technologies may only focus on the coordinates of the touch point while ignoring the complete signal distribution of the touch area, leading to information loss or inaccuracy in complex touch scenarios.

[0034] Secondly, in terms of touch area recognition and feature extraction, this application can identify one or more touch areas and extract the geometric features of each touch area as well as the spatial relationships between multiple touch areas to construct touch structure information. This enables the system to understand more complex gestures, such as multi-finger collaborative operation and palm touch, rather than just simple single-point or two-point touch. Existing technologies often struggle to effectively handle multi-finger or complex-shaped touches and are prone to misjudgment.

[0035] Furthermore, in terms of gesture intent recognition, this application compares the touch structure information and its changes over time with multiple preset gesture patterns, and determines the operator's intended gesture within a preset tolerance range of differences. This mechanism greatly enhances the robustness and accuracy of gesture recognition. By introducing a tolerance range of differences, the system can adapt to individual differences and inaccuracies in operator gestures, effectively avoiding recognition failures caused by minor deviations. In contrast, existing technologies often require high operational precision in gesture recognition; even slight deviations in the operator's gestures can lead to recognition failures or misjudgments.

[0036] Finally, this application can respond to intentional gestures, execute corresponding industrial operation commands, and provide a visual presentation of the operation results. This forms a closed-loop, efficient human-computer interaction process, ensuring the real-time nature of the operation and the intuitiveness of the feedback.

[0037] In summary, this application achieves more accurate and robust human-machine interaction control in complex industrial environments by introducing refined touch structure information, dynamic gesture comparison, and difference tolerance mechanisms. Its technological advancement lies in the deep understanding of operator intentions and the effective response to environmental interference, thereby bringing significant efficiency and precision improvements to precision industrial production.

[0038] In practical industrial applications, operator gestures may be somewhat ambiguous or uncertain due to factors such as the operating environment, personal habits, wearing of protective gloves, or equipment status. Relying solely on the comparison of physical gesture features may lead to insufficient accuracy in intent recognition, especially in complex or multi-tasking scenarios, which can easily result in misjudgments, thereby affecting the efficiency of human-computer interaction and the safety of industrial operations.

[0039] In response, this application further proposes a more refined and robust method for determining intention gestures. This method integrates physical confidence and semantic consistency by introducing aggregation processing of secondary touch areas, generation of physical gesture hypothesis sets, acquisition of task context information, and semantic consistency evaluation, in order to more accurately determine the operator's intention gestures.

[0040] The steps described above, which compare the touch structure information and its changes over time with multiple preset gesture patterns and determine the operator's intended gesture within a preset tolerance range, include: Secondary touch areas are identified from the signal distribution, and these secondary touch areas are aggregated to obtain virtual touch areas. Based on the virtual touch area, a set of physical gesture hypotheses is generated. The set of physical gesture hypotheses contains multiple fuzzy gesture hypotheses, and each fuzzy gesture hypothesis has a physical confidence score. Obtain current task context information; Based on task context information, semantic consistency is evaluated for each fuzzy gesture hypothesis in the physical gesture hypothesis set to obtain a semantic consistency score. By combining physical confidence scores and semantic consistency scores, the operator's intentional gestures can be determined.

[0041] Specifically, secondary touch areas refer to weaker or scattered signal regions sensed by the edges, slight contact, or adjacent areas of the finger or touch tool, in addition to the primary touch points, within the original capacitive signal distribution. These areas may not constitute independent primary touch points, but their presence and distribution can provide additional detailed information about the operator's gestures. Aggregation processing of secondary touch areas refers to merging or associating these scattered secondary touch areas based on their spatial proximity, signal strength, or temporal continuity to form one or more more representative virtual touch areas. For example, when an operator performs a pinch gesture, in addition to the two primary touch points, the edges of their fingers may generate multiple secondary touch areas. Aggregation processing can merge these secondary areas into the virtual touch area formed by the two primary touch points, thus more accurately reflecting the overall shape of the pinch gesture.

[0042] The virtual touch area can be understood as one or more logical touch areas formed by abstracting and modeling the original touch signals (including primary touch points and aggregated secondary touch areas). These virtual touch areas can more accurately reflect the operator's actual touch intentions, especially when the physical touch signals are unclear or noisy. Based on these virtual touch areas, the system can generate a set of physical gesture hypotheses. The set of physical gesture hypotheses contains multiple fuzzy gesture hypotheses, each representing a possible operator gesture, such as pinch, swipe, or click, but its specific parameters (such as direction, speed, and size) may have a certain degree of ambiguity. Each fuzzy gesture hypothesis has a physical confidence score, which quantifies the degree of matching between the hypothesis and the actual physical touch signals (virtual touch areas). A higher score indicates a higher degree of matching, and correspondingly, a higher physical confidence score.

[0043] Current task context information specifically refers to various contextual data related to the current human-computer interaction task, such as the type of currently displayed interface elements, the industrial operation instructions being executed, the operator's role and permissions, equipment operating status, historical operation sequences, and expected operation procedures. This information can be obtained through internal system states, sensor data, or user configuration files. Based on the task context information, a semantic consistency evaluation is performed on each fuzzy gesture hypothesis in the physical gesture hypothesis set. The purpose is to determine whether the physical gesture hypothesis has reasonable semantics in the current task context. For example, in a valve control interface, a sliding gesture hypothesis might be evaluated as adjusting the valve opening or switching the valve state. Its semantic consistency score will be calculated based on the current valve state and the expected operation. The higher the semantic consistency score, the more reasonable the gesture hypothesis is in the current context.

[0044] Ultimately, by combining physical confidence scores and semantic consistency scores, the system can determine the operator's intended gesture. This combination can employ various methods such as weighted averaging, logical judgment, or machine learning models. The aim is to comprehensively consider the physical morphological characteristics of the gesture and its semantic rationality in the current task context, thereby making a more accurate judgment that better reflects the operator's true intention.

[0045] This application, by introducing the recognition and aggregation of secondary touch areas, can more comprehensively capture the operator's touch intentions, especially in multi-finger touch or complex gesture scenarios. By generating virtual touch areas, it can effectively handle the ambiguity of touch signals and avoid misjudgments caused by insufficient recognition of a single touch point. Based on this, a set of physical gesture hypotheses is constructed and assigned a physical confidence score, quantifying the probability of gestures purely based on physical features, providing a foundation for subsequent intention judgment. Furthermore, by acquiring current task context information and performing semantic consistency evaluation on the physical gesture hypotheses, a contextual understanding of the operator's intentions is introduced. This mechanism enables the system not only to recognize the physical form of gestures but also to understand their actual meaning in a specific operational context, thereby effectively avoiding misjudgments caused by gesture ambiguity or lack of context. Finally, by combining the physical confidence score and the semantic consistency score, the system can comprehensively consider the physical performance of the gesture and its logical rationality in the current operational environment, thus making more accurate and reliable intention gesture judgments.

[0046] Through the above technical solution, this application can significantly improve the accuracy and robustness of operator intention gesture recognition in the human-machine interaction control method of LCD screens. Especially in industrial environments, facing complex and ever-changing operating scenarios and potentially ambiguous gestures, this solution effectively reduces the misrecognition rate and improves the reliability of human-machine interaction by comprehensively considering the physical gesture characteristics and the semantic information of the task context. As a result, operators can execute industrial operation instructions more efficiently and safely, reducing potential risks and repetitive operations caused by gesture misjudgment, thereby optimizing the overall user experience and system performance. In addition, by introducing the concepts of secondary touch areas and virtual touch areas, this application can better adapt to the touch habits of different operators and the influence of external factors such as protective gloves, further enhancing the adaptability and usability of the system.

[0047] In practical industrial applications, human-machine interface elements and their associated control command categories may be frequently updated or changed. If the semantic evaluation rules are static or cannot be adaptively adjusted, it may lead to a deviation in the system's understanding of the operator's intentions, reducing the efficiency and accuracy of human-machine interaction.

[0048] To this end, the steps described above, which evaluate the semantic consistency of each fuzzy gesture hypothesis in the physical gesture hypothesis set based on task context information to obtain a semantic consistency score, include: Get the type of the current interactive interface element and the category of the control command associated with the interface element; For newly introduced interface elements or interface elements whose associated control instruction categories have changed, semantic evaluation rules are generated or adjusted based on the functional description and expected operation mode of the newly introduced interface elements or interface elements whose associated control instruction categories have changed, combined with the interaction behaviors related to the newly introduced interface elements or interface elements whose associated control instruction categories have changed in the operator's historical operation log. Monitor the success rate and frequency of repeated operations on interface elements; Based on the success rate of the operation and the frequency of repeated operations, the weights or logic of the semantic evaluation rules are adjusted to obtain a semantic consistency score.

[0049] Specifically, it retrieves the current interactive interface element type and the associated control command category. The interactive interface element type can refer to buttons, sliders, switches, text input boxes, etc., while the control command category refers to the specific industrial operation triggered when these elements are operated, such as equipment start / stop, parameter adjustment, mode switching, etc.

[0050] Furthermore, for newly introduced interface elements or those whose associated control command categories have changed, semantic evaluation rules are generated or adjusted based on the functional description and expected operation mode of the newly introduced interface elements or those whose associated control command categories have changed, combined with the operator's historical operation logs showing interaction behaviors related to the newly introduced interface elements or those whose associated control command categories have changed. Specifically, when the system detects a new interface element or a change in the function of an existing interface element, it analyzes its expected operation method and purpose, and dynamically constructs or updates the rule set for evaluating gesture semantics by referring to the operator's past operating habits in similar situations. For example, for a newly introduced emergency stop button, the system will generate corresponding semantic evaluation rules based on its high priority and expected single-confirmation operation mode, combined with the operator's gesture characteristics in other emergency operations.

[0051] Subsequently, the success rate and frequency of repeated operations of interface elements were monitored. The success rate refers to the proportion of times an operator successfully triggers the expected instruction through a specific gesture, while the frequency of repeated operations reflects the number of times an operator repeatedly performs the same or similar gestures within a short period when attempting to complete an operation. These metrics directly reflect the effectiveness of the current semantic evaluation rules and the operator's understanding of the interface elements.

[0052] Finally, based on the success rate and frequency of repeated operations, the weights or logic of the semantic evaluation rules are adjusted to obtain a semantic consistency score. For example, if the success rate of an operation on a certain interface element is low or the frequency of repeated operations is high, it indicates that the current semantic evaluation rule may not be accurate enough or does not match the operator's actual intention. In this case, the system will reduce the weight of the rule or modify its logic accordingly to improve the accuracy of subsequent evaluations.

[0053] This application addresses the problem of insufficient adaptability of traditional semantic evaluation rules in the face of dynamically changing industrial human-machine interaction environments by introducing a dynamic monitoring and feedback mechanism for interface element types, control command categories, historical operation logs, operation success rates, and repetitive operation frequencies. Specifically, by acquiring current interface element information, the system can understand the context of the operation; by analyzing the functional descriptions, expected operation modes, and historical interaction behaviors of newly introduced or changed interface elements, the system can generate or adjust more accurate initial semantic evaluation rules for these dynamic elements. Furthermore, by continuously monitoring operation success rates and repetitive operation frequencies, the system can evaluate the effectiveness of the current semantic evaluation rules in real time. When a deviation is found between the rules and the actual operator's intent (e.g., low success rate or high repetitive operation frequency), the system can adaptively adjust the weights or logic of the rules, thereby making the semantic consistency evaluation results more accurately reflect the operator's true intent and effectively avoiding misjudgments caused by rigid rules.

[0054] Through the above technical solution, this application can achieve more accurate recognition of operator intentions and gestures, especially when industrial control interfaces are frequently updated or operators are learning new functions. This solution significantly improves the system's adaptability to complex and changing operational scenarios by dynamically generating and adjusting semantic evaluation rules, reducing the risk of misoperation and enhancing the efficiency of human-computer interaction and user experience. Furthermore, by combining the operator's historical operation logs and real-time operation feedback, the system can better understand the operator's learning curve and habits, thereby providing a more personalized and intelligent interactive experience and effectively reducing repeated attempts and cognitive load caused by rule mismatches.

[0055] In some embodiments, the steps of monitoring the success rate and repetitive operation frequency of interface elements described above may include: Obtain execution status feedback for industrial operation instructions; Monitor the state changes of interface elements related to commands on the human-computer interaction interface; Determine whether the execution status feedback of the instruction and the state changes of the interface elements meet expectations; When the execution status feedback of the instruction or the status change of the interface element does not meet expectations, identify repeated operations performed by the operator on the same or related interface elements within a preset time window. Mark repeated operations as repeated attempts; Based on the execution status feedback of instructions, changes in the state of interface elements, and the marking of repeated attempts, determine the success rate of operations and the frequency of repeated operations for interface elements.

[0056] Specifically, obtaining execution status feedback for industrial operation instructions refers to the system receiving information from the industrial control layer regarding the execution results of specific operation instructions, such as whether the instruction was successfully executed, failed, timed out, or partially completed. Its purpose is to provide direct evidence for evaluating the effectiveness of the operation. Monitoring the status changes of interface elements related to the instruction on the human-machine interface can be understood as the system continuously observing visual or auditory feedback changes of interface elements (such as buttons, indicator lights, and numerical display areas) associated with specific industrial operation instructions. For example, when the operator clicks the start button, the system should detect that the start button becomes grayed out and unavailable, or the corresponding equipment status indicator light changes from red to green. Its purpose is to verify the execution of the instruction at the user interface level.

[0057] Determining whether the execution status feedback and interface element state changes of an instruction meet expectations is accomplished by comparing the actual received execution status feedback and monitored interface element state changes with a preset successful execution pattern for a specific instruction. For example, if the expected result of a start instruction is successful execution and the start indicator light illuminates, the system checks whether both conditions are met simultaneously. The purpose is to identify whether the operation was completed as expected. Furthermore, when the execution status feedback or interface element state changes do not meet expectations, the system activates a monitoring mechanism that continuously tracks whether the operator performs repeated operations on the same interface element (such as clicking the start button again) or other related interface elements (such as clicking the reset button and then clicking the start button again) within a preset time window of, for example, 5 or 10 seconds. The purpose is to capture retrying attempts made by the operator due to unsuccessful operations. Thus, identified repeated operations are internally marked as repeated attempts, which helps distinguish between a one-time successful operation and success or failure after multiple attempts. The purpose is to provide accurate raw data for subsequent success rate and frequency calculations.

[0058] Based on the execution status feedback of instructions, changes in the state of interface elements, and the marking of repeated attempts, the success rate and frequency of repeated operations for interface elements are determined. The success rate can be calculated by statistically analyzing the ratio of the number of successfully executed instructions to the total number of operation attempts (including repeated attempts) for a specific interface element within a certain period. The frequency of repeated operations can be quantified by statistically analyzing the average or total number of repeated attempts made by the operator for a specific interface element within a specific time period. The purpose is to quantify the efficiency and effectiveness of operator interaction with interface elements.

[0059] This application comprehensively evaluates the actual effectiveness of operations by acquiring execution status feedback of industrial operation instructions and monitoring state changes of interface elements related to the instructions on the human-machine interface. When the operation result is found to be unsatisfactory, the system can intelligently identify repeated operations performed by the operator on the same or related interface elements within a preset time window and mark them as repeated attempts. It is precisely because of this accurate identification and marking of repeated attempts that the system can more accurately calculate the success rate and frequency of repeated operations of interface elements. This mechanism can distinguish between a one-time successful operation and the success or failure after multiple attempts, thus providing more refined and reliable data support for subsequent adjustments to semantic evaluation rules.

[0060] The aforementioned technical solutions enable a more precise quantification of the efficiency and effectiveness of operator interactions with specific interface elements. This meticulous monitoring mechanism not only identifies whether an operation is successful but also captures the operator's repeated attempts when encountering difficulties, thus providing richer and more insightful data for adjusting evaluation rules. Consequently, it more accurately reflects the operator's learning curve and operating habits in actual operation, making the adjustment of semantic evaluation rules more closely aligned with real-world operating scenarios and improving the adaptability and intelligence level of the human-computer interaction system.

[0061] In practical applications, for newly introduced interface elements or interface elements that lack sufficient historical operation data, directly adjusting rules based on limited success rates and repetition frequencies may lead to evaluation bias or inaccurate adjustments due to insufficient accumulation of operation data.

[0062] In response, this application further proposes a scheme for dynamically adjusting semantic evaluation rules by monitoring the differences between the operator's first interaction behavior and repeated operations for interface elements that lack sufficient historical operation data.

[0063] The steps described above, which adjust the weights or logic of the semantic evaluation rules based on the success rate and frequency of repeated operations to obtain a semantic consistency score, include: Identify UI elements that lack sufficient historical operation data; Based on the functional descriptions and expected operation modes of interface elements that lack sufficient historical operation data, select an initial semantic evaluation rule from the preset general operation logic templates that matches the interface elements that lack sufficient historical operation data. Monitor operator interaction behavior when using interface elements for the first time without sufficient historical operation data. Interaction behavior includes gesture trajectory, touch pressure, and operation duration. The analysis examines the differences between repetitive operations performed by operators on interface elements lacking sufficient historical operation data. These differences include the similarity of gesture trajectories and the range of variation in touch pressure. The initial semantic evaluation rules are adjusted based on the convergence observed during repeated operations. Experienced operators or engineers are allowed to confirm or fine-tune the adjusted semantic evaluation rules.

[0064] Specifically, identifying UI elements lacking sufficient historical operation data refers to UI elements for which the system has not recorded a sufficient number of valid operation logs within a certain time window, or entirely new UI elements appearing for the first time after a system update. The purpose is to differentiate between UI elements with sufficient historical data and those lacking it, in order to adopt more suitable rule adjustment strategies. Specifically, based on the functional description and expected operation mode of the UI element lacking sufficient historical operation data, an initial semantic evaluation rule matching the UI element lacking sufficient historical operation data is selected from a preset general operation logic template. The functional description can be understood as the design purpose and expected response of the UI element, such as whether a button is used for confirmation or cancellation, or whether a slider is used to adjust brightness or volume. Expected operation mode refers to the types of gestures and interaction habits that users typically use when interacting with the UI element, such as clicking, swiping, and pinching. The general operation logic template can be a series of predefined standard semantic evaluation rule sets applicable to different types of UI elements, such as corresponding initial evaluation rules for different control types like buttons, switches, sliders, and text input boxes. The purpose is to provide a reasonable starting evaluation benchmark for new UI elements, avoiding learning from scratch.

[0065] This study monitors operator interactions when they first use interface elements lacking sufficient historical data. Interaction behaviors include gesture trajectories, touch pressure, and operation duration. Gesture trajectories are the paths of the operator's fingers or stylus on the touch panel; touch pressure is the magnitude and spatial distribution of pressure applied by the operator during touch; and operation duration is the length of time from the start to the end of the touch. This data is used to capture the raw, unlearned interaction characteristics of operators when they first attempt to interact with new interface elements. Furthermore, the study analyzes the differences between repeated operations performed by operators on interface elements lacking sufficient historical data. These differences include the similarity of gesture trajectories and the range of touch pressure variation. The similarity of gesture trajectories quantifies the closeness of gesture paths across different operations, for example, through calculation using the Dynamic Time Warping (DTW) algorithm. The range of touch pressure variation reflects the stability and consistency of pressure applied by the operator in repeated operations. Its purpose is to assess an operator's mastery of the expected operating patterns of interface elements by observing the learning curve and proficiency of the operator in repetitive operations.

[0066] The initial semantic evaluation rules are adjusted based on the convergence observed in repeated operations. Convergence refers to the trend of an operator's repetitive operational behaviors (such as gesture trajectories and touch pressure) gradually stabilizing and becoming consistent with the increase in the number of operations. Good convergence indicates that the operator has mastered the expected operation of the interface element, and the semantic evaluation rules associated with this operation mode can be strengthened. Conversely, poor convergence may mean that the operator is confused about the operation of the interface element, in which case the rules may need to be adjusted to accommodate a wider range of operation variations, or the system may need to be prompted for further guidance. Allowing experienced operators or engineers to confirm or fine-tune the adjusted semantic evaluation rules means that, to ensure the accuracy and security of rule adjustments, the system can provide an interface that allows experienced operators or engineers with professional knowledge to manually review, confirm, or fine-tune the automatically adjusted semantic evaluation rules. The purpose is to combine human experience and domain knowledge to further improve the accuracy and robustness of the rules, especially in critical industrial control scenarios.

[0067] This application effectively addresses the problem of inaccurate rule adjustments when data is insufficient by introducing a special processing mechanism for interface elements lacking sufficient historical operation data. Specifically, firstly, by identifying these special interface elements, the system avoids blindly applying evaluation logic based on large amounts of historical data. Secondly, by selecting initial semantic evaluation rules from a general operation logic template, a reasonable starting point is provided for new interface elements, avoiding inefficient learning from scratch. Furthermore, by monitoring the operator's interaction behavior during first use, the system can capture the operator's original intentions and habits. Based on this, by analyzing the similarity of gesture trajectories and the range of changes in touch pressure during repeated operations, the system can quantify the operator's learning adaptation process and proficiency, thereby judging their mastery of the expected operation patterns of interface elements. It is precisely this dynamic perception of the operator's learning process that allows the system to make targeted adjustments to the initial semantic evaluation rules based on the convergence exhibited by repeated operations, making them more aligned with actual operating habits. Finally, by allowing experienced operators or engineers to confirm or fine-tune the rules, the accuracy and security of the rules are further guaranteed, ensuring precise human-computer interaction control even in data-sparse scenarios.

[0068] Through the above technical solution, this application effectively addresses the problem of insufficient accuracy in adjusting semantic evaluation rules for newly introduced LCD screen interface elements or those lacking sufficient historical operational data in industrial control scenarios. Specifically, this solution dynamically monitors the operator's initial interaction behavior and the convergence of repeated operations, enabling it to learn and adapt to the operator's actual operating habits for new interface elements more quickly and accurately. This allows for the generation or adjustment of more reasonable and effective semantic evaluation rules even with insufficient data accumulation. Consequently, it significantly improves the adaptability and robustness of the human-machine interaction control system when facing interface updates or new function launches, reduces the operational error rate caused by inaccurate rules, shortens the debugging time before system deployment, and enhances the overall system intelligence level and user experience.

[0069] In real-world industrial applications, operators often wear protective gloves, and the surface of the LCD screen may have a special coating. These factors can cause nonlinear changes and local distortions in the original touch pressure signal, thus affecting the accuracy of the touch pressure distribution data and consequently the accurate assessment of operator interaction behavior. This results in the generation or adjustment of the initial semantic evaluation rules being inaccurate.

[0070] In this regard, this application further proposes that the steps for monitoring the touch pressure of an operator when using an interface element for the first time, where there is a lack of sufficient historical operation data, include: The original capacitance signals generated by all sensing areas on the touch panel are acquired, and the original capacitance signals are spatially sampled to obtain the initial touch pressure distribution data. Obtain information about the material of the protective gloves currently worn by the operator; Based on the material information of the protective gloves and the pre-stored dielectric response characteristics of each area of ​​the touch panel under different protective glove materials, the initial touch pressure distribution data is regionally corrected to compensate for the nonlinear changes and local distortions caused by the protective gloves and display coatings. Multiple local pressure peak regions in the corrected touch pressure distribution are identified, and boundary analysis is performed on each local pressure peak region to determine its shape and size; Based on the shape, size, and corrected touch pressure distribution data of each local pressure peak region, the pressure centroid and total pressure of each local pressure peak region are calculated, thereby quantifying the nonlinear characteristics and distortion of the touch pressure distribution to ensure the validity of the touch pressure distribution data.

[0071] Specifically, acquiring the raw capacitance signals of all sensing areas on the touch panel refers to real-time acquisition of the capacitance value of each sensing unit through the touch panel's sensor array. These capacitance values ​​reflect the impact of touch events on the electric field. Subsequently, spatial sampling of the raw capacitance signals can be understood as mapping the discrete capacitance signal data onto a two-dimensional spatial grid, forming a preliminary, unprocessed touch pressure distribution map, i.e., initial touch pressure distribution data. Its purpose is to initially capture the spatial characteristics of touch events.

[0072] Obtaining information about the material of the protective gloves currently worn by the operator can be achieved in various ways, such as reading glove information via RFID tags, manual input by the operator, or automatic identification of the glove type using image recognition technology. The material information for the protective gloves can include, but is not limited to, cotton, rubber, leather, or synthetic materials.

[0073] Based on the material information of the protective gloves and the pre-stored dielectric response characteristics of different areas of the touch panel under different protective glove materials, the initial touch pressure distribution data is regionally corrected. The purpose is to eliminate or reduce the interference of the protective gloves and display coating on the touch signal. The pre-stored dielectric response characteristics can be a lookup table or a correction model, obtained through experiments or simulations, describing the influence of different glove materials and display coatings on the capacitive signal in different areas. Regional correction can employ weighted averaging, interpolation, or machine learning-based adaptive correction algorithms to compensate for the nonlinear variations and local distortions caused by the protective gloves and display coating.

[0074] Furthermore, identifying multiple local pressure peak regions in the corrected touch pressure distribution refers to using peak detection algorithms to find points or regions in the corrected pressure distribution map where the pressure value is significantly higher than the surrounding area. Boundary analysis is performed on each local pressure peak region, using techniques such as image segmentation and edge detection to accurately determine the shape (e.g., circular, elliptical, or irregular shape) and size (e.g., area or aspect ratio) of each touch point.

[0075] Therefore, based on the shape and size of each local pressure peak region and the corrected touch pressure distribution data, the pressure centroid and total pressure of each local pressure peak region are calculated. The pressure centroid represents the central location of the touch force, while the total pressure reflects the intensity of the touch. By quantifying the nonlinear characteristics and distortion of the touch pressure distribution, such as by calculating the skewness, kurtosis, or KL divergence of the pressure distribution with respect to the ideal distribution, the validity of the touch pressure data can be ensured, providing a more reliable input for subsequent gesture recognition and intent determination.

[0076] This application's solution first acquires the original capacitance signal and performs spatial sampling to form initial touch pressure distribution data, laying the foundation for subsequent processing. Given the significant impact of protective gloves and display coatings on touch signals in industrial environments, this solution further acquires information about the material of the protective gloves worn by the operator and, combined with pre-stored dielectric response characteristics, performs regional correction on the initial touch pressure distribution data. It is precisely this targeted correction that effectively compensates for the nonlinear variations and local distortions caused by protective gloves and display coatings, thereby obtaining pressure distribution data that more closely approximates real-world touch conditions. Based on this, by identifying local pressure peak regions and performing boundary analysis, as well as calculating the pressure centroid and total pressure, this solution can quantify the nonlinear characteristics and distortion degree of the touch pressure distribution, ensuring the validity of the acquired touch pressure distribution data. This effective data is crucial for accurately evaluating operator interaction behavior, especially gesture trajectories and touch pressure, thus providing a solid foundation for subsequent generation or adjustment of semantic evaluation rules.

[0077] Through the above technical solution, this application effectively solves the problem of distorted and inaccurate touch pressure distribution data in industrial environments due to factors such as operators wearing protective gloves and display screen coatings. By introducing the acquisition of protective glove material information and regional correction based on dielectric response characteristics, this application significantly improves the accuracy and reliability of touch pressure distribution data. Therefore, when monitoring the interaction behavior of operators using interface elements for the first time when there is insufficient historical operation data, more realistic and effective touch pressure data can be obtained, providing high-quality input for subsequent generation or adjustment of semantic evaluation rules. This not only improves the adaptability and robustness of human-computer interaction control methods but also enables the system to learn and adapt more quickly and accurately when facing new interface elements or new operators, improving the overall system intelligence level and user experience.

[0078] In practical applications, repetitive operations by operators can be influenced by various factors, such as environmental noise, equipment drift, or the operator's own learning and adaptation process. Failure to effectively distinguish these differences from different sources may lead to misjudgments of operator intent, thereby affecting the accuracy and efficiency of semantic evaluation rule adjustments. To address this, this application proposes a more refined analysis method to deeply analyze the differences between repetitive operations performed by operators on interface elements lacking sufficient historical operation data, thereby more accurately distinguishing deviations caused by different factors.

[0079] The above analysis examines the differences between repetitive operations performed by operators on interface elements lacking sufficient historical operation data. These differences include the similarity of gesture trajectories and the range of variations in touch pressure. The steps involved are as follows: Obtain the gesture trajectory data for each repeated operation; Preprocess the gesture trajectory data to remove instantaneous noise and jitter; The preprocessed gesture trajectory is segmented; Calculate the rate of change of direction and curvature of each segment; Obtain the touch pressure distribution data for each repeated operation; Regional corrections are performed on the touch pressure distribution data to compensate for nonlinear drift caused by wear on protective gloves and localized contamination of the display screen; Identify multiple local pressure peak regions in the corrected pressure distribution; Calculate the centroid of pressure and total pressure for each local pressure peak region; Construct feature vectors for the directional change rate and curvature of the gesture trajectory, as well as the pressure centroid and total pressure of the touch pressure distribution; Calculate the distance between feature vectors of different repeated operations to quantify the similarity of gesture trajectories; Calculate the total pressure fluctuation amplitude in the local pressure peak region during each repeated operation; Analyze the trend of total pressure fluctuation amplitude over time to quantify the range of touch pressure variation; Get the duration of an operator's actions in a repetitive operation; Compare the similarity of gesture trajectories, the range of changes in touch pressure, and the changes in operation duration; Based on the comparison results, the differences between deviations caused by environmental or equipment factors and differences in the operator's learning and adaptation process can be distinguished.

[0080] Specifically, acquiring gesture trajectory data for each repeated operation refers to the system recording the sequence of continuous position points of the operator's fingers or stylus on the screen during each repeated operation on the touch panel. Preprocessing the gesture trajectory data to remove instantaneous noise and jitter can be achieved using algorithms such as smoothing filtering, median filtering, or Kalman filtering to eliminate unexpected data fluctuations caused by sensor errors or slight operator tremors. Segmenting the preprocessed gesture trajectory involves dividing a complete trajectory into several sub-segments based on significant changes in its direction, velocity, or curvature to facilitate the extraction of local features. Calculating the rate of change of direction and curvature of each segment involves using mathematical methods, such as finite difference or curve fitting, to quantify the degree of turning and curvature of each trajectory segment. These parameters reflect the fine characteristics of the gesture.

[0081] The process of acquiring touch pressure distribution data for each repeated operation involves the system recording the capacitance signal strength of each sensing area on the touch panel during each operation, forming a two-dimensional pressure distribution map. Regional correction of the touch pressure distribution data compensates for nonlinear drift caused by wear on protective gloves and localized contamination of the display screen. This can be achieved by adjusting the pressure values ​​of specific areas based on a preset compensation model or real-time calibration data to eliminate interference from external factors on pressure sensing. Identifying multiple local pressure peak areas in the corrected pressure distribution involves using a peak detection algorithm to find discrete areas with relatively high pressure values ​​in the pressure distribution map. These areas typically correspond to the actual contact points of the operator's finger or stylus. Calculating the pressure centroid and total pressure of each local pressure peak area involves calculating the average position (centroid) of each identified peak area in two-dimensional space and the sum of all pressure values ​​within that area. These parameters reflect the precise position and force of the touch.

[0082] Constructing feature vectors for the directional change rate and curvature of gesture trajectories, as well as the pressure centroid and total pressure of touch pressure distribution, involves combining the extracted gesture and pressure features into a multi-dimensional vector for subsequent quantitative comparison. Calculating the distance between feature vectors of different repetitive operations quantifies the similarity of gesture trajectories. Algorithms such as Euclidean distance, cosine similarity, or Dynamic Time Warping (DTW) can be used to assess the closeness of gesture features between different operations. Calculating the total pressure fluctuation amplitude of the local pressure peak region in each repetitive operation refers to the difference between the maximum and minimum touch pressure values ​​recorded during a single operation. Analyzing the trend of the total pressure fluctuation amplitude over time quantifies the range of touch pressure variation. Statistical analysis methods, such as calculating standard deviation, variance, or trend lines, can be used to describe the stability and regularity of pressure changes.

[0083] Furthermore, obtaining the duration of an operator's repetitive actions refers to recording the time interval from the start to the end of the action. Comparing the similarity of gesture trajectories, the range of changes in touch pressure, and the changes in action duration involves comprehensively analyzing these multi-dimensional data to find the correlations and patterns of change among them. Based on the comparison results, distinguishing between deviations caused by environmental or equipment factors and differences in the operator's learning and adaptation process involves classifying these changes by setting thresholds, pattern recognition, or machine learning models. For example, continuous and regular trajectory convergence may indicate learning and adaptation, while random and irregular fluctuations may point to environmental interference.

[0084] This application addresses the limitations of the aforementioned basic solutions in distinguishing the sources of differences in repetitive operations by introducing multi-dimensional and refined data analysis steps. Specifically, by preprocessing and segmenting gesture trajectory data and calculating the rate of change of direction and curvature, the dynamic characteristics of gestures can be captured more accurately, rather than relying solely on the original trajectory. Simultaneously, regional correction is performed on the touch pressure distribution data, and the centroid and total pressure of local pressure peak areas are extracted, effectively compensating for external interference and enabling the pressure data to more realistically reflect the operator's touch force and position. By constructing feature vectors from these refined gesture and pressure features and calculating their distances, the similarity of gesture trajectories between different repetitive operations can be quantified, thereby identifying the stability or trend of operator gestures. Furthermore, by analyzing the changes in the amplitude of total pressure fluctuations and the duration of operations, the description of operational behavior is further enriched. Finally, by comprehensively comparing these multi-dimensional data, the system can more effectively identify which changes are due to the operator's adaptive adjustments when learning new interface elements (manifested as the gradual convergence of gestures and pressure), and which are due to random or irregular deviations caused by external factors such as environmental noise, equipment drift, or wear and tear on protective gloves. This ability to differentiate is crucial for accurately assessing an operator's learning curve and adjusting semantic evaluation rules.

[0085] Through the above technical solution, this application can not only quantify the similarity of gesture trajectories and the range of touch pressure variations in detail, but also effectively reduce the impact of external interference on data analysis by introducing a multi-dimensional feature extraction and correction mechanism. More importantly, this application can intelligently distinguish between the internal changes in the operator's learning and adaptation process and external deviations caused by environmental or equipment factors based on comprehensive comparison results. Therefore, it can avoid misjudging the operator's intentions due to external interference, making the adjustment of semantic evaluation rules more accurate and efficient, thereby significantly improving the adaptability and user experience of the human-computer interaction system, especially when the operator first encounters new interface elements or operates in a complex industrial environment.

[0086] In actual industrial operating environments, subjective factors such as the operator's physiological state, cognitive load, and emotional state may significantly affect gesture trajectory, touch pressure, and operation duration, leading to misjudgments by the system when distinguishing these deviations.

[0087] In response, this application further proposes the following steps for comparing the similarity of gesture trajectories, the range of changes in touch pressure, and the changes in operation duration: Acquire the operator's physiological status information, including heart rate, skin conductivity, and eye movement data; Acquire operator cognitive load information, including task completion time, error rate, and complexity of the operation sequence; Acquire the operator's emotional state information, which includes facial expression recognition results and voice tone analysis results; Based on physiological state information, cognitive load information, and emotional state information, generate subjective state assessment results for operators; Based on the operator's subjective state assessment results, corrections are made to the similarity of gesture trajectories, the range of changes in touch pressure, and the changes in operation duration in order to eliminate deviations caused by operator subjective factors. By comparing the similarity of the corrected gesture trajectories, the range of changes in the corrected touch pressure, and the changes in the duration of the corrected operation, the deviations caused by environmental or equipment factors can be distinguished from the differences in the operator's learning and adaptation process.

[0088] Specifically, acquiring an operator's physiological state information refers to the real-time collection of the operator's heart rate, skin conductivity, and eye movement data using non-invasive sensors (such as wearable devices). Heart rate can reflect the operator's physiological arousal level, skin conductivity is related to the degree of emotional excitement, while eye movement data can reveal the operator's focus of attention and cognitive processes.

[0089] Obtaining operator cognitive load information can be understood as quantifying their cognitive burden by analyzing their behavioral data during task execution. Task completion time, error rate, and the complexity of the operation sequence are key indicators for measuring cognitive load. For example, excessively long task completion time, increased error rate, or hesitation in complex operation sequences may all indicate that the operator is experiencing a high cognitive load.

[0090] The acquisition of operators' emotional state information is achieved through facial expression recognition technology and voice tone analysis technology. Facial expression recognition can capture operators' basic emotions such as joy, anger, sorrow, and happiness, while voice tone analysis can extract emotional features from the operator's speech, such as speech rate, pitch, and volume changes.

[0091] Furthermore, based on physiological state information, cognitive load information, and emotional state information, subjective assessment results of the operator's state can be generated. This typically involves fusing and analyzing the aforementioned multimodal data, for example, by using machine learning models to perform pattern recognition on the data, thereby deriving a comprehensive assessment that reflects the operator's current level of fatigue, stress, focus, or emotional stability.

[0092] Therefore, based on the operator's subjective state assessment results, corrections can be made for the similarity of gesture trajectories, the range of touch pressure variations, and the duration of operations. For example, when the assessment results indicate that the operator is under high cognitive load or emotional stress, the system can assign lower weights to the similarity of gesture trajectories, the range of touch pressure variations, and the duration of operations collected concurrently, or adjust them through a preset compensation model to eliminate biases caused by these subjective factors. The purpose of this correction is to distinguish between changes in the operator's internal state and external environmental or equipment factors, as well as the operator's own learning and adaptation process.

[0093] Ultimately, by comparing the similarity of the corrected gesture trajectories, the range of changes in touch pressure after calibration, and the changes in the duration of the corrected operations, the system can more accurately distinguish between deviations caused by environmental or equipment factors and differences in the operator's learning and adaptation process. By removing the interference of the operator's subjective factors, the system can obtain purer and more objective interactive behavior data, thereby making more accurate judgments.

[0094] Through the above technical solution, this application can more accurately understand the operator's true intentions and learning process, avoiding misjudgments caused by fluctuations in the operator's subjective state. This not only improves the robustness and accuracy of the human-computer interaction control method, but also makes the adjustment of semantic evaluation rules more scientific and reasonable, thereby significantly improving the efficiency and safety of industrial operations. In addition, real-time monitoring and feedback of the operator's subjective state also helps to optimize the operator's workload management and improve their work experience.

[0095] In real-world industrial environments, operators typically need to wear various protective equipment, such as protective gloves and protective clothing. This protective equipment may cause nonlinear interference to the signal transmission path and bioimpedance of physiological sensors, leading to distortion or inaccuracy of the raw physiological signals acquired by the sensors.

[0096] In this regard, this application further proposes the steps for obtaining the operator's physiological state information, including heart rate, skin conductivity, and eye movement data, as described above: Obtain information on the material and wearing method of the protective equipment worn by the operator; Acquire contact status information between the physiological sensor and the operator's skin, including contact area and contact pressure; Based on the material information, wearing method information, and contact status information between the physiological sensors and the operator's skin of the protective equipment, the corresponding compensation parameters are selected from the preset compensation parameter table. Based on compensation parameters, the raw physiological signals collected by the physiological sensors are corrected to compensate for the nonlinear interference caused by the protective equipment on the signal transmission path and bioimpedance. The stability of the corrected physiological signal is monitored, including the amplitude of signal fluctuation and the degree of drift. When the stability of the calibrated physiological signal is lower than the preset threshold, the operator is prompted to adjust the way the protective equipment is worn or check the wearing of the physiological sensors.

[0097] Specifically, acquiring information about the material and wearing method of the protective equipment worn by operators aims to identify external factors that may interfere with physiological signals. Different protective equipment materials, such as rubber, leather, and fabric, as well as different wearing methods, such as tightness, looseness, and coverage area, can all affect the signal transmission characteristics between physiological sensors and the skin. For example, some materials may have a high dielectric constant, leading to signal attenuation in capacitive sensors; while a tight wearing method may alter blood flow to the skin, affecting the accuracy of heart rate sensors.

[0098] Furthermore, information on the contact state between the physiological sensor and the operator's skin is acquired. This contact state information includes the contact area and contact pressure, the purpose of which is to quantify the degree of coupling between the sensor and the organism. Contact area and contact pressure are key factors affecting the quality of physiological signal acquisition, such as bioelectrical and photoelectric signals. For example, insufficient contact area or pressure may result in weak signals and high noise; while excessive pressure may obstruct local blood circulation, affecting the accuracy of the signal.

[0099] Based on this, according to the material information of the protective equipment, the wearing method information, and the contact state information between the physiological sensors and the operator's skin, the corresponding compensation parameter is selected from a preset compensation parameter table. This compensation parameter table pre-stores typical deviation patterns of physiological signals that may occur under different types of protective equipment, wearing methods, and contact states, along with corresponding correction coefficients. By looking up the table, a customized basis can be provided for subsequent signal correction.

[0100] Subsequently, based on compensation parameters, the raw physiological signals acquired by the physiological sensors are corrected to compensate for the nonlinear interference caused by the protective equipment on the signal transmission path and bioimpedance. This correction process can employ various signal processing techniques, such as filtering, gain adjustment, and baseline drift correction. Its core purpose is to eliminate or reduce the systematic errors and random noise introduced by the protective equipment, thereby restoring the true form of the physiological signals.

[0101] In addition, monitoring the stability of the corrected physiological signal, including the amplitude of signal fluctuations and the degree of drift, aims to assess the correction effect and the continued reliability of the signal. If the corrected signal still exhibits excessive fluctuations or significant drift, it indicates that the signal quality may still be poor.

[0102] When the stability of the calibrated physiological signal falls below a preset threshold, the operator is prompted to adjust the way protective equipment is worn or check the wearing condition of the physiological sensors. This feedback mechanism can promptly remind the operator to optimize the sensor wearing conditions, thereby ensuring the accuracy and reliability of subsequent physiological data collection and avoiding errors in subsequent analysis due to inaccurate physiological data.

[0103] Through the above technical solution, this application can significantly improve the accuracy and reliability of acquiring operator physiological state information in industrial environments. Due to effective compensation for nonlinear interference introduced by protective equipment, the collected physiological signals can more realistically reflect physiological indicators such as the operator's heart rate, skin conductivity, and eye movements, thus providing a high-quality data foundation for subsequent operator subjective state assessment. This makes the aforementioned operator subjective state assessment results more accurate, and can more accurately correct for variations in the similarity of gesture trajectories, the range of touch pressure changes, and the duration of operation, thereby more reliably distinguishing between deviations caused by environmental or equipment factors and differences in the operator's learning and adaptation process.

[0104] Based on the same inventive concept, this application also discloses a liquid crystal display human-computer interaction control system, such as... Figure 2 As shown, the system includes: Signal acquisition module 1 is used to acquire the original capacitance signals generated by all sensing areas on the touch panel to form a signal distribution that reflects touch events; The structure recognition module 2 is used to identify one or more touch areas from the signal distribution, and extract the geometric features of each touch area and the spatial relationship between multiple touch areas to construct touch structure information; The gesture comparison module 3 is used to compare the touch structure information and its changes over time with a variety of preset gesture patterns, and determine the operator's intention gesture within a preset difference tolerance range. The instruction execution module 4 is used to respond to intentional gestures, execute corresponding industrial operation instructions, and provide a visual presentation of the operation results.

[0105] The system proposed in this application acquires the raw capacitive signals generated by all sensing areas on the touch panel through a signal acquisition module, forming a signal distribution reflecting touch events. A structure recognition module identifies the touch area from the signal distribution and extracts its geometric features and spatial relationships to construct touch structure information. A gesture comparison module compares the touch structure information and its changes over time with preset gesture patterns to determine the operator's intended gesture within a tolerance range. Finally, an instruction execution module responds to the intended gesture, executes industrial operation instructions, and provides visual presentation. This application can effectively address the challenges of complex industrial environments and improve the accuracy and efficiency of human-computer interaction through more accurate and intelligent gesture recognition and intent judgment.

[0106] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application.

Claims

1. A human-computer interaction control method for a liquid crystal display screen, characterized in that, include: Acquire the raw capacitance signals generated by all sensing areas on the touch panel to form a signal distribution that reflects touch events; From the signal distribution, one or more touch areas are identified, and the geometric features of each touch area and the spatial relationship between multiple touch areas are extracted to construct touch structure information; The touch structure information and its changes over time are compared with multiple preset gesture patterns, and the operator's intention gesture is determined within a preset difference tolerance range. In response to the intended gesture, the corresponding industrial operation command is executed, and a visual presentation of the operation result is provided.

2. The human-computer interaction control method for a liquid crystal display screen according to claim 1, characterized in that, The step of comparing the touch structure information and its changes over time with multiple preset gesture patterns, and determining the operator's intended gesture within a preset difference tolerance range, includes: Secondary touch areas are identified from the signal distribution, and these secondary touch areas are aggregated to obtain virtual touch areas. Based on the virtual touch area, a set of physical gesture hypotheses is generated. The set of physical gesture hypotheses contains multiple fuzzy gesture hypotheses, and each fuzzy gesture hypothesis has a physical confidence score. Obtain current task context information; Based on the task context information, a semantic consistency evaluation is performed on each fuzzy gesture hypothesis in the physical gesture hypothesis set to obtain a semantic consistency score. The operator's intention gesture is determined by combining the physical confidence score and the semantic consistency score.

3. The human-computer interaction control method for a liquid crystal display screen according to claim 2, characterized in that, The step of evaluating the semantic consistency of each fuzzy gesture hypothesis in the physical gesture hypothesis set based on the task context information to obtain a semantic consistency score includes: Obtain the current interactive interface element type and the control instruction category associated with the interface element; For newly introduced interface elements or interface elements whose associated control instruction categories have changed, semantic evaluation rules are generated or adjusted based on the functional description and expected operation mode of the newly introduced interface elements or interface elements whose associated control instruction categories have changed, combined with the interaction behaviors related to the newly introduced interface elements or interface elements whose associated control instruction categories have changed in the operator's historical operation log. Monitor the success rate and frequency of repeated operations of the interface elements; The weights or logic of the semantic evaluation rules are adjusted based on the success rate of the operation and the frequency of repeated operations to obtain the semantic consistency score.

4. The human-computer interaction control method for a liquid crystal display screen according to claim 3, characterized in that, The steps for monitoring the success rate and frequency of repeated operations of the interface elements include: Obtain execution status feedback for industrial operation instructions; Monitor the state changes of interface elements related to the command on the human-computer interaction interface; Determine whether the execution status feedback of the instruction and the state changes of the interface elements meet expectations; When the execution status feedback of the instruction or the state change of the interface element does not meet expectations, the operator is identified as performing repeated operations on the same or related interface elements within a preset time window. Mark the repeated operation as a repeated attempt; Based on the execution status feedback of the instruction, the status changes of the interface element, and the markers of repeated attempts, the success rate of the operation and the frequency of repeated operations of the interface element are determined.

5. The human-computer interaction control method for a liquid crystal display screen according to claim 3, characterized in that, The step of adjusting the weights or logic of the semantic evaluation rules based on the operation success rate and the frequency of repeated operations to obtain the semantic consistency score includes: Identify UI elements that lack sufficient historical operation data; Based on the functional description and expected operation mode of the interface element that lacks sufficient historical operation data, an initial semantic evaluation rule matching the interface element that lacks sufficient historical operation data is selected from the preset general operation logic template. Monitor the interaction behavior of the operator when using the interface element for the first time, which lacks sufficient historical operation data; the interaction behavior includes gesture trajectory and touch pressure. The analysis distinguishes between repetitive operations performed by operators on interface elements lacking sufficient historical operation data, including the similarity of gesture trajectories and the range of variation in touch pressure. The initial semantic evaluation rules are adjusted based on the convergence exhibited in the repeated operations. Experienced operators or engineers are allowed to confirm or fine-tune the adjusted semantic evaluation rules.

6. The human-computer interaction control method for a liquid crystal display screen according to claim 5, characterized in that, The monitoring of the interaction behavior when an operator first uses the interface element lacking sufficient historical operation data includes gesture trajectory and touch pressure. The step of monitoring the touch pressure when an operator first uses the interface element lacking sufficient historical operation data includes: The original capacitance signals of all sensing areas on the touch panel are acquired, and the original capacitance signals are spatially sampled to obtain the initial touch pressure distribution data. Obtain information about the material of the protective gloves currently worn by the operator; Based on the material information of the protective gloves and the pre-stored dielectric response characteristics of each area of ​​the touch panel under different protective glove materials, the initial touch pressure distribution data is regionally corrected to compensate for the nonlinear changes and local distortions caused by the protective gloves and display coating. Multiple local pressure peak regions in the corrected touch pressure distribution are identified, and boundary analysis is performed on each local pressure peak region to determine its shape and size; Based on the shape, size, and corrected touch pressure distribution data of each local pressure peak region, the pressure centroid and total pressure of each local pressure peak region are calculated, thereby quantifying the nonlinear characteristics and distortion of the touch pressure distribution to ensure the validity of the touch pressure data.

7. A human-computer interaction control method for a liquid crystal display screen according to claim 5, characterized in that, The step of analyzing the differences between repetitive operations performed by the operator on interface elements lacking sufficient historical operation data, including the similarity of gesture trajectories and the range of variation in touch pressure, includes: Obtain the gesture trajectory data for each repeated operation; The gesture trajectory data is preprocessed to remove instantaneous noise and jitter; The preprocessed gesture trajectory is segmented; Calculate the rate of change of direction and curvature of each segment; Obtain the touch pressure distribution data for each repeated operation; The touch pressure distribution data is regionally corrected to compensate for nonlinear drift caused by wear of protective gloves and local contamination of the display screen; Identify multiple local pressure peak regions in the corrected pressure distribution; Calculate the centroid of pressure and total pressure for each local pressure peak region; Construct feature vectors for the directional change rate and curvature of the gesture trajectory, as well as the pressure centroid and total pressure of the touch pressure distribution; Calculate the distance between the feature vectors of different repeated operations to quantify the similarity of gesture trajectories; Calculate the total pressure fluctuation amplitude in the local pressure peak region during each repeated operation; Analyze the trend of the total pressure fluctuation amplitude over time to quantify the range of touch pressure variation; Get the duration of an operator's actions in a repetitive operation; Compare the similarity of the gesture trajectories, the range of change in the touch pressure, and the change in the duration of the operation; Based on the comparison results, the deviations caused by environmental or equipment factors are distinguished from the differences in the operator's learning and adaptation process.

8. The human-computer interaction control method for a liquid crystal display screen according to claim 7, characterized in that, The step of comparing the similarity of the gesture trajectory, the range of change of the touch pressure, and the change of the operation duration includes: Acquire the operator's physiological state information, including heart rate, skin conductivity, and eye movement data; Acquire operator cognitive load information, which includes task completion time, error rate, and complexity of the operation sequence; The operator's emotional state information is obtained, including facial expression recognition results and voice tone analysis results; Based on the physiological state information, the cognitive load information, and the emotional state information, an operator subjective state assessment result is generated. Based on the operator's subjective state assessment results, the similarity of the gesture trajectory, the range of change of the touch pressure, and the change of the operation duration are corrected to eliminate deviations caused by operator subjective factors. The similarity of the corrected gesture trajectory, the range of change of the corrected touch pressure, and the change of the corrected operation duration are compared to distinguish the deviation caused by environmental or equipment factors from the differences in the operator's learning and adaptation process.

9. A human-computer interaction control method for a liquid crystal display screen according to claim 8, characterized in that, The step of acquiring the operator's physiological state information, including heart rate, skin conductivity, and eye movement data, includes: Obtain information on the material and wearing method of the protective equipment worn by the operator; Acquire contact status information between the physiological sensor and the operator's skin, the contact status information including contact area and contact pressure; Based on the material information of the protective equipment, the wearing method information, and the contact status information between the physiological sensor and the operator's skin, the corresponding compensation parameter is selected from the preset compensation parameter table; Based on the compensation parameters, the raw physiological signals collected by the physiological sensors are corrected to compensate for the nonlinear interference caused by the protective equipment on the signal transmission path and bioimpedance. The stability of the corrected physiological signal is monitored, including the amplitude of signal fluctuation and the degree of drift. When the stability of the corrected physiological signal is lower than a preset threshold, the operator is prompted to adjust the way the protective equipment is worn or to check the wearing of the physiological sensors.

10. A human-computer interaction control system for a liquid crystal display screen, characterized in that, The system includes: The signal acquisition module is used to acquire the raw capacitance signals generated by all sensing areas on the touch panel to form a signal distribution that reflects touch events. The structure recognition module is used to identify one or more touch areas from the signal distribution, and extract the geometric features of each touch area and the spatial relationship between multiple touch areas to construct touch structure information; The gesture comparison module is used to compare the touch structure information and its changes over time with a variety of preset gesture patterns, and determine the operator's intention gesture within a preset difference tolerance range. The instruction execution module is used to execute the corresponding industrial operation instructions in response to the intention gesture and to provide a visual presentation of the operation results.

Images