Gesture-based computer-aided engineering software intelligent agent application human-machine interaction method

Abstract

The application discloses a kind of gesture-based computer-aided engineering software intelligent agent application human-computer interaction method, and the structured command stream including translation, rotation, scaling, point selection and other operation types and parameters is output by pre-trained deep neural network. The operation precision is significantly improved by smoothing and correcting the recognition parameters through the secondary fine correction link. The CAE intelligent agent receives the corrected command stream, calls the software API to execute the operation, and feeds back the next step suggestion to the user after obtaining the execution state. The application solves the problem that natural language is difficult to accurately describe high-dimensional space constraints, and breaks the closed loop from instruction understanding to actual operation execution of the intelligent agent. At the same time, the visual interaction without external device is adopted, which completely eliminates the dependence of mouse and keyboard on the flat workbench, adapts to outdoor, clean workshop and high-altitude work and other limited working conditions, eliminates the operation failure and precision deficiency problem caused by the environmental constraints of traditional external devices, and realizes full-process, high-precision CAE simulation contactless control.

Description

Technical Field

[0001] This invention relates to the field of gesture recognition and computer-aided engineering software intelligent agent interaction technology, and in particular to a gesture-based human-computer interaction method for computer-aided engineering software intelligent agent applications. Background Technology

[0002] With the rapid development of large-scale model technology, agent technology based on large vision models (LVM) has been deployed on a large scale in multiple civilian and industrial scenarios. For example, in the consumer sector, agents can accurately identify plant and animal species and provide popular science information; in the transportation sector, fully automated driving (FSD) technology based on visual perception has enabled advanced driver assistance systems; and in the industrial surveying sector, mine modeling technology based on drone aerial photography and visual reconstruction is becoming increasingly mature. These applications demonstrate that agent technology possesses powerful environmental perception and semantic understanding capabilities.

[0003] However, in the field of Computer-Aided Engineering (CAE), existing intelligent agent technologies remain at a superficial level of dialogue and interaction, unable to deeply penetrate the core modeling and simulation (M&S) process. CAE modeling and simulation heavily rely on fine-grained manipulation of the graphical user interface (GUI), with typical scenarios including: Model assembly stage: High-precision spatial pose adjustment (such as translation and rotation) is required for the geometry of the parts; Working condition setup stage: It is necessary to accurately select geometric surfaces, lines, and points in complex 3D scenes in order to apply physical boundary conditions and loads.

[0004] Such operations are characterized by high dimensionality, strong interactivity, and strong spatial constraints. It is difficult to achieve precise control by relying solely on natural language dialogue, which results in existing CAE agents being "able to talk but unable to do" in actual engineering and unable to independently complete closed-loop simulation tasks.

[0005] On the other hand, traditional CAE software mainly relies on the mouse and keyboard to complete the above operations. Although the mouse has high precision in a desktop environment, its use is limited by the physical environment: in outdoor work, clean rooms, high-altitude work, or scenarios without desks, users often find it difficult to obtain a flat work surface, resulting in mouse control failure or a significant decrease in precision.

[0006] In summary, existing CAE interaction technologies suffer from a dual bottleneck: on the one hand, intelligent agents lack the ability to understand and execute non-linguistic spatial operations; on the other hand, traditional peripherals are severely constrained by the physical environment. Therefore, there is an urgent need for a novel human-computer interaction solution that is independent of physical peripherals and capable of accurately performing complex spatial operations, in order to bridge the technological gap between language understanding and action execution in CAE intelligent agents. Summary of the Invention

[0007] This invention provides a gesture-based human-computer interaction method for computer-aided engineering software intelligent agents, in order to overcome the above-mentioned technical problems.

[0008] To achieve the above objectives, the technical solution of the present invention is as follows: A gesture-based human-computer interaction method for computer-aided engineering software intelligent agents includes the following steps: S1. Import the 3D model to be operated on into the computer-aided engineering software; S2. Capture user gesture operation images through a camera, and preprocess the user gesture operation images to obtain hand posture timing data. S3. Input the hand posture timing data into a pre-trained gesture recognition deep neural network, and output a structured operation command stream, which includes operation command type and corresponding gesture recognition parameters; perform secondary fine correction processing on the gesture recognition parameters in the operation command stream to obtain the processed operation command stream; The operation command types include panning command, zooming command, rotating command, selection command, and confirmation command; The secondary fine-tuning correction process for the gesture recognition parameters in the operation command stream includes: Retrieve the original hand posture time series data corresponding to the generated gesture recognition parameters, use the preset skeleton model to calculate the three-dimensional coordinates of key hand points, and then generate specific feature vectors; After performing coordinate transformation on the specific feature vector, a specific filtering algorithm is used for further processing; S4. The intelligent agent in the computer-aided engineering software receives the processed operation command stream, calls the corresponding function interface of the computer-aided engineering software to perform the operation according to the processed operation command stream, and obtains the operation execution result status. S5. The agent in the computer-aided engineering software provides feedback to the user on the next operation suggestion based on the execution result status and the built-in structured knowledge graph; the structured knowledge graph is constructed by the agent using a combination of retrieval enhancement generation and knowledge graph-based retrieval enhancement generation techniques, combined with existing technical documents.

[0009] Furthermore, the pre-trained gesture recognition deep neural network includes: an attention layer, an intermediate hidden space layer, and a fully connected layer; The specific steps for inputting the hand pose timing data into a pre-trained gesture recognition deep neural network and outputting an operation command stream include: The attention layer weights the local features in the hand pose time series data that are discriminative for translation, rotation, scaling, selection and confirmation operations, while suppressing noise interference, to obtain weighted and enhanced hand feature data. The intermediate hidden space layer extracts the temporal and spatial features from the weighted and enhanced hand feature data, and outputs a highly robust gesture feature vector. The gesture feature vector is classified and regressed through the fully connected layer, and adaptive inference is performed based on the nonlinear mapping relationship learned in pre-training, thereby outputting the operation command stream.

[0010] Furthermore, a second, refined correction process is performed on the gesture recognition parameters corresponding to the translation command, including: Retrieve the original hand posture time series data corresponding to the generated translation gesture recognition parameters, use the MediaPipeHands skeleton model to solve the three-dimensional coordinates of 21 hand key points, extract the wrist reference point, the geometric center of the fingertip cluster and the palm normal vector, and generate a refined translation posture feature vector. The initial and final coordinates of the wrist are recalculated based on the refined translational posture feature vector. After mapping through the preset camera-screen-model three-dimensional coordinate system, high-precision start and end coordinate parameters are obtained. A Kalman filter and exponential smoothing fusion algorithm is used to denoise, interpolate, limit amplitude, and constrain acceleration of the high-precision start and end coordinate parameters to complete coordinate smoothing correction, thereby obtaining the processed translation gesture recognition parameters.

[0011] Furthermore, a second, refined correction process is performed on the gesture recognition parameters corresponding to the rotation command, including: Retrieve the original hand posture time series data corresponding to the generated rotation gesture recognition parameters, use the MediaPipeHands skeleton model to solve the three-dimensional coordinates of 21 hand key points, extract the center of the fingertip cluster, the center of wrist rotation and the normal vector of the rotation plane, and generate a refined rotation posture feature vector. Based on the refined rotational posture feature vector, the rotational coordinate axis and rotational angle are recalculated. After mapping through the preset camera-screen-model three-dimensional coordinate system, high-precision rotational axis and rotational angle parameters are obtained. First-order hysteresis filtering and multi-frame mean filtering are used to denoise, interpolate, limit, and constrain the high-precision rotation axis and rotation angle parameters to complete parameter smoothing correction, thereby obtaining the processed rotation gesture recognition parameters.

[0012] Furthermore, a second, refined correction process is performed on the gesture recognition parameters corresponding to the zoom command, including: Retrieve the original hand posture time series data corresponding to the generated scaling gesture recognition parameters, use the MediaPipeHands skeleton model to solve the three-dimensional coordinates of 21 hand key points, extract the bounding box of fingertip clusters, palm opening and fingertip distance change features, and generate a refined scaling gesture feature vector. The hand opening and closing change is recalculated based on the refined scaling posture feature vector. After mapping through the preset camera-screen-model three-dimensional coordinate system, the high-precision scaling factor parameter is obtained. Logarithmic transformation, amplitude constraint, and rate smoothing algorithm are used to denoise, interpolate, and constrain the amplitude and rate of change of the high-precision scaling factor parameters to complete parameter smoothing correction, thereby obtaining the processed scaling gesture recognition parameters.

[0013] Furthermore, a second, refined correction process is performed on the gesture recognition parameters corresponding to the selected command, including: Retrieve the original hand posture time series data corresponding to the generated point selection gesture recognition parameters, use the MediaPipeHands skeleton model to solve the three-dimensional coordinates of 21 hand key points, extract the three-dimensional coordinates of the index fingertip, palm posture and pointing vector, and generate a refined point selection gesture feature vector. The fingertip spatial coordinates are recalculated based on the refined point selection posture feature vector. After mapping through the preset camera-screen-model three-dimensional coordinate system, high-precision screen X and Y axis coordinate parameters are obtained. By employing position prediction filtering, jitter suppression, and subpixel correction algorithms, the X and Y axis coordinate parameters of the high-precision screen are denoised, interpolated, and subjected to positioning error constraints to achieve parameter smoothing correction, thereby obtaining the processed point-selection gesture recognition parameters. Furthermore, a second, refined correction process is performed on the gesture recognition parameters corresponding to the confirmation command, including: Retrieve the original hand posture time series data corresponding to the generated confirmation gesture recognition parameters, use the MediaPipeHands skeleton model to solve the three-dimensional coordinates of 21 hand key points, extract the fingertip aggregation point and palm posture features corresponding to the confirmation gesture, and generate a refined confirmation gesture feature vector. Based on the refined confirmed posture feature vector, the confirmation trigger coordinate parameters are recalculated, and after mapping through the preset camera-screen-model three-dimensional coordinate system, high-precision confirmation position parameters are obtained. An anti-shake filtering and trigger threshold constraint algorithm is used to denoise, interpolate, and trigger boundary constraints on the high-precision confirmation position parameters to complete parameter smoothing correction, thereby obtaining the processed confirmation gesture recognition parameters.

[0014] Beneficial effects: This invention relies on an interactive architecture that combines gesture acquisition with neural network instruction recognition and secondary parameter refinement correction. This overcomes the shortcomings of natural language in accurately describing high-dimensional spatial constraints, breaks down the technical barriers between instruction recognition and actual execution of intelligent agents, and solves the pain point that traditional intelligent agents can only perform semantic interaction and cannot implement precise simulation operations. On the other hand, it adopts a visual gesture interaction method without external devices, eliminating the dependence of mouse and keyboard on flat working environments. It is suitable for various restricted working conditions such as outdoor, cleanroom, and high-altitude operations, and eliminates the control failure and insufficient accuracy problems caused by the use environment of traditional peripherals. Ultimately, it can accurately complete the closed-loop control of the entire CAE simulation without physical peripherals, thus broadening the scope of engineering application of CAE intelligent agents. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is the first flowchart of the human-computer interaction method for the application of gesture-based computer-aided engineering software intelligent agents in this invention; Figure 2 This is a second flowchart of the human-computer interaction method for a gesture-based computer-aided engineering software intelligent agent in an embodiment of the present invention; Figure 3 This is a schematic diagram of various translation gestures used in the gesture training dataset in this embodiment of the invention; Figure 4 This is a schematic diagram of the shrinking gesture used in the gesture training dataset in this embodiment of the invention; Figure 5 This is a schematic diagram of the magnified gestures used in the gesture training dataset in this embodiment of the invention; Figure 6 This is a schematic diagram of the counterclockwise and clockwise rotation gestures used in the gesture training dataset in this embodiment of the invention; Figure 7 This is a schematic diagram of the point selection gestures used in the gesture training dataset in this embodiment of the invention; Figure 8 This is a schematic diagram of an actual application interaction scenario in an embodiment of the present invention.

[0017] In the diagram: 1. User; 2. CAE intelligent agent host; 3. Camera; 4. Microphone; 5. Display; 6. Speaker. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] This embodiment provides a gesture-based human-computer interaction method for computer-aided engineering software intelligent agents, such as... Figure 1 and Figure 2 As shown, the specific steps include: S1. Import the 3D model to be operated on into the CAE software; S2. Capture user gesture operation images through a camera and preprocess the user gesture operation images to obtain high signal-to-noise ratio hand posture time series data. Specifically, the preprocessing operation includes performing foreground segmentation, dynamic background suppression, global illumination normalization, and lens distortion correction on the user gesture operation image in sequence.

[0020] S3. Input the hand posture timing data into a pre-trained gesture recognition deep neural network, and output a structured operation command stream, which includes operation command type and corresponding gesture recognition parameters; perform secondary fine correction processing on the gesture recognition parameters in the operation command stream to obtain the processed operation command stream; The operation command types include panning command, zooming command, rotating command, selection command, and confirmation command; in practice, command types can be added as needed. Specifically, the gesture recognition parameters corresponding to the translation command, zoom command, rotate command, point selection command, and confirm command include: the coordinates of the start and end points of the translation, the zoom ratio coefficient, the rotation axis such as the x, y, z axes and the rotation angle along the x, y, z axes, and the coordinates of the selected spatial point; The secondary fine-tuning correction process for the gesture recognition parameters in the operation command stream includes: Retrieve the original hand posture time series data corresponding to the generated gesture recognition parameters, use the preset skeleton model to calculate the three-dimensional coordinates of key hand points, and then generate specific feature vectors; After performing coordinate transformation on the specific feature vector, a specific filtering algorithm is used for further processing; S4. The CAE agent receives the processed operation command stream, calls the corresponding CAE software's function API to perform the operation according to the processed operation command stream, and obtains the operation execution result status. Specifically, in this embodiment, the processed operation command stream is encapsulated according to the CAE software interaction protocol, and the encapsulated operation script is output to the intelligent agent through asynchronous communication. The intelligent agent is responsible for sending this series of scripts to the script driver API of the CAE software and executing the corresponding operations one by one.

[0021] Specifically, the CAE software mentioned in this embodiment is a general term for this type of software and is not limited to ANSYS, Abaqus, Comsol, or domestic CAE software INTESIM, etc.

[0022] Specifically, in this embodiment, for example: after the CAE agent obtains the view angle adjustment command, it calls the View_rotate_API function and executes the function; in practice, the API functions corresponding to the software's operation commands include: CAE_ViewPan(x1,y1,z1,x2,y2,z2); CAE_ViewZoom(factor); CAE_ViewRotate(axis, angle); CAE_PickEntity(screenX, screenY) etc.

[0023] S5. The agent in the computer-aided engineering software provides feedback to the user on the next operation suggestion based on the execution result status and the built-in structured knowledge graph. The structured knowledge graph is constructed by the agent using a combination of RAG (Retrieval-Augmented Generation) and GraphRAG (Retrieval-Augmented Generation based on knowledge graphs) and existing technical documents.

[0024] Specifically, for example, after the CAE agent executes the View_rotate_API command, it obtains the command execution result status return value of 0 or 1, where 0 indicates successful execution and 1 indicates failure. Then, based on the execution result of the previous command, the agent provides feedback to the user with suggestions for the next operation.

[0025] In this embodiment, the CAE agent has a built-in domain knowledge base and knowledge graph for CAE modeling and simulation scenarios to support user intent understanding and operation process planning. Specifically, the CAE intelligent agent adopts a technical solution that combines RAG and GraphRAG to vectorize and store technical documents such as CAE software manuals, simulation specifications, and engineering cases, and extracts entities, attributes, and relationships from the documents to construct a structured knowledge graph. During user interaction, the CAE agent first performs preliminary information retrieval in the vector knowledge base based on cosine similarity calculation, matching the technical content and historical cases most relevant to the user's needs. Then, combined with semantic analysis capabilities, it performs logical reasoning and path mining on the nodes and edges in the knowledge graph, extracting the operational logic structure and constraints that conform to the current simulation goal, and finally transforming the user's fuzzy needs into a standardized CAE modeling and simulation operation list that can be executed step by step. Through the above design, the CAE agent can automatically complete missing parameters and correct unreasonable settings when the user's intention is incomplete or uncertain. It also guides the user to complete the entire process from geometric modeling, boundary condition setting, mesh generation to solving and post-processing in a human-computer question-and-answer manner, significantly improving the executability and success rate of CAE simulation tasks. In a specific embodiment, the pre-trained gesture recognition deep neural network includes: an attention layer, an intermediate hidden space layer, and a fully connected layer; In the pre-training phase of the gesture recognition deep neural network, multi-dimensional temporal parameter joint supervised training is performed. The complete process and constraint logic include: (1) Pre-collect a gesture training dataset, including hand posture sample data with time-series annotation. The hand posture sample data is the coordinate sequence of key hand points corresponding to continuous frame images, and fully records the trajectory of the spatial position of hand joints, palm base and fingertips changing over time from the start frame to the end frame. (2) Numerical calculations were performed on the collected hand posture time series data, and three types of basic gesture motion parameters were extracted as supervision ground truth: the first is the gesture movement distance, which is calculated based on the spatial coordinate difference of the hand reference key points in adjacent frames / first and last frames; the second is the gesture rotation angle, which is calculated based on the temporal change of the hand bone vector, and the angle change of the palm flip, wrist rotation, and finger bending; the third is the action duration, which is calculated based on the total number of consecutive frames corresponding to the complete gesture action from the start frame to the end frame, and converted into standard time length. (3) The calculated gesture movement distance, rotation angle and action duration are used as the ground truth labels of the fully connected layer to construct the basic regression supervision term, which directly constrains the prediction output of the fully connected layer for the three types of motion parameters; (4) Based on the calculated gesture movement distance and action duration, further derive and calculate the user's limb movement speed: with the total gesture movement distance as the total displacement and the action duration as the time denominator, obtain the overall average movement speed of the gesture. This speed characterizes the speed of the user's hand limb movement. (5) The calculated limb movement speed is added as an additional loss constraint term and embedded into the overall loss function of the network: on the one hand, the network is fitted with the numerical distribution of distance, angle and duration by using the basic true value supervision; on the other hand, the speed loss constraint is used to constrain the coupling relationship between the three to avoid the network independently predicting the three types of parameters and ignoring the physical logic of motion. (6) The network weights are updated by combining the base true value regression loss and the limb movement speed constraint loss through backpropagation. This drives the gesture recognition deep neural network to autonomously learn the inherent nonlinear coupling mapping relationship between gesture movement distance, rotation angle, and movement duration, and capture the linkage change law between translation amplitude, rotation amplitude, and movement speed under different gesture movements. (7) After completing the pre-training, the resulting gesture recognition deep neural network has the ability to adapt gesture parameter mapping. It can synchronously and accurately output three types of linkage parameters that match the physical logic of human movement: movement distance, rotation angle, and action duration, based on the input hand temporal posture, thus eliminating the deviation of single-parameter prediction from the real limb movement law.

[0026] The specific steps for inputting the hand pose timing data into a pre-trained gesture recognition deep neural network and outputting an operation command stream include: The attention layer weights the local features in the hand pose time series data that are discriminative for translation, rotation, scaling, selection and confirmation operations, while suppressing irrelevant noise interference from background, non-hand limbs and motion blur, thereby enhancing the expression of effective features and obtaining weighted and enhanced hand feature data. The intermediate hidden space layer extracts the temporal and spatial features from the weighted and enhanced hand feature data, and outputs a highly robust gesture feature vector. Specifically, the intermediate hidden space layer can adopt a temporal convolutional network (TCN), a long short-term memory network (LSTM), or a Transformer network structure to capture the continuity and long-distance dependencies of gesture actions. The gesture feature vector is classified and regressed through the fully connected layer, and adaptive inference is performed based on the nonlinear mapping relationship learned in pre-training, thereby outputting the operation command stream.

[0027] Specifically, to ensure the generalization ability and robustness of the gesture recognition deep neural network in complex engineering scenarios, this embodiment employs the following three training strategies to optimize the gesture recognition deep neural network: (1) Construction of multi-form gesture dataset For different types of operation commands, gesture training datasets with significant differences are constructed to improve the ability of deep neural networks for gesture recognition to adapt to different user habits.

[0028] Specifically, such as Figure 3 As shown, for translation operations, operation videos containing various finger postures such as two fingers, three fingers, and five fingers are captured; for example... Figure 4 and Figure 5 The image shows a diagram illustrating the zoom-in and zoom-out gestures. Figure 6 The image shows a dataset of rotation gestures. For rotation operations, a further dataset covering various hand postures, such as natural hand bending, three-finger pinching, and fist clenching, can be constructed. Figure 7 The diagram illustrates a point-and-click gesture. For the gestures described above, the sample diversity can be expanded based on actual conditions to meet the gesture recognition needs of various groups and usage scenarios. Through the above multi-form data collection, the model is ensured to possess high recognition accuracy and generalization performance when faced with different gesture expressions.

[0029] (2) Adaptive gesture parameter mapping mechanism In the training process of a deep neural network for gesture recognition, the speed of the user's limb movements is introduced as a key training factor. The gesture recognition deep neural network learns the non-linear mapping relationship between gesture movement distance, rotation angle, and action duration, and adaptively infers the corresponding operation execution parameters, including translation distance, scaling ratio, and rotation angle. This mechanism effectively eliminates parameter deviations caused by differences in the speed of user movements, ensuring a consistent user experience when using gesture interaction for users with varying abilities.

[0030] (3) Robustness optimization based on data augmentation To improve the stability of gesture recognition deep neural networks in complex environments, various data augmentation processes are applied to the input images during training, including but not limited to: randomly adjusting image resolution; randomly changing image brightness; and randomly occluding local areas.

[0031] The above enhancement methods significantly improve the adaptability of the gesture recognition deep neural network to changes in lighting, occlusion interference, and device differences, thereby improving the overall robustness of the gesture recognition system.

[0032] In specific implementation, the steps for performing secondary fine-tuning correction on the gesture recognition parameters in the operation command stream include: (1) Perform secondary fine-tuning correction on the gesture recognition parameters corresponding to the translation command, including: Retrieve the original hand posture time series data corresponding to the generated translation gesture recognition parameters, use the MediaPipeHands skeleton model to solve the three-dimensional coordinates of 21 hand key points, extract the wrist reference point, the geometric center of the fingertip cluster and the palm normal vector, and generate a refined translation posture feature vector. Based on the refined translational posture feature vector, the precise initial and final wrist coordinates are recalculated. After mapping through the preset camera-screen-model three-dimensional coordinate system, high-precision start and end coordinate parameters are obtained. A Kalman filter and exponential smoothing fusion algorithm is used to perform noise reduction, interpolation, amplitude limiting, and acceleration constraints on the high-precision start and end coordinate parameters to complete coordinate smoothing correction, thereby obtaining the processed translation gesture recognition parameters.

[0033] Specifically, the coordinates of the hand's 3D key points in the camera coordinate system are acquired by the camera. First, the forward projection from the camera coordinate system to the screen's 2D pixel coordinate system is completed. Then, through interactive inverse mapping and rigid body transformation, the 2D operation quantities of the screen gestures are converted into control parameters (translation, rotation, scaling coordinates, angles) in the local coordinate system of the 3D model. Since any single-axis rotation in 3D space will change the angle between the other two axes, the 2D screen operations (rotation, angle) generated by the gestures can be used to completely calculate the three-axis posture parameters required for the control of the 3D model, thereby meeting the interaction requirements.

[0034] Specifically, since the hand posture data processed in this embodiment is extremely small in scale, containing only the spatial coordinate temporal information of dozens of key hand points, the overall computational load is extremely low: the entire set of motion parameter calculation and network prediction links will not introduce significant delays, and have almost no negative impact on the overall real-time performance of the dynamic gesture recognition system.

[0035] (2) Perform secondary fine-tuning correction on the gesture recognition parameters corresponding to the rotation command, including: Retrieve the original hand posture time series data corresponding to the generated rotation gesture recognition parameters, use the MediaPipeHands skeleton model to solve the three-dimensional coordinates of 21 hand key points, extract the center of the fingertip cluster, the center of wrist rotation and the normal vector of the rotation plane, and generate a refined rotation posture feature vector. Based on the refined rotational posture feature vector, the precise rotational coordinate axes (X / Y / Z) and rotation angles are recalculated. After mapping through the preset camera-screen-model three-dimensional coordinate system, high-precision rotational axis and rotation angle parameters are obtained. First-order hysteresis filtering and multi-frame mean filtering are used to denoise, interpolate, limit, and constrain the high-precision rotation axis and rotation angle parameters to complete parameter smoothing correction, thereby obtaining the processed rotation gesture recognition parameters.

[0036] (3) Perform secondary fine-tuning correction on the gesture recognition parameters corresponding to the zoom command, including: Retrieve the original hand posture time series data corresponding to the generated scaling gesture recognition parameters, use the MediaPipeHands skeleton model to solve the three-dimensional coordinates of 21 hand key points, extract the bounding box of fingertip clusters, palm opening and fingertip distance change features, and generate a refined scaling gesture feature vector. Based on the refined scaling posture feature vector, the precise change in palm opening and closing is recalculated, and after being mapped through a preset camera-screen-model three-dimensional coordinate system, a high-precision scaling factor parameter is obtained. Logarithmic transformation, amplitude constraint, and rate smoothing algorithm are used to denoise, interpolate, and constrain the amplitude and rate of change of the high-precision scaling factor parameters to complete parameter smoothing correction, thereby obtaining the processed scaling gesture recognition parameters.

[0037] (4) Perform secondary fine-tuning correction on the gesture recognition parameters corresponding to the selected command, including: Retrieve the original hand posture time series data corresponding to the generated point selection gesture recognition parameters, use the MediaPipeHands skeleton model to solve the three-dimensional coordinates of 21 hand key points, extract the three-dimensional coordinates of the index fingertip, palm posture and pointing vector, and generate a refined point selection gesture feature vector. Based on the refined point-selection posture feature vector, the precise fingertip spatial coordinates are recalculated, and after mapping through the preset camera-screen-model three-dimensional coordinate system, high-precision screen X and Y axis coordinate parameters are obtained. By employing position prediction filtering, jitter suppression, and subpixel correction algorithms, the X and Y axis coordinate parameters of the high-precision screen are denoised, interpolated, and subjected to positioning error constraints to achieve parameter smoothing correction, thereby obtaining the processed point-selection gesture recognition parameters. (5) Perform secondary fine-tuning correction on the gesture recognition parameters corresponding to the confirmation command, including: Retrieve the original hand posture time series data corresponding to the generated confirmation gesture recognition parameters, use the MediaPipeHands skeleton model to solve the three-dimensional coordinates of 21 hand key points, extract the fingertip aggregation point and palm posture features corresponding to the confirmation gesture, and generate a refined confirmation gesture feature vector. Based on the refined confirmation posture feature vector, the precise confirmation trigger coordinate parameters are recalculated, and after mapping through the preset camera-screen-model three-dimensional coordinate system, high-precision confirmation position parameters are obtained. An anti-shake filtering and trigger threshold constraint algorithm is used to denoise, interpolate, and trigger boundary constraints on the high-precision confirmation position parameters to complete parameter smoothing correction, thereby obtaining the processed confirmation gesture recognition parameters.

[0038] Specifically, in this embodiment, the confirmation gesture adopts a single-finger tap triggering judgment logic: when the duration of the single-finger continuous tap state exceeds a preset time threshold (such as 0.5 seconds), it is determined to be a valid confirmation gesture; the time threshold is not a fixed value, and can be adjusted and optimized according to different interaction scenarios and device sampling frame rates during the offline testing phase.

[0039] Specifically, the output parameters of gesture recognition deep neural networks inherently suffer from accuracy defects. For example, pre-trained gesture recognition deep neural networks focus on gesture category classification, and the gesture parameters such as start and end coordinates, rotation angle, scaling factor, and screen position output by inference are coarse-grained estimates. These are susceptible to image noise, hand tremors, and imaging distortion, resulting in large deviations between the coordinates and quantized parameters. Directly feeding these parameters into a CAE can easily lead to viewport drift, scaling jumps, and mis-selection. Therefore, this embodiment uses finely recalculated features based on the original time-series data and performs multi-coordinate system mapping to replace the coarse parameters of the gesture recognition deep neural network. This significantly improves the numerical accuracy of translation coordinates, rotation angles, scaling factors, and screen position picking, reducing CAE manipulation position deviations.

[0040] Specifically, CAE software has stringent requirements for the stability of control parameters: CAE 3D model manipulation is a high-precision industrial interaction, and the original network parameters contain random jitter and abrupt changes, which cannot match the parameter smoothness and positioning accuracy specifications of CAE interface calls, and must be filtered, corrected, and optimized. Therefore, this embodiment uses correction processing to adapt the standardized parameters to the CAE native API input specifications, reducing the probability of command call errors, pickup failures, and model misoperations, and improving the overall stability of gesture-controlled CAE.

[0041] In practical engineering applications, CAE modeling and simulation are typically characterized by long processes, high parameter coupling, and complex operational logic. Taking a typical cantilever beam static analysis as an example, the complete process includes seven main stages: geometric modeling, material definition, boundary condition setting, load application, mesh generation, solution calculation, and post-processing. Each stage contains a large number of fine-grained configuration items. For example, the mesh generation stage requires further clarification of specific parameters such as mesh size, element type, and patching strategy.

[0042] To address the aforementioned complexities, the CAE agent in this embodiment is equipped with user intent understanding capabilities. This function, as the core of the agent, is used to transform the user's high-level requirements into a standardized, step-by-step work list. When the user's input intent information is incomplete or ambiguous, the agent can proactively ask the user questions through human-computer question-and-answer interaction to clarify and complete the ambiguous information. This allows the agent to accurately drive the CAE software to complete the entire simulation task without requiring the user to be proficient in the underlying software operations.

[0043] like Figure 8As shown, in this embodiment, user 1 views the model on display 5 and makes corresponding gestures, and receives brief voice commands from user 1 via microphone 4. CAE intelligent agent host 2 receives visual signals by connecting to camera 3, and receives command feedback from intelligent agent by connecting to speaker 6. Example 1: Users need to perform static analysis of cantilever beams. The interaction process is as follows: User voice input: "Help me do a static analysis of the cantilever beam"; The agent asks: "Please enter the length, width, and height dimensions of the cantilever beam." The user replied: "Length 100mm, Width 20mm, Height 20mm"; The agent automatically calls the geometric modeling API to generate the corresponding model and then asks: "How should the boundaries and loads be set?" Users can adjust the model's viewpoint by rotating the model to locate the fixed end of the cantilever beam; users can select the fixed end face by tapping the face and then input "add a fixed constraint to this face" via voice input. The agent invokes the boundary setting API to apply constraints. The user rotates the model again, selects the side to be loaded by tapping, and inputs "add 10MPa pressure to this side" via voice. The intelligent agent completes the payload setup; The agent proactively prompts: "Next, we will configure the mesh. How do I set the cell type and mesh size?" The user responded via voice: "Use hexahedral units, with the size set to 2 millimeters." The agent completes the grid partitioning; The subsequent solution and post-processing steps are automatically guided by the intelligent agent according to the standard process, without the need for the use of a mouse and keyboard.

[0044] Beneficial effects of this invention: End-to-end contactless interaction closed loop: A complete interaction chain is constructed from gesture image acquisition, semantic parsing, command execution to view feedback, realizing contactless control throughout the entire process. Users can complete complex operations in CAE software without relying on traditional peripherals such as mice, keyboards or touchpads, significantly reducing the interaction threshold in special work scenarios (such as cleanrooms, wearing protective equipment, and high-altitude operations); Multi-level feature extraction and joint optimization: Through a three-level network structure of attention layer - intermediate hidden space layer - fully connected layer, this design achieves high-precision calculation of fine-grained parameters such as translation distance, rotation angle, and scaling ratio while ensuring the accuracy of gesture recognition, and outputs continuous action parameters, taking into account both classification and regression tasks. Standardized command flow and high system scalability: The recognition results are uniformly encapsulated into a standardized command flow decoupled from the CAE software. The agent only needs to call the corresponding API interface to complete the operation. This design makes the system highly scalable: when adding a new gesture type or adapting to a new CAE software, only the command mapping library needs to be updated, without reconstructing the underlying recognition model, which greatly reduces maintenance costs. Strong robustness and environmental adaptability: By introducing dynamic background suppression and illumination normalization in the preprocessing stage, and combining it with attention mechanism to filter noise, the impact of complex illumination, dynamic background and motion blur on recognition stability is effectively overcome, ensuring continuous and reliable operation in variable engineering environments.

[0045] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A gesture-based human-computer interaction method for computer-aided engineering software intelligent agents, characterized in that: The specific steps include: S1. Import the 3D model to be operated on into the computer-aided engineering software; S2. Capture user gesture operation images through a camera, and preprocess the user gesture operation images to obtain hand posture timing data. S3. Input the hand posture timing data into a pre-trained gesture recognition deep neural network, and output a structured operation command stream, which includes operation command type and corresponding gesture recognition parameters; perform secondary fine correction processing on the gesture recognition parameters in the operation command stream to obtain the processed operation command stream; The operation command types include panning command, zooming command, rotating command, selection command, and confirmation command; The secondary fine-tuning correction process for the gesture recognition parameters in the operation command stream includes: Retrieve the original hand posture time series data corresponding to the generated gesture recognition parameters, use the preset skeleton model to calculate the three-dimensional coordinates of key hand points, and then generate specific feature vectors; After performing coordinate transformation on the specific feature vector, a specific filtering algorithm is used for further processing; S4. The intelligent agent in the computer-aided engineering software receives the processed operation command stream, calls the corresponding function interface of the computer-aided engineering software to perform the operation according to the processed operation command stream, and obtains the operation execution result status. S5. The agent in the computer-aided engineering software provides feedback to the user on the next operation suggestion based on the execution result status and the built-in structured knowledge graph; the structured knowledge graph is constructed by the agent using a combination of retrieval enhancement generation and knowledge graph-based retrieval enhancement generation techniques, combined with existing technical documents.

2. The human-computer interaction method for gesture-based computer-aided engineering software intelligent agents according to claim 1, characterized in that, The pre-trained gesture recognition deep neural network includes: an attention layer, an intermediate hidden space layer, and a fully connected layer; The specific steps for inputting the hand pose timing data into a pre-trained gesture recognition deep neural network and outputting an operation command stream include: The attention layer weights the local features in the hand pose time series data that are discriminative for translation, rotation, scaling, selection and confirmation operations, while suppressing noise interference, to obtain weighted and enhanced hand feature data. The intermediate hidden space layer extracts the temporal and spatial features from the weighted and enhanced hand feature data, and outputs a highly robust gesture feature vector. The gesture feature vector is classified and regressed through the fully connected layer, and adaptive inference is performed based on the nonlinear mapping relationship learned in pre-training, thereby outputting the operation command stream.

3. The human-computer interaction method for gesture-based computer-aided engineering software intelligent agents according to claim 2, characterized in that, A secondary fine-tuning correction process is performed on the gesture recognition parameters corresponding to the translation command, including: Retrieve the original hand posture time series data corresponding to the generated translation gesture recognition parameters, use the MediaPipe Hands skeleton model to solve the three-dimensional coordinates of 21 hand key points, extract the wrist reference point, the geometric center of the fingertip cluster and the palm normal vector, and generate a refined translation posture feature vector. The initial and final coordinates of the wrist are recalculated based on the refined translational posture feature vector. After mapping through the preset camera-screen-model three-dimensional coordinate system, high-precision start and end coordinate parameters are obtained. A Kalman filter and exponential smoothing fusion algorithm is used to denoise, interpolate, limit amplitude, and constrain acceleration of the high-precision start and end coordinate parameters to complete coordinate smoothing correction, thereby obtaining the processed translation gesture recognition parameters.

4. The human-computer interaction method for gesture-based computer-aided engineering software intelligent agents according to claim 2, characterized in that, A secondary fine-tuning process is performed on the gesture recognition parameters corresponding to the rotation command, including: Retrieve the original hand posture time series data corresponding to the generated rotation gesture recognition parameters, use the MediaPipe Hands skeleton model to solve the three-dimensional coordinates of 21 hand key points, extract the center of the fingertip cluster, the center of wrist rotation and the normal vector of the rotation plane, and generate a refined rotation posture feature vector. Based on the refined rotational posture feature vector, the rotational coordinate axis and rotational angle are recalculated. After mapping through the preset camera-screen-model three-dimensional coordinate system, high-precision rotational axis and rotational angle parameters are obtained. First-order hysteresis filtering and multi-frame mean filtering are used to denoise, interpolate, limit, and constrain the high-precision rotation axis and rotation angle parameters to complete parameter smoothing correction, thereby obtaining the processed rotation gesture recognition parameters.

5. The human-computer interaction method for gesture-based computer-aided engineering software intelligent agents according to claim 2, characterized in that, A secondary fine-tuning correction process is performed on the gesture recognition parameters corresponding to the zoom command, including: Retrieve the original hand posture time series data corresponding to the generated scaling gesture recognition parameters, use the MediaPipe Hands skeleton model to solve the three-dimensional coordinates of 21 hand key points, extract the bounding box of fingertip clusters, palm opening and fingertip distance change features, and generate a refined scaling gesture feature vector. The hand opening and closing change is recalculated based on the refined scaling posture feature vector. After mapping through the preset camera-screen-model three-dimensional coordinate system, the high-precision scaling factor parameter is obtained. Logarithmic transformation, amplitude constraint, and rate smoothing algorithm are used to denoise, interpolate, and constrain the amplitude and rate of change of the high-precision scaling factor parameters to complete parameter smoothing correction, thereby obtaining the processed scaling gesture recognition parameters.

6. The human-computer interaction method for gesture-based computer-aided engineering software intelligent agents according to claim 2, characterized in that, A secondary fine-tuning correction process is performed on the gesture recognition parameters corresponding to the selected command, including: Retrieve the original hand posture time series data corresponding to the generated point selection gesture recognition parameters, use the MediaPipe Hands skeleton model to solve the three-dimensional coordinates of 21 hand key points, extract the three-dimensional coordinates of the index fingertip, palm posture and pointing vector, and generate a refined point selection gesture feature vector. The fingertip spatial coordinates are recalculated based on the refined point selection posture feature vector. After mapping through the preset camera-screen-model three-dimensional coordinate system, high-precision screen X and Y axis coordinate parameters are obtained. By employing position prediction filtering, jitter suppression, and subpixel correction algorithms, the X and Y axis coordinate parameters of the high-precision screen are denoised, interpolated, and subjected to positioning error constraints to achieve parameter smoothing correction, thereby obtaining the processed point-selection gesture recognition parameters.

7. The human-computer interaction method for gesture-based computer-aided engineering software intelligent agents according to claim 2, characterized in that, A secondary fine-tuning correction process is performed on the gesture recognition parameters corresponding to the confirmation command, including: Retrieve the original hand posture time series data corresponding to the generated confirmation gesture recognition parameters, use the MediaPipe Hands skeleton model to solve the three-dimensional coordinates of 21 hand key points, extract the fingertip aggregation point and palm posture features corresponding to the confirmation gesture, and generate a refined confirmation gesture feature vector. Based on the refined confirmed posture feature vector, the confirmation trigger coordinate parameters are recalculated, and after mapping through the preset camera-screen-model three-dimensional coordinate system, high-precision confirmation position parameters are obtained. An anti-shake filtering and trigger threshold constraint algorithm is used to denoise, interpolate, and trigger boundary constraints on the high-precision confirmation position parameters to complete parameter smoothing correction, thereby obtaining the processed confirmation gesture recognition parameters.

Images