A robot welding action generation method, system and electronic device

Abstract

The application relates to the technical field of intelligent manufacturing and computer vision, and discloses a robot welding action generation method and system and electronic equipment. The application realizes adaptive elimination of visual redundancy in the time sequence dimension through global time sequence merging, guarantees complete reservation of key visual detail features in a complex welding scene through diversity perception pruning, realizes collaborative simplification of text and visual features through attention weight screening pruning, and completes dynamic adaptive optimization of bimodal features through semantic perception pruning. The application solves the defect that the prior art cannot adaptively process visual features, removes redundant features layer by layer, reduces the calculation load, greatly improves the real-time performance of welding work, accurately reserves the core visual and text semantic features required for welding action generation, fully guarantees the control accuracy of the welding action, and significantly improves the adaptive perception capability and work reliability of the robot welding system in a complex scene.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing and computer vision interdisciplinary technology, specifically to a method, system and electronic device for generating robot welding actions. Background Technology

[0002] The continuous advancement of intelligent manufacturing technology has driven the deep evolution of industrial robot welding processes from traditional teach-and-write programming to autonomy and intelligence, becoming a key support for achieving efficient and stable welding operations in the high-end equipment manufacturing field. In complex and ever-changing industrial environments, welding operations often face multiple challenges, such as workpiece pose deviations in unstructured environments, strong arc light interference, and smoke and dust obstruction. Traditional teach-and-write methods are difficult to adapt flexibly. Therefore, robot welding control technology based on visual feedback has become a key area of ​​research and application in the industry. Its core objective is to achieve autonomous trajectory planning and motion control of welding robots through machine vision perception and intelligent algorithms, in order to meet the stringent requirements of modern manufacturing for welding accuracy, operational efficiency, and environmental robustness.

[0003] Currently, mainstream vision-guided welding control solutions first collect video information including the weld, workpiece, and surrounding environment using vision sensors deployed at the welding site. Then, deep neural networks are used to extract high-dimensional semantic features from the scene. Finally, the extracted features are mapped to the robot's motion space to generate corresponding welding trajectories and control commands. These solutions can achieve stable operation under structured, low-interference standard conditions. However, when facing complex welding scenarios with high dynamics, multiple occlusions, and significant variations in working conditions, the contradiction between computational efficiency and motion generation accuracy becomes increasingly prominent, becoming a core bottleneck restricting the practical application of the technology.

[0004] In existing technologies, the methods for addressing the conflict between computing power and accuracy still have significant limitations. They are difficult to adaptively adjust feature processing strategies according to the dynamic changes in the welding scenario, and cannot ensure the complete preservation of key details while effectively eliminating redundant information. Consequently, it is difficult to balance the dual requirements of real-time operation and motion control accuracy, which restricts the large-scale application of vision-guided welding technology in complex industrial scenarios. Summary of the Invention

[0005] This invention provides a method, system, and electronic device for generating robot welding motions, in order to solve the technical problem that existing technologies cannot adaptively process visual features and are difficult to balance real-time operation and motion control accuracy.

[0006] In a first aspect, the present invention provides a method for generating robot welding actions, the method comprising: Collect continuous video frames of the target welding operation scene, extract and project features to generate a frame token sequence, respond to the input welding task text command by word segmentation and projection to generate a text feature sequence; The frame token sequence is globally temporally merged to obtain a visual feature sequence; The visual feature sequence is subjected to diversity perception pruning to obtain a diversity-pruned visual feature sequence. Based on the diverse pruned visual feature sequence, attention weight filtering and pruning are performed on the text feature sequence to obtain the pruned text feature sequence; The diverse pruned visual feature sequence and the pruned text feature sequence are fused together, and semantic-aware pruning is performed on the fused dual-modal fused feature sequence to obtain an adaptive pruned feature sequence. The adaptive pruning feature sequence is subjected to action semantic decoding to generate and output the welding robot action control instructions.

[0007] This invention acquires continuous video frames of a target welding operation scene, extracts features, and projects them to generate a frame token sequence. Simultaneously, it segments and projects the welding task text instructions to generate a text feature sequence. First, it performs global temporal merging on the frame token sequence to regularize temporal visual features and remove redundant temporal information to obtain a visual feature sequence. Then, it performs diversity-aware pruning on the visual feature sequence, selecting features based on the spatial distribution and semantic uniqueness of visual features to retain sparse but crucial semantic details in the welding scene, avoiding the loss of core visual information. Subsequently, based on the diversity-pruned visual feature sequence, it performs attention weight pruning on the text feature sequence, achieving collaborative adaptation between text and visual features, removing redundant text features irrelevant to the welding task, and improving the matching degree and processing efficiency of the dual-modal features. Finally, it fuses the two types of pruned features and performs semantic-aware pruning on the dual-modal fused feature sequence, adaptively adjusting feature retention according to the dynamic characteristics of the welding scene. The invention utilizes a multi-dimensional, hierarchical adaptive feature processing system. This system achieves adaptive removal of temporal-dimensional visual redundancy through global temporal merging, ensures the complete preservation of key visual details in complex welding scenarios through diversity-based perceptual pruning, achieves collaborative simplification of text and visual features through attention-weighted filtering pruning, and completes dynamic adaptive optimization of dual-modal features through semantic-perceptual pruning. This system overcomes the limitations of existing technologies in adaptively processing visual features. By progressively removing redundant features and reducing computational load to significantly improve the real-time performance of welding operations, it accurately retains the core visual and textual semantic features required for welding action generation, fully guaranteeing the control precision of welding actions. This achieves a high-efficiency balance between real-time performance and motion control precision, significantly improving the adaptive perception capability and operational reliability of the robot welding system in complex scenarios.

[0008] In one optional implementation, the step of acquiring continuous video frames of the target welding operation scene, extracting and projecting features to generate a frame token sequence, and segmenting and projecting the input welding task text command to generate a text feature sequence includes: Acquire continuous video frames of the target welding operation scene; Spatial features are obtained by extracting features from the continuous video frames using a visual encoder; The spatial features are projected to generate a frame token sequence; The input welding task text command is segmented by a word segmenter to obtain a text word sequence; The text word sequence is projected to generate a text feature sequence.

[0009] By extracting and projecting visual spatial features from welding site video footage and segmenting and projecting welding process text instructions, standardized and structured encoding of visual and text modalities is achieved. This transforms the original video images and natural language instructions into a unified feature vector sequence that the model can recognize, eliminating the heterogeneity of dual-modal data and enabling accurate structured representation of welding visual scene information and text process instruction information. This lays a regular and unified feature input foundation for subsequent temporal merging, diversity pruning, dual-modal feature fusion, and adaptive optimization, avoiding subsequent feature processing failures and modal fusion misalignments caused by messy original data formats and mismatched feature dimensions. This ensures that all subsequent pruning and optimization steps are executed in an orderly and accurate manner.

[0010] In one optional implementation, the step of performing global temporal merging on the frame token sequence to obtain a visual feature sequence includes: Calculate the feature cosine similarity between two adjacent frame tokens in the frame token sequence; Frame tokens with a feature cosine similarity not lower than a preset similarity threshold are marked as static frame tokens, and frame tokens with a feature cosine similarity lower than the preset similarity threshold are marked as dynamic frame tokens. Average pooling is performed on each consecutive segment of the static frame token to obtain multiple compressed static frame tokens; The compressed static frame tokens and the dynamic frame tokens are concatenated and fused to obtain a visual feature sequence.

[0011] By accurately distinguishing between static and dynamic transient welding intervals in the welding process using inter-frame feature cosine similarity, the static frame tokens corresponding to welding backgrounds with no significant changes over long periods and uniform, stable welding are compressed and merged using average pooling. Short-term key dynamic frame tokens such as arc initiation, arc termination, droplet transfer, arc spatter, and welding torch posture adjustment are completely preserved without compression. This effectively solves the technical drawbacks of traditional fixed sliding window segmentation methods, which cannot adapt to the temporal changes in welding videos, easily disrupt the temporal continuity of welding operations, and have a large amount of redundant feature computation. While significantly compressing the number of redundant frame tokens in the temporal dimension, reducing basic computational overhead, and improving subsequent processing speed, the core transient dynamic details that determine welding quality during the welding process are completely preserved, achieving the dual goals of simplifying welding temporal features and not losing key dynamic information.

[0012] In one optional implementation, performing diversity-perceptual pruning on the visual feature sequence to obtain a diversity-pruned visual feature sequence includes: Calculate the cosine distance between any two different frame token features in the visual feature sequence, and construct a cosine distance matrix; Select one frame token feature from the visual feature sequence as the retained frame token feature and add it to the feature retention set; Use the unselected frame token features as the remaining frame token features; For each remaining frame token feature, based on the cosine distance matrix, query the cosine distance between the remaining frame token feature and each retained frame token feature in the feature retention set, and take the minimum value as the minimum cosine distance of the remaining frame token feature. Among all the remaining frame token features, the remaining frame token feature with the largest minimum cosine distance is selected as the retained frame token feature and added to the feature retention set; Jump to execute the step for each of the remaining frame token features until the number of retained frame token features reaches a preset threshold. By integrating all the retained frame token features in the feature retention set, a diverse pruned visual feature sequence is obtained.

[0013] This approach employs a diversity perception pruning strategy based on the feature cosine distance matrix combined with the farthest point iterative sampling strategy. It abandons the traditional single-mode approach of selecting features solely based on the high or low attention weights, and actively selects visual frame token features with significant semantic differences and sparse distribution. It forcibly retains long-tail detail features with low attention weights but crucial for welding defect identification, effectively avoiding the problem of the model over-focusing on strong saliency features such as bright arcs and bright molten pools while ignoring weakly saliency key details such as dim weld edges, micropores, and fine cracks. This completely eliminates the semantic blind spots in welding visual perception. On the basis of further reducing the number of visual features and reducing the computational power consumption of model inference, it comprehensively improves the model's perception and recognition accuracy of subtle defects and edge conditions in complex welding scenarios, and avoids the loss of key welding detail information due to feature pruning.

[0014] In one optional implementation, the step of performing attention weight filtering and pruning on the text feature sequence based on the diversity-pruned visual feature sequence to obtain the pruned text feature sequence includes: The pruning threshold of the text feature sequence is determined based on the number of frame token features in the diverse pruned visual feature sequence. Calculate the attention weight value of each text feature within the text feature sequence; Text features with attention weight values ​​lower than the pruning threshold are removed to obtain a pruned text feature sequence.

[0015] By matching the number of pruned visual features to set a specific pruning threshold for text features, and accurately quantifying the correlation between each text word and the core welding visual scene through cross-modal attention weights, redundant text features with no actual welding guidance significance and weak semantic correlation are accurately eliminated. Only core text instruction features that are strongly related to welding visual conditions and process control are retained. This achieves accurate adaptation between the scale of text features and the scale of visual features, and deep matching of dual-modal semantics. This reduces the amount of invalid computation in the dual-modal feature fusion stage, improves the efficiency and accuracy of feature fusion, and ensures that the retained text features can effectively guide the generation of welding robot actions. It avoids redundant text features interfering with the accuracy of action decoding, and achieves collaborative simplification and simultaneous quality and efficiency improvement of visual and text dual-modal features.

[0016] In one optional implementation, the process of fusing the diverse pruned visual feature sequence and the pruned text feature sequence, and then performing semantic-aware pruning on the fused bimodal fused feature sequence to obtain an adaptive pruned feature sequence, includes: The diverse pruned visual feature sequence and the pruned text feature sequence are fused to obtain a dual-modal fusion feature sequence. Calculate the semantic convergence score of the dual-modal fused feature sequence; The corresponding adaptive retention rate is determined based on the semantic convergence score; Based on the adaptive retention rate, hybrid selection pruning and key-value cache pruning are performed sequentially to obtain an adaptive pruning feature sequence.

[0017] The simplified visual and textual features are fused to form a unified dual-modal fusion feature. The semantic complexity and feature redundancy of the current welding condition are quantified by the semantic convergence score. The feature retention pruning intensity is dynamically and adaptively matched under different conditions. For simple welding conditions, the pruning intensity is automatically increased, which greatly saves inference computing power. For complex welding conditions such as irregular welds and strong arc light interference, the pruning intensity is automatically reduced, which retains sufficient feature semantic information. The computing resources are intelligently and dynamically allocated according to the complexity of the welding condition, which takes into account both the real-time performance of model inference and the integrity of semantic representation of complex conditions. At the same time, the dual optimization of hybrid selection pruning and key-value cache pruning is combined to complete adaptive simplification at both the feature level and the model cache level, which is suitable for the actual operation deployment requirements of industrial welding robots with low latency and high precision.

[0018] In one optional implementation, the hybrid selection pruning specifically involves: The total number of features in the dual-modal fusion feature sequence is counted, and the target number of retained features is calculated and rounded based on the adaptive retention rate. The number of target retained features is divided into the number of relevance selections N and the number of diversity completions M; Calculate the attention weight of each feature in the dual-modal fusion feature sequence and sort them in descending order. Select the top N features to form a subset of relevant features. The relevant feature subset is removed from the dual-modal fusion feature sequence to obtain the remaining features to be completed. For each of the remaining features to be completed, calculate the cosine distance between the remaining feature to be completed and all features in the subset of related features, and take the minimum value as the minimum feature distance of the remaining feature to be completed. Sort the features by minimum feature distance from largest to smallest, and select the top M features to form a diverse feature subset; The correlation feature subset and the diversity feature subset are merged to obtain the hybrid pruned feature sequence.

[0019] A hybrid pruning model combining relevance screening and diversity completion is adopted. It not only selects core features that are strongly related to the welding action decoding task through attention weights to ensure the basic accuracy of the welding action control, but also completes niche semantic details that are easily ignored by the attention mechanism through feature cosine distance sampling. This makes up for the shortcomings of single attention screening, which only focuses on salient features and misses edge welding details. It takes into account the task-adaptive relevance of dual-modal fusion features and the diversity of scene semantic coverage. While continuously compressing the number of features and reducing the inference computation pressure, it comprehensively covers the semantic information of the entire welding scene, avoiding problems such as perception bias, action decoding deviation and welding control illusion. This further enhances the accuracy and stability of action generation under complex welding conditions.

[0020] In one optional implementation, the key-value cache pruning specifically involves: Establish the corresponding index relationship between each feature in the dual-modal fusion feature sequence and the Key and Value entries in the key-value cache, and record the original index number of each feature; Extract the original index numbers of all retained features from the hybrid pruned feature sequence to form a retained index set; Iterate through all Key and Value entries in the key-value cache, and filter out valid Keys and Values ​​that match the retention features according to the retention index set; Delete redundant Key and Value entries in the key-value cache that do not fall into the reserved index set; By integrating the valid Key and valid Value with the corresponding hybrid pruned feature sequence, an adaptive pruned feature sequence is obtained.

[0021] Based on feature hybrid pruning, precise pruning of the Transformer model key-value cache is performed simultaneously. A one-to-one correspondence between features and cache entries is established through the original feature index number. Invalid key and value cache entries corresponding to pruned redundant features are precisely deleted, and only core cache data matching valid features are retained. There is no need to retrain or fine-tune the original welding video model. It is a plug-and-play lightweight optimization method in the inference stage, which significantly reduces the memory usage and data read / write latency during model inference. It improves the response speed of welding robot motion generation from the hardware computing power level, adapts to the high real-time welding operation requirements in industrial sites, and ensures that the cached data and pruned features are completely matched without affecting the model decoding inference logic and welding motion control accuracy. It achieves a balance between lightweight deployment and high-precision control.

[0022] In a second aspect, the present invention provides a robot welding motion generation system, the device comprising: The visual text preprocessing module is used to acquire continuous video frames of the target welding operation scene, extract and project features, generate a frame token sequence, respond to the input welding task text command, segment and project, and generate a text feature sequence. A global temporal merging module is used to perform global temporal merging on the frame token sequence to obtain a visual feature sequence; A diversity-perceived pruning module is used to perform diversity-perceived pruning on the visual feature sequence to obtain a diversity-pruned visual feature sequence. The text feature attention pruning module is used to perform attention weight filtering and pruning on the text feature sequence based on the diversity pruning visual feature sequence to obtain the pruned text feature sequence; The semantic adaptive decoding and pruning module is used to fuse the diverse pruning visual feature sequence and the pruning text feature sequence, and to perform semantic-aware pruning on the fused dual-modal fused feature sequence to obtain an adaptive pruning feature sequence. The welding action decoding output module is used to perform action semantic decoding on the adaptive pruning feature sequence, generate and output welding robot action control commands.

[0023] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the robot welding action generation method of the first aspect or any corresponding embodiment described above. Attached Figure Description

[0024] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of the first process of a robot welding action generation method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a second process for generating robot welding actions according to an embodiment of the present invention; Figure 3 This is a structural block diagram of a robot welding motion generation system according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating the overall architecture of the robot welding motion generation system according to an embodiment of the present invention. Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0028] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0029] This invention provides a method, system, and electronic device for generating robot welding actions. In complex industrial welding scenarios characterized by high dynamics and numerous interferences, traditional vision-guided welding solutions, employing fixed-parameter feature processing strategies, struggle to retain key details while eliminating redundant information. This makes it difficult to adapt to the dynamic complexity of welding conditions, hindering the balance between computational efficiency and motion generation accuracy. This invention addresses this by acquiring continuous video frames of the target welding operation scenario, extracting and projecting features to generate a frame token sequence. Simultaneously, it segments and projects the input welding task text commands to generate a text feature sequence. Global temporal merging of the frame token sequence removes temporal redundancy, yielding a visual feature sequence. Based on this visual feature sequence, diversity perception pruning is performed to select and retain sparse, crucial semantic features in the welding scenario, resulting in a diversity-pruned visual feature sequence. This visual feature sequence is then used to perform attention weight pruning on the text feature sequence, eliminating irrelevant redundant text features to achieve dual-modal feature co-adaptation. Finally, the two types of pruned features are fused, and semantic perception pruning is applied to the dual-modal fused feature sequence. The feature retention ratio is adaptively adjusted based on the dynamic characteristics of the scenario, and the welding robot motion control commands are output via motion semantic decoding.

[0030] This invention, through global temporal merging and diversity-aware pruning, simultaneously eliminates redundant information and retains core visual details from both temporal and spatial dimensions. This fundamentally avoids the problems of wasted computing power or loss of key features caused by fixed parameter strategies, effectively solving the technical defects of existing technologies that cannot adaptively process visual features. Simultaneously, through the collaborative optimization of text feature attention-based pruning and bimodal semantic-aware pruning, it achieves precise adaptation of text commands to visual scenes and dynamic control of feature processing, balancing computational efficiency with the semantic integrity of action generation. Compared to traditional methods, this solution significantly improves the real-time performance and control accuracy of welding action generation, accurately depicting the action detail requirements in complex welding scenarios, and providing efficient and reliable technical support for the autonomous operation of welding robots in unstructured industrial environments.

[0031] According to an embodiment of the present invention, a method for generating robot welding actions is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0032] This embodiment provides a method for generating robot welding actions. Figure 1 This is a flowchart of a robot welding action generation method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps: Step S101: Collect continuous video frames of the target welding operation scene and perform feature extraction and projection to generate a frame token sequence. Response to the input welding task text command, segment and project the text to generate a text feature sequence.

[0033] It should be noted that the target welding operation scenario is a welding operation environment that includes the weld seam, molten pool, welding torch, workpiece, and workshop background; continuous video frames are a time-series image sequence of the entire welding process acquired and recorded by an industrial camera at a fixed frame rate; feature extraction and projection is the process of extracting high-dimensional semantic features from video frames and mapping them to a unified feature space to generate a structured representation; the frame token sequence is a structured feature vector sequence aligned with the time sequence, formed by projecting the spatial features of the video frames; the welding task text instructions are text instruction information describing the weld seam type, welding path, welding torch posture, and welding process parameters; word segmentation is the process of dividing the continuous text instructions of the welding task into discrete semantic words; and the text feature sequence is a structured semantic feature vector sequence generated by feature projection of text words.

[0034] In this embodiment of the invention, the welding robot acquires continuous video frames of the target welding operation scene through its onboard industrial vision sensor. After performing basic preprocessing on the video frames, it extracts frame-level features using a pre-trained visual feature extraction network and projects them to generate a frame token sequence. At the same time, it receives externally input welding task text instructions, segments them by a word segmenter, and then performs feature encoding and projection through a text encoder to generate a text feature sequence.

[0035] Step S102: Perform global temporal merging on the frame token sequence to obtain the visual feature sequence.

[0036] It should be noted that global temporal merging is a temporal feature processing method that compresses static interval features of the frame token sequence, fully preserves dynamic interval features, and combines global segmentation optimization; the visual feature sequence is a simplified frame token sequence that retains both the common features of the static background and the detailed features of the welding dynamic event after global temporal merging.

[0037] In this embodiment of the invention, the feature similarity between adjacent frame tokens in the frame token sequence is calculated. The frame tokens are then time-sequentially grouped and merged based on the similarity results. Continuous frame tokens that are in a stable welding stage and have high feature similarity are aggregated to remove temporal redundancy. At the same time, key frame tokens whose scene states have changed are retained, and finally, a visual feature sequence with temporal alignment and duplicate information removed is obtained.

[0038] Step S103: Perform diversity perception pruning on the visual feature sequence to obtain a diversity-pruned visual feature sequence.

[0039] It should be noted that diversity perception pruning is a feature selection method that relies on feature space distance sampling and actively retains long-tail features with low attention weight but unique semantics; the diversity pruning visual feature sequence is a visual feature sequence that is evenly distributed in the feature space and completely covers the full semantics of the welding scene after diversity perception pruning.

[0040] In this embodiment of the invention, the diversity measure of each feature vector in the visual feature sequence is calculated, such as the cosine similarity between features, the distribution distance in the feature space, or the information entropy. Based on the preset diversity screening threshold, redundant features with semantic homogeneity and low information contribution are eliminated, while features with significant differences in distribution in the feature space and carrying key welding scene information are retained, and finally, a diversity-pruned visual feature sequence is obtained.

[0041] Step S104: Based on the diversity pruning visual feature sequence, the text feature sequence is pruned by attention weight filtering to obtain the pruned text feature sequence.

[0042] It should be noted that attention weight pruning is a pruning method that selects text features related to the welding task based on the cross-modal attention correlation between text features and visual features; the pruned text feature sequence is a concise text feature sequence that removes irrelevant and redundant words and retains only those semantically strongly related to the welding scene.

[0043] In this embodiment of the invention, cross-modal attention weights are calculated based on the diverse pruned visual feature sequences and text feature sequences. These weights characterize the degree of correlation between each text feature and the key visual feature. According to the preset attention weight threshold, text features that are strongly related to the visual information of the welding task are selected, and redundant text features with low correlation and no substantial guiding significance are removed. Finally, a pruned text feature sequence that is adapted to the visual feature sequence is obtained.

[0044] Step S105: Fuse the diverse pruned visual feature sequence and the pruned text feature sequence, and perform semantic-aware pruning on the fused dual-modal fused feature sequence to obtain the adaptive pruned feature sequence.

[0045] It should be noted that the dual-modal fusion feature sequence is a feature sequence that aligns and merges the dimensions of the visual feature sequence and the text feature sequence, while carrying both visual semantics and text instruction semantics; semantic-aware pruning is an adaptive pruning mechanism that dynamically adjusts the pruning intensity based on the semantic convergence score during the large model decoding stage, taking into account both feature relevance and diversity; and adaptive pruning feature sequence is a lightweight feature sequence that, after semantic-aware pruning, adaptively adjusts the feature size according to the complexity of the welding condition and adapts to the model's inference computing power.

[0046] In this embodiment of the invention, diverse pruned visual feature sequences and pruned text feature sequences are fused across modalities through feature concatenation or weighted fusion to obtain a unified-dimensional bimodal fused feature sequence. The semantic convergence score of the bimodal fused feature sequence is calculated. This score is quantified based on feature similarity and distribution clustering, and is used to characterize the semantic clustering degree and information redundancy level of the current feature sequence. According to a preset score-retention rate mapping relationship, the corresponding adaptive retention rate is determined by the semantic convergence score. A higher semantic convergence score indicates higher feature redundancy, and a lower retention rate is set accordingly. Based on the adaptive retention rate, hybrid selection pruning is sequentially performed on the bimodal fused feature sequence to filter and retain core semantic features. Simultaneously, the key-value cache is pruned to remove cache entries corresponding to the pruned features, ultimately resulting in a lightweight adaptive pruned feature sequence.

[0047] Step S106: Perform action semantic decoding on the adaptive pruning feature sequence to generate and output the welding robot action control instructions.

[0048] Action semantic decoding is the decoding process that maps bimodal fusion features to the robot motion control space and outputs motion parameters; welding robot motion control commands are standardized control commands that include parameters such as welding torch posture, welding speed, wire feed speed, welding current and voltage, and can directly drive robot operation.

[0049] In this embodiment of the invention, the adaptive pruning feature sequence is input into the pre-trained action semantic decoder. The decoder maps the welding scene semantic information carried in the fused dual-modal features to the motion control space of the welding robot, generating a continuous action sequence including welding trajectory, motion speed, welding torch posture and welding parameters. The action sequence is then converted into a standard control instruction format that can be recognized by the robot controller, and finally output to the control unit of the welding robot to drive the robot to perform the corresponding welding operation.

[0050] The robot welding motion generation method provided in this embodiment generates standardized frame token sequences and text feature sequences through dual-modal parallel acquisition and encoding, providing a standardized and complete input foundation for subsequent feature processing. By leveraging global temporal merging of the frame token sequences, it can eliminate temporal redundant features in stable welding stages and retain key frames of scene changes, effectively reducing feature size and avoiding interference from temporal repetition information, thus initially improving feature processing efficiency. Diversity-aware pruning can filter out semantically homogenized and low-contribution redundant information in visual features, accurately retaining core detail features of the welding scene and ensuring the effectiveness and representativeness of visual features. Text feature filtering and pruning based on cross-modal attention weights enables collaborative adaptation between text features and key visual features, eliminating irrelevant redundant text features, reducing computational overhead caused by invalid features, and improving dual-modal feature matching. The system employs a hierarchical, adaptive feature processing logic. By combining bimodal feature fusion with semantic convergence scores to determine the adaptive retention rate, and then sequentially performing hybrid selection pruning and key-value cache pruning, the feature retention scale can be dynamically adjusted based on the degree of semantic redundancy. This maximizes feature simplification while fully preserving core semantic information, achieving a balance between feature lightweighting and information integrity. Finally, standardized robot control commands are generated through action semantic decoding, accurately mapping bimodal semantic information to the robot's action space. This ensures the accuracy and executability of welding action output. The overall system overcomes the limitations of existing technologies in adaptively processing visual features through hierarchical, adaptive, and end-to-end feature processing logic. It significantly improves the real-time performance of welding operations while maintaining precise motion control, effectively balancing operational efficiency and control performance. This significantly enhances the robot's adaptive operation capability and reliability in complex welding scenarios.

[0051] This embodiment provides a method for generating robot welding actions. Figure 2 This is a flowchart of a robot welding action generation method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps: Step S201: Collect continuous video frames of the target welding operation scene and perform feature extraction and projection to generate a frame token sequence. Response to the input welding task text command, segment and project the text to generate a text feature sequence.

[0052] Specifically, step S201 includes: Step S2011: Acquire continuous video frames of the target welding operation scene.

[0053] In this embodiment of the invention, the welding robot acquires high-resolution continuous video frames of the target welding operation scene through an industrial camera. The video stream contains a large amount of spatiotemporal data, including both relatively static backgrounds (such as workpiece holders and workshop environment) and highly dynamic welding core areas (such as molten pool flow, arc spatter, and welding torch trajectory adjustment), fully covering the visual information of the entire welding operation process.

[0054] Step S2012: Extract features from consecutive video frames using a visual encoder to obtain spatial features.

[0055] In this embodiment of the invention, the acquired continuous video frames are input into a pre-trained visual encoder. The encoder extracts spatial features from the video frame by frame through a multi-layer feature extraction structure. These features include general features of the static background area and dynamic detail features of the welding core area, resulting in frame-level spatial features corresponding to each video frame.

[0056] Step S2013: Project the spatial features to generate a frame token sequence.

[0057] In this embodiment of the invention, the spatial features of each frame are mapped to a unified feature dimension through a linear projection layer to generate a frame token sequence that is temporally aligned with the video frames. Each frame token corresponds to a structured feature representation of a video frame.

[0058] Step S2014: In response to the input welding task text instruction, the text is segmented by a word segmenter to obtain a text word sequence.

[0059] In this embodiment of the invention, welding task text instructions (such as weld type, welding path requirements, target welding torch posture, etc.) are received from external input. The text instructions are segmented by a preset word segmenter to divide the continuous text into discrete text word sequences and extract key semantic units directly related to the welding task.

[0060] Step S2015: Project the text word sequence to generate a text feature sequence.

[0061] In this embodiment of the invention, the text word sequence is input into the text embedding layer for feature projection, and the discrete words are mapped to the feature space of the same dimension as the frame token to generate a text feature sequence. Each text feature corresponds to the semantic representation of a word, thereby realizing the structured encoding of welding task text instructions.

[0062] The visual encoder is a convolutional neural network or Transformer visual model used to extract spatial features from video frames; spatial features are high-dimensional image semantic feature vectors representing weld contours, molten pool states, and workpiece positions in video frames; the word segmenter is a dedicated text processing module that adapts to welding terminology and performs precise word segmentation of task text instructions; the text word sequence is a sequence of discrete text semantic units obtained after segmentation by the word segmenter.

[0063] Step S202: Perform global temporal merging on the frame token sequence to obtain the visual feature sequence.

[0064] Specifically, step S202 includes: Step S2021: Calculate the feature cosine similarity between two adjacent frame tokens in the frame token sequence.

[0065] In this embodiment of the invention, the frame token sequence is traversed, and the cosine similarity of adjacent frame tokens in the feature space is calculated sequentially according to time. This similarity is used to measure the consistency of feature distribution between adjacent frames, thereby reflecting the degree of dynamic change in the welding scene. The higher the similarity, the more stable the inter-frame features are, corresponding to the steady period of the welding process (such as the uniform linear welding segment, the static background without change stage); the lower the similarity, the more drastic the inter-frame feature changes are, corresponding to the abrupt change period of the welding process (such as arc initiation, arc termination, droplet transfer, arc spatter, welding torch trajectory adjustment or fault jamming stage).

[0066] Step S2022: Mark the frame tokens with feature cosine similarity not lower than the preset similarity threshold as static frame tokens, and mark the frame tokens with feature cosine similarity lower than the preset similarity threshold as dynamic frame tokens.

[0067] In this embodiment of the invention, frame tokens with a feature cosine similarity not lower than a preset similarity threshold are marked as static frame tokens, and frame tokens with a feature cosine similarity lower than the preset similarity threshold are marked as dynamic frame tokens. Based on the merging profit function and dynamic programming algorithm, the frame token sequence is divided into a static compression interval and a dynamic retention interval. Constructing the merged revenue function: Defining the merged revenue function Quantization and merging of a continuous static frame interval The computational resources that can be saved are expressed in the following function form:

[0068] in, static frame interval The number of frame tokens included (i.e., the interval length). For interval The average cosine similarity of all adjacent static frame tokens (reflecting scene stability). For interval The number of dynamic frame tokens (dynamic events) interspersed within the interval. The logic of this function is: the longer the interval, the higher the stability between frames, the fewer the interspersed dynamic events, the higher the benefit of merging the interval, and the more computing power is saved.

[0069] To maximize the cumulative merged revenue of the entire frame token sequence, a dynamic programming algorithm is constructed: Initialize the state array With backtracking array : Indicates processing up to the th The maximum cumulative merged profit that can be obtained when the number of frames is Token; Record number The optimal split point corresponding to each state.

[0070] State transition calculation: Traverse the frame token sequence, for each position Backtracking forward through all possible starting points of the interval ( If the interval For static frame token intervals (i.e., average similarity within the interval) Then calculate its consolidated income. And update the state transition equation:

[0071] Backtracking to determine the optimal segment: After traversal, start from the end of the sequence. Backtracking The array is used to find all the optimal time segmentation points in turn, and the frame token sequence is accurately divided into two types of intervals: static compression interval (corresponding to long-span stable scenarios with high merging benefits, such as static backgrounds and uniform welding stages of tens of seconds) and dynamic retention interval (corresponding to the stage where dynamic frame tokens are located and the features change drastically). This solves the defect of traditional sliding window technology that easily cuts off the continuity of the welding process.

[0072] Step S2023: Perform average pooling on each consecutive static frame token segment to obtain multiple compressed static frame tokens.

[0073] In this embodiment of the invention, for each static compression interval obtained by dynamic programming, a time-dimensional average pooling operation is performed. The average value of all frame tokens within the interval is calculated according to the feature dimension to generate a single compressed static frame token. For example, hundreds of frame tokens corresponding to a stable welding background image lasting tens of seconds are compressed into 1-2 compressed static frame tokens. This token aggregates the common features of all static frames within the interval, which not only fully preserves the semantic information of the background but also significantly reduces the feature volume of static redundant information. For the frame tokens of the dynamically preserved interval, no pooling compression is performed, and all the words of the original sequence are fully preserved. This ensures that the feature information of key dynamic events such as millisecond-level droplet transition and spatter anomalies is not lost. This achieves accurate physical modeling of the stable characteristics of video segments and solves the memory explosion problem caused by excessive data volume in traditional models in long-term welding tasks.

[0074] Step S2024: The compressed static frame tokens and dynamic frame tokens are concatenated and fused to obtain a visual feature sequence.

[0075] In this embodiment of the invention, the compressed static frame token and dynamic frame token are rearranged and spliced ​​according to the original welding scene time sequence. The dynamic frame token is retained in its entirety without compression, ensuring that the feature information of key dynamic events such as millisecond-level droplet transition and spatter anomalies is not lost. The resulting visual feature sequence after splicing is significantly reduced in length, while taking into account both the semantic coverage of the static background and the preservation of details of the dynamic core events. This greatly reduces the amount of unnecessary data at the input end and solves the memory explosion problem caused by excessive data in traditional models in long-term welding tasks.

[0076] Feature cosine similarity is a quantitative indicator that measures the consistency of feature vectors of two frame tokens in the feature space, with a value range of [-1, 1]. The higher the value, the higher the feature similarity. The preset similarity threshold is a pre-set cosine similarity threshold used to distinguish between static frame tokens and dynamic frame tokens. Static frame tokens are those with a feature cosine similarity not lower than the preset threshold, corresponding to the stable and uniform welding stage and scenes with no change in the static background. Dynamic frame tokens are those with a feature cosine similarity lower than the preset threshold, corresponding to scenes such as arc initiation, arc termination, droplet transfer, and sudden changes in welding torch posture. Average pooling is an operation method that calculates the average value of continuous static frame tokens according to the feature dimension, compressing the features of multiple frames into a single feature representation. Compressed static frame tokens are simplified feature vectors generated after aggregating common features of static intervals through average pooling. Splicing and fusion is an integration method that rearranges and combines compressed static frame tokens and dynamic frame tokens according to the original video temporal sequence to form a temporally coherent feature sequence.

[0077] Step S203: Perform diversity perception pruning on the visual feature sequence to obtain a diversity-pruned visual feature sequence.

[0078] It should be noted that this step adopts a sampling strategy based on the farthest point of diversity coverage to solve the problem that traditional pruning is easily dominated by high-brightness arc light and ignores the characteristics of small defects in the weld. Specifically, the above step S203 includes: Step S2031: Calculate the cosine distance between any two different frame token features in the visual feature sequence and construct a cosine distance matrix.

[0079] In this embodiment of the invention, each frame token in the visual feature sequence is regarded as an independent feature vector. The cosine similarity between each pair of feature vectors is calculated and converted into cosine distance (the larger the distance, the more significant the semantic difference between the two features). All distance values ​​are then organized into a two-dimensional cosine distance matrix. Each element in the matrix corresponds to the degree of semantic difference between a pair of frame tokens, providing a basis for subsequent diversity sampling and avoiding the need to rely solely on feature amplitudes such as arc brightness for screening.

[0080] Step S2032: Select a frame token feature from the visual feature sequence as the retained frame token feature and add it to the feature retention set.

[0081] In this embodiment of the invention, an initial frame token is selected and added to the feature retention set as the sampling starting point. The initial frame token can be randomly selected or a frame token with a high proportion of weld seam area can be selected first. Without complex calculations, the diversity sampling process can be started.

[0082] Step S2033: Use the unselected frame token features as the remaining frame token features.

[0083] In this embodiment of the invention, all frame tokens in the visual feature sequence that are not added to the feature retention set are marked as remaining frame token features and used as candidate objects for subsequent iterative sampling. No additional processing is performed, and they directly enter the next round of screening.

[0084] Step S2034: For each remaining frame token feature, based on the cosine distance matrix, query the cosine distance between the remaining frame token feature and each retained frame token feature in the feature retention set, and take the minimum value as the minimum cosine distance of the remaining frame token feature.

[0085] In this embodiment of the invention, each remaining frame token is traversed, and its distance value with all selected frame tokens in the feature retention set is queried from the cosine distance matrix. The minimum value among them is taken as the "minimum cosine distance" of the remaining feature. This value reflects the "least similarity" between the feature and the currently selected features. The larger the value, the more unique the feature is and the more it can supplement the overall semantic diversity.

[0086] Step S2035: Among all remaining frame token features, select the remaining frame token feature with the largest minimum cosine distance as the retained frame token feature and add it to the feature retention set.

[0087] In this embodiment of the invention, the feature with the largest "minimum cosine distance" is selected from the remaining frame tokens and added to the feature retention set. That is, the feature with the largest difference from the currently selected feature is selected each time. This forces the retention of long-tail visual words (such as weld edges, micro-cracks, pores, etc.) that are sparsely distributed but semantically unique in the feature space, avoiding the dominance of strong salient features such as bright arc light in the selection process and preventing the model from generating visual blind spots.

[0088] Step S2036: Jump to execute the steps for each remaining frame token feature until the number of retained frame token features reaches a preset threshold.

[0089] In this embodiment of the invention, the iterative process of steps S2033 to S2035 is repeated, with each iteration adding a unique feature until the number of frame tokens in the feature retention set reaches a preset upper limit. This upper limit can be configured in advance according to hardware computing power and task accuracy requirements to ensure that the system processing budget is not exceeded and to avoid the risk of computing power overload.

[0090] Step S2037: Integrate all retained frame token features in the feature retention set to obtain a diverse pruned visual feature sequence.

[0091] In this embodiment of the invention, the frame tokens in the feature retention set are rearranged according to their chronological order in the original visual feature sequence to obtain a diversity-pruned visual feature sequence. This sequence not only simplifies the number of features but also fully covers the complete semantics of the welding scene, ensuring that weld details are not lost.

[0092] Cosine distance is an indicator that quantifies the semantic difference between two feature vectors in the feature space. It is obtained by subtracting the cosine similarity from 1, and its value ranges from [0,2]. The larger the value, the more significant the semantic difference. The cosine distance matrix is ​​a two-dimensional data table composed of the pairwise cosine distances between any two frame tokens in the visual feature sequence. The feature retention set is a set used to store the features of the frame tokens that have been determined to be retained during the iterative screening process. The remaining frame token features are candidate frame tokens in the visual feature sequence that have not been included in the feature retention set and are waiting for diversity screening. The minimum cosine distance is the minimum value among the cosine distances between a single remaining frame token and all retained features in the feature retention set, representing the semantic difference between this feature and the retained features. The preset quantity threshold is the upper limit of the total number of visual features retained after diversity pruning, which is preset according to the hardware computing power and welding accuracy.

[0093] Step S204: Based on the diversity pruning visual feature sequence, the text feature sequence is pruned by attention weight filtering to obtain the pruned text feature sequence.

[0094] This step achieves collaborative adaptation between text features and key visual features, eliminates irrelevant text redundancy, and reduces unnecessary computational overhead. Specifically, step S204 includes: Step S2041: Determine the pruning threshold of the text feature sequence based on the number of frame token features in the diversity pruning visual feature sequence.

[0095] In this embodiment of the invention, the target number of text features to be retained is first determined based on the number of visual features after diversity pruning (for example, the number of text features to be retained can be set to 1 or 2 times the number of visual features to achieve the adaptation of the number of text and visual features). Then, the pruning threshold is determined accordingly. For example, the quantile value of the attention weight of the text features is set as the threshold, and only the top N text features with the highest weight are retained. The scale of the text features can be controlled without complicated calculations.

[0096] Step S2042: Calculate the attention weight value of each text feature in the text feature sequence.

[0097] In this embodiment of the invention, the diverse pruning visual feature sequence is assumed to contain several visual feature vectors, and the text feature sequence contains the first... The text feature vectors are Calculate the cross-modal attention association score between each text feature and all visual features. The formula for calculating the attention weight of a single text feature is as follows:

[0098] In the formula, For the first Attention weight values ​​corresponding to each text feature The token visual feature vectors of each frame in the diverse pruned visual feature sequence are used; the maximum value of the correlation score between the text feature and all visual features is taken as the final attention weight of the text feature. The weight value directly represents the degree of correlation between the current text semantics and the key features of the welding visual scene. The higher the correlation, the larger the weight value.

[0099] Step S2043: Remove text features whose attention weight values ​​are lower than the pruning threshold to obtain the pruned text feature sequence.

[0100] In this embodiment of the invention, based on the pruning threshold determined in step S2041, text features with attention weights below the threshold are removed, and only high-weight text features are retained, resulting in a pruned text feature sequence. This sequence eliminates text redundancy irrelevant to the current welding task, improves the matching degree between text features and visual key features, reduces invalid computation, and ensures that text instructions can effectively guide the generation of subsequent welding actions.

[0101] The pruning threshold is set based on the number of visual features pruned for diversity, and is used as the critical value of attention weight to filter effective text features. The attention weight value is a quantitative value of the cross-modal semantic correlation between text features and visual features. The higher the value, the stronger the guidance correlation of text words to the welding scene.

[0102] Step S205: Fuse the diverse pruned visual feature sequence and the pruned text feature sequence, and perform semantic-aware pruning on the fused dual-modal fused feature sequence to obtain the adaptive pruned feature sequence.

[0103] This step is the semantic adaptive decoding pruning process within the large model. It achieves adaptive pruning rate adjustment by dynamically calculating the semantic convergence score, and combines hybrid selection pruning and key-value cache pruning to reduce inference overhead. Specifically, step S205 includes: Step S2051: The diverse pruned visual feature sequence and the pruned text feature sequence are fused to obtain a dual-modal fused feature sequence.

[0104] In this embodiment of the invention, after aligning the diverse pruning visual feature sequence and the pruning text feature sequence according to the feature dimension, they are fused into a dual-modal fusion feature sequence with a unified dimension by feature splicing; or the two types of features are weighted and fused according to a preset weight, so that each fused feature simultaneously carries the visual semantic information and text instruction information of the welding scene.

[0105] Step S2052: Calculate the semantic convergence score of the dual-modal fusion feature sequence.

[0106] In this embodiment of the invention, the semantic convergence score is obtained by weighted fusion of the attention deterministic signal and the representation stability signal, and is quantified and solved using standardized calculation formulas: 1. Calculate attention deterministic signals Suppose that the dual-modal fusion feature sequence contains a total of The model's decoding layer outputs the attention probability distribution corresponding to each feature, which is a fused feature. ,satisfy .

[0107] (1) Information entropy of attention distribution Information entropy The smaller the entropy value, the more focused the attention and the more certain the semantic focus; the larger the entropy value, the more divergent the attention and the higher the semantic uncertainty.

[0108] (2) Top-K cumulative attention probability Select the one with the highest attention probability Calculate the cumulative probability based on the given features: in, The attention probabilities are sorted in descending order. The higher the value, the higher the concentration of key features and the stronger the certainty of attention.

[0109] (3) Normalization yields the attention deterministic signal After normalizing the information entropy and cumulative probability to the [0,1] interval, they are weighted and fused:

[0110] For fixed weighting coefficients, The normalized information entropy, This represents the cumulative probability after normalization. The closer it is to 1, the higher the certainty of attention.

[0111] 2. Calculate the stability signal (1) Interlayer feature representation bias Let the current layer be the The fusion features are The corresponding positional feature of the previous layer is Calculate the average cosine distance between floors:

[0112] The smaller the value, the smaller the variation in features between adjacent network layers, and the more stable the representation.

[0113] (2) Average similarity of features within the layer Calculate the average cosine similarity between all pairwise features of the current layer: The larger the value, the more severe the homogenization of semantic features within the layer, and the higher the feature redundancy.

[0114] (3) Normalization yields a signal characterizing stability. After normalizing inter-layer bias and intra-layer similarity to [0,1], the layers are fused: For fixed weighting coefficients, For the corresponding normalized values; The closer it is to 1, the more stable the feature representation and the stronger the semantic redundancy.

[0115] 3. The semantic convergence score is obtained by fusion. The two signals are weighted and fused to obtain the final semantic convergence score. :

[0116] in For preset fixed weights and satisfying ; The value range is [0,1]. The higher the score, the more convergent the semantics of the dual-modal fusion feature sequence and the higher the feature redundancy, providing a quantitative basis for subsequent adaptive retention rate matching.

[0117] Step S2053: Determine the corresponding adaptive retention rate based on the semantic convergence score.

[0118] In this embodiment of the invention, a preset score-retention rate mapping relationship is established. The higher the semantic convergence score, the simpler the current welding condition (such as standard weld seam tracking) and the higher the semantic redundancy. The corresponding feature retention rate is lower (the pruning is more aggressive). Conversely, the lower the score, the more complex the condition (such as untaught corners or strong arc light interference) and the higher the semantic uncertainty. The corresponding retention rate is higher (the pruning is more conservative).

[0119] This can be achieved through segmented mapping or linear mapping: for example, when the score is below the low threshold (e.g., 0.3), the retention rate is set to 0.8~1.0; when the score is in the middle range (0.3~0.7), the retention rate is set to 0.5~0.8; when the score is above the high threshold (e.g., 0.7), the retention rate is set to 0.3~0.5, thus automatically adjusting the pruning intensity according to the complexity of the working conditions.

[0120] Step S2054: Based on the adaptive retention rate, perform hybrid selection pruning and key-value cache pruning sequentially to obtain the adaptive pruning feature sequence.

[0121] This step first filters features through hybrid selection pruning, and then simultaneously prunes the key-value cache. In some optional implementations, hybrid selection pruning specifically involves: Step a1: Count the total number of features in the dual-modal fusion feature sequence, calculate and round the target number of retained features based on the adaptive retention rate.

[0122] In this embodiment of the invention, the total number of features in the dual-modal fusion feature sequence is set to be... The adaptive retention rate obtained from semantic convergence score matching is The formula for calculating the number of features to be retained is:

[0123] in, This indicates a floor operation to ensure that the calculated target number of retained features is a positive integer. By multiplying the total number of features by the adaptive retention rate and then rounding, the total number of features to be retained is accurately determined, providing a benchmark value for the subsequent division of the number of relevance selections and the number of diversity completions.

[0124] Step a2: Divide the number of target retained features into the number of relevance selections N and the number of diversity completions M.

[0125] In this embodiment of the invention, the number of target retentions is allocated according to a preset ratio, for example, N accounts for 70% of the target number and M accounts for 30%, or it is dynamically adjusted according to the current decoding task; N is used to retain features that are strongly related to the current query, and M is used to supplement long-tail semantic features that are ignored by the attention mechanism, taking into account both query relevance and semantic integrity.

[0126] Step a3: Calculate the attention weight of each feature in the dual-modal fusion feature sequence and sort them in descending order. Select the top N features to form a subset of relevant features.

[0127] In this embodiment of the invention, let the query vector at the decoding end be... The first in the dual-modal fusion feature sequence The fused feature vectors are First, calculate the original attention score between the individual fused feature and the query vector: The standardized attention weights are then obtained by solving the problem using Softmax normalization.

[0128] in, This represents the total number of features in the dual-modal fusion feature sequence. For the first Each fusion feature corresponds to an attention weight; the larger the weight value, the higher the semantic relevance of the fusion feature to the current welding task decoding query. All fusion features are then assigned attention weights. Arrange the features in descending order from largest to smallest, select the top N features, and combine them to form a subset of relevant features.

[0129] Step a4: Remove the relevant feature subset from the dual-modal fusion feature sequence to obtain the remaining features to be completed.

[0130] In this embodiment of the invention, the selected relevant features are removed from the original dual-modal fusion feature sequence, and the remaining features are the remaining features to be completed. These features are mostly flat-tailed distribution features that are not focused on by the attention mechanism, such as the weld edge in a dimly lit area, small pores and other details that are easily covered by arc light.

[0131] Step a5: For each remaining feature to be completed, calculate the cosine distance between the remaining feature to be completed and all features in the subset of related features, and take the minimum value as the minimum feature distance of the remaining feature to be completed.

[0132] In this embodiment of the invention, for each remaining feature to be completed, all features within the subset of relevant features are traversed, the cosine distance is calculated, and the minimum value is taken as the minimum feature distance of that feature. The larger the value, the more significant the semantic difference between that feature and the selected relevant features, and the higher the value of supplementing diversity.

[0133] Step a6: Sort the features by minimum feature distance from largest to smallest, and select the top M features to form a diversity feature subset.

[0134] In this embodiment of the invention, the remaining features to be completed are arranged in descending order of minimum feature distance, and the top M features are selected as a subset of diverse features. These features are semantically unique background or detailed information, which can prevent the model from creating visual blind spots due to excessive focus on bright arc light and ensure that the small defect features of the weld are not lost.

[0135] Step a7: Merge the relevant feature subset and the diversity feature subset to obtain the hybrid pruned feature sequence.

[0136] In this embodiment of the invention, the subset of relevant features and the subset of diversity features are rearranged according to the time / position order of the original sequence to obtain a hybrid pruned feature sequence. This sequence retains the core task-related features and supplements the long-tail semantic features that are easily ignored, achieving a balance between semantic integrity and feature simplification.

[0137] In some optional implementations, key-value cache pruning specifically involves: Step b1: Establish the corresponding index relationship between each feature in the dual-modal fusion feature sequence and the Key and Value entries in the key-value cache, and record the original index number of each feature.

[0138] In this embodiment of the invention, an original index number is assigned to each feature in the dual-modal fusion feature sequence, and a one-to-one mapping relationship is established with the corresponding Key and Value entries in the Transformer key-value cache (KV-Cache), providing a basis for subsequent cache filtering.

[0139] Step b2: Extract the original index numbers of all retained features in the feature sequence after hybrid pruning to form a retained index set.

[0140] In this embodiment of the invention, all features in the hybrid pruned feature sequence are traversed, their original index numbers are extracted, and they are integrated into a retained index set to determine the location of the cache entries that need to be retained.

[0141] Step b3: Traverse all Key and Value entries in the key-value cache, and filter out valid Keys and Values ​​that match the retention characteristics based on the retention index set.

[0142] In this embodiment of the invention, the KV-Cache is filtered according to the reserved index set, and only the Key and Value entries whose indexes belong to the reserved index set are retained, while the remaining entries are marked as redundant and to be deleted.

[0143] Step b4: Delete redundant Key and Value entries in the key-value cache that do not fall into the reserved index set.

[0144] In this embodiment of the invention, redundant Key and Value entries in the KV-Cache are physically deleted, so that the cache only retains the valid entries corresponding to the hybrid pruned features, which greatly reduces memory usage and inference latency.

[0145] Step b5: Integrate the valid Key and valid Value with the corresponding hybrid pruned feature sequence to obtain the adaptive pruned feature sequence.

[0146] In this embodiment of the invention, the selected valid Key and Value are integrated with the hybrid pruned feature sequence to form the final adaptive pruned feature sequence; the KV-Cache corresponding to this sequence has been pruned synchronously, significantly reducing the memory overhead of the inference process, while not affecting the coherence of subsequent decoding generation.

[0147] The semantic convergence score is a comprehensive score that quantifies the degree of semantic aggregation and redundancy of dual-modal fusion features by fusing attention deterministic signals and representation stability signals. The score ranges from [0,1], with higher scores indicating greater feature redundancy. The adaptive retention rate is the feature retention ratio dynamically matched based on the semantic convergence score. The retention rate is low in simple high-convergence scenarios and high in complex low-convergence scenarios. Hybrid selection pruning is a feature selection strategy that combines attention relevance filtering and feature distance diversity completion, simultaneously considering task relevance and semantic integrity. Key-value cache pruning is a cache optimization method that simultaneously cleans up redundant Key and Value entries in the Transformer inference key-value cache, reducing memory usage and inference latency.

[0148] The target retained feature count is the total number of features retained after multiplying the total number of bimodal fusion features by the adaptive retention rate and rounding down. The relevance selection count N is the number of features in the target retained count allocated for filtering and decoding strongly correlated core features for queries. The diversity completion count M is the number of features in the target retained count allocated for supplementing niche semantic features that are easily ignored by attention. The relevance feature subset is the feature set that focuses on the core semantics of welding, consisting of the top N features selected in descending order of attention weight. The remaining features to be completed are the remaining fusion features with potential semantic supplementation value after removing the relevance feature subset. The minimum feature distance is the minimum cosine distance between the remaining features to be completed and all features in the relevance feature subset. The diversity feature subset is the feature set that supplements long-tail detail semantics, consisting of the top M features selected in descending order of minimum feature distance. The hybrid pruned feature sequence is a simplified feature sequence that merges the relevance feature subset and the diversity feature subset, while satisfying both task relevance and semantic diversity.

[0149] The key-value cache is a dedicated cache structure for storing key and value vectors during the inference phase of the Transformer model, used to accelerate attention computation. Key and value entries are key and value vector storage units in the key-value cache that correspond one-to-one with the bimodal fusion features. The original index number is the fixed position number of each feature in the bimodal fusion feature sequence, used to establish the mapping relationship between features and cache entries. The retained index set is the collection of original index numbers of all retained features after hybrid pruning. Valid key and valid value are available cache entries in the key-value cache whose indices fall into the retained index set and match the retained features. Redundant key and value entries are invalid cache entries in the key-value cache whose indices do not fall into the retained index set and correspond to pruned features, and are awaiting deletion.

[0150] Step S206: Perform action semantic decoding on the adaptive pruning feature sequence to generate and output the welding robot action control instructions.

[0151] In this embodiment of the invention, the adaptively pruned feature sequence is input into a pre-trained action semantic decoder (such as the decoding layer of a large language model). The decoder maps the features to the action control space of the welding robot based on the bimodal semantic information carried by the fused features. The decoder analyzes visual scene information (such as weld position, molten pool state, and defect features) and text instruction information (such as welding path and process parameter requirements) in the features to generate a continuous action sequence including welding torch posture, welding speed, wire feed speed, and welding current / voltage. For simple working conditions (such as standard weld seam tracking), the decoder quickly generates basic control commands based on concise feature sequences; for complex working conditions (such as untaught corners and strong interference scenarios), the decoder uses fully preserved detailed features to generate precise adjustment commands, such as correcting the welding torch posture and adjusting the welding speed to avoid defects. The generated motion sequence is converted into a standard control instruction format (such as industrial robot communication protocol instructions or G code) that can be recognized by the welding robot controller, and output to the robot control unit to drive the robot to perform the corresponding welding operation, thereby achieving high-precision adaptive welding control.

[0152] This invention has the following advantages: Balancing global welding timing structure with local transient details: This invention, through global temporal merging of frame tokens combined with a dynamic programming global optimal segmentation mechanism, can intelligently compress a large number of redundant frame tokens in long-term static backgrounds and uniform welding intervals during the welding process into a small number of feature words. Simultaneously, it fully preserves transient key details such as arc initiation, arc termination, molten droplet transfer, arc spatter, and sudden changes in welding torch posture, which only occur briefly. Compared to traditional fixed sliding window segmentation schemes, this invention completely solves the technical problem that fixed windows cannot adapt to the temporal stability of welding videos and easily disrupt the temporal continuity of welding operations, thus fully preserving the spatiotemporal semantic structure of the entire welding process.

[0153] Eliminating the semantic blind spot in welding caused by the flat-tailed distribution of features: This invention employs a dual optimization strategy of diversity perception pruning and bimodal hybrid selection pruning to proactively retain visual and bimodal fusion feature words with low attention weights but unique semantic distributions. This avoids the model from over-focusing on high-frequency salient features such as bright arcs and bright molten pools, while ignoring low-weight long-tail detail features such as weld edge contours, micropores, and fine cracks. This effectively eliminates semantic perception blind spots in the model and significantly improves the accuracy of environmental detail, weld defect identification, and scene understanding under complex welding conditions.

[0154] Achieve adaptive dynamic allocation of computing power to balance inference real-time performance and control precision: This invention introduces a semantic convergence score to quantify the semantic aggregation degree and redundancy level of dual-modal fusion features, and adaptively and dynamically adjusts the feature retention rate and pruning intensity based on the convergence score. For simple welding conditions such as standard regular welds, the semantic convergence is high and the feature redundancy is large, so aggressive pruning can be performed to reduce the feature volume and significantly save inference computing power. For complex conditions such as irregular welds, corner welding, and strong arc light interference, the pruning intensity is automatically reduced and more effective features are retained to preserve semantic details, ensure the accuracy of action decoding and control, and realize intelligent optimal allocation of computing resources among different welding conditions and different samples.

[0155] Low deployment cost, high versatility, and no need for model retraining and fine-tuning: This invention's entire suite of global temporal merging, diversity-aware pruning, cross-modal attention-based pruning, semantic-aware adaptive pruning, and key-value caching pruning schemes are all deployed during the model inference stage. No retraining or parameter fine-tuning of the welding video large language model, bimodal feature encoding model, or action semantic decoding model is required, making it a plug-and-play inference acceleration and optimization architecture. It saves the high costs of large-scale working condition sample training and model fine-tuning, and is compatible with various Transformer-based welding bimodal perception models, adapting to multiple types of industrial welding robot operation scenarios, demonstrating strong engineering feasibility and versatility.

[0156] This embodiment also provides a robot welding motion generation system, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0157] This embodiment provides a robot welding motion generation system, such as Figure 3 As shown, it includes: The visual text preprocessing module 301 is used to acquire continuous video frames of the target welding operation scene and perform feature extraction and projection to generate a frame token sequence. It also responds to the input welding task text command by word segmentation and projection to generate a text feature sequence. The global temporal merging module 302 is used to perform global temporal merging on the frame token sequence to obtain a visual feature sequence. The diversity coverage perception pruning module 303 is used to perform diversity perception pruning on the visual feature sequence to obtain a diversity-pruned visual feature sequence. The text feature attention pruning module 304 is used to perform attention weight filtering and pruning on the text feature sequence based on the diversity pruning visual feature sequence to obtain the pruned text feature sequence. The semantic adaptive decoding and pruning module 305 is used to fuse diverse pruned visual feature sequences and pruned text feature sequences, and to perform semantic-aware pruning on the fused dual-modal fused feature sequences to obtain adaptive pruned feature sequences. The welding action decoding output module 306 is used to perform action semantic decoding on the adaptive pruning feature sequence, generate and output welding robot action control instructions.

[0158] Please see Figure 4 , Figure 4 This demonstrates the complete path of the welding video input stream, from compression by a pre-filling module to fusion with text features before entering the large model, and the subsequent layer-by-layer semantic-aware adaptive pruning during the decoding stage. Welding video frames are first visually extracted and projected to generate a frame token sequence. Static and dynamic frame tokens are then divided by calculating inter-frame feature similarity. High-similarity static frame tokens corresponding to stable welding phases and static backgrounds are merged and compressed, while low-similarity dynamic frame tokens such as those related to arc initiation, spatter, and sudden changes in welding torch posture are fully preserved, forming a lightweight fused frame token sequence. Then, a cosine distance matrix is ​​constructed, and a farthest-point sampling strategy is used for diversity-aware pruning, forcibly retaining semantically unique long-tail visual words and eliminating semantic blind spots caused by bright arc light, resulting in a concise and semantically complete diversity-pruned visual feature sequence. Simultaneously, welding task text instructions are segmented and feature-projected to generate a text feature sequence, and redundant text is filtered and eliminated through cross-modal attention weights. This feature yields a pruned text feature sequence. After fusing the two types of features, a bimodal fusion feature sequence is formed. Once inside the large model, the semantic convergence score, which is a weighted fusion of attention deterministic signals and representation stability signals, is calculated to dynamically determine the adaptive retention rate of the current layer. Based on this retention rate, hybrid selection pruning and key-value cache pruning are performed sequentially. Hybrid selection pruning first selects the features with the highest relevance to the current decoding query through attention weights, and then completes the semantically unique features that are easily ignored by attention through cosine distance sampling. Simultaneously, the key and value of the corresponding redundant entries in the key-value cache are pruned, finally yielding an adaptive pruned feature sequence. After action semantic decoding, welding robot control instructions containing information such as welding torch posture and welding speed are generated. This realizes the full-process optimization of video input from pre-compression, bimodal fusion to adaptive inference inside the large model. While taking into account the welding temporal structure and transient details, it eliminates semantic blind spots and realizes dynamic allocation of computing power.

[0159] In some alternative implementations, the visual text preprocessing module 301 includes: A continuous video frame unit is used to acquire continuous video frames of the target welding operation scene; Spatial feature unit, used to extract spatial features from consecutive video frames by a visual encoder; The frame token sequence unit is used to project spatial features to generate a frame token sequence. The text word sequence unit is used to respond to the input welding task text instructions. It is segmented by a word segmenter to obtain a text word sequence. The text feature sequence unit is used to project the text word sequence to generate a text feature sequence.

[0160] In some optional implementations, the global timing merging module 302 includes: The feature cosine similarity unit is used to calculate the feature cosine similarity between two adjacent frame tokens in the frame token sequence. The marking unit is used to mark frame tokens with feature cosine similarity not lower than a preset similarity threshold as static frame tokens, and to mark frame tokens with feature cosine similarity lower than a preset similarity threshold as dynamic frame tokens. The average pooling unit is used to perform average pooling on each consecutive segment of static frame tokens to obtain multiple compressed static frame tokens. The visual feature sequence unit is used to concatenate and fuse the compressed static frame tokens with the dynamic frame tokens to obtain the visual feature sequence.

[0161] In some alternative implementations, the diversity coverage-aware pruning module 303 includes: The cosine distance matrix unit is used to calculate the cosine distance between any two different frame token features in the visual feature sequence and construct the cosine distance matrix. The first selection unit is used to select a frame token feature from the visual feature sequence as the retained frame token feature and add it to the feature retention set; The Remaining Frame Token Feature Unit is used to take the unselected frame token features as the remaining frame token features. The processing unit is used to query the cosine distance between each remaining frame token feature and each retained frame token feature in the feature retention set based on the cosine distance matrix, and take the minimum value as the minimum cosine distance of the remaining frame token feature. The second selection unit is used to select the remaining frame token feature with the largest minimum cosine distance from all remaining frame token features as the retained frame token feature and add it to the feature retention set. The jump unit is used to jump to execute the steps for each remaining frame token feature until the number of retained frame token features reaches a preset threshold. The integration unit is used to integrate all retained frame token features in the feature retention set to obtain a diverse pruned visual feature sequence.

[0162] In some optional implementations, the text feature attention pruning module 304 includes: The pruning threshold unit is used to determine the pruning threshold of the text feature sequence based on the number of frame token features in the diverse pruning visual feature sequence. Attention weight value unit, used to calculate the attention weight value of each text feature in the text feature sequence; The pruned text feature sequence unit is used to remove text features whose attention weight values ​​are lower than the pruning threshold, thus obtaining the pruned text feature sequence.

[0163] In some alternative implementations, the semantic adaptive decoding pruning module 305 includes: The dual-modal fusion feature sequence unit is used to fuse diverse pruned visual feature sequences with pruned text feature sequences to obtain a dual-modal fusion feature sequence. The semantic convergence score unit is used to calculate the semantic convergence score of the bimodal fused feature sequence. An adaptive retention rate unit is used to determine the corresponding adaptive retention rate based on the semantic convergence score; The adaptive pruning feature sequence unit is used to perform hybrid selection pruning and key-value cache pruning sequentially based on the adaptive retention rate to obtain the adaptive pruning feature sequence.

[0164] In some alternative implementations, hybrid selective pruning specifically involves: The total number of features in the dual-modal fusion feature sequence is counted, and the target number of retained features is obtained by combining the adaptive retention rate and rounding. The number of target retained features is divided into the number of relevance selections N and the number of diversity completions M; Calculate the attention weight of each feature in the dual-modal fusion feature sequence and sort them in descending order. Select the top N features to form a subset of relevant features. The relevant feature subset is removed from the dual-modal fusion feature sequence to obtain the remaining features to be completed. For each remaining feature to be completed, calculate the cosine distance between the remaining feature to be completed and all features in the subset of relevant features, and take the minimum value as the minimum feature distance of the remaining feature to be completed. Sort the features by minimum feature distance from largest to smallest, and select the top M features to form a diverse feature subset; By merging the subsets of relevance features and the subsets of diversity features, a hybrid pruned feature sequence is obtained.

[0165] In some optional implementations, key-value cache pruning specifically involves: Establish the corresponding index relationship between each feature in the dual-modal fusion feature sequence and the key and value entries in the key-value cache, and record the original index number of each feature; Extract the original index numbers of all retained features from the feature sequence after hybrid pruning to form a retained index set; Iterate through all Key and Value entries in the key-value cache, and filter out valid Keys and Values ​​that match the retention characteristics based on the retained index set; Delete redundant key and value entries in the key-value cache that do not fall into the reserved index set; By integrating valid keys and valid values ​​with the corresponding hybrid pruned feature sequences, an adaptive pruned feature sequence is obtained.

[0166] The robot welding action generation system provided in this embodiment of the invention can execute the robot welding action generation method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method. Further functional descriptions of the various modules and units are the same as in the corresponding embodiments described above, and will not be repeated here.

[0167] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0168] The following is a detailed reference. Figure 5 This diagram illustrates a suitable structural schematic for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from memory 508 into random access memory (RAM) 503. RAM 503 also stores various programs and data required for the operation of the electronic device. The processor 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0169] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0170] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a memory 508, or installed from a ROM 502. When the computer program is executed by the processor 501, it performs the functions defined in the robot welding action generation method of the embodiments of the present invention.

[0171] Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0172] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for generating robot welding actions, characterized in that, The method includes: Collect continuous video frames of the target welding operation scene, extract and project features to generate a frame token sequence, respond to the input welding task text command by word segmentation and projection to generate a text feature sequence; The frame token sequence is globally temporally merged to obtain a visual feature sequence; The visual feature sequence is subjected to diversity perception pruning to obtain a diversity-pruned visual feature sequence. Based on the diverse pruned visual feature sequence, attention weight filtering and pruning are performed on the text feature sequence to obtain the pruned text feature sequence; The diverse pruned visual feature sequence and the pruned text feature sequence are fused together, and semantic-aware pruning is performed on the fused dual-modal fused feature sequence to obtain an adaptive pruned feature sequence. The adaptive pruning feature sequence is subjected to action semantic decoding to generate and output the welding robot action control instructions.

2. The method according to claim 1, characterized in that, The process involves acquiring continuous video frames of the target welding operation scene, extracting and projecting features to generate a frame token sequence, and responding to the input welding task text command by word segmentation and projection to generate a text feature sequence, including: Acquire continuous video frames of the target welding operation scene; Spatial features are obtained by extracting features from the continuous video frames using a visual encoder; The spatial features are projected to generate a frame token sequence; The input welding task text command is segmented by a word segmenter to obtain a text word sequence; The text word sequence is projected to generate a text feature sequence.

3. The method according to claim 1, characterized in that, The step of performing global temporal merging on the frame token sequence to obtain a visual feature sequence includes: Calculate the feature cosine similarity between two adjacent frame tokens in the frame token sequence; Frame tokens with a feature cosine similarity not lower than a preset similarity threshold are marked as static frame tokens, and frame tokens with a feature cosine similarity lower than the preset similarity threshold are marked as dynamic frame tokens. Average pooling is performed on each consecutive segment of the static frame token to obtain multiple compressed static frame tokens; The compressed static frame tokens and the dynamic frame tokens are concatenated and fused to obtain a visual feature sequence.

4. The method according to claim 1, characterized in that, The process of performing diversity-perceptual pruning on the visual feature sequence to obtain a diversity-pruned visual feature sequence includes: Calculate the cosine distance between any two different frame token features in the visual feature sequence, and construct a cosine distance matrix; Select one frame token feature from the visual feature sequence as the retained frame token feature and add it to the feature retention set; Use the unselected frame token features as the remaining frame token features; For each remaining frame token feature, based on the cosine distance matrix, query the cosine distance between the remaining frame token feature and each retained frame token feature in the feature retention set, and take the minimum value as the minimum cosine distance of the remaining frame token feature. Among all the remaining frame token features, the remaining frame token feature with the largest minimum cosine distance is selected as the retained frame token feature and added to the feature retention set; Jump to execute the step for each of the remaining frame token features until the number of retained frame token features reaches a preset threshold. By integrating all the retained frame token features in the feature retention set, a diverse pruned visual feature sequence is obtained.

5. The method according to claim 1, characterized in that, The step of performing attention weight filtering and pruning on the text feature sequence based on the diverse pruned visual feature sequence to obtain the pruned text feature sequence includes: The pruning threshold of the text feature sequence is determined based on the number of frame token features in the diverse pruned visual feature sequence. Calculate the attention weight value of each text feature within the text feature sequence; Text features with attention weight values ​​lower than the pruning threshold are removed to obtain a pruned text feature sequence.

6. The method according to any one of claims 1-5, characterized in that, The process of fusing the diverse pruned visual feature sequence and the pruned text feature sequence, and then performing semantic-aware pruning on the fused dual-modal fused feature sequence to obtain an adaptive pruned feature sequence includes: The diverse pruned visual feature sequence and the pruned text feature sequence are fused to obtain a dual-modal fusion feature sequence. Calculate the semantic convergence score of the dual-modal fused feature sequence; The corresponding adaptive retention rate is determined based on the semantic convergence score; Based on the adaptive retention rate, hybrid selection pruning and key-value cache pruning are performed sequentially to obtain an adaptive pruning feature sequence.

7. The method according to claim 6, characterized in that, The hybrid selection pruning specifically refers to: The total number of features in the dual-modal fusion feature sequence is counted, and the target number of retained features is calculated and rounded based on the adaptive retention rate. The number of target retained features is divided into the number of relevance selections N and the number of diversity completions M; Calculate the attention weight of each feature in the dual-modal fusion feature sequence and sort them in descending order. Select the top N features to form a subset of relevant features. The relevant feature subset is removed from the dual-modal fusion feature sequence to obtain the remaining features to be completed. For each of the remaining features to be completed, calculate the cosine distance between the remaining feature to be completed and all features in the subset of related features, and take the minimum value as the minimum feature distance of the remaining feature to be completed. Sort the features by minimum feature distance from largest to smallest, and select the top M features to form a diverse feature subset; The correlation feature subset and the diversity feature subset are merged to obtain the hybrid pruned feature sequence.

8. The method according to claim 7, characterized in that, The key-value cache pruning specifically involves: Establish the corresponding index relationship between each feature in the dual-modal fusion feature sequence and the Key and Value entries in the key-value cache, and record the original index number of each feature; Extract the original index numbers of all retained features from the hybrid pruned feature sequence to form a retained index set; Iterate through all Key and Value entries in the key-value cache, and filter out valid Keys and Values ​​that match the retention features according to the retention index set; Delete redundant Key and Value entries in the key-value cache that do not fall into the reserved index set; By integrating the valid Key and valid Value with the corresponding hybrid pruned feature sequence, an adaptive pruned feature sequence is obtained.

9. A robot welding motion generation system, characterized in that, The system includes: The visual text preprocessing module is used to acquire continuous video frames of the target welding operation scene, extract and project features, generate a frame token sequence, respond to the input welding task text command, segment and project, and generate a text feature sequence. A global temporal merging module is used to perform global temporal merging on the frame token sequence to obtain a visual feature sequence; A diversity-perceived pruning module is used to perform diversity-perceived pruning on the visual feature sequence to obtain a diversity-pruned visual feature sequence. The text feature attention pruning module is used to perform attention weight filtering and pruning on the text feature sequence based on the diversity pruning visual feature sequence to obtain the pruned text feature sequence; The semantic adaptive decoding and pruning module is used to fuse the diverse pruning visual feature sequence and the pruning text feature sequence, and to perform semantic-aware pruning on the fused dual-modal fused feature sequence to obtain an adaptive pruning feature sequence. The welding action decoding output module is used to perform action semantic decoding on the adaptive pruning feature sequence, generate and output welding robot action control commands.

10. An electronic device, characterized in that, include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the method of any one of claims 1 to 8.

Images