A welding process visual language understanding and decision-making method based on action chain reasoning

Abstract

The application relates to the technical field of intelligent manufacturing and industrial robot control, and discloses a welding process visual language understanding and decision-making method based on action chain reasoning, which comprises the following steps: acquiring a welding video stream, sequentially performing sparse sampling, dense sampling and action segmentation on the welding video stream to obtain structured visual information; acquiring a language instruction, and performing consistency matching on the language instruction and the structured visual information to obtain a matching result; selecting and executing a corresponding reasoning prediction strategy according to the matching result to obtain a prediction result, wherein the prediction result comprises stage recognition results, abnormality determination results and process decision suggestions. The application realizes visual language understanding and decision-making of the welding process through action chain reasoning, outputs stage recognition, abnormality determination and process decision suggestions, solves problems such as insufficient flexibility of a traditional welding scheme, general model reasoning illusion and long-time sequence forgetting, realizes reliable decision-making based on visual evidence, and has high efficiency, interpretability and scene adaptability.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing and industrial robot control technology, specifically to a visual language understanding and decision-making method for welding processes based on action chain reasoning. Background Technology

[0002] Welding processes typically involve complex timing actions and stringent technological requirements. This necessitates that the control system not only perceive the environment but also understand the operational procedures and make decisions like a skilled worker. Existing technologies commonly used in handling welding path planning and action decisions include: teaching and offline programming for welding path planning, passive tracking based on traditional methods and vision, and end-to-end control based on a large model using a universal visual language.

[0003] Existing technologies can handle simple, repetitive assembly line operations, but when dealing with long sequences, multiple action combinations, and complex welding tasks that require a high degree of cognitive involvement, they often need to build a large comprehensive model or rely on relatively fixed rules, making it difficult to meet the needs of intelligentization. They face fundamental challenges in understanding unstructured environments and making complex process decisions, and cannot achieve high-quality, adaptive welding automation. Summary of the Invention

[0004] This invention provides a visual language understanding and decision-making method for welding processes based on action chain reasoning, in order to solve the problems of reasoning illusion, forgetting of long sequence action logic, and unreliable decision-making.

[0005] In a first aspect, the present invention provides a visual language understanding and decision-making method for welding processes based on action chain reasoning, the method comprising: The welding video stream is acquired, and sparse sampling, dense sampling, and action segmentation are performed sequentially on the welding video stream to obtain structured visual information. The language instructions are acquired, and the language instructions are matched with the structured visual information to obtain the matching results. Based on the matching results, the corresponding inference and prediction strategy is selected and executed to obtain the prediction results, which include: stage identification results, anomaly determination results, and process decision suggestions.

[0006] The present invention provides a visual language understanding and decision-making method for welding processes based on action chain reasoning. This method achieves visual language understanding and decision-making during the welding process through action chain reasoning. First, the welding video stream is transformed into structured visual information through sparse sampling, dense sampling, and action segmentation. Then, consistency matching between language instructions and visual information is performed. Finally, the corresponding reasoning strategy is executed according to the matching result, outputting stage identification, anomaly judgment, and process decision suggestions. This effectively solves problems such as insufficient flexibility in traditional welding schemes, the illusion of reasoning in general large models, and long-term forgetting. It achieves reliable decision-making based on visual evidence, combining high efficiency, interpretability, and scenario adaptability, significantly improving the intelligence level and production stability of complex welding operations.

[0007] In one optional implementation, the welding video stream is sequentially subjected to sparse sampling, dense sampling, and motion segmentation to obtain structured visual information, including: Sparse sampling is performed on the welding video stream to obtain the global context; Keyframe regions are determined based on global context, and dense sampling is performed on keyframe regions to capture fine movements. Action segmentation is performed based on global context and fine-grained actions. The continuous welding process is discretized into action atoms and then represented in a structured manner to obtain structured visual information.

[0008] The visual language understanding and decision-making method for welding process based on action chain reasoning provided by this invention can quickly acquire the global welding process, accurately capture key details, and finally convert continuous video into standardized visual information, avoiding the computational redundancy of full sampling. At the same time, it can fully preserve the welding sequence, spatial and action features, providing a reliable and queryable structured foundation for subsequent language matching and decision-making, improving the overall understanding efficiency and accuracy, and adapting to the real-time processing needs of long industrial welding videos.

[0009] In one alternative implementation, keyframe regions are determined based on global context, and these regions are densely sampled to capture fine-grained movements, including: Coarse-grained stage information is determined based on the global context. This information includes: stage distribution, stage coarse-grained start and end times, and welding action type. The coarse-grained boundary of the action switching stage is determined based on the coarse-grained stage information, and the abrupt change point of the welding action is determined by combining the inter-frame optical flow changes. Within the preset time range of the stage coarse-grained boundary and the abrupt change point of the welding action, the key frame region is determined, and the key frame region is densely sampled to capture fine movements.

[0010] The visual language understanding and decision-making method for welding process based on action chain reasoning provided by this invention combines stage boundaries and optical flow mutation to locate key frame regions, achieves accurate focusing of high-value frames, significantly reduces invalid calculations, and clearly captures short-term key actions such as arc initiation, arc extinguishing, and electrode switching through dense sampling, making up for the lack of details in sparse sampling, making action segmentation more accurate and action atoms more complete, and effectively improving the ability to identify short-term welding anomalies and state switching.

[0011] In one optional implementation, the language instructions are matched with structured visual information to obtain a matching result, including: From four perspectives—action type, temporal sequence, spatial location, and technological constraints—a lightweight evaluator is used to compare the degree of conformity between verbal instructions and structured visual information. If the structured visual information contains action requirements, time intervals, spatial locations, and process constraints that correspond to the language instructions, then the matching result is consistent; otherwise, the matching result is inconsistent.

[0012] The visual language understanding and decision-making method for welding processes based on action chain reasoning provided by this invention performs lightweight matching from four dimensions: action type, time sequence, spatial location, and process constraints. The dimensions are comprehensive and closely match the welding process logic. The matching rules are clear and efficient. The lightweight evaluator does not require complex calculations and can quickly determine whether the instructions and visual information match, achieving a response time of seconds for simple tasks. At the same time, it strictly relies on visual evidence and does not depend on language priors, suppressing reasoning illusions from the source and ensuring that the matching results are true and reliable.

[0013] In one optional implementation, a corresponding inference prediction strategy is selected and executed based on the matching result to obtain the prediction result, including: If the matching results are consistent, the stage recognition results, anomaly judgment results, and process decision suggestions are directly output based on the structured visual information as the prediction results. If the matching results are inconsistent, deep reasoning is performed using multiple agents based on language instructions and structured visual information to obtain the prediction result.

[0014] The visual language understanding and decision-making method for welding processes based on action chain reasoning provided by this invention adopts a diversion strategy of consistent direct output and inconsistent deep reasoning, which takes into account both efficiency and reliability. Consistent scenarios directly output results to meet the real-time requirements of the pipeline; inconsistent scenarios initiate deep reasoning to resolve ambiguity and conflict scenarios. This avoids over-computation for simple tasks while ensuring rigorous decision-making in complex welding scenarios, balancing real-time performance and accuracy, and adapting to the intelligent decision-making needs of flexible welding production in multiple scenarios.

[0015] In one alternative implementation, based on language instructions and structured visual information, deep reasoning using multiple agents is employed to obtain prediction results, including: A planning agent is used to break down language instructions into multiple sub-propositions; The tool agent converts each sub-proposition into a query operation, and retrieves structured visual information based on the query operation to obtain visual evidence, which includes: action atoms and corresponding keyframes. By using a summarizing agent, visual evidence is logically judged based on sub-propositions to obtain prediction results.

[0016] The visual language understanding and decision-making method for welding processes based on action chain reasoning provided by this invention, through the division of labor and cooperation of three intelligent agents—planning, tooling, and summarizing—decomposes complex instructions into verifiable sub-propositions, accurately retrieves visual evidence, and completes logical judgments. Each step of reasoning relies on action atoms and keyframes, making decisions traceable and logic verifiable. This completely solves the problems of illusion and temporal forgetting in general large models, significantly improves the understanding depth and decision credibility of complex welding scenarios, and meets the high safety and high reliability requirements of industrial operations.

[0017] In one alternative implementation, a blackboard mechanism is used to record and display language instructions, sub-propositions, time intervals for each sub-proposition, visual evidence, and prediction results.

[0018] The visual language understanding and decision-making method for welding process based on action chain reasoning provided by this invention utilizes a blackboard mechanism to achieve full-process information sharing and visualization. All agents collaborate based on unified data, avoiding state conflicts and information errors. The recorded content covers the entire chain of instructions, propositions, evidence, and results. Each decision can be traced back to a specific frame and action atom, greatly improving the interpretability of the system. At the same time, it facilitates manual verification, debugging, and process optimization, and lowers the threshold for industrial implementation.

[0019] Secondly, the present invention provides a visual language understanding and decision-making device for welding processes based on action chain reasoning, the device comprising: The visual data processing module is used to acquire welding video streams and sequentially perform sparse sampling, dense sampling, and motion segmentation on the welding video streams to obtain structured visual information. The consistency matching module is used to acquire language instructions and perform consistency matching between the language instructions and structured visual information to obtain the matching results; The inference and prediction module is used to select and execute the corresponding inference and prediction strategy based on the matching results to obtain prediction results, including: stage identification results, anomaly judgment results, and process decision suggestions.

[0020] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the method described in the first aspect or any corresponding embodiment thereof.

[0021] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the method described in the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0022] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of an application scenario according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the first type of visual language understanding and decision-making method for welding process based on action chain reasoning according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a second process for a visual language understanding and decision-making method for welding processes based on action chain reasoning, according to an embodiment of the present invention. Figure 4 This is a structural block diagram of a welding process visual language understanding and decision-making device based on action chain reasoning according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0026] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0027] As an optional application scenario of this invention, such as Figure 1 As shown, the visual language understanding and decision-making system for the welding process may include at least one terminal device and at least one server. Figure 1 The system is illustrated in the example, which includes a computer 101, a mobile terminal 102, and a server 103, and the terminal devices such as the computer 101 and the mobile terminal 102 are connected to the server 103 through a network 110.

[0028] Specifically, the terminal device can be a smartphone, tablet, laptop, PDA, desktop computer, game console, smart TV, smart wearable device, in-vehicle terminal, VR (Virtual Reality) device, AR (Augmented Reality) device, etc. Server 103 can be a standalone physical server, a server cluster, a distributed system, or a cloud server providing cloud services. Network 110 can be a wired or wireless network, examples of which include, but are not limited to, the Internet, corporate intranet, local area network, wide area network, mobile communication network, and combinations thereof.

[0029] Existing solutions mainly employ the following technical approaches when handling welding path planning and motion decisions: Traditional teaching and offline programming mode: This is currently the most common method. The welding path is recorded point-by-point by a handheld teach pendant, or offline trajectory planning is performed using computer-aided design (CAD) software. The robot strictly follows the preset program instructions, lacking environmental perception and understanding capabilities. This method is highly dependent on the precision of the tooling and the experience of skilled workers, resulting in poor reproducibility. When the dimensional deviation of the welded workpiece is large or the working environment changes slightly, the preset program cannot adapt automatically, requiring a shutdown and manual re-teaching. It lacks flexibility, and the operation process is difficult to program, making it unsuitable for flexible production needs involving multiple varieties and small batches.

[0030] Passive tracking based on traditional machine vision involves adding laser vision or structured light sensors to the end effector of a welding robot. Image processing algorithms extract weld features, such as the center and edges of the bevel, and correct the robot's trajectory in real time. This approach primarily focuses on the extraction and correction of geometric features. While it endows the robotic arm with visual capabilities to some extent, its planning method remains simple, focusing only on local geometric position correction. It cannot understand the contextual logic of welding actions, such as when to pause or when to adjust posture. It lacks semantic understanding of the entire welding process. When encountering complex weld discontinuities or obstacles obstructing the welding process, it often fails to make decisions due to a lack of capability.

[0031] End-to-end control based on a generalized visual-language model (VLM): With the development of artificial intelligence, attempts are being made to use a generalized visual-language model to directly understand welding scene images or videos, and generate operation suggestions in natural language form or directly map them into control commands, attempting to use the generalization ability of the large model to solve the problem of understanding unstructured scenes. Although theoretically it has high-level semantic understanding capabilities, it suffers from serious reasoning illusion problems in industrial practice. First, existing models are highly susceptible to language priors, often generating responses based on probability distributions in the training data, resulting in erroneous inferences that do not conform to current visual evidence. For example, it may infer that welding has started when there is no arc. Second, when processing long sequences of actions, there is contextual forgetting, making it impossible to accurately grasp the temporal relationship of welding actions, leading to confused decision-making logic. Finally, there is a lack of rigorous evidence verification mechanisms, and the generated decisions lack traceability, which is unacceptable for welding operations that require high reliability and safety, severely restricting its application in actual production.

[0032] To address the aforementioned issues, this invention provides a visual language understanding and decision-making method for welding processes based on action chain reasoning. By employing action chain reasoning, visual and instruction consistency matching, and multi-agent reasoning, the method aims to improve the interpretability, scenario adaptability, and reliability of decision-making.

[0033] According to an embodiment of the present invention, a visual language understanding and decision-making method for welding processes based on action chain reasoning is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0034] This embodiment provides a visual language understanding and decision-making method for welding processes based on action chain reasoning, which can be used in the aforementioned computer system. Figure 2This is a flowchart of a visual language understanding and decision-making method for welding processes based on action chain reasoning according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps: Step S201: Obtain the welding video stream, and sequentially perform sparse sampling, dense sampling, and action segmentation on the welding video stream to obtain structured visual information.

[0035] Specifically, the welding video stream of the target area is acquired in real time using video acquisition equipment, and the unstructured welding video stream is transformed into a structured, queryable set of atomic evidence using an action chain paradigm modeling module. The continuous welding video stream is equivalent to a signal sequence on the time axis. A sampling strategy from sparse to dense is adopted, performing sparse sampling, dense sampling, and action segmentation sequentially to obtain structured visual information as an evidence base for subsequent inference. This effectively digitizes the visual signal, and the structured visual information is structured action atoms. Each action atom includes metadata in four key dimensions: action type (e.g., arc initiation, linear motion, oscillation, arc extinguishing, idle stroke, etc.), time interval, spatial coordinates, and confidence score.

[0036] Step S202: Obtain language instructions and perform consistency matching between language instructions and structured visual information to obtain matching results.

[0037] Specifically, a two-stage assertion-driven inference module is established based on a vision-centric collaborative multi-agent architecture. The first stage is a consistency screening stage, and the second stage is a deep inference stage. Language commands can originate from manually input detection tasks, pre-set welding process specifications, quality inspection rules, or judgment requests issued by the upper-level scheduling system. Based on the language commands, the current structured visual information is retrieved, and the matching consistency between the language commands and the current structured visual information is calculated. If the consistency requirement is met, the matching result is considered consistent. Then, based on the current visual input and language commands, the prediction result is directly generated and output, ensuring real-time performance for simple tasks.

[0038] Step S203: Select and execute the corresponding inference prediction strategy based on the matching results to obtain the prediction results, which include: stage identification results, anomaly judgment results, and process decision suggestions.

[0039] Specifically, if the matching result is consistent, the prediction result is directly output. The prediction result includes, but is not limited to: stage identification result, anomaly judgment result and / or process decision suggestion, such as "the current stage is oscillating bar" or "a broken arc was detected in the interval between 12.5 seconds and 13.2 seconds". These are just examples, but are not limited to them.

[0040] If the matching results are inconsistent, it indicates that an accurate prediction cannot be obtained directly from the current structured visual information. There may be a risk of illusion or logical conflict. The risk of illusion refers to the model misjudging when there is a lack of visual evidence, such as misjudging "arc ignition" when a stable arc has not yet appeared. Logical conflict refers to contradictory conclusions, such as judging "continuous and stable welding throughout the entire process" in the first part, and then identifying "re-ignition of the arc" in the second part. This is just an example and is not a limitation. At this time, the second stage of deep reasoning process is automatically triggered, which performs deep reasoning based on language instructions and the current structured visual information to obtain the prediction result.

[0041] The visual language understanding and decision-making method for welding processes based on action chain reasoning provided in this embodiment achieves visual language understanding and decision-making in the welding process through action chain reasoning. First, the welding video stream is transformed into structured visual information through sparse sampling, dense sampling, and action segmentation. Then, consistency matching between language instructions and visual information is completed. Finally, the corresponding reasoning strategy is executed according to the matching result, and stage identification, anomaly judgment, and process decision suggestions are output. This effectively solves the problems of insufficient flexibility of traditional welding schemes, the illusion of reasoning in general large models, and long-term forgetting. It realizes reliable decision-making based on visual evidence, and has the advantages of high efficiency, interpretability, and scene adaptability, which greatly improves the intelligence level and production stability of complex welding operations.

[0042] This embodiment provides a visual language understanding and decision-making method for welding processes based on action chain reasoning, which can be used in the aforementioned computer system. Figure 3 This is a flowchart of a visual language understanding and decision-making method for welding processes based on action chain reasoning according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps: Step S301: Obtain the welding video stream, and sequentially perform sparse sampling, dense sampling, and action segmentation on the welding video stream to obtain structured visual information.

[0043] Specifically, step S301 includes: Step S3011: Sparsely sample the welding video stream to obtain the global context.

[0044] Specifically, the global context is coarse-grained stage boundary information obtained by sparse sampling of the entire welding video. It is used to characterize the stage distribution of the entire welding process on the time axis, the approximate start and end times of each stage, and the main welding action types. For example, sparse sampling identifies the 0 to 4 seconds of the video stream as the empty stroke alignment and the 4 to 6 seconds as the arc ignition. This is just an example and is not a limitation.

[0045] Step S3012: Determine the keyframe region based on the global context, and perform intensive sampling on the keyframe region to capture fine motion.

[0046] Specifically, based on the global context obtained by sparse sampling, the most valuable time period in the video is accurately located, and high-resolution sampling is performed to capture details such as welding action switching and short-term anomalies, providing fine-grained visual data for subsequent action segmentation.

[0047] In some alternative implementations, keyframe regions are determined based on global context, and these regions are densely sampled to capture fine-grained motion, including: Step a1: Determine coarse-grained stage information based on the global context. The coarse-grained stage information includes: stage distribution, stage coarse-grained start and end time, and welding action type.

[0048] Specifically, based on the global context obtained through sparse sampling, the entire welding video is divided into several continuous large stages to obtain coarse-grained stage information. Stage distribution refers to which welding stages the video includes, such as idle travel, arc initiation, linear motion, oscillating motion, and arc termination. The coarse-grained start and end times of each stage are the approximate start and end times. The welding action type corresponds to the core welding action of each stage. Only large-scale, low-precision stage division is performed, without focusing on millisecond-level details, with the aim of quickly grasping the temporal outline of the entire welding process.

[0049] Step a2: Determine the coarse-grained boundary of the action switching stage based on the coarse-grained stage information, and determine the abrupt change point of the welding action by combining the inter-frame optical flow changes.

[0050] Specifically, based on the stage division, the switching moments between two stages are identified, such as the time points from the idle stroke to the arc initiation, or from linear welding to oscillating welding. These points are the stage boundaries. Then, the horizontal and vertical optical flow components of each pixel are calculated frame by frame to obtain the temporal motion energy. When the abrupt change in motion energy exceeds a set threshold, it is determined as an action abrupt change point, corresponding to instantaneous actions such as sudden stop of the welding torch, arc initiation, arc termination, and arc termination. The stage boundaries and action abrupt change points are merged to obtain the time set of most likely critical actions, which is used for the next step of locating keyframe regions.

[0051] Step a3: Within the preset time range of the stage coarse-grained boundary and the abrupt change point of the welding action, determine the key frame region and perform dense sampling on the key frame region to capture fine actions.

[0052] Specifically, keyframe regions are local high-value time segments further determined based on the global context and optical flow abrupt change points. For example, in a welding video stream, sparse sampling can first obtain coarse-grained stage information such as 0 to 4 seconds for empty travel alignment, 4 to 6 seconds for arc initiation, 6 to 18 seconds for linear welding, 18 to 24 seconds for oscillating welding, and 24 to 30 seconds for arc termination. Subsequently, keyframe regions are determined near 4, 18, and 24 seconds by combining optical flow abrupt change points, and dense sampling is performed near 4.6, 18.3, and 24.2 seconds to obtain fine-grained evidence of action switching. Fine-grained actions include continuous frame sequences, precise start and end times of actions, relative positions of the welding torch and weld, and arc / molten pool states.

[0053] The temporal motion energy used to determine the keyframe region can be expressed as:

[0054] in, and These represent adjacent frames at the pixel level. The horizontal and vertical optical flow components at the location, and These represent the height and width of the video frame image, respectively.

[0055] The set of abrupt change points in motion obtained from time-space motion energy can be represented as:

[0056]

[0057] in, and These represent the mean and standard deviation of the kinetic energy difference between adjacent time points, respectively. For threshold coefficient, The set of stage boundaries obtained by sparse sampling. This is the final set of boundaries used to determine the keyframe region.

[0058] The welding process visual language understanding and decision-making method based on action chain reasoning provided in this embodiment combines stage boundaries and optical flow mutations to locate key frame regions, achieving precise focusing of high-value frames, significantly reducing invalid calculations, and densely sampling to clearly capture short-term key actions such as arc initiation, arc extinguishing, and electrode switching, compensating for the lack of details in sparse sampling, making action segmentation more accurate and action atoms more complete, and effectively improving the ability to identify short-term welding anomalies and state switching.

[0059] Step S3013: Perform action segmentation based on global context and fine-grained actions, discretize the continuous welding process into action atoms, and perform structured representation to obtain structured visual information.

[0060] Specifically, based on the coarse-grained stage segmentation results and the dense sampling results of key frame regions, action segmentation is performed, and the continuous welding process is discretized into action atoms. Sparse sampling obtains stage boundaries, action type candidates, and scene background information, while dense sampling obtains continuous frames of key moments, fine time boundaries of action start and end, relative position changes of welding torch and weld, and changes of arc light and molten pool, thereby generating action atoms (including action type, time interval, spatial location, and confidence level).

[0061] To facilitate a structured representation of the continuous welding process, each action atom can be formally represented as:

[0062] in, Indicates the first One action atom; Indicates the type of action; Indicates the time interval during which the action occurs; Indicates the spatial coordinates or region corresponding to the action; This represents the confidence level of the action, and 0 ≤ ≤1.

[0063] The welding process visual language understanding and decision-making method based on action chain reasoning provided in this embodiment quickly acquires the global welding process, accurately captures key details, and finally converts continuous video into standardized visual information, avoiding computational redundancy of full sampling. At the same time, it fully preserves the welding sequence, spatial and action features, providing a reliable and queryable structured foundation for subsequent language matching and decision-making, improving the overall understanding efficiency and accuracy, and adapting to the real-time processing needs of long industrial welding videos.

[0064] Step S302: Obtain the language instructions and perform consistency matching between the language instructions and the structured visual information to obtain the matching result.

[0065] Specifically, step S302 includes: Step S3021: From four perspectives—action type, temporal sequence, spatial location, and process constraints—a lightweight evaluator is used to compare the degree of conformity between verbal instructions and structured visual information.

[0066] Specifically, the structured visual information is preferably structured visual evidence output by the action chain paradigm modeling module, including but not limited to a set of time-stamped keyframes, a set of action atoms within the corresponding time interval, and spatial region information corresponding to the action atoms. Language instructions can originate from manually input detection tasks, preset welding process specifications, quality inspection rules, or judgment requests issued by the upper-level scheduling system. Matching consistency is used to characterize the degree of conformity between the current visual input and the language instructions in terms of action type, temporal sequence, spatial location, and process constraints.

[0067] The lightweight evaluator employs fast computational logic, avoiding complex reasoning and focusing solely on determining the existence and conformity of evidence. It compares whether the welding actions required by the language instructions (e.g., arc initiation, electrode movement, arc extinguishing, arc breaking) match the types of action atoms; whether the sequence of actions required by the language instructions aligns with the time intervals of the action atoms; whether the welding area, welding torch posture, and weld position defined by the language instructions match the spatial coordinates of the action atoms; and whether the process requirements specified by the language instructions, such as electrode movement speed, arc length, and dwell time, match the visual evidence.

[0068] Step S3022: If the structured visual information contains action requirements, time intervals, spatial locations, and process constraints that match the language instructions, then the matching result is consistent; otherwise, the matching result is inconsistent.

[0069] Specifically, if the structured visual information contains action atoms, corresponding time intervals, corresponding spatial positions, and process constraints that completely match the language instructions, it indicates that the visual evidence fully supports the instruction description, without conflict or omission, and the matching result is consistent.

[0070] If any of the following conditions are not met: missing / incorrect action type, reversed timing, spatial mismatch, process exceeding limits, or lack of corresponding visual evidence, it is judged as inconsistent, which may pose a risk of inference illusion or logical conflict, and the matching result is inconsistent.

[0071] For example, when the language instruction is "determine whether there is an arc break during the welding process", if the current visual input retrieves arc-starting action atoms, continuous bar movement action atoms, and arc-extinguishing action atoms corresponding to the disappearance of the arc light midway, the consistency evaluator determines that the current visual input has a high matching relationship with the "existence of arc break" determination task; if the visual input only shows that the welding torch is approaching the weld seam without a stable arc light and a continuous molten pool, the system does not directly determine that welding has started based on language priors, thereby suppressing inference illusions.

[0072] The visual language understanding and decision-making method for welding processes based on action chain reasoning provided in this embodiment performs lightweight matching from four dimensions: action type, time sequence, spatial location, and process constraints. The dimensions are comprehensive and closely match the welding process logic. The matching rules are clear and efficient. The lightweight evaluator does not require complex calculations and can quickly determine whether the instructions and visual information match, achieving a response time of seconds for simple tasks. At the same time, it strictly relies on visual evidence and does not depend on language priors, suppressing reasoning illusions from the source and ensuring that the matching results are true and reliable.

[0073] Step S303: Select and execute the corresponding inference prediction strategy based on the matching results to obtain the prediction results, which include: stage identification results, anomaly judgment results, and process decision suggestions.

[0074] Specifically, step S303 includes: In step S3031, if the matching results are consistent, the stage recognition results, anomaly judgment results, and process decision suggestions are directly output based on the structured visual information as prediction results.

[0075] Specifically, when the lightweight evaluator determines that the language instruction is completely consistent with the structured visual information, it does not perform any additional complex calculations and directly outputs the prediction result based on the existing action atoms, keyframes, and spatiotemporal information.

[0076] For example, the prediction results include: "The current stage is oscillating bar movement" (stage identification result); "A broken arc was detected in the interval between 12.5 seconds and 13.2 seconds" (anomaly judgment result); "It is recommended to restart the arc and adjust the bar movement speed" (process decision suggestion). These are just examples and are not limited to this.

[0077] In step S3032, if the matching results are inconsistent, deep reasoning is performed using multiple agents based on language instructions and structured visual information to obtain the prediction result.

[0078] Specifically, the risk of illusion or logical conflict refers to inconsistencies between model output and visual evidence, or contradictory conclusions regarding the relationship between actions in a long sequence. For example, based on prior experience, a model might mistakenly determine that "arc has been initiated," even though a stable arc and molten pool do not exist within the corresponding time interval; or the model might determine "continuous and stable welding throughout the entire process" in the earlier stages, but then identify "re-initiating the arc" in the later stages, creating a logical conflict. These are just examples and are not the only possibilities. When inconsistencies such as missing actions, temporal contradictions, spatial mismatches, process out-of-bounds errors, or insufficient evidence occur, the system automatically enters a highly reliable deep reasoning process, utilizing a planning agent, a tool agent, and a summarizing agent to collaboratively complete the deep reasoning.

[0079] The visual language understanding and decision-making method for welding processes based on action chain reasoning provided in this embodiment adopts a diversion strategy of consistent direct output and inconsistent deep reasoning, which takes into account both efficiency and reliability. Consistent scenarios directly output results to meet the real-time requirements of the pipeline; inconsistent scenarios initiate deep reasoning to resolve ambiguity and conflict scenarios. This avoids over-computation for simple tasks while ensuring rigorous decision-making in complex welding scenarios, balancing real-time performance and accuracy, and adapting to the intelligent decision-making needs of flexible welding production in multiple scenarios.

[0080] In some optional implementations, step S3032 above includes: Step b1: Use a planning agent to break down language instructions into multiple sub-propositions.

[0081] Specifically, the planner is responsible for breaking down complex long-sequence tasks into a series of verifiable logical propositions based on welding process requirements. For example, "determining whether there is an arc break during the welding process" can be broken down into sub-propositions such as "whether there is an arc ignition action", "whether there is a continuous bar motion action after arc ignition", "whether there is an unexpected arc extinction during continuous bar motion", and "whether arc re-ignition occurs after arc extinction".

[0082] Step b2: Use the tool agent to convert each sub-proposition into a query operation, and retrieve structured visual information based on the query operation to obtain visual evidence, which includes: action atoms and corresponding keyframes.

[0083] Specifically, the tool agent is responsible for converting each sub-proposition into a query operation for the action chain evidence base, retrieving the action atomic evidence corresponding to arc initiation, bar movement, arc extinguishing, and arc re-initiation on the timeline.

[0084] Step b3: Using the summarizing agent, the visual evidence is logically judged based on the sub-propositions to obtain the prediction result.

[0085] Specifically, the intelligent agent is responsible for collecting visual evidence returned by the tool agent, performing logical evaluation, and generating the final decision suggestion. Logical evaluation is performed on the retrieval results. When an arc-initiating action is retrieved first, followed by a stable motion action, an arc-extinguishing action is retrieved in between, and a re-initiating action is subsequently retrieved, the decision suggestion "Restart after arc breakage exists" is output.

[0086] No. Each welding stage to be verified can be formally represented as: (5) in, and They represent the first The start and end times of each welding stage to be verified. This indicates the welding object corresponding to this stage. This indicates the type of action atom corresponding to this stage. This indicates the environmental context information of this stage.

[0087] The corresponding action assertion to be verified can be represented as: (6) in, Indicates the first An action assertion to be verified, This indicates the welding object corresponding to the action assertion. This indicates the type of the action atom corresponding to the action assertion. This indicates the time constraint interval corresponding to the action assertion. Based on keyframe evidence and action atoms within the time interval, the action assertion is... Perform truth value determination and then perform logical evaluation of each subproposition accordingly.

[0088] The visual language understanding and decision-making method for welding processes based on action chain reasoning provided in this embodiment, through the division of labor and cooperation of three intelligent agents—planning, tools, and summarizing—decomposes complex instructions into verifiable sub-propositions, accurately retrieves visual evidence, and completes logical judgments. Each step of reasoning relies on action atoms and keyframes, making decisions traceable and logic verifiable. This completely solves the problems of illusion and temporal forgetting in general large models, significantly improves the depth of understanding and decision credibility of complex welding scenarios, and meets the high safety and high reliability requirements of industrial operations.

[0089] In some alternative implementations, a blackboard mechanism is used to record and display language instructions, sub-propositions, time intervals for each sub-proposition, visual evidence, and prediction results.

[0090] Specifically, the blackboard mechanism is a shared intermediate state recording area for multiple agents. It stores the overall problem, a list of sub-propositions, the time intervals for each sub-proposition, retrieved action atoms, evidence frame numbers, evidence confidence levels, current logical evaluation results, and items to be supplemented for retrieval. This ensures the traceability of the collaborative process and allows for continued supplementary retrieval when evidence is insufficient. It ensures that each decision step is supported by explicit visual evidence, effectively suppressing the illusions caused by linguistic priors. Each agent shares the state by reading and writing to the blackboard mechanism, making each step of reasoning traceable to the corresponding visual evidence and triggering supplementary retrieval when evidence is insufficient.

[0091] The visual language understanding and decision-making method for welding processes based on action chain reasoning provided in this embodiment utilizes a blackboard mechanism to achieve full-process information sharing and visualization. All agents collaborate based on unified data, avoiding state conflicts and information errors. The recorded content covers the entire chain of instructions, propositions, evidence, and results. Each decision can be traced back to a specific frame and action atom, greatly improving the interpretability of the system. At the same time, it facilitates manual verification, debugging, and process optimization, reducing the threshold for industrial implementation.

[0092] This embodiment also provides a visual language understanding and decision-making device for welding processes based on action chain reasoning. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0093] This embodiment provides a visual language understanding and decision-making device for welding processes based on action chain reasoning, such as... Figure 4 As shown, it includes: The visual data processing module 401 is used to acquire the welding video stream, and sequentially perform sparse sampling, dense sampling and action segmentation on the welding video stream to obtain structured visual information. The consistency matching module 402 is used to acquire language instructions and perform consistency matching between the language instructions and structured visual information to obtain the matching result. The inference prediction module 403 is used to select and execute the corresponding inference prediction strategy based on the matching results to obtain prediction results, including: stage identification results, anomaly judgment results, and process decision suggestions.

[0094] The welding process visual language understanding and decision-making device based on action chain reasoning provided in this embodiment of the invention can execute the welding process visual language understanding and decision-making method based on action chain reasoning provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution. Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0095] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0096] The following is a detailed reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from memory 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device. The processor 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0097] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0098] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a memory 508, or installed from a ROM 502. When the computer program is executed by the processor 501, it performs the functions defined in the action chain reasoning-based visual language understanding and decision-making method for welding processes according to embodiments of the present invention.

[0099] Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0100] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the visual language understanding and decision-making method for welding processes based on action chain reasoning shown in the above embodiments is implemented.

[0101] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A visual language understanding and decision-making method for welding processes based on action chain reasoning, characterized in that, The method includes: The welding video stream is acquired, and sparse sampling, dense sampling, and action segmentation are performed sequentially on the welding video stream to obtain structured visual information. Obtain language instructions and perform consistency matching between the language instructions and the structured visual information to obtain matching results; Based on the matching results, the corresponding inference prediction strategy is selected and executed to obtain the prediction results, which include: stage identification results, anomaly determination results, and process decision suggestions.

2. The method according to claim 1, characterized in that, The welding video stream is sequentially subjected to sparse sampling, dense sampling, and action segmentation to obtain structured visual information, including: The welding video stream is sparsely sampled to obtain the global context; The keyframe region is determined based on the global context, and the keyframe region is densely sampled to capture fine movements. Action segmentation is performed based on the global context and fine-grained actions, the continuous welding process is discretized into action atoms, and then structured representation is performed to obtain structured visual information.

3. The method according to claim 2, characterized in that, Based on the global context, keyframe regions are determined, and these regions are densely sampled to capture fine movements, including: Based on the global context, coarse-grained stage information is determined, which includes: stage distribution, stage coarse-grained start and end times, and welding action type. The coarse-grained boundary of the action switching stage is determined based on the coarse-grained stage information, and the abrupt change point of the welding action is determined by combining the inter-frame optical flow changes. Within a preset time range of the coarse-grained boundary of the stage and the abrupt change point of the welding action, a key frame region is determined, and the key frame region is densely sampled to capture fine actions.

4. The method according to claim 1, characterized in that, The language instructions are matched with the structured visual information to obtain a matching result, including: From four perspectives—action type, temporal sequence, spatial location, and process constraints—a lightweight evaluator is used to compare the degree of conformity between the language instructions and the structured visual information. If the structured visual information contains action requirements, time intervals, spatial locations, and process constraints that match the language instructions, then the matching result is consistent; otherwise, the matching result is inconsistent.

5. The method according to claim 4, characterized in that, Based on the matching results, a corresponding inference prediction strategy is selected and executed to obtain prediction results, including: If the matching results are consistent, the stage recognition results, anomaly judgment results, and process decision suggestions are directly output based on the structured visual information as prediction results. If the matching results are inconsistent, deep reasoning is performed using multiple agents based on the language instructions and the structured visual information to obtain the prediction result.

6. The method according to claim 5, characterized in that, Based on the language instructions and the structured visual information, deep reasoning using multiple agents is employed to obtain prediction results, including: The language instructions are broken down into multiple sub-propositions using a planning agent; The tool agent converts each sub-proposition into a query operation, and retrieves the structured visual information based on the query operation to obtain visual evidence, which includes: action atoms and corresponding keyframes; By using a summarizing agent, the visual evidence is logically judged based on the sub-propositions to obtain the prediction result.

7. The method according to claim 6, characterized in that, The blackboard mechanism records and displays the language instructions, sub-propositions, time intervals of each sub-proposition, visual evidence, and prediction results.

8. A visual language understanding and decision-making device for welding processes based on action chain reasoning, characterized in that, The device includes: The visual data processing module is used to acquire the welding video stream and sequentially perform sparse sampling, dense sampling and action segmentation on the welding video stream to obtain structured visual information. The consistency matching module is used to acquire language instructions and perform consistency matching between the language instructions and the structured visual information to obtain a matching result; The inference prediction module is used to select and execute the corresponding inference prediction strategy based on the matching result to obtain the prediction result, which includes: stage identification result, anomaly judgment result, and process decision suggestion.

9. An electronic device, characterized in that, include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method of any one of claims 1 to 7.

Images