Cantilevered ship sub-assembly structure recognition vision sensing system based on image sensing

Abstract

The application discloses a cantilever type ship small group structure identification visual sensing system based on image sensing, and relates to the technical field of ship building and manufacturing. The visual sensing processing center is communicatively connected with the following modules, wherein: an image acquisition and filtering engine is used to acquire image data of the ship small group structure by using an image sensor, and to perform filtering operation on the acquired image data and integrate to form a comprehensive image set. The application acquires detailed image data of the surface of the ship small group structure by using a high-resolution image sensor, and performs filtering processing such as denoising, contrast enhancement and edge sharpening, thereby effectively improving the image quality. The double-branch classification network can accurately distinguish the weld and the defect, the welding trajectory is predicted and smoothed by combining the LSTM time sequence network and the Kalman filter, thereby significantly improving the accuracy and real-time performance of the welding path, and avoiding welding path errors caused by false identification and workpiece displacement.

Description

Technical Field

[0001] This invention relates to the field of shipbuilding and manufacturing technology, and specifically to a visual sensing system for recognizing the modular structure of cantilevered ships based on image sensing. Background Technology

[0002] Shipbuilding involves a large number of complex structural workpieces, such as elbow plates and T-sections. The accurate identification and positioning of these workpieces is a prerequisite for the precision requirements of welding. Moreover, with the expansion of shipbuilding scale and the improvement of construction precision, traditional manual welding and simple mechanical welding can no longer meet the production needs of high efficiency and high quality. Automated welding equipment has become the key to improving the efficiency and quality of shipbuilding. Through devices such as 3D smart cameras, workpiece information can be automatically scanned and identified, providing accurate data support for welding tasks.

[0003] In existing technologies, the surface of steel used in ships often has defects such as rust, oil stains, and scratches. When traditional image processing extracts welds through threshold segmentation or edge detection, it is easy to misjudge surface defects as weld features, triggering false recognition and leading to incorrect welding paths. Furthermore, the workpiece may undergo slight displacement due to thermal deformation during the welding process. Traditional vision systems need to track this displacement through inter-frame difference or optical flow methods. The computational delay will cause the welding path to lag, thus affecting the weld quality. Therefore, how to construct a dual-branch classification network to reduce false recognition, improve the accuracy of welding paths, reduce delay, and improve the real-time performance of welding paths is the problem that this invention aims to solve. To this end, an image sensing-based visual sensing system for recognizing the cantilevered ship group assembly structure is proposed. Summary of the Invention

[0004] The purpose of this invention is to provide a visual sensing system for recognizing the cantilevered ship group assembly structure based on image sensing, so as to solve the problems mentioned in the background art.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] A visual sensing system for recognizing the modular assembly structure of cantilevered ships based on image sensing includes a visual sensing processing center. The visual sensing processing center is communicatively connected to the following modules:

[0007] The image acquisition and filtering engine is used to acquire image data of the ship's assembly structure using a high-resolution image sensor, including information such as welds, rust, oil stains, and scratches on the workpiece surface, and to filter the acquired image data and integrate it into a comprehensive image set.

[0008] The dual-branch classification module is used to construct a dual-branch classification network that includes extracting weld geometric features and identifying defect textures, and to fuse the identification results of the two branches to distinguish between welds and defects in order to determine the location of the weld.

[0009] The trajectory prediction and compensation module is used to introduce a Long Short-Term Memory (LSTM) time-series network to determine the welding trajectory and smooth the predicted welding trajectory to perform displacement compensation.

[0010] The welding path planning module is used to plan a precise welding path based on the determined weld position and the welding trajectory after displacement compensation, so as to generate a welding path that meets the actual welding requirements.

[0011] The welding execution controller is used to transmit the planned welding path to the robotic welding system, and control the robotic welding system to perform welding operations according to the planned welding path, so as to realize the automated welding process.

[0012] A further improvement to the technical solution of the present invention is that the image acquisition and filtering engine specifically includes:

[0013] Using high-resolution image sensors mounted on the robotic welding system, images of the ship's assembly structure are acquired according to preset parameters, capturing details of welds, rust, oil stains, and scratches to ensure coverage of critical areas and obtain complete and comprehensive image data;

[0014] The acquired image data is filtered through processes including denoising, contrast enhancement, and edge sharpening to remove noise interference and irrelevant information, highlight weld and defect features, improve image purity and reliability, and enhance image clarity.

[0015] Based on the acquisition sequence and location information, the filtered image data are arranged in an orderly manner and integrated into a comprehensive image set, reflecting the overall condition of the ship group's assembly structure, effectively improving image quality and enhancing clarity and contrast.

[0016] A further improvement of the technical solution of the present invention is that the dual-branch classification module includes a dual-branch classification network construction unit and a fusion decision unit;

[0017] The dual-branch classification network construction unit is used to construct a dual-branch classification network. One branch is used to extract the geometric features of the weld, and learns the geometric information of the weld, including its shape and size, through a convolutional neural network structure. The other branch is used to identify defect textures, and captures defect texture features, including rust, oil stains and scratches, using different network layers.

[0018] The fusion decision unit is used to fuse the identification results of the two branches, combine the geometric features of the weld and the texture features of the defect, and determine whether the target area in the image is a weld or a defect, thereby determining the location of the weld.

[0019] A further improvement to the technical solution of the present invention is that the dual-branch classification network construction unit specifically includes:

[0020] Design a neural network architecture with two independent branches, one for extracting weld geometric features and the other for identifying defect textures. During initialization, assign parameters for the number of network layers and neurons to the two branches.

[0021] The filtered image data is input into two branches for independent training. The branch for extracting weld geometric features adjusts parameters through forward and backward propagation to reduce extraction errors, while the branch for identifying defect textures optimizes the loss function to improve recognition accuracy. The two branches are trained independently, focusing on their respective tasks to improve performance.

[0022] After independent training is completed, the two branches are integrated into a dual-branch classification network. The connections and parameters between the branches are adjusted, and the joint optimization algorithm is used to synthesize the output results. The overall network is then fine-tuned, and the fine-tuned output results are transmitted to the fusion decision unit for the final judgment of welds and defects.

[0023] A further improvement to the technical solution of the present invention is that the fusion decision unit specifically includes:

[0024] We receive weld geometric features and defect texture features from two independent branches, normalize the spatial coordinates of the weld geometric features, and standardize the numerical range of the two types of features to avoid features with excessively large values ​​dominating the fusion result.

[0025] A weighted summation strategy is adopted to deeply integrate weld geometric features and defect texture features through learnable weights. The weights are optimized during the fusion process, and the complementarity of the two types of features is automatically learned to improve the ability to distinguish between welds and defects.

[0026] The fused multimodal features are input into the decision classification layer, and the fully connected network determines whether the target region is a weld or a defect. The classifier outputs the category probability, and the final category is determined based on the maximum probability criterion. If it is a weld, the precise location is located by combining the geometric features of the weld; if it is a defect, the defect type is analyzed, and the final classification result and weld location coordinates are output.

[0027] A further improvement of the technical solution of the present invention is that the trajectory prediction and compensation module includes an LSTM temporal network prediction unit and a displacement smoothing compensation unit;

[0028] The LSTM temporal network prediction unit is used to introduce a Long Short-Term Memory (LSTM) temporal network to predict the changing trend of the weld position in the current frame based on the determined weld position and the weld position information in historical frames, thereby determining the welding trajectory of the weld.

[0029] The displacement smoothing compensation unit is used to smooth the predicted welding trajectory using a Kalman filter algorithm to remove noise interference, thereby achieving displacement compensation of the welding trajectory, reducing the delay of the welding path, and making the welding path more real-time and accurate.

[0030] A further improvement to the technical solution of this invention is that the LSTM temporal network prediction unit specifically includes:

[0031] We acquire weld position information from current and historical frames from a visual sensing system, extract precise coordinates from multiple consecutive frames to form a position coordinate sequence, perform differential processing on the position coordinates to obtain a position change sequence, combine the position coordinate sequence and the position change sequence to construct a temporal feature vector and divide it into a training set and a test set.

[0032] The temporal feature vector is input into the LSTM network. The long-term dependencies in the long sequence data are processed through the input gate, forget gate and output gate. The mean squared error loss function is used. The parameters are updated through backpropagation and Adam optimization algorithm to learn the mapping relationship between the historical frame and the current frame position.

[0033] The latest historical frame weld position information is input into the trained LSTM network, which outputs the predicted coordinates and position change trend of the weld position in the current frame. Based on this, the welding trajectory of the weld is dynamically planned and output.

[0034] A further improvement to the technical solution of the present invention is that the displacement smoothing compensation unit specifically includes:

[0035] Based on the initial state predicted by the welding trajectory, the state variables, observation variables, and system parameters of the Kalman filter are set. The state variables include the position and velocity information of the welding head in two-dimensional or three-dimensional space, the observation variables are the trajectory coordinates collected by the sensor in real time, and the system parameters include the state transition matrix, the observation matrix, the process noise covariance matrix, and the observation noise covariance matrix.

[0036] During the welding process, the Kalman filter iterates step by step. In each step, based on the optimal state estimate and state transition matrix of the previous moment, the state variables and covariance matrix of the current moment are predicted. At the same time, the uncertainty of the prediction is corrected by combining the process noise covariance matrix. Using the observation value of the current moment (the trajectory coordinates collected by the sensor), the predicted state is mapped to the observation space through the observation matrix, and then the Kalman gain is calculated. Finally, the predicted state is corrected by combining the Kalman gain to obtain the optimal state estimate of the current moment, and the covariance matrix is ​​updated to reflect the corrected uncertainty.

[0037] After iterative updates, the optimal state estimate output by the Kalman filter is the smoothed welding trajectory coordinates. Based on this, the smoothed welding trajectory is compared with the original predicted welding trajectory, the displacement deviation is calculated, and a compensation command is generated. By adjusting the motion control parameters of the welding equipment, the actual path of the welding head is corrected in real time.

[0038] A further improvement to the technical solution of the present invention is that the welding path planning module specifically includes:

[0039] It receives input data from a vision sensing system and a Kalman filter, including the determined weld position and the welding trajectory after displacement compensation. It integrates the weld position and the compensated welding trajectory to form a complete welding path dataset.

[0040] Based on the integrated welding path dataset, a geometry-based path planning method is used to generate a preliminary welding path. The generated welding path is then optimized by combining the dynamic characteristics of the welding process and the physical characteristics of the workpiece. By analyzing the weld topology, the welding trajectory is decomposed into multiple basic motion units, and the transition path between units is optimized. At the same time, the spatial posture of the path is dynamically adjusted by combining the welding process parameters to ensure that the welding torch and the workpiece always maintain the optimal relative position during the welding process, thus ensuring that the generated path meets the actual welding requirements.

[0041] After the welding path is optimized, the final welding path is converted into an instruction set that can be executed by the robot welding system, including the motion coordinates, speed and acceleration parameters of each axis, and a time-seriesd robot welding path file is generated and transmitted to the welding execution controller.

[0042] A further improvement to the technical solution of the present invention is that the welding execution controller specifically includes:

[0043] The welding execution controller receives the robot welding path file output from the welding path planning module through a preset communication interface. It performs integrity verification on the received robot welding path file to ensure that the data transmission is error-free. Then, it parses the robot welding path file to extract the sequence instructions for each axis movement, ensuring that the controller can accurately understand the various requirements of the planned path.

[0044] After the path is parsed, the welding execution controller sends control commands to the robot welding system based on the parsed data. Through real-time communication with each drive module of the robot, it precisely controls the movement of each axis. At the same time, it monitors the robot's motion status in real time, including parameters such as position, speed and acceleration, and compares them with the planned path in real time. Once a deviation is detected, the control commands are adjusted immediately to ensure that the robot performs welding operations according to the planned path.

[0045] During the welding process, the robot continuously monitors the welding quality in real time, acquiring quality parameters such as weld width, depth, reinforcement height, welding current, and voltage. These parameters are compared with preset quality standards. If any deviation in weld quality is detected, the control strategy is immediately adjusted. Simultaneously, the welding controller records and stores various data and quality inspection results during the welding process for subsequent quality traceability and analysis.

[0046] Due to the adoption of the above technical solution, the technical progress achieved by this invention compared to the prior art is as follows:

[0047] 1. This invention provides a visual sensing system for recognizing cantilevered ship assembly structures based on image sensing. It acquires detailed image data of the surface of the ship assembly structure through a high-resolution image sensor and performs filtering processes such as noise reduction, contrast enhancement, and edge sharpening to effectively improve image quality. It can accurately distinguish between welds and defects by using a dual-branch classification network. It combines an LSTM temporal network and a Kalman filter to predict and smooth the welding trajectory, which significantly improves the accuracy and real-time performance of the welding path and avoids welding path errors caused by false recognition and workpiece displacement.

[0048] 2. This invention provides a visual sensing system for recognizing the assembly structure of cantilevered ships based on image sensing. By introducing an LSTM time-series network and combining the weld position of historical frames to predict the displacement of the current frame, and using a Kalman filter algorithm to smooth the predicted welding trajectory, the computational delay can be reduced, noise interference can be removed, and displacement compensation of the welding trajectory can be achieved, making the welding path more real-time and accurate. This effectively addresses the small displacements of the workpiece caused by thermal deformation during the welding process and ensures stable welding quality. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0050] Figure 1 This is a data flow diagram of the image sensing-based visual sensing system for recognizing the modular structure of cantilevered ships according to the present invention.

[0051] Figure 2 This is a schematic diagram illustrating the workflow of the image sensing-based visual sensing system for recognizing the cantilevered ship group assembly structure according to the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] Example 1, as Figure 1 , Figure 2 As shown, this invention provides a visual sensing system for recognizing the modular structure of a cantilevered ship based on image sensing, including a visual sensing processing center. The visual sensing processing center is communicatively connected to the following modules, wherein:

[0054] The image acquisition and filtering engine utilizes a high-resolution image sensor to acquire image data of the ship's sub-assembly structure, including information such as welds, rust, oil stains, and scratches on the workpiece surface. It filters the acquired image data and integrates it into a comprehensive image set, effectively improving image quality, enhancing image clarity and contrast, and reducing subsequent processing errors caused by image quality issues. Using a high-resolution image sensor mounted on a robotic welding system, it acquires images of the ship's sub-assembly structure according to preset parameters, capturing details of welds, rust, oil stains, and scratches to ensure coverage of key areas and obtain complete and comprehensive image data. The acquired image data undergoes filtering processes including noise reduction, contrast enhancement, and edge sharpening to remove noise interference and irrelevant information, highlighting weld and defect features, improving image purity and reliability, and enhancing image clarity. Based on the acquisition sequence and location information, the filtered image data is arranged in an orderly manner and integrated into a comprehensive image set, reflecting the overall condition of the ship's sub-assembly structure, effectively improving image quality and enhancing clarity and contrast.

[0055] The image acquisition and filtering engine works as follows: Utilizing high-resolution image sensors deployed on the robotic welding system, and based on preset acquisition parameters including resolution, frame rate, and exposure time, it acquires image data of the ship assembly structure, capturing various details on the workpiece surface, including the shape and orientation of welds, the distribution of rust, the area covered by oil, and the depth of scratches. This ensures coverage of key parts of the entire ship assembly structure, avoiding blind spots and guaranteeing complete and comprehensive image data. The workpiece types include elbow plates and T-plates. The profiles and other components can be installed with one or two stiffeners. When there are two stiffeners, the included angle between the stiffeners should be 75° to 105°, and the distance between the ends should meet the requirements for the welding torch to pass through. The workpiece dimensions should be ≥0.4 meters in length (enough to be fixed on the welding platform by itself without auxiliary positioning), the weld position on the part should be ≥10mm from the edge of the base plate, and the part height should be ≤700mm. The surface condition of the plate should be 15μm to 25μm thick anti-corrosion primer (without affecting welding). The workpiece condition should be that the assembly and positioning welding is completed, the weld corner is less than 3mm, the assembly gap between plates is ≤2mm, and the angle between the stiffener and the main plate is 90°±1. 5°, workpiece assembly error ≤ ±2mm; workpiece placement requirements: workpieces must maintain a gap > 35mm, welds must be within the platform range, base plates are allowed to extend beyond the platform, workpieces must be horizontal during welding, and cannot be tilted; welding types include horizontal fillet welds, corner welds, and intermittent welds, excluding assembly welds, and the maximum allowable workpiece height is 700mm; the acquired image data is filtered, including noise reduction, contrast enhancement, and edge sharpening, to eliminate noise interference and irrelevant information. Noise reduction removes random noise from the image data, contrast enhancement improves the clarity of the image data, making the characteristics of welds and defects more obvious, and edge sharpening highlights key features in the image to reduce the negative impact of noise on image quality and improve image purity and reliability; the filtered image data is integrated, and the image data is arranged in an orderly manner according to the acquisition order and location information, combining the scattered individual image data into a comprehensive image set to comprehensively and accurately reflect the overall condition of the ship group assembly structure, effectively improving image quality, clarity, and contrast;

[0056] The dual-branch classification module is used to construct a dual-branch classification network that includes extracting weld geometric features and identifying defect textures, and to fuse the identification results of the two branches to distinguish between welds and defects in order to determine the location of the weld. The dual-branch classification module includes a dual-branch classification network construction unit and a fusion decision unit.

[0057] The dual-branch classification network construction unit is used to build a dual-branch classification network. One branch is used to extract weld geometric features, learning the geometric information of the weld, including its shape and size, through a convolutional neural network structure. The other branch is used to identify defect textures, capturing defect texture features including rust, oil stains, and scratches using different network layers. A neural network architecture with two independent branches is designed, one for extracting weld geometric features and the other for identifying defect textures. During initialization, parameters for the number of network layers and neurons are assigned to the two branches. Filtered image data is input into the two branches for independent training. The weld geometric feature extraction branch adjusts parameters through forward and backward propagation to reduce extraction errors, while the defect texture identification branch optimizes the loss function to improve recognition accuracy. The two branches are trained independently, focusing on their respective tasks to improve performance. After independent training, the two branches are integrated into a dual-branch classification network. The connections and parameters between the branches are adjusted, and a joint optimization algorithm is used to synthesize the output results. The overall network is then fine-tuned, and the fine-tuned output results are transmitted to the fusion decision unit for the final weld and defect judgment.

[0058] The specific work of the dual-branch classification network construction unit is as follows: Design a neural network architecture with two independent branches. For the two tasks of extracting weld geometric features and recognizing defect textures, design independent branch network structures for each. During the initialization phase, assign parameters for the number of network layers and neurons to the two branches. Specifically, for the branch extracting weld geometric features, determine the structure of the convolutional neural network based on the weld shape and size characteristics, including the number and arrangement of convolutional and pooling layers to ensure effective learning of the weld's geometric information. For the branch recognizing defect textures, design corresponding network layers based on the characteristics of rust, oil stains, and scratches, utilizing the characteristics of different network layers... The system captures defect texture features. One branch extracts weld geometry features, aiming to learn the shape and size of the weld. The input layer receives filtered image data containing weld shape and size information. Multiple convolutional layers extract local features from the image data, each followed by a ReLU activation function to introduce non-linearity. Pooling layers reduce the spatial dimensionality of the feature map, decreasing computation while preserving important features. Fully connected layers flatten the feature maps extracted by the convolutional and pooling layers, performing classification or regression to output the weld geometry features. Another branch identifies defect textures, such as rust, oil stains, and scratches. The input layer receives... After filtering, the image data contains defect textures such as rust, oil stains, and scratches. Multiple convolutional layers are used to extract texture features from the images, with each convolutional layer followed by a ReLU activation function. Pooling layers are used to reduce the spatial dimension of the feature maps, and fully connected layers are used to flatten the feature maps extracted by the convolutional and pooling layers. Classification or regression is then performed through the fully connected layers to output the defect texture features. The filtered ship assembly structure image data is assigned to two branches. The weld geometry feature extraction branch receives image data containing weld shape and size information. Forward and backward propagation training is performed using a convolutional neural network structure. During training, the network parameters are adjusted to ensure that the network accurately learns the geometric features of the weld. To reduce errors in extracting weld geometry information, the defect texture identification branch is trained using defect texture image data containing rust, oil, and scratches. Different network layers are used to capture and enhance defect texture features. By optimizing the loss function, the accuracy of this branch in identifying defect textures is improved. The two branches are trained independently to improve the performance of their respective tasks. After the two branches have been trained independently, they are integrated into a complete dual-branch classification network. The connection method and parameters between the two branches are further adjusted. Through a joint optimization algorithm, the output results of the two branches are combined to fine-tune the entire dual-branch classification network. The fine-tuned output results of the two branches are then transmitted to the fusion decision unit.

[0059] The fusion decision unit is used to fuse the recognition results of the two branches, combining weld geometric features and defect texture features to determine whether the target area in the image is a weld or a defect, thereby determining the weld location. It receives weld geometric features and defect texture features from the two independent branches, performs spatial coordinate normalization on the weld geometric features, and standardizes the numerical range of the two types of features to avoid features with excessively large values ​​dominating the fusion result. A weighted summation strategy is adopted, deeply integrating weld geometric features and defect texture features through learnable weights. The weights are optimized during the fusion process, automatically learning the complementarity of the two types of features to improve the ability to distinguish between welds and defects. The fused multimodal features are input into the decision classification layer, where a fully connected network determines whether the target area is a weld or a defect. The classifier outputs the class probability, and the final class is determined based on the maximum probability criterion. If it is a weld, the precise location is located by combining the weld geometric features; if it is a defect, the defect type is analyzed, and the final classification result and weld location coordinates are output.

[0060] The specific tasks of the fusion decision unit are as follows: It receives extracted feature information from two independent branch networks. The weld geometry feature branch provides geometric information about the weld shape and size, while the defect texture feature branch provides texture features of rust, oil stains, and scratches. The unit then aligns and preprocesses the weld geometry and defect texture features. Specifically, it normalizes the spatial coordinates of the geometric features to ensure their correspondence with the texture features in the image coordinate system. Simultaneously, it standardizes the numerical ranges of the two types of features to prevent one type of feature from dominating the fusion result due to excessively large values. After preprocessing, a weighted summation strategy is used to deeply integrate the weld geometry and defect texture features. The weighted summation is achieved through... The system learns the importance of weighting the two types of features, highlighting key information, optimizing the fusion weights during the fusion process, and automatically learning the complementarity of the two types of features to improve the ability to distinguish between welds and defects. The fused multimodal features are input to the decision classification layer, which is composed of a fully connected network to determine whether the target area is a weld or a defect. The classifier learns the mapping relationship between the fused features and the label, outputs the probability of the weld or defect category, and determines the final category based on the maximum probability criterion. At the same time, it combines the spatial coordinate information in the weld's geometric features to further locate the precise position of the weld in the image. If a defect is detected, the defect type is analyzed through texture features. Finally, the classification result and the weld position coordinates are output.

[0061] The trajectory prediction and compensation module is used to introduce a Long Short-Term Memory (LSTM) time-series network to determine the welding trajectory and smooth the predicted welding trajectory to perform displacement compensation.

[0062] The welding path planning module is used to plan a precise welding path based on the determined weld position and the welding trajectory after displacement compensation, so as to generate a welding path that meets the actual welding requirements, avoid welding path errors caused by false identification and workpiece displacement, and improve welding quality and efficiency.

[0063] The welding execution controller is used to transmit the planned welding path to the robotic welding system, control the robotic welding system to perform welding operations according to the planned welding path, realize the automated welding process, ensure the consistency and stability of weld quality, and improve the welding level of the entire ship assembly structure.

[0064] Example 2, as Figure 1 , Figure 2 As shown, based on Embodiment 1, the present invention provides a technical solution: preferably, the trajectory prediction compensation module includes an LSTM time-series network prediction unit and a displacement smoothing compensation unit;

[0065] The LSTM temporal prediction unit is used to introduce a Long Short-Term Memory (LSTM) temporal network. Based on the determined weld position and combined with the weld position information in historical frames, it predicts the change trend of the weld position in the current frame and determines the welding trajectory of the weld. It obtains the weld position information of the current and historical frames from the visual sensing system, extracts the precise coordinates of multiple consecutive frames to form a position coordinate sequence, performs differential processing on the position coordinates to obtain the position change sequence, combines the position coordinate sequence and the position change sequence to construct a temporal feature vector and divides it into training and test sets. The temporal feature vector is input into the LSTM network, and the long-term dependencies in the long sequence data are processed through the input gate, forget gate and output gate. The mean squared error loss function is used, and the parameters are updated through backpropagation and Adam optimization algorithm. The mapping relationship between the positions of historical frames and the current frame is learned. The weld position information of the latest historical frame is input into the trained LSTM network, and the predicted coordinates and position change trend of the weld position in the current frame are output. Based on this, the welding trajectory of the determined weld is dynamically planned and output.

[0066] The specific tasks of the LSTM temporal network prediction unit are as follows: It acquires the weld position information from the current and historical frames in the visual sensing system, extracts the precise coordinates of the weld from multiple consecutive frames (including horizontal and vertical coordinates), forms a position coordinate sequence, and performs differential processing on the position coordinates to obtain a position change sequence, capturing the dynamic information of weld movement. The position coordinate sequence and the position change sequence are combined to construct a complete temporal feature vector. The training and test sets are then divided according to time sequence to ensure that the training data contains sufficient historical temporal information. The prepared temporal feature vector is then input into the LSTM network for training. The LSTM network includes an input gate, a forget gate, and an output gate to handle long-term dependencies in long-sequence data. During the process, the LSTM network adjusts its internal weight parameters to minimize the error between the predicted and actual values. It uses a mean squared error loss function to measure the degree of deviation between the predicted and actual positions. Through backpropagation and Adam optimization algorithms, the network parameters are updated step by step, enabling the LSTM network to learn the mapping relationship between weld position information in historical frames and the current frame position. Based on the trained LSTM network, the latest historical frame weld position information is input to predict the weld position in the current frame. According to the learned temporal patterns, the LSTM network outputs the predicted coordinates and position change trend of the weld position in the current frame. Combining the prediction results, the welding trajectory of the weld is dynamically planned, and then the determined welding trajectory of the weld is output.

[0067] The displacement smoothing compensation unit is used to smooth the predicted welding trajectory using a Kalman filter algorithm, removing noise interference to achieve displacement compensation of the welding trajectory, reduce welding path delay, and make the welding path more real-time and accurate. Based on the initial state of the predicted welding trajectory, the state variables, observation variables, and system parameters of the Kalman filter are set. The state variables include the position and velocity information of the welding head in two-dimensional or three-dimensional space; the observation variables are the trajectory coordinates acquired by the sensor in real time; and the system parameters include the state transition matrix (describing the dynamic characteristics of the trajectory, such as a uniform or uniformly accelerated model), the observation matrix (corresponding to the state variables and observation variables), the process noise covariance matrix (characterizing the uncertainty of the system model, such as mechanical vibration or control error), and the observation noise covariance matrix (reflecting sensor measurement error). During the welding process, the Kalman filter runs iteratively according to the time step. In each step, based on the optimal state estimate and state of the previous moment... The transition matrix predicts the current state variables and covariance matrix. It also incorporates the process noise covariance matrix to correct prediction uncertainties. Using the current observations (trajectory coordinates acquired by sensors), the predicted state is mapped to the observation space via the observation matrix. The Kalman gain is then calculated (balancing the reliability of the predicted and observed values). Finally, the Kalman gain is used to correct the predicted state, yielding the optimal state estimate for the current moment. The covariance matrix is ​​updated to reflect the corrected uncertainty. After iterative updates, the optimal state estimate output by the Kalman filter is the smoothed welding trajectory coordinates. Based on this, the smoothed welding trajectory is compared with the original predicted welding trajectory to calculate the displacement deviation and generate compensation commands. By adjusting the motion control parameters of the welding equipment, the actual path of the welding head is corrected in real time, thereby reducing path deviations caused by system delays or noise interference and ensuring a high degree of consistency between the welding path and the theoretical design.

[0068] The welding path planning module specifically includes:

[0069] The system receives input data from a vision sensing system and a Kalman filter, including the determined weld position and the welding trajectory after displacement compensation. The weld position and the compensated welding trajectory are integrated to form a complete welding path dataset, containing all positional information that the welding head needs to follow during the welding process. Based on the integrated welding path dataset, a geometry-based path planning method is used to generate a preliminary welding path. The generated welding path is then optimized by considering the dynamic characteristics of the welding process and the physical characteristics of the workpiece. By analyzing the weld topology, the welding trajectory is decomposed into multiple basic motion units, optimizing the transition paths between units to reduce frequent starts and stops and sudden changes in direction of the welding head, thus improving motion stability. Simultaneously, the spatial attitude of the path is dynamically adjusted based on welding process parameters to ensure that the welding torch and workpiece maintain the optimal relative position during the welding process, ensuring that the generated path meets actual welding requirements. After optimizing the welding path, the final welding path is converted into an executable instruction set for the robot welding system, including the motion coordinates, velocity, and acceleration parameters of each axis, and a time-seriesd robot welding path file is generated and transmitted to the welding execution controller.

[0070] The welding execution controller specifically includes:

[0071] The welding execution controller receives the robot welding path file output from the welding path planning module via a preset communication interface. This file contains the motion coordinates, velocity, and acceleration parameters that each robot axis must follow during the welding process. The welding execution controller performs an integrity check on the received robot welding path file to ensure error-free data transmission. Then, it parses the robot welding path file to extract the sequence instructions for each axis's movement, ensuring the controller accurately understands the requirements of the planned path. After path parsing, the welding execution controller sends control commands to the robot welding system based on the parsed data. Through real-time communication with each drive module of the robot, it precisely controls the movement of each axis and simultaneously monitors the robot's motion status in real time, including position, velocity, and acceleration. The parameters are compared with the planned path in real time. Once a deviation is detected, the control commands are immediately adjusted to ensure that the robot performs welding operations according to the planned path. During the welding process, the welding execution controller continuously monitors the welding quality in real time. Through various sensors integrated into the robot welding system, it acquires quality parameters during the welding process, including weld width, weld depth, weld reinforcement height, welding current, and voltage. The quality parameters are compared and analyzed with preset quality standards. Once a deviation in weld quality is detected, the control strategy is immediately adjusted, such as fine-tuning the robot's motion path or welding parameters. At the same time, the welding execution controller records and stores various data and quality inspection results during the welding process for subsequent quality traceability and analysis.

[0072] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A visual sensing system for recognizing the modular structure of a cantilevered ship based on image sensing, comprising a visual sensing processing center, characterized in that, The visual sensing processing center has the following communication connection modules, wherein: The image acquisition and filtering engine is used to acquire image data of the ship's modular structure using image sensors, and to filter and integrate the acquired image data to form a comprehensive image set. The dual-branch classification module is used to construct a dual-branch classification network that includes extracting weld geometric features and identifying defect textures, and to fuse the identification results of the two branches to distinguish between welds and defects in order to determine the location of the weld. The trajectory prediction and compensation module is used to introduce a long short-term memory network (LSTM) to determine the welding trajectory and smooth the predicted welding trajectory to perform displacement compensation. The trajectory prediction and compensation module includes an LSTM time-series network prediction unit and a displacement smoothing compensation unit. The LSTM temporal network prediction unit is used to introduce a long short-term memory (LSTM) temporal network to predict the changing trend of the weld position in the current frame based on the determined weld position and the weld position information in historical frames, thereby determining the welding trajectory of the weld. Specifically, this includes: We acquire weld position information from current and historical frames from a visual sensing system, extract precise coordinates from multiple consecutive frames to form a position coordinate sequence, perform differential processing on the position coordinates to obtain a position change sequence, combine the position coordinate sequence and the position change sequence to construct a temporal feature vector and divide it into a training set and a test set. The temporal feature vector is input into the LSTM network. The long-term dependencies in the long sequence data are processed through the input gate, forget gate and output gate. The mean squared error loss function is used. The parameters are updated through backpropagation and Adam optimization algorithm to learn the mapping relationship between the historical frame and the current frame position. The latest historical frame weld position information is input into the trained LSTM network, which outputs the predicted coordinates and position change trend of the weld position in the current frame. Based on this, the welding trajectory of the determined weld is dynamically planned and output. The displacement smoothing compensation unit is used to smooth the predicted welding trajectory using a Kalman filter algorithm to remove noise interference, thereby achieving displacement compensation of the welding trajectory. Specifically, it includes: Based on the initial state predicted by the welding trajectory, the state variables, observation variables, and system parameters of the Kalman filter are set. The state variables include the position and velocity information of the welding head in two-dimensional or three-dimensional space, the observation variables are the trajectory coordinates collected by the sensor in real time, and the system parameters include the state transition matrix, the observation matrix, the process noise covariance matrix, and the observation noise covariance matrix. During the welding process, the Kalman filter runs iteratively according to the time step. In each step, based on the optimal state estimate and state transition matrix of the previous moment, the state variables and covariance matrix of the current moment are predicted. At the same time, the uncertainty of the prediction is corrected by combining the process noise covariance matrix. Using the observation value of the current moment, the predicted state is mapped to the observation space through the observation matrix. Then, the Kalman gain is calculated to correct the predicted state and obtain the optimal state estimate of the current moment. After iterative updates, the optimal state estimate output by the Kalman filter is the smoothed welding trajectory coordinates. Based on this, the smoothed welding trajectory is compared with the original predicted welding trajectory, the displacement deviation is calculated, and a compensation command is generated. By adjusting the motion control parameters of the welding equipment, the actual path of the welding head is corrected in real time. The welding path planning module is used to generate a welding path that meets the actual welding requirements based on the determined weld position and the welding trajectory after displacement compensation. The welding execution controller is used to transmit the planned welding path to the robotic welding system and control the robotic welding system to perform welding operations according to the planned welding path.

2. The image sensing-based visual sensing system for recognizing the cantilevered ship assembly structure according to claim 1, characterized in that: The image acquisition and filtering engine specifically includes: Using the image sensor mounted on the robotic welding system, images of the ship's assembly structure are acquired according to preset parameters, capturing details of welds, rust, oil stains and scratches, and obtaining complete and comprehensive image data; The acquired image data is filtered to remove noise interference and irrelevant information, highlighting weld and defect features; Based on the acquisition sequence and location information, the filtered image data are arranged in an orderly manner and integrated into a comprehensive image set to reflect the overall condition of the ship group's stand-up structure.

3. The image sensing-based visual sensing system for recognizing the cantilevered ship assembly structure according to claim 1, characterized in that: The dual-branch classification module includes a dual-branch classification network construction unit and a fusion decision unit; The dual-branch classification network construction unit is used to construct a dual-branch classification network. One branch is used to extract the geometric features of the weld, and learns the geometric information of the weld, including its shape and size, through a convolutional neural network structure. The other branch is used to identify defect textures, and captures defect texture features, including rust, oil stains and scratches, using different network layers. The fusion decision unit is used to fuse the identification results of the two branches, combine the geometric features of the weld and the texture features of the defect, and determine whether the target area in the image is a weld or a defect, thereby determining the location of the weld.

4. The image sensing-based visual sensing system for recognizing the cantilevered ship assembly structure according to claim 3, characterized in that: The dual-branch classification network construction unit specifically includes: Design a neural network architecture with two independent branches, one for extracting weld geometric features and the other for identifying defect textures. During initialization, assign parameters for the number of network layers and neurons to the two branches. The filtered image data is input into two branches for independent training. The branch for extracting weld geometric features adjusts parameters through forward and backward propagation, while the branch for identifying defect textures optimizes the loss function. After independent training is completed, the two branches are integrated into a dual-branch classification network. The connections and parameters between the branches are adjusted, and the joint optimization algorithm is used to synthesize the output results. The overall network is then fine-tuned, and the fine-tuned output results are transmitted to the fusion decision unit.

5. The image sensing-based visual sensing system for recognizing the cantilevered ship assembly structure according to claim 3, characterized in that: The fusion decision-making unit specifically includes: Weld geometric features and defect texture features are received from two independent branches. Spatial coordinates of the weld geometric features are normalized, and the numerical ranges of the two types of features are standardized. A weighted summation strategy is adopted to deeply integrate weld geometric features and defect texture features through learnable weights. The weights are optimized during the fusion process, and the complementarity of the two types of features is automatically learned. The fused multimodal features are input into the decision classification layer, and the fully connected network determines whether the target region is a weld or a defect. The classifier outputs the category probability, and the final category is determined based on the maximum probability criterion. If it is a weld, the precise location is located by combining the geometric features of the weld; if it is a defect, the defect type is analyzed, and the final classification result and weld location coordinates are output.

6. The image sensing-based visual sensing system for recognizing the cantilevered ship assembly structure according to claim 1, characterized in that: The welding path planning module specifically includes: It receives input data from a vision sensing system and a Kalman filter, including the determined weld position and the welding trajectory after displacement compensation. It integrates the weld position and the compensated welding trajectory to form a complete welding path dataset. Based on the integrated welding path dataset, a geometry-based path planning method is used to generate a preliminary welding path. The generated welding path is then optimized by combining the dynamic characteristics of the welding process and the physical characteristics of the workpiece. By analyzing the weld topology, the welding trajectory is decomposed into multiple basic motion units, and the transition path between units is optimized. At the same time, the spatial posture of the path is dynamically adjusted by combining the welding process parameters to keep the welding torch and the workpiece in the best relative position during the welding process. After the welding path is optimized, the final welding path is converted into an instruction set that can be executed by the robot welding system, including the motion coordinates, speed and acceleration parameters of each axis, and a time-seriesd robot welding path file is generated and transmitted to the welding execution controller.

7. The image sensing-based visual sensing system for recognizing the cantilevered ship assembly structure according to claim 6, characterized in that: The welding execution controller specifically includes: The welding execution controller receives the robot welding path file output from the welding path planning module through a preset communication interface, performs integrity verification on the received robot welding path file, and parses the robot welding path file to extract the sequence instructions for each axis movement. After the path is parsed, the welding execution controller sends control commands to the robot welding system based on the parsed data. Through real-time communication with each drive module of the robot, it precisely controls the movement of each axis. At the same time, it monitors the robot's motion status in real time, including parameters such as position, speed and acceleration, and compares them with the planned path in real time. Once a deviation is detected, the control commands are adjusted immediately. During the welding process, the robot continuously monitors the welding quality in real time, acquiring quality parameters such as weld width, depth, reinforcement height, welding current, and voltage. The quality parameters are compared and analyzed with preset quality standards. If any deviation in weld quality is detected, the control strategy is immediately adjusted. At the same time, all data and quality inspection results during the welding process are recorded and stored.

Images