Ship pipe flange intelligent identification and assembly system

Abstract

The application discloses a kind of ship pipe flange intelligent identification assembly systems, the present application relates to ship intelligent manufacturing technical field, comprising: bimodal heterogeneous data acquisition unit, the bimodal heterogeneous data acquisition unit includes first camera and second camera;The first camera is used to collect the global image data of the scene where flange and steel pipe are located;The second camera is used to collect the local fine image data of flange end face and steel pipe end portion.The ship pipe flange intelligent identification assembly system, the ship pipe flange intelligent identification assembly system is combined by constructing the technical framework of bimodal heterogeneous data acquisition and multi-level fusion verification, realizes from image acquisition, multi-level check, the full-process automation closed-loop control of fetching execution to group and reinspection;Under the complex illumination and dynamic environment of ship workshop, effectively reduce the uncertainty and operating error caused by manual participation, improve the consistency and reliability of flange assembly process.

Description

Technical Field

[0001] This invention relates to the field of intelligent ship manufacturing technology, specifically to an intelligent identification and assembly system for ship pipe flanges. Background Technology

[0002] Today, the shipbuilding industry, as a typical complex equipment manufacturing industry, relies heavily on the precision and efficient processing and assembly of components. In the manufacturing of ship piping systems, the precise assembly of flanges and steel pipes is a crucial process, as its quality directly determines the sealing performance, structural strength, and operational safety and reliability of the entire piping system. Traditional flange assembly operations heavily depend on manual experience; workers must manually identify and select flanges and assemble them with steel pipes. This method is not only labor-intensive and inefficient, but also prone to assembly errors due to human factors, making it difficult to meet the ever-increasing demands of modern shipbuilding for precision, efficiency, and consistency.

[0003] To improve the intelligence level of ship pipe fitting assembly, some technological explorations have been carried out in related fields. For example, patent document with authorized publication number "CN119131488A" describes "an intelligent identification device and method for prefabricated ship pipe fittings based on visual inspection." This solution collects pipe fitting images through edge devices, uses a deep learning model to identify the structural features and text codes of the pipe fittings, including flanges and elbows; and compares the identification results with an engineering database, ultimately providing workers with pipe fitting model and assembly process information on an interactive interface, thus assisting manual identification and assembly to a certain extent. However, this solution mainly focuses on pipe fitting identification and information retrieval. Its final execution links, such as flange gripping, handling, and assembly with steel pipes, still rely on manual operation, failing to form a closed-loop automated process from identification to assembly. In actual production, manual assembly of flanges and steel pipes, especially when handling large or heavy pipe fittings, still suffers from unstable positioning accuracy, high operational risks, and difficulty in further improving efficiency.

[0004] To address the shortcomings of the existing technologies, this invention aims to provide an intelligent identification and assembly system for ship pipe flanges. This system can not only automatically identify the flange type, position, and angle, but also drive a robot to complete precise grasping and automatic assembly, thereby overcoming the defects of low efficiency, poor accuracy, and high risk of traditional manual assembly, and truly realizing the intelligentization and automation of the ship pipe flange assembly process. Summary of the Invention

[0005] The purpose of this invention is to provide an intelligent identification and assembly system for ship pipe flanges to solve the problems mentioned in the background art.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a ship pipe flange intelligent identification and assembly system, comprising: a dual-modal heterogeneous data acquisition unit, wherein the dual-modal heterogeneous data acquisition unit includes a first camera and a second camera; the first camera is used to acquire global image data of the scene in which the flange and steel pipe are located; the second camera is used to acquire local fine image data of the flange end face and the end of the steel pipe;

[0007] A multi-level fusion verification module is connected to the first camera and the second camera respectively. It is used to receive the global image data and the local fine image data, and sequentially perform data layer fusion, feature layer cross-validation and decision layer spatiotemporal consistency verification on the global image data and the local fine image data to generate high-confidence assembly guidance data.

[0008] A robot control unit, which is connected to the multi-level fusion verification module, is used to receive the assembly guidance data;

[0009] A robot execution unit, connected to the robot control unit, is used to perform a gripping action on the flange and an assembly action between the flange and the steel pipe according to the assembly guidance data;

[0010] The assembly and re-inspection module is connected to the second camera and the robot control unit. When the robot execution unit moves the flange to the end of the steel pipe, the assembly and re-inspection module triggers the second camera to acquire image data of the flange-steel pipe mating clearance. Based on the image data of the mating clearance, the module calculates the coaxiality deviation value and the clearance deviation value, and then sends the coaxiality deviation value and the clearance deviation value to the robot control unit. The robot control unit generates a fine-tuning command based on the coaxiality deviation value and the clearance deviation value and sends it to the robot execution unit until the mating clearance meets the preset process standard. The calculation results of the assembly and re-inspection module are fed back to the robot control unit in real time, and the single feedback cycle is consistent with the image acquisition cycle of the second camera.

[0011] Preferably, the first camera is an industrial camera equipped with a high dynamic range imaging mode; a variable angle pulse illumination system is provided on one side of the first camera, the variable angle pulse illumination system being used to automatically adjust the incident angle and pulse width of the illumination light according to the shooting angle of the first camera and the scene lighting conditions. The illumination trigger signal of the variable angle pulse illumination system is synchronized with the exposure trigger signal of the first camera, and the illumination duration is matched with the camera exposure duration.

[0012] Preferably, the second camera is an industrial camera equipped with a coaxial polarized light illumination system; the illumination light emitted by the coaxial polarized light illumination system is coaxial with the optical axis of the second camera, used to suppress reflective interference from the flange metal surface. The polarizer and analyzer of the coaxial polarized light illumination system are arranged orthogonally to each other, which can filter out glare generated by specular reflection from the metal surface.

[0013] Preferably, the multi-level fusion verification module includes:

[0014] The data layer fusion submodule is used to perform pixel-level registration of the global image data and the local fine image data acquired by the first camera and the second camera at the same time node to generate a super-resolution fused image containing global contour information and local texture information.

[0015] A feature layer cross-validation submodule is provided, which is pre-configured with a first deep learning model and a second deep learning model. The feature layer cross-validation submodule simultaneously inputs the super-resolution fused image into both the first and second deep learning models. The first deep learning model is used to identify the flange type and the coarse spatial orientation of the flange relative to the steel pipe. The second deep learning model is used to identify the bolt hole positions on the flange end face and the fine angle of the flange end face relative to the steel pipe end face. The feature layer cross-validation submodule compares the first identification result output by the first deep learning model with the second identification result output by the second deep learning model. If the first identification result and the second identification result are within a preset feature deviation threshold range, the first identification result and the second identification result are fused and output to the decision layer spatiotemporal consistency verification submodule. If the first identification result and the second identification result exceed the feature deviation threshold range, secondary feature extraction and logical reasoning of the corresponding difference regions in the super-resolution fused image are triggered until consistency is achieved.

[0016] A decision-level spatiotemporal consistency verification submodule, connected to the feature-level cross-validation submodule, is used to receive the fused recognition result. Before the robot execution unit performs the grasping action, the decision-level spatiotemporal consistency verification submodule triggers the first camera and the second camera to perform a second synchronous image acquisition. The second recognition result obtained after processing by the data-level fusion submodule and the feature-level cross-validation submodule after the second synchronous image acquisition is compared with the current flange predicted pose predicted by the kinematic prediction model based on the recognition result obtained after the first synchronous image acquisition. If the deviation between the second recognition result and the current flange predicted pose is less than a preset dynamic grasping threshold, the second recognition result is output as the assembly guidance data to the robot control unit. If the deviation between the second recognition result and the current flange predicted pose is greater than or equal to the dynamic grasping threshold, the data is rejected, and the first camera and the second camera are re-triggered to perform the next synchronous image acquisition and processing process until the results of two consecutive independent image acquisition events and the prediction event are consistent in the spatiotemporal dimension. The processing flow for the second synchronous image acquisition is completely consistent with that for the first acquisition, ensuring that the comparison benchmark for the two recognition results is consistent.

[0017] Preferably, the kinematic prediction model is a neural network model trained based on historical assembly data. This model is used to estimate the spatial position of the flange at the current moment in real time based on the flange pose obtained from the previous identification, the robot's motion trajectory, and the operating speed of the conveying mechanism. The kinematic prediction model employs a long short-term memory recurrent neural network architecture, which can fit and extrapolate the temporal pose change patterns of the flange.

[0018] Preferably, the robot control unit includes:

[0019] The virtual-real registration module is pre-loaded with a lightweight digital twin model of the scene being captured. The virtual-real registration module is used to receive the assembly guidance data and update the lightweight digital twin model in real time according to the assembly guidance data. At the same time, it calculates the dynamic coordinate offset between the virtual pose of the flange in the lightweight digital twin model and the actual pose of the flange in the assembly guidance data.

[0020] The trajectory planning module, connected to the virtual-real registration module, calculates the optimal motion trajectory for the robot's end effector to move from its current position to the flange gripping position based on the dynamic coordinate offset and a preset robot kinematic model, and sends the optimal motion trajectory to the robot execution unit. The trajectory planning uses a fifth-order polynomial interpolation algorithm to generate a smooth trajectory, and the motion parameters at each interpolation point in the trajectory satisfy the operational constraints of the robot's joint motors.

[0021] Preferably, the assembly re-inspection module is specifically used for: triggering the second camera to continuously capture multiple images of the mating area between the flange end face and the steel pipe end face when the robot execution unit moves the flange to the end of the steel pipe; the assembly re-inspection module calculates multiple coaxiality deviation values ​​and multiple gap deviation values ​​in different directions based on the multiple captured image data, and performs average processing on the multiple coaxiality deviation values ​​and multiple gap deviation values; the assembly re-inspection module sends the averaged coaxiality deviation values ​​and gap deviation values ​​to the robot control unit; the robot control unit generates multi-axis fine-tuning commands based on the averaged coaxiality deviation values ​​and gap deviation values ​​and sends them to the robot execution unit; the robot execution unit continuously fine-tunes the pose of the flange in multiple degrees of freedom according to the multi-axis fine-tuning commands; simultaneously, during the fine-tuning process, the assembly re-inspection module continuously triggers the second camera to perform image acquisition and deviation value calculation, forming a closed-loop control loop of visual measurement and robot fine-tuning. The calculation of coaxiality deviation values ​​and gap deviation values ​​uses a sub-pixel edge detection algorithm to extract contour features, improving the accuracy of parameter calculation.

[0022] Preferably, the assembly and re-inspection module is pre-set with preset process standards; the preset process standards include the maximum allowable coaxiality deviation and the maximum clearance deviation when assembling the flange and the steel pipe. The preset process standards are stored in the form of a structured data table, and the corresponding parameters can be called according to the type and specifications of the flange.

[0023] Preferably, the system further includes:

[0024] A data self-optimization module is connected to the multi-level fusion verification module and the robot control unit. This module stores high-confidence assembly guidance data generated by the multi-level fusion verification module and robot motion data actually executed by the robot control unit during each successful assembly process. The module also performs periodic incremental training on its internal kinematic prediction model based on the stored high-confidence assembly guidance data and robot motion data to optimize the model's prediction accuracy. This incremental training process is performed during system idle standby and does not consume system resources for normal assembly operations.

[0025] Preferably, both the first camera and the second camera are equipped with a full-color status indicator; the full-color status indicator is used to display different colors of status light in real time according to the verification results of the multi-level fusion verification module, so as to indicate the current operating status of image acquisition and data verification. The display status of the full-color status indicator corresponds one-to-one with the operating status of each link in the system, and the status switching is synchronized with the trigger signal of the corresponding link.

[0026] This invention provides an intelligent identification and assembly system for ship pipe flanges. It has the following advantages:

[0027] This intelligent identification and assembly system for ship pipe flanges achieves fully automated closed-loop control from image acquisition, multi-level verification, capture execution to assembly re-inspection by constructing a technical architecture that combines dual-modal heterogeneous data acquisition with multi-level fusion verification. The system utilizes a first camera equipped with high dynamic range imaging and variable angle pulse illumination to acquire global scene images, while a second camera equipped with coaxial polarized light illumination acquires detailed local images of the flange end face. Data layer fusion generates a super-resolution fused image. A feature layer cross-validation submodule runs two deep learning models in parallel to extract the global type, spatial orientation, and local bolt hole positions and end face angles of the flange. When inconsistencies arise in the recognition results, secondary feature extraction is triggered until consistency is achieved. A decision layer spatiotemporal consistency verification submodule triggers a second image acquisition before grasping and spatiotemporally compares the actual pose with the predicted pose calculated by the kinematic prediction model. Only when the deviation between the two independent event results is less than the dynamic grasping threshold is high-confidence assembly guidance data output. This effectively reduces the uncertainty and operational errors caused by human intervention in the complex lighting and dynamic environment of a shipyard, improving the consistency and reliability of the flange assembly process.

[0028] This intelligent identification and assembly system for ship pipe flanges integrates a data self-optimization module and a full-color status indicator. The data self-optimization module records high-confidence data and actual robot motion data in real time during successful assembly. When the system is idle, it automatically triggers incremental training to periodically update the kinematic prediction model, enabling the model to continuously adapt to changes in characteristics caused by long-term operation, such as wear of the conveyor mechanism and changes in robot joint clearance. This ensures that the prediction accuracy and verification pass rate are maintained throughout the system's entire lifecycle. The full-color status indicator installed on the first and second cameras switches between multiple colors and flashing modes in real time based on the status signals generated by the multi-level fusion verification module at different stages. On-site operators can intuitively understand the operating status of each stage, such as image acquisition, feature verification, spatiotemporal verification, pairing measurement, and abnormal alarms, without having to check the control cabinet screen. This provides clear visual cues for automatic re-identification or manual intervention and also serves as a safety confirmation signal before the robot performs a grasping action, further improving the human-machine collaboration efficiency and operational safety of the system in the ship pipe processing workshop. Attached Figure Description

[0029] Figure 1 This is a data flow diagram between modules of an intelligent identification and assembly system for ship pipe flanges according to the present invention;

[0030] Figure 2This is a flowchart of the multi-level fusion verification module of a ship pipe flange intelligent identification and assembly system according to the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Please see Figure 1 and Figure 2 The present invention provides a technical solution: a ship pipe flange intelligent identification and assembly system, comprising: a dual-modal heterogeneous data acquisition unit, the dual-modal heterogeneous data acquisition unit including a first camera and a second camera; the first camera is used to acquire global image data of the scene where the flange and steel pipe are located; the second camera is used to acquire local fine image data of the flange end face and the end of the steel pipe;

[0033] The multi-level fusion verification module is connected to the first camera and the second camera respectively. It is used to receive global image data and local fine image data, and sequentially perform data layer fusion, feature layer cross-validation and decision layer spatiotemporal consistency verification on the global image data and local fine image data to generate high-confidence assembly guidance data.

[0034] The robot control unit is connected to the multi-level fusion verification module and is used to receive assembly guidance data.

[0035] The robot execution unit is connected to the robot control unit and is used to perform the gripping action on the flange and the assembly action between the flange and the steel pipe according to the assembly guidance data.

[0036] The assembly and re-inspection module is connected to the second camera and the robot control unit. When the robot execution unit moves the flange to the end of the steel pipe, the assembly and re-inspection module triggers the second camera to acquire image data of the fit gap between the flange and the steel pipe. Based on the image data of the fit gap, the module calculates the coaxiality deviation value and the gap deviation value, and then sends the coaxiality deviation value and the gap deviation value to the robot control unit. The robot control unit generates a fine-tuning command based on the coaxiality deviation value and the gap deviation value and sends it to the robot execution unit until the fit gap meets the preset process standard.

[0037] The first camera is an industrial camera equipped with a high dynamic range imaging mode; a variable angle pulse illumination system is set on one side of the first camera. The variable angle pulse illumination system is used to automatically adjust the incident angle and pulse width of the illumination light according to the shooting angle of the first camera and the scene lighting conditions.

[0038] It should be further explained that the first camera is an industrial digital camera with a built-in high dynamic range imaging mode. The image sensor of this camera can simultaneously and clearly present the image details of the strong light reflection area and the shadow area in the scene through multiple exposure synthesis or wide dynamic response under single exposure. For example, in the environment of a shipyard, it can simultaneously capture the highlight part of the unprocessed metal flange and the texture of the steel pipe end in the backlight position.

[0039] A variable-angle pulse illumination system is fixedly mounted on one side of the housing of the first camera. This illumination system includes a ring array light source composed of multiple high-brightness light-emitting diodes. The ring array light source is connected to the housing of the first camera via a two-dimensional motorized rotating bracket. The two-dimensional motorized rotating bracket can adjust the orientation of the ring array light source in two degrees of freedom: horizontal and pitch, so that the incident angle of the illumination light relative to the optical axis of the first camera can continuously change within the range of 0 to 60 degrees. The variable-angle pulse illumination system also has a built-in pulse width modulation controller. The pulse width modulation controller is connected to the exposure trigger signal output terminal of the first camera. It can output a pulse with adjustable width during the camera exposure according to the frame synchronization signal of the first camera to drive the ring array light source to light up instantaneously. The pulse width is adjustable within the range of 10 microseconds to 1000 microseconds, thereby achieving pulse illumination that is precisely synchronized with the camera exposure cycle.

[0040] To achieve automatic adjustment based on shooting angle and scene lighting conditions, the image data processing unit of the first camera integrates an ambient light analysis module. This module performs grayscale histogram analysis on the preview image captured in real time by the first camera, calculates the ratio of overexposed to underexposed areas and the overall average grayscale value to determine the intensity and direction distribution of the current ambient light. Simultaneously, an attitude sensor is fixed on the housing of the first camera. This attitude sensor detects the tilt angle and orientation of the first camera relative to the horizontal plane in real time. The ambient light analysis module sends the analyzed ambient light parameters and the camera attitude parameters detected by the attitude sensor to an illumination control unit. The illumination control unit calculates the optimal illumination incident angle that maximizes the uniformity of reflected light on the flange surface and the optimal pulse width that minimizes motion blur under the current ambient light and camera attitude, based on a preset illumination angle lookup table or through an online optimization algorithm. Then, the illumination control unit sends a drive signal to the two-dimensional electric rotating bracket to adjust the ring array light source to the optimal illumination incident angle and sends a pulse width setting value to the pulse width modulation controller to set the optimal pulse width.

[0041] During subsequent image acquisition, each time the first camera triggers a shot, the lighting control unit controls the ring array light source to perform synchronous pulse illumination according to the latest calculated optimal illumination incident angle and optimal pulse width. This ensures stable acquisition of high-quality global images of the flange and steel pipe under complex lighting conditions, providing reliable raw image data for the subsequent multi-level fusion verification module.

[0042] The second camera is an industrial camera equipped with a coaxial polarized light illumination system. The illumination light emitted by the coaxial polarized light illumination system is coaxial with the optical axis of the second camera, used to suppress reflective interference from the flange metal surface. It should be further noted that the second camera specifically employs a high-resolution industrial digital camera with a built-in global shutter. The image sensor pixel size of this camera is in the range of 1.6 micrometers to 3.4 micrometers, with a resolution of no less than 5 million pixels, used to clearly capture the microscopic structural details of the flange end face bolt holes, sealing grooves, and characters engraved on the end face.

[0043] A coaxial polarized light illumination system is fixedly installed between the front end of the lens of the second camera and the camera housing. The coaxial polarized light illumination system includes a semi-transparent and semi-reflective beam splitter, a polarizer, an analyzer, and an area array light source.

[0044] A semi-transparent, semi-reflective beam splitter is positioned at a 45-degree angle in the optical path of the second camera lens, with its beam-splitting surface facing the optical axis of the second camera lens. An array light source is composed of multiple high color rendering white light-emitting diodes (LEDs). This array light source is fixedly mounted on one side of the second camera housing, with its emitting surface facing the side of the semi-transparent, semi-reflective beam splitter and perpendicular to the optical axis of the second camera. A polarizer is an optical polarizing film attached to the emitting surface of the array light source, with its polarization direction set along a first direction in the horizontal plane. An analyzer is an annular optical polarizer embedded at the front of the second camera lens and located between the semi-transparent, semi-reflective beam splitter and the subject being photographed, with its polarization direction set along a second direction perpendicular to the first direction in the horizontal plane.

[0045] The illumination light emitted by the area array light source first passes through a polarizer to form linearly polarized light polarized along the first direction. After the linearly polarized light is incident on a semi-transparent and semi-reflective beam splitter, a portion of the light is reflected by the semi-transparent and semi-reflective beam splitter. The reflected light then illuminates the metal surface of the flange end face perpendicularly downward along the optical axis of the second camera.

[0046] When linearly polarized light irradiates the flange end face, it undergoes specular reflection when it encounters a smooth metal surface. The reflected light retains its original first-direction polarization characteristics. When the specularly reflected light returns along the original path and passes through the semi-transparent and semi-reflective beam splitter again, some of the light passes through the semi-transparent and semi-reflective beam splitter and enters the analyzer upward. Since the polarization direction of the analyzer is the second direction perpendicular to the first direction, the specularly reflected light is blocked by the analyzer and cannot reach the second camera image sensor.

[0047] When linearly polarized light irradiates the flange end face, it undergoes diffuse reflection when it encounters non-smooth areas on the flange surface, such as the edge of bolt holes, the bottom of sealing grooves, or the pits of stamped characters. The polarization direction of the diffusely reflected light is randomly depolarized and transformed into unpolarized or partially polarized light. When the diffusely reflected light returns along the original path and is transmitted through a semi-transparent beam splitter into the analyzer, the portion of light perpendicular to the second direction is absorbed by the analyzer, while the portion of light parallel to the second direction passes through the analyzer and enters the lens of the second camera, where it is imaged on the image sensor.

[0048] By arranging the polarizer and analyzer with orthogonal polarization directions, glare from direct specular reflection from the metal surface in the illumination path is effectively filtered out, while diffuse reflection from the microstructural features of the flange end face is preserved and imaged. This clearly presents the edge contour of the bolt hole, the depth variation of the sealing groove, and the stroke details of the stamped characters in the image, providing low-noise, high-contrast raw image data for the second deep learning model in the subsequent multi-level fusion verification module to identify the bolt hole position and fine angle of the end face.

[0049] The multi-level fusion verification module includes:

[0050] The data layer fusion submodule is used to perform pixel-level registration of global image data and local fine image data acquired by the first camera and the second camera at the same time node, and generate a super-resolution fused image containing global contour information and local texture information.

[0051] The feature layer cross-validation submodule is pre-configured with a first deep learning model and a second deep learning model. It simultaneously inputs the super-resolution fused image into both models. The first deep learning model identifies the flange type and its coarse spatial orientation relative to the steel pipe. The second deep learning model identifies the bolt hole positions on the flange end face and the fine angles between the flange end face and the steel pipe end face. The submodule compares the first identification result output by the first deep learning model with the second identification result output by the second deep learning model. If the first and second identification results are within a preset feature deviation threshold, they are fused and output to the decision layer spatiotemporal consistency verification submodule. If the first and second identification results exceed the feature deviation threshold, secondary feature extraction and logical reasoning are triggered for the corresponding difference regions in the super-resolution fused image until consistency is achieved.

[0052] The decision-level spatiotemporal consistency verification submodule, connected to the feature-level cross-validation submodule, receives the fused recognition results. Before the robot execution unit performs the grasping action, the decision-level spatiotemporal consistency verification submodule triggers the first and second cameras to perform a second synchronous image acquisition. The second recognition result, obtained after processing by the data-level fusion submodule and the feature-level cross-validation submodule, is then compared with the current flange predicted pose predicted by the kinematic prediction model based on the recognition result obtained after the first synchronous image acquisition for spatiotemporal consistency. If the deviation between the second recognition result and the current flange predicted pose is less than a preset dynamic grasping threshold, the second recognition result is output as assembly guidance data to the robot control unit. If the deviation between the second recognition result and the current flange predicted pose is greater than or equal to the dynamic grasping threshold, the data is rejected, and the first and second cameras are re-triggered for the next synchronous image acquisition and processing cycle, until the results of two consecutive independent image acquisition events and the predicted event are consistent in the spatiotemporal dimension.

[0053] It should be further explained that the multi-level fusion verification module is internally integrated with a data layer fusion sub-module, a feature layer cross-validation sub-module, and a decision layer spatiotemporal consistency verification sub-module.

[0054] The data layer fusion submodule receives global image data acquired by the first camera and local fine image data acquired by the second camera. The data layer fusion submodule first timestamps the global image data and local fine image data, extracts image pairs acquired at the same trigger time, and then uses a phase-correlation-based registration algorithm to calculate the translation and rotation offset between the two images. Based on the offset, the local fine image data is projected and transformed to the pixel coordinate system of the global image data to achieve pixel-level accurate registration. After registration, the data layer fusion submodule performs wavelet domain fusion of the low-frequency contour information of the global image data and the high-frequency texture information of the local fine image data to generate a super-resolution fused image that simultaneously contains the global spatial layout relationship of the flange steel pipe and the microscopic details of the flange end face bolt holes and sealing surface.

[0055] The feature layer cross-validation submodule receives the super-resolution fused image. Internally, the feature layer cross-validation submodule contains a first deep learning model and a second deep learning model. The first deep learning model employs a target detection architecture based on a region convolutional neural network. This model is trained to locate the overall outline of the flange, the relative positional relationship between the flange and the steel pipe, and the approximate orientation angle of the flange in the super-resolution fused image. The output of the first deep learning model is a first recognition result containing the flange type label, flange center coordinates, flange outer diameter, and the tilt angle of the flange plane relative to the horizontal plane. The second deep learning model employs a pixel-level segmentation architecture based on a fully convolutional network. This model is trained to finely segment the edge contour of each bolt hole on the flange end face, the direction of the flange end face sealing groove, and the digital character area engraved on the end face in the super-resolution fused image. The output of the second deep learning model is a second recognition result containing the center coordinates of each bolt hole, the pitch circle diameter of the bolt hole, the deflection angle of the flange end face relative to the steel pipe end face, and the character encoding on the end face.

[0056] The feature layer cross-validation submodule compares the first recognition result with the second recognition result. Specifically, it compares the deviation value of the flange center coordinates, the matching degree of the flange outer diameter and the bolt hole pitch circle diameter, and the consistency between the flange tilt angle in the first recognition result and the flange end face deflection angle in the second recognition result. If the deviation values ​​of the above comparison results are all less than the preset feature deviation threshold, the feature layer cross-validation submodule determines that the first recognition result and the second recognition result have passed mutual verification. It then combines the flange type label in the first recognition result and the bolt hole coordinates, end face deflection angle, and character encoding in the second recognition result to form a fused recognition result and outputs it to the decision layer spatiotemporal consistency verification submodule. If the deviation value of any comparison result exceeds the preset feature deviation threshold range, the feature layer cross-validation submodule determines that there is a feature inconsistency. At this time, the feature layer cross-validation submodule extracts the image patch of the difference area between the first recognition result and the second recognition result in the super-resolution fused image. It then re-inputs the difference image patch into the first deep learning model and the second deep learning model for secondary feature extraction and performs logical reasoning based on the secondary extraction results until the output results of the two models reach a consensus or exceed the preset iteration limit, at which point it is marked as a manual intervention state.

[0057] The decision-level spatiotemporal consistency verification submodule connects with the feature-level cross-validation submodule to receive the fused recognition results. This submodule integrates a kinematic prediction model, a temporal prediction model built on a long short-term memory recurrent neural network. This model uses flange pose data from multiple consecutive moments during the historical assembly process, encoder feedback values ​​from each robot joint angle, and encoder pulse counts from the conveyor mechanism as training samples to learn and predict the spatial pose change pattern of the flange under the motion state of the conveyor mechanism. Before the robot execution unit performs the grasping action, the decision-level spatiotemporal consistency verification submodule, based on the flange pose from the last successfully acquired and fused first recognition result before the current moment, the robot motion trajectory recorded by the robot control unit from that moment to the current moment, and the operating speed of the conveyor mechanism, calls the kinematic prediction model to calculate the predicted pose of the flange at the current moment. Simultaneously, the submodule sends a second synchronization trigger signal to the first and second cameras to acquire the second global image data and the second local fine image data at the current moment, and calls the data-level fusion submodule and the feature-level cross-validation submodule to process the second acquired data. After obtaining the second identification result, the decision-level spatiotemporal consistency verification submodule compares the actual flange pose in the second identification result with the current flange predicted pose output by the kinematic prediction model. It calculates the center coordinate deviation, angle deviation, and feature point matching degree between the two. If all deviation values ​​are less than the preset dynamic grasping threshold, the decision-level spatiotemporal consistency verification submodule determines the second identification result is reliable and outputs it as assembly guidance data to the robot control unit. If any deviation value is greater than or equal to the dynamic grasping threshold, the decision-level spatiotemporal consistency verification submodule determines the second identification result is valid. If the data is deemed unreliable due to environmental disturbances, the system refuses to output it to the robot control unit and immediately re-triggers the first and second cameras for another synchronized image acquisition. Simultaneously, it updates the input state of the kinematic prediction model and predicts the new current pose again. This process of second acquisition, fusion, and comparison is repeated until the deviations in spatial position and angle between the results of two consecutive independent image acquisition events and the prediction events are consistently less than the dynamic grasping threshold. Only then is the latest fusion recognition result output as assembly guidance data, thereby ensuring that the data used to guide the robot's grasping has high confidence with dual verification in both spatiotemporal dimensions.

[0058] The kinematic prediction model is a neural network model trained based on historical assembly data. The kinematic prediction model is used to estimate the spatial position of the flange at the current moment in real time based on the flange pose obtained from the previous identification, the robot's motion trajectory, and the operating speed of the conveying mechanism.

[0059] It should be further explained that the kinematic prediction model is specifically constructed as a time-series prediction model based on a long short-term memory recurrent neural network architecture. This model includes an input layer, two stacked long short-term memory hidden layers, and a fully connected output layer.

[0060] The feature vector received by the input layer is composed of three parts of data. The first part is the flange pose data obtained by the previous synchronous image acquisition and fusion recognition. This pose data includes the three-dimensional coordinates of the flange center in the world coordinate system and the orientation angle of the flange plane normal vector relative to the three coordinate axes of the world coordinate system. The second part is the three-dimensional spatial position sequence and attitude angle sequence of the robot end effector recorded and stored by the robot control unit from the time of the previous image acquisition to the current prediction time. The third part is the cumulative value of displacement pulses and running direction flag of the conveyor belt or roller conveyor fed back by the encoder of the conveyor mechanism in the corresponding time period.

[0061] The long short-term memory hidden layer contains several memory units. Each memory unit is equipped with a forget gate, an input gate, and an output gate to selectively retain or discard relevant states in historical time series information. The two stacked hidden layers sequentially extract features and model temporal dependencies from the input time series data. The first hidden layer outputs the hidden state sequence at each time step as the input of the second hidden layer. The second hidden layer outputs a hidden state vector containing the encoding features of the flange motion law at the last time step.

[0062] The fully connected output layer maps the hidden state vector output by the second hidden layer to the predicted pose of the flange at the current moment. This predicted pose also includes the predicted three-dimensional coordinates of the flange center and the three direction angles of the flange plane normal vector.

[0063] During the offline training phase, the kinematic prediction model extracts data from several consecutive time periods from the system's historical assembly records to form a training sample set. The input of each training sample is the previous real acquisition pose, the robot's motion trajectory at each intermediate moment, and the displacement of the conveyor mechanism. The output is the real acquisition of the flange pose at the end of the time period as a label. The model training uses mean square error as the loss function and updates the network weight parameters through the backpropagation algorithm and the adaptive moment estimation optimizer until the loss converges.

[0064] During the online prediction phase, the decision-making layer spatiotemporal consistency verification submodule acquires the latest previous recognition pose, the sequence of actual motion trajectory points cached by the robot control unit since that moment, and the real-time cumulative value of the encoder of the transmission mechanism in real time. After normalizing and aligning the above data according to the format required by the input layer, it is input into the kinematic prediction model. The model forward calculates to obtain the flange prediction pose output at the current moment. This prediction pose is then used to compare the spatiotemporal consistency with the actual pose in the second recognition result obtained by the fusion recognition of the second synchronous image acquisition, in order to determine the credibility of the second recognition result.

[0065] The robot control unit includes:

[0066] The virtual-real registration module is pre-loaded with a lightweight digital twin model of the scene being captured. The virtual-real registration module is used to receive assembly guidance data and update the lightweight digital twin model in real time according to the assembly guidance data. At the same time, it calculates the dynamic coordinate offset between the virtual pose of the flange in the lightweight digital twin model and the actual pose of the flange in the assembly guidance data.

[0067] The trajectory planning module, connected to the virtual-real registration module, is used to calculate the optimal motion trajectory of the robot end effector from its current position to the flange gripping position based on the dynamic coordinate offset and the preset robot kinematic model, and then send the optimal motion trajectory to the robot execution unit.

[0068] It should be further explained that the robot control unit integrates a virtual-real registration module and a trajectory planning module.

[0069] The virtual-real registration module is pre-built with a lightweight digital twin model of the grasping scene. This lightweight digital twin model is constructed by combining three-dimensional mesh and parametric geometry. The model includes simplified geometric representations of the robot base, robot links, robot end effector, conveyor roller surface, and flange grasping station. Each geometry in the model is given initial spatial coordinates and kinematic constraints consistent with the physical world.

[0070] The virtual-real registration module is connected to the decision-level spatiotemporal consistency verification submodule of the multi-level fusion verification module, and receives the second identification result as the assembly guidance data output in real time. The second identification result includes the three-dimensional coordinate value of the flange center in the world coordinate system at the current moment, the direction angle of the flange plane normal vector, and the center coordinates of at least three non-collinear bolt holes on the flange end face.

[0071] Based on the received second recognition result, the virtual-real registration module first updates the pose parameters of the corresponding flange virtual model in the lightweight digital twin model to the spatial position and orientation consistent with the second recognition result, aligning the virtual flange with the real flange in the physical world in digital space. Then, the virtual-real registration module calculates the spatial vector difference between the current position of the virtual flange in the lightweight digital twin model and the current position of the robot end effector virtual model. Simultaneously, considering the dynamic offset caused by the continuous movement of the conveyor mechanism, it uses forward kinematics to calculate the end position corresponding to each joint angle of the robot end effector virtual model, thereby obtaining the dynamic coordinate offset that needs to be compensated when the robot end effector moves from the current position to the grasping position. This dynamic coordinate offset includes a three-dimensional position offset component and a rotational offset component around the three coordinate axes. The virtual-real registration module sends the calculated dynamic coordinate offset to the trajectory planning module in real time.

[0072] The trajectory planning module is connected to the virtual-real registration module. The trajectory planning module stores the robot's kinematic model, which is established using the standard DH parameter method. It includes the geometric parameters and motion range constraints of each joint of the robot. After receiving the dynamic coordinate offset sent by the virtual-real registration module, the trajectory planning module uses the dynamic coordinate offset as the target position input and performs inverse kinematics calculation in combination with the robot's kinematic model to obtain the target joint angles required by each joint of the robot to reach the grasping position. The trajectory planning module further uses a fifth-order polynomial interpolation algorithm to plan a smooth motion trajectory between the current joint angles of the robot and the target joint angles. The angle, angular velocity, and angular acceleration of this motion trajectory at each interpolation point in the joint space are continuous and meet the maximum speed and maximum torque constraints of the robot's joint motors. At the same time, the trajectory planning module calls a lightweight digital twin model for collision detection during the planning process to ensure that the robot body, end effector, and surrounding fixed obstacles and other pipes on the conveying mechanism always maintain a safe distance during the entire motion process.

[0073] The trajectory planning module converts the planned optimal motion trajectory into a pulse command sequence that the robot controller can recognize. This sequence is then sent in real time to the servo driver of the robot execution unit via the industrial Ethernet bus. This drives the joint motors to move along the planned trajectory, enabling the robot end effector to move precisely to the flange location and perform the grasping action.

[0074] The assembly and re-inspection module is specifically used for: when the robot execution unit moves the flange to the end of the steel pipe, triggering the second camera to continuously take multiple pictures of the mating area between the flange end face and the steel pipe end face; the assembly and re-inspection module calculates multiple coaxiality deviation values ​​and multiple clearance deviation values ​​in different directions based on the multiple image data, and performs average processing on the multiple coaxiality deviation values ​​and multiple clearance deviation values; the assembly and re-inspection module sends the averaged coaxiality deviation values ​​and clearance deviation values ​​to the robot control unit; the robot control unit generates multi-axis fine-tuning commands based on the averaged coaxiality deviation values ​​and clearance deviation values ​​and sends them to the robot execution unit; the robot execution unit continuously fine-tunes the pose of the flange in multiple degrees of freedom according to the multi-axis fine-tuning commands; at the same time, during the fine-tuning process, the assembly and re-inspection module continuously triggers the second camera to acquire images and calculate deviation values, forming a closed-loop control loop of visual measurement and robot fine-tuning.

[0075] It should be further explained that the assembly and re-inspection module performs the following operations: When the robot execution unit moves the grasped flange to the predetermined assembly position at the end of the steel pipe according to the optimal motion trajectory planned by the trajectory planning module, the assembly and re-inspection module first sends a continuous shooting trigger signal to the second camera. The second camera continuously takes multiple shots of the mating area between the flange end face and the steel pipe end face at a preset acquisition frequency, for example, continuously acquiring five to ten frames of images within 0.5 seconds.

[0076] The pairing re-inspection module receives the mating area images captured by the second camera each time. For each frame of the image, it uses a sub-pixel localization algorithm based on edge detection to extract the outer circular contour of the flange end face and the outer circular contour of the steel pipe end face, and calculates the two-dimensional plane deviation between the centers of the two contours as the coaxiality deviation value corresponding to the frame of the image. At the same time, it extracts the pixel width of the gap between the flange end face and the steel pipe end face at multiple sampling points in the circumferential direction and converts it into the actual physical distance. It calculates the average value of the gap values ​​at these sampling points as the gap deviation value corresponding to the frame of the image.

[0077] The pairing and re-inspection module performs an arithmetic mean operation on multiple coaxiality deviation values ​​obtained from continuous shooting to obtain a mean coaxiality deviation value, and performs an arithmetic mean operation on multiple gap deviation values ​​to obtain a mean gap deviation value. The mean operation can eliminate random errors in single measurements caused by instantaneous shaking of the second camera or local reflections on the flange surface.

[0078] The assembly and re-inspection module encapsulates the averaged coaxiality deviation and gap deviation values ​​according to a preset data format and sends them to the robot control unit via real-time industrial Ethernet.

[0079] After receiving the averaged coaxiality deviation value and the averaged gap deviation value, the robot control unit compares them with the preset process standards inside the assembly re-inspection module. The preset process standards include the maximum coaxiality deviation threshold and the maximum gap deviation threshold allowed when assembling the flange and the steel pipe. If the averaged coaxiality deviation value is less than the maximum coaxiality deviation threshold and the averaged gap deviation value is less than the maximum gap deviation threshold, the robot control unit determines that the assembly is qualified and controls the robot execution unit to release the gripper to complete the assembly.

[0080] If either the averaged coaxiality deviation or the averaged gap deviation exceeds the corresponding threshold, the robot control unit generates a multi-axis fine-tuning command containing minute adjustments to the robot end effector in multiple degrees of freedom through inverse kinematics calculation based on the magnitude and direction of the averaged coaxiality deviation and the averaged gap deviation. Specifically, the multi-axis fine-tuning command includes translational adjustments along the X, Y, and Z axes of the world coordinate system and rotational adjustments around the X, Y, and Z axes. The robot control unit sends the multi-axis fine-tuning command to the robot execution unit, which drives each joint motor to perform minute incremental movements based on the current position, thereby achieving online correction of the flange pose.

[0081] During the fine-tuning process performed by the robot execution unit, the assembly and re-inspection module continuously triggers the second camera to acquire real-time images and calculate deviation values ​​of the mating area. The latest calculated averaged coaxiality deviation value and averaged gap deviation value are fed back to the robot control unit in real time. The robot control unit continuously updates the multi-axis fine-tuning command based on the latest deviation value and sends it to the robot execution unit. This process is repeated to form a closed-loop control loop between visual measurement and robot fine-tuning. When the averaged coaxiality deviation value and averaged gap deviation value fed back for several consecutive times are consistently less than the corresponding preset process standard threshold, the robot control unit determines that the assembly accuracy meets the standard and controls the robot execution unit to complete the final assembly action.

[0082] The assembly and re-inspection module has preset process standards. These preset standards include the maximum allowable coaxiality deviation and maximum clearance deviation when assembling the flange and steel pipe. It should be further noted that the assembly and re-inspection module is equipped with non-volatile memory containing these preset process standards. These standards are stored in a structured data table. Each row in the table corresponds to a combination of flange type and specifications. Each row contains two core fields: the first field is the maximum coaxiality deviation value, which represents the allowable offset in three-dimensional space between the center axis of the flange end face and the center axis of the steel pipe end face after assembly; the second field is the maximum clearance deviation value, which represents the allowable fluctuation range of the clearance size at several measurement points evenly distributed along the circumferential direction between the flange end face and the steel pipe end face after assembly.

[0083] The specific values ​​of the maximum coaxiality deviation and the maximum clearance deviation are determined based on the ship piping system design specifications, the nominal pressure rating of the flange, the type of gasket, and the process parameters such as the steel pipe wall thickness. For example, for a marine carbon steel flange with a nominal pressure of PN16, the maximum coaxiality deviation is set to 0.5 mm, and the maximum clearance deviation is set to a clearance difference of no more than 0.3 mm between any two points along the circumference. For a high-pressure flange with a nominal pressure of PN40, the maximum coaxiality deviation is set to 0.2 mm, and the maximum clearance deviation is set to a clearance difference of no more than 0.1 mm between any two points along the circumference.

[0084] When the assembly re-inspection module makes a deviation judgment, it first extracts the type identifier and specification parameters of the flange to be assembled from the assembly guidance data output by the multi-level fusion verification module, or obtains the flange model corresponding to the current work order from the robot control unit. Then, based on the model information, it queries the preset process standard data table in the memory and reads the corresponding maximum coaxiality deviation value and maximum clearance deviation value as the judgment threshold for whether the assembly is qualified or not.

[0085] The assembly re-inspection module compares the calculated averaged coaxiality deviation value with the read maximum coaxiality deviation value, and simultaneously compares the calculated averaged gap deviation value with the read maximum gap deviation value. If the averaged coaxiality deviation value is less than or equal to the maximum coaxiality deviation value and the averaged gap deviation value is less than or equal to the maximum gap deviation value, the assembly re-inspection module outputs an assembly pass signal to the robot control unit. If the averaged coaxiality deviation value is greater than the maximum coaxiality deviation value or the averaged gap deviation value is greater than the maximum gap deviation value, the assembly re-inspection module outputs an assembly fail signal to the robot control unit along with the current deviation value. Based on this fail signal, the robot control unit triggers the aforementioned multi-axis fine-tuning action or records the abnormal status for operator handling.

[0086] The system also includes:

[0087] The data self-optimization module is connected to the multi-level fusion verification module and the robot control unit. The data self-optimization module is used to store the high-confidence assembly guidance data generated by the multi-level fusion verification module and the robot motion data actually executed by the robot control unit during each successful assembly process. The data self-optimization module is also used to perform periodic incremental training on the internal kinematic prediction model based on the stored high-confidence assembly guidance data and robot motion data in order to optimize the prediction accuracy of the kinematic prediction model.

[0088] It should be further explained that the system also includes a data self-optimization module, which is connected to the decision-level spatiotemporal consistency verification sub-module of the multi-level fusion verification module and the virtual-real registration module of the robot control unit.

[0089] The data self-optimization module is equipped with a data storage unit and an incremental training unit. The data storage unit uses a circular buffer to record in real time the assembly guidance data finally output by the decision-making layer spatiotemporal consistency verification submodule and the robot motion data actually executed and fed back by the robot control unit during each successful assembly process. The assembly guidance data specifically includes the reliable three-dimensional coordinates of the flange center, the direction angle of the flange plane normal vector, and the center coordinates of the bolt holes on the flange end face after spatiotemporal consistency verification. The robot motion data specifically includes the actual angle values ​​fed back by the angle encoders of each joint of the robot in each control cycle during the entire process from the grasping start point to the assembly completion point, as well as the expected angle command values ​​generated by the trajectory planning module and sent to the servo driver.

[0090] After each assembly, the data storage unit compresses and indexes the above data as a complete training sample according to the preset data format and stores it, while automatically removing the earliest stored sample to keep the storage capacity within the set upper limit.

[0091] The incremental training unit is connected to the data storage unit. The incremental training unit has a pre-built kinematic prediction model network structure that is the same as that in the decision layer spatiotemporal consistency verification submodule, namely, a temporal prediction model architecture based on long short-term memory recurrent neural network. The incremental training unit automatically triggers an incremental training process when the system is in standby idle state or after each completion of a preset number of assembly tasks. During incremental training, a batch of multiple training samples stored in the latest batch are randomly selected from the data storage unit to form a small batch dataset. The previous recognition pose, the robot's actual motion trajectory at the intermediate time and the displacement of the conveyor mechanism of each sample in the small batch dataset are used as inputs. The actual collection pose at the end of the time period corresponding to the sample is used as the label. The same loss function and optimizer as offline training are used for several rounds of iterative training. During the training process, some network weights at the bottom layer of the kinematic prediction model are frozen and only the weights of the fully connected layers at the top layer are updated to speed up the convergence speed and prevent overfitting.

[0092] After completing one round of incremental training, the incremental training unit sends the updated network weight parameters to the decision layer spatiotemporal consistency verification submodule via the system bus. The decision layer spatiotemporal consistency verification submodule receives and loads these new weight parameters, replacing the original kinematic prediction model weights, thereby realizing the real-time update and optimization of the kinematic prediction model in online operation.

[0093] Through the periodic incremental training mechanism of the aforementioned data self-optimization module, the kinematic prediction model can continuously absorb the actual data generated during the latest assembly process, gradually adapt to the changes in motion characteristics caused by wear of the transmission mechanism, the changes in joint clearance caused by long-term operation of the robot, and the potential impact of slow changes in ambient light on the recognition pose. This results in a gradual reduction in the deviation between the subsequently predicted flange pose and the actual pose, and a gradual increase in the pass rate of spatiotemporal consistency verification, thereby maintaining a high confidence output of assembly guidance data throughout the entire system lifecycle.

[0094] Both the first and second cameras are equipped with full-color status indicators; the full-color status indicators are used to display different colors of status light in real time according to the verification results of the multi-level fusion verification module, so as to indicate the current image acquisition and data verification operation status.

[0095] It should be further explained that a full-color status indicator is fixedly installed on the back or side of the housing of the first camera and the second camera respectively. The full-color status indicator uses a high-brightness red, green and blue three-color light-emitting diode array as the light-emitting element. The surface of the light-emitting diode array is covered with a milky white light-diffusing mask to make the emitted light uniform and soft. The control input terminal of the full-color status indicator is connected to the digital output port of the image data processing unit of the first camera and the second camera respectively, or connected to the general input and output interface of the decision layer spatiotemporal consistency verification submodule of the multi-level fusion verification module.

[0096] The full-color status indicator is configured to switch the display color and flashing mode in real time according to the status signals generated by the multi-level fusion verification module at different stages during the verification process. Specifically, when the first camera or the second camera completes a single image acquisition and successfully transmits the image data to the multi-level fusion verification module, the corresponding full-color status indicator displays a solid blue light, indicating to the operator that the current camera image acquisition is normal.

[0097] When the multi-level fusion verification module is performing data layer fusion and feature layer cross-verification, the full-color status indicators corresponding to the first and second cameras will simultaneously display a yellow flashing state, with the flashing frequency set to twice per second, to remind the operator that the system is in the process of image processing and feature comparison. At this time, please do not move the pipes or obstruct the camera's field of view.

[0098] When the feature layer cross-validation submodule determines that the recognition results of the first deep learning model and the second deep learning model have passed mutual verification, the two full-color status indicators simultaneously switch to a solid green state, indicating to the operator that the current recognition result is valid and will soon enter the decision layer for verification.

[0099] When the feature layer cross-validation submodule determines that the first recognition result and the second recognition result exceed the feature deviation threshold range and triggers secondary feature extraction, the two full-color status indicators simultaneously display a red flashing state, with the flashing frequency set to five times per second, prompting the operator that the current recognition result has a conflict and needs to be reshot or manually checked.

[0100] When the decision-making spatiotemporal consistency verification submodule determines that the deviation between the second identification result and the current flange predicted pose output by the kinematic prediction model is less than the dynamic grasping threshold and outputs the assembly guidance data, the two full-color status indicators simultaneously switch to a green constant-on state and remain so until the robot execution unit starts to move.

[0101] When the decision-making spatiotemporal consistency verification submodule determines that the deviation between the second recognition result and the predicted pose is greater than or equal to the dynamic capture threshold and refuses to output data, the two full-color status indicators simultaneously display a slow red flashing state, with the flashing frequency set to once per second, to prompt the operator that there is a disturbance in the current environment that makes the recognition result unreliable and the system is performing automatic re-recognition.

[0102] During the assembly and re-inspection module's measurement and fine-tuning of the flange and steel pipe fit clearance, the full-color status indicator corresponding to the second camera displays a solid blue light, indicating to the operator that a precision assembly measurement is currently being performed. Once the assembly is completed and the deviation value meets the preset process standard, the full-color status indicator corresponding to the second camera returns to the standby status display consistent with the first camera.

[0103] With the real-time status feedback provided by the full-color status indicator, on-site operators can intuitively understand the current operating stage of the system, the confidence level of the recognition results, and whether there are any abnormalities without having to look at the control cabinet screen. This facilitates timely responses when the system automatically re-identifies or when manual intervention is required. At the same time, the final solid green state displayed by the full-color status indicator also serves as a visual confirmation signal before the robot performs the grasping action, ensuring that the operator knows that the equipment is about to start moving and maintains a safe distance.

[0104] A smart identification and assembly system for ship pipe flanges employs a technical architecture combining dual-modal heterogeneous data acquisition and multi-level fusion verification. First, a first camera equipped with high dynamic range imaging and variable-angle pulse illumination acquires a global scene image. Simultaneously, a second camera equipped with coaxial polarized light illumination acquires a detailed local image of the flange end face. A data layer fusion submodule performs pixel-level registration of the two types of images to generate a super-resolution fused image.

[0105] The feature layer cross-validation submodule extracts the global type, spatial orientation, local bolt hole position, and end face angle of the flange through two parallel deep learning models. When the recognition results are inconsistent, it triggers secondary feature extraction until a consensus is reached.

[0106] The decision-making layer spatiotemporal consistency verification submodule triggers a second synchronous image acquisition before the robot performs the grasping action. It compares the actual pose obtained by fusion recognition with the current flange prediction pose calculated based on the kinematic prediction model in spatiotemporal. Only when the deviation values ​​of the results of the two independent events in the spatial and temporal dimensions are less than the dynamic grasping threshold will high-confidence assembly guidance data be output.

[0107] The virtual-real registration module in the robot control unit updates the pose of the virtual flange in the lightweight digital twin model in real time based on the assembly guidance data and calculates the dynamic coordinate offset. The trajectory planning module plans the optimal motion trajectory based on the offset and drives the robot execution unit to complete the flange gripping.

[0108] When the flange moves to the end of the steel pipe, the assembly and re-inspection module triggers the second camera to take multiple consecutive pictures of the mating clearance. The coaxiality deviation value and clearance deviation value obtained from multiple measurements are averaged and compared with the preset process standard. If the value exceeds the threshold, a multi-axis fine-tuning command is generated to drive the robot execution unit to perform continuous fine-tuning in multiple degrees of freedom and continuously feed back the image measurement results to form a closed-loop control loop until the mating clearance meets the preset process standard requirements.

[0109] This enables fully automated closed-loop control of the entire process, from image acquisition, multi-level verification, capture execution to assembly and re-inspection, in the complex lighting and dynamic environment of the shipyard. It effectively reduces the uncertainty and operational errors caused by human intervention and improves the consistency and reliability of the flange assembly process.

[0110] The system also integrates a data self-optimization module and a full-color status indicator. The data self-optimization module records high-confidence assembly guidance data and actual robot motion data in real time during each successful assembly process. When the system is idle, it automatically triggers incremental training to periodically update the network weights of the kinematic prediction model, enabling the model to continuously adapt to long-term changes in characteristics caused by wear of the transmission mechanism, changes in robot joint clearance, and slow drift of ambient light. This ensures that the deviation between the predicted pose and the actual pose remains within a small range, thereby maintaining a high pass rate for spatiotemporal consistency verification throughout the system's entire lifecycle.

[0111] The full-color status indicators installed on the first and second cameras switch between various colors and flashing modes in real time based on the status signals generated by the multi-level fusion verification module during the verification process. These modes include solid blue, flashing yellow, solid green, fast red, slow red, and solid cyan. On-site operators can intuitively understand the operational status of each stage, such as image acquisition, feature verification, spatiotemporal verification, pairing measurement, and abnormal alarms, without having to look at the control cabinet screen. This provides clear visual cues for automatic re-identification or manual intervention and also serves as a safety confirmation signal before the robot performs a grasping action, further improving the efficiency of human-machine collaboration and operational safety in the shipbuilding pipe processing workshop.

[0112] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0113] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A smart identification and assembly system for ship pipe flanges, characterized in that, include: A dual-modal heterogeneous data acquisition unit includes a first camera and a second camera; the first camera is used to acquire global image data of the scene where the flange and steel pipe are located; the second camera is used to acquire local fine image data of the flange end face and the end of the steel pipe. A multi-level fusion verification module is connected to the first camera and the second camera respectively. It is used to receive the global image data and the local fine image data, and sequentially perform data layer fusion, feature layer cross-validation and decision layer spatiotemporal consistency verification on the global image data and the local fine image data to generate high-confidence assembly guidance data. A robot control unit, which is connected to the multi-level fusion verification module, is used to receive the assembly guidance data; A robot execution unit, connected to the robot control unit, is used to perform a gripping action on the flange and an assembly action between the flange and the steel pipe according to the assembly guidance data; The assembly and re-inspection module is connected to the second camera and the robot control unit. When the robot execution unit moves the flange to the end of the steel pipe, the assembly and re-inspection module triggers the second camera to acquire image data of the fit gap between the flange and the steel pipe, calculates the coaxiality deviation value and the gap deviation value based on the image data of the fit gap, and then sends the coaxiality deviation value and the gap deviation value to the robot control unit. The robot control unit generates a fine-tuning command based on the coaxiality deviation value and the clearance deviation value and sends it to the robot execution unit until the mating clearance meets the preset process standard.

2. The intelligent identification and assembly system for ship pipe flanges according to claim 1, characterized in that: The first camera is an industrial camera equipped with a high dynamic range imaging mode; a variable angle pulse illumination system is provided on one side of the first camera, which is used to automatically adjust the incident angle and pulse width of the illumination light according to the shooting angle of the first camera and the scene lighting conditions.

3. The intelligent identification and assembly system for ship pipe flanges according to claim 2, characterized in that: The second camera is an industrial camera equipped with a coaxial polarized light illumination system; the illumination light emitted by the coaxial polarized light illumination system is coaxial with the optical axis of the second camera, and is used to suppress reflective interference from the flange metal surface.

4. The intelligent identification and assembly system for ship pipe flanges according to claim 3, characterized in that: The multi-level fusion verification module includes: The data layer fusion submodule is used to perform pixel-level registration of the global image data and the local fine image data acquired by the first camera and the second camera at the same time node to generate a super-resolution fused image containing global contour information and local texture information. A feature layer cross-validation submodule is provided, which is pre-configured with a first deep learning model and a second deep learning model. The feature layer cross-validation submodule simultaneously inputs the super-resolution fused image into both the first and second deep learning models. The first deep learning model is used to identify the flange type and the coarse spatial orientation of the flange relative to the steel pipe. The second deep learning model is used to identify the bolt hole positions on the flange end face and the fine angle of the flange end face relative to the steel pipe end face. The feature layer cross-validation submodule compares the first identification result output by the first deep learning model with the second identification result output by the second deep learning model. If the first identification result and the second identification result are within a preset feature deviation threshold range, the first identification result and the second identification result are fused and output to the decision layer spatiotemporal consistency verification submodule. If the first identification result and the second identification result exceed the feature deviation threshold range, secondary feature extraction and logical reasoning of the corresponding difference regions in the super-resolution fused image are triggered until consistency is achieved. A decision-level spatiotemporal consistency verification submodule, connected to the feature-level cross-validation submodule, is used to receive the fused recognition result. Before the robot execution unit performs the grasping action, the decision-level spatiotemporal consistency verification submodule triggers the first camera and the second camera to perform a second synchronous image acquisition. The second recognition result obtained after processing by the data-level fusion submodule and the feature-level cross-validation submodule after the second synchronous image acquisition is compared with the current flange predicted pose predicted by the kinematic prediction model based on the recognition result obtained after the first synchronous image acquisition. If the deviation between the second recognition result and the current flange predicted pose is less than a preset dynamic grasping threshold, the second recognition result is output as the assembly guidance data to the robot control unit. If the deviation between the second recognition result and the current flange predicted pose is greater than or equal to the dynamic grasping threshold, the data is rejected, and the first camera and the second camera are re-triggered to perform the next synchronous image acquisition and processing process until the results of two consecutive independent image acquisition events and the prediction event are consistent in the spatiotemporal dimension.

5. The intelligent identification and assembly system for ship pipe flanges according to claim 4, characterized in that: The kinematic prediction model is a neural network model trained based on historical assembly data; the kinematic prediction model is used to estimate the spatial position of the flange at the current moment in real time based on the flange pose obtained from the previous identification, the robot's motion trajectory, and the operating speed of the conveying mechanism.

6. The intelligent identification and assembly system for ship pipe flanges according to claim 5, characterized in that: The robot control unit includes: The virtual-real registration module is pre-loaded with a lightweight digital twin model of the scene being captured. The virtual-real registration module is used to receive the assembly guidance data and update the lightweight digital twin model in real time according to the assembly guidance data. At the same time, it calculates the dynamic coordinate offset between the virtual pose of the flange in the lightweight digital twin model and the actual pose of the flange in the assembly guidance data. The trajectory planning module, which is connected to the virtual-real registration module, is used to calculate the optimal motion trajectory of the robot end effector from the current position to the flange gripping position based on the dynamic coordinate offset and the preset robot kinematic model, and send the optimal motion trajectory to the robot execution unit.

7. The intelligent identification and assembly system for ship pipe flanges according to claim 6, characterized in that: The assembly re-inspection module is specifically used for: triggering the second camera to continuously capture multiple images of the mating area between the flange end face and the steel pipe end face when the robot execution unit moves the flange to the end of the steel pipe; calculating multiple coaxiality deviation values ​​and multiple clearance deviation values ​​of the mating clearance in different directions based on the multiple captured image data, and averaging the multiple coaxiality deviation values ​​and multiple clearance deviation values; and sending the averaged coaxiality deviation values ​​and clearance deviation values ​​to the robot control unit. The robot control unit generates multi-axis fine-tuning commands based on the averaged coaxiality deviation and gap deviation values ​​and sends them to the robot execution unit. The robot execution unit continuously fine-tunes the flange pose in multiple degrees of freedom according to the multi-axis fine-tuning commands. At the same time, during the fine-tuning process, the pairing re-inspection module continuously triggers the second camera to perform image acquisition and deviation value calculation, forming a closed-loop control loop of visual measurement and robot fine-tuning.

8. The intelligent identification and assembly system for ship pipe flanges according to claim 7, characterized in that: The assembly and re-inspection module is pre-set with the preset process standards; the preset process standards include the maximum coaxiality deviation and the maximum clearance deviation allowed when assembling the flange and the steel pipe.

9. The intelligent identification and assembly system for ship pipe flanges according to claim 8, characterized in that: The system also includes: A data self-optimization module is connected to the multi-level fusion verification module and the robot control unit. The data self-optimization module stores high-confidence assembly guidance data generated by the multi-level fusion verification module and robot motion data actually executed by the robot control unit during each successful assembly process. The data self-optimization module also performs periodic incremental training on the internal kinematic prediction model based on the stored high-confidence assembly guidance data and the robot motion data to optimize the prediction accuracy of the kinematic prediction model.

10. The intelligent identification and assembly system for ship pipe flanges according to claim 9, characterized in that: Both the first camera and the second camera are equipped with a full-color status indicator; the full-color status indicator is used to display different colors of status light in real time according to the verification result of the multi-level fusion verification module, so as to indicate the current image acquisition and data verification operation status.

Images