Tube sheet workpiece welding method, apparatus and system, and computer device and storage medium

Abstract

A tube sheet workpiece welding method, comprising: acquiring global information obtained by means of performing collection on a tube sheet workpiece (S202), wherein the tube sheet workpiece comprises a plurality of tube holes; on the basis of the global information, determining position information of each tube hole (S204); for each tube hole, on the basis of the position information of the tube hole, determining an information collection pose for the tube hole (S206); on the basis of tube hole information obtained by means of performing collection on the tube hole in the information collection pose, performing fitting to obtain an expected welding trajectory of the tube hole (S208); and controlling a welding robot to weld the tube sheet workpiece according to the expected welding trajectory of each tube hole (S210).

Description

Tube-sheet workpiece welding method, device, system, computer equipment and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application number 2023115469783, filed with the Patent Office of China on November 20, 2023, entitled “Tube Sheet Workpiece Welding Method and System,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates to the field of welding processing technology, and in particular to a tube-sheet workpiece welding method, device, system, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0004] Tube sheets are devices used to separate or connect the flows of different fluids. They are key components for ensuring fluid flow control and processing, and are widely used in equipment such as reactors, distillation towers, heat exchangers, boilers, and gas turbines. In practical applications, tube sheets need to be welded to heat exchange tubes.

[0005] Traditionally, welding in tubesheets is performed manually due to the large number and dense distribution of tube holes. Technological advancements have led to the emergence of semi-automatic welding methods using pneumatic expansion welding machines or teachable welding robots. However, these methods all require manual intervention, resulting in high labor costs.

[0006] Summary of the Invention

[0007] According to various embodiments of the present application, a tube-sheet workpiece welding method, apparatus, system, computer equipment, computer-readable storage medium, and computer program product are provided.

[0008] In a first aspect, the present application provides a tube sheet workpiece welding method. The method comprises:

[0009] Acquire global information collected for a tube sheet workpiece; the tube sheet workpiece includes a plurality of tube holes;

[0010] Based on the global information, the position information of each pore is determined;

[0011] For each pore, based on the position information of the pore, determine the information collection posture for the pore;

[0012] Based on the pipe hole information collected for the pipe hole in the information collection posture, fitting the expected welding trajectory of the pipe hole; and

[0013] According to the desired welding trajectory of each tube hole, the welding robot is controlled to weld the tube sheet workpiece.

[0014] In a second aspect, the present application also provides a tube sheet workpiece welding device. The device comprises:

[0015] A global information acquisition module is used to acquire global information collected for a tube sheet workpiece; the tube sheet workpiece includes a plurality of tube holes;

[0016] A position information determination module, configured to determine the position information of each pore based on the global information;

[0017] An information collection posture determination module is used to determine, for each pore, an information collection posture for the pore based on the pore's position information;

[0018] A trajectory fitting module is used to fit the desired welding trajectory of the pipe hole based on the pipe hole information collected for the pipe hole in the information collection posture; and

[0019] The welding module is used to control the welding robot to weld the tube sheet workpiece according to the desired welding trajectory of each tube hole.

[0020] In a third aspect, the present application also provides a tube-sheet workpiece welding system. The system includes a controller, a global acquisition device, a local acquisition device, and a welding robot connected to the controller;

[0021] The global acquisition device is used to collect global information of the tube sheet workpiece to obtain global information of the tube sheet workpiece; the tube sheet workpiece includes a plurality of tube holes;

[0022] The local collection device is used to collect information from each pore separately to obtain the pore information of each pore;

[0023] Welding robots are used to weld tube-to-sheet workpieces; and

[0024] The controller is used to implement the above-mentioned tube sheet workpiece welding method.

[0025] In a fourth aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the following method are implemented:

[0026] Acquire global information collected for a tube sheet workpiece; the tube sheet workpiece includes a plurality of tube holes;

[0027] Based on the global information, the position information of each pore is determined;

[0028] For each pore, based on the position information of the pore, determine the information collection posture for the pore;

[0029] Based on the pipe hole information collected for the pipe hole in the information collection posture, fitting the expected welding trajectory of the pipe hole; and

[0030] According to the desired welding trajectory of each tube hole, the welding robot is controlled to weld the tube sheet workpiece.

[0031] In a fifth aspect, the present application further provides a computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by a processor, implement the steps of the following method:

[0032] Acquire global information collected for a tube sheet workpiece; the tube sheet workpiece includes a plurality of tube holes;

[0033] Based on the global information, the position information of each pore is determined;

[0034] For each pore, based on the position information of the pore, determine the information collection posture for the pore;

[0035] Based on the pipe hole information collected for the pipe hole in the information collection posture, fitting the expected welding trajectory of the pipe hole; and

[0036] According to the desired welding trajectory of each tube hole, the welding robot is controlled to weld the tube sheet workpiece.

[0037] In a sixth aspect, the present application further provides a computer program product. The computer program product includes computer-readable instructions, which, when executed by a processor, implement the steps of the following method:

[0038] Acquire global information collected for a tube sheet workpiece; the tube sheet workpiece includes a plurality of tube holes;

[0039] Based on the global information, the position information of each pore is determined;

[0040] For each pore, based on the position information of the pore, determine the information collection posture for the pore;

[0041] Based on the pipe hole information collected for the pipe hole in the information collection posture, fitting the expected welding trajectory of the pipe hole; and

[0042] According to the desired welding trajectory of each tube hole, the welding robot is controlled to weld the tube sheet workpiece.

[0043] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without any creative work.

[0045] FIG1 is a diagram showing an application environment of a tube sheet workpiece welding method according to an embodiment;

[0046] FIG2 is a schematic flow chart of a tube sheet workpiece welding method according to one embodiment;

[0047] FIG3 is a system schematic diagram of a tube sheet workpiece welding process according to one embodiment;

[0048] FIG4 is a schematic diagram showing the effect of pore macro positioning and recognition in one embodiment;

[0049] FIG5 is a schematic diagram of the working posture calculation principle of a three-dimensional camera in another embodiment;

[0050] FIG6 is a schematic flow chart of a tube sheet workpiece welding method according to another embodiment;

[0051] FIG7 is a structural block diagram of a tube sheet workpiece welding system according to one embodiment;

[0052] FIG8 is a structural block diagram of a tube sheet workpiece welding device according to one embodiment;

[0053] FIG9 is a diagram showing the internal structure of a computer device in one embodiment.

[0054] Explanation of the accompanying symbols: 1-global acquisition device; 2-robotic arm; 3-local acquisition device; 4-tube sheet workpiece; 41-tube sheet end face; 411-tube hole center; 412-tube hole end face; 5-tooling table; 6-ground rail; 7-welding machine; 8-C-type column; 9-control cabinet; 10-welding wire barrel; 11-gun cleaning station. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0056] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0057] In one embodiment, the tube sheet workpiece welding method provided in the present application can be applied in an application environment as shown in FIG1 . The application environment may include, for example, a terminal 101, a controller 102, and a welding robot 103. The terminal 101 and the controller 102, as well as the controller 102 and the welding robot 103, may be connected in a wired or wireless manner to achieve data transmission. Among them, the terminal 101 may be, but is not limited to, various desktop computers, laptops, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. In a specific embodiment, the terminal 101 may include one or more information collection devices for collecting information on the tube sheet workpiece.

[0058] The controller 102 can be a hardware module that includes various processing chips and their peripheral circuits, and has logical operation capabilities. The processing chip can be a single-chip microcomputer, a DSP (Digital Signal Processor) chip, or an FPGA (Field Programmable Gate Array) chip. The welding robot 103 is an industrial robot that includes a welding arm and is capable of welding.

[0059] Specifically, during tubesheet welding, controller 102 can obtain global information collected about the tubesheet from terminal 101 and, based on this global information, determine the positional information of each tube hole on the tubesheet. Controller 102 then determines an information collection pose for each tube hole based on the positional information. Based on the tube hole information collected by terminal 101 at this information collection pose, controller 102 fits the desired welding trajectory for that tube hole. After obtaining the desired welding trajectory for each tube hole, controller 102 controls the welding robot to weld the tubesheet according to the desired welding trajectory for each tube hole.

[0060] In one embodiment, as shown in FIG2 , a tube sheet workpiece welding method is provided. The method is described by taking the controller 102 in FIG1 as an example, and includes the following steps:

[0061] Step S202: Acquire global information collected for the tube sheet workpiece.

[0062] The tubesheet workpiece comprises multiple tube holes. In practical applications, a tubesheet workpiece is obtained by drilling holes slightly larger than the outer diameter of the heat exchange tubes in a circular steel plate. The heat exchange tubes are inserted into the tube holes in the tubesheet workpiece and welded to secure the heat exchange tubes. Global information refers to information encompassing the entire tubesheet workpiece. In a specific application, global information includes 3D point cloud information and 2D image information of the tubesheet end face. The tubesheet end face refers to the end face of the tubesheet workpiece that requires welding, such as tubesheet end face 41 of tubesheet workpiece 4 in Figure 3.

[0063] Specifically, a global acquisition device can be configured to collect global information about the tubesheet workpiece. The controller can then retrieve the global information collected from the global acquisition device. The global acquisition device is a device capable of collecting information and can include at least one of a camera, radar, or laser. In practical applications, the installation position and orientation of the tubesheet workpiece and the global acquisition device can be adjusted so that the signal transmission direction of the global acquisition device is perpendicular to the tubesheet end face of the tubesheet workpiece, enabling more convenient acquisition of global information about the tubesheet workpiece.

[0064] In one specific embodiment, the tubesheet workpiece welding method further includes: obtaining a field of view of a global acquisition device; and adjusting a fixed position of the global acquisition device based on the position of the tubesheet workpiece within the field of view to ensure that the tubesheet workpiece is within the field of view of the global acquisition device. In this embodiment, step S202 includes obtaining global information collected by the global acquisition device regarding the tubesheet workpiece.

[0065] In actual applications, the size and tube hole distribution of the tube sheet workpiece to be welded may vary. To address this, the controller can adjust the fixed position of the global acquisition device before acquiring global information to ensure that the tube sheet workpiece is contained within the global acquisition device's field of view. Specifically, the controller can acquire an image captured by the global acquisition device at its current position to determine the field of view at that position. Then, based on the tube sheet workpiece's position within this field of view, the controller adjusts the fixed position of the global acquisition device to ensure that the tube sheet workpiece is contained within the field of view of the global acquisition device. The controller then acquires the global information captured by the global acquisition device for the tube sheet workpiece.

[0066] In this embodiment, before obtaining the global information, the position of the global information acquisition device is adjusted so that the tube sheet workpiece is included in the field of view of the global acquisition device. Therefore, only one information acquisition is required to obtain the global information of the tube sheet workpiece, which can adapt to the welding requirements of different batches of tube sheet workpieces and realize flexible automatic welding.

[0067] Step S204: determining the position information of each pore based on the global information.

[0068] The position information of the tube hole refers to information that can characterize the position of the tube hole. This position information can be, for example, the geometric center or center of gravity of the tube hole. For example, if the cross-section of the tube hole on the tube sheet end face is circular, the position information of the tube hole can refer to the position of the hole center. Specifically, the controller can locate the tube holes in the tube sheet workpiece based on the global information of the tube sheet workpiece and determine the position information of each tube hole.

[0069] Taking the example of global information consisting of 3D point cloud information and 2D image information, the controller can perform pore identification on the 2D image information, determine the pore region in the 2D image where the pore exists, and, based on the mapping relationship between the 2D image information and the 3D point cloud information, determine the 3D point cloud region corresponding to the pore region in the 3D point cloud information. Pore identification is then performed on the 3D point cloud region to determine the pore's location information. It is understood that this location information refers to the 3D position of the pore and can be represented by the 3D coordinates of the pore's center. Pore identification can be performed based on at least one of pore shape or pore size, for example. Specifically, a worker can send the pore shape or size information to the controller via a terminal, and the controller can then perform pore identification based on this shape or size information.

[0070] In a specific implementation, taking the case of a circular pore as an example, the controller can perform circle detection based on 2D image information, determine the pore region containing a circular structure in the 2D image, and further determine a 3D point cloud region that is mapped to the pore region. The controller then performs 3D circle detection on the 3D point cloud region, fitting the circular structure parameters to obtain the pore location information. Determining the pore region based on 2D image information before further detecting the 3D location information ensures the accuracy of the location information while improving efficiency.

[0071] Step S206 : For each pore, determine the information collection posture for the pore based on the position information of the pore.

[0072] Specifically, after determining the position information of each tube hole in the tube sheet workpiece, the tube hole information can be collected for each tube hole separately. The information collection posture refers to the position and posture of the information collection device when collecting information from the tube hole. In this embodiment, the device for collecting information from the tube hole is called a local collection device. Correspondingly, the information collection posture for the tube hole refers to the information collection posture of the local collection device for the tube hole. As mentioned above, if the tube sheet workpiece includes multiple tube holes, a local collection device can be configured to realize the tube hole information collection for multiple tube holes by changing the information collection posture; when the number of tube holes is large, in order to improve the information collection efficiency, multiple local collection devices can also be configured to collect tube hole information. Furthermore, the local collection device and the global collection device can be integrated into the same terminal or set separately.

[0073] In a specific embodiment, as shown in Figure 3, the global acquisition device 1 and the local acquisition device 3 can be separate devices located at different locations. The global acquisition device 1 is located above the C-shaped column 8 to collect global information about the tube sheet workpiece. The local acquisition device 3 is located at the end of the welding robot's mechanical arm 2 and can adjust its position to capture information. In this embodiment, the position of the local acquisition device 3 can be represented by the position information of the mechanical arm 2.

[0074] Step S208: fitting the expected welding trajectory of the pipe hole based on the pipe hole information collected in the information collection posture.

[0075] The expected welding trajectory of the pipe hole is the trajectory of the edge of the pipe hole. Specifically, the controller can determine the information collection posture for each pipe hole, and obtain the pipe hole information collected for the pipe hole under the information collection posture, and then perform edge fitting based on the pipe hole information to obtain the expected welding trajectory of the pipe hole. The specific method of fitting to obtain the expected welding trajectory is not unique. For example, the controller can determine multiple position points of the pipe hole edge based on the pipe hole information, and then smoothly connect each position point to obtain the expected welding trajectory of the pipe hole. For another example, the controller can determine the expected welding trajectory of the pipe hole based on the pipe hole shape and multiple position points of the pipe hole edge, and perform edge fitting based on the pipe hole shape.

[0076] In step S210 , the welding robot is controlled to weld the tube sheet workpiece according to the desired welding trajectory of each tube hole.

[0077] A welding robot refers to an industrial robot that includes a welding arm and is capable of welding. In a specific implementation, as shown in Figure 3 , the welding robot may include a robotic arm 2, a welding machine 7, a control cabinet 9, a wire drum 10, and a gun cleaning station 11. These components can be fixed to a C-shaped column 8 to follow the C-shaped column 8 as it moves on a floor rail 6. Arc striking and wire feeding are performed at the end of the robotic arm 2 to weld the tubesheet workpiece 4. The tubesheet workpiece 4 can be fixed to a workbench 5. It should be noted that the mechanical structure and assembly method shown in Figure 3 are not exclusive and can be adjusted based on actual application requirements. For example, the welding robot can be configured as a front-mounted or side-mounted design depending on the size of the tubesheet workpiece 4; the C-shaped column can be replaced with a gantry design, and the floor rail can be replaced with an overhead rail, etc. Furthermore, multiple welding arms can be used to weld a single tubesheet workpiece, or a single welding arm can be used to weld multiple tubesheet workpieces.

[0078] Specifically, the controller can control the welding robot to weld the tube sheet workpiece according to the desired welding trajectory for each tube hole. It should be noted that determining the desired welding trajectory and performing welding can be performed alternately; alternatively, the welding robot can be controlled to perform welding after the desired welding trajectory for each tube hole has been determined. For example, the controller can first fit the desired welding trajectory for each tube hole, then control the welding robot to weld that tube hole. Then, the controller can fit the desired welding trajectory for the next tube hole, and so on, until all tube holes in the tube sheet workpiece are welded. It can be understood that when all tube holes in the tube sheet workpiece are welded, the welding of the tube sheet workpiece is also complete.

[0079] In one embodiment, step S210 includes: obtaining a welding sequence for each pipe hole; and controlling a welding robot to weld each pipe hole in sequence according to a desired welding trajectory for each pipe hole.

[0080] The welding sequence for the tube holes can be from outside to inside, clockwise, counterclockwise, radially, or circumferentially, among other options. Specifically, the controller can determine the welding sequence for each tube hole and then, based on the desired welding trajectory for each tube hole, control the welding robot to weld each tube hole in the welding sequence. Welding according to this welding sequence allows workers to adjust the welding sequence based on the distribution of tube holes in the tube sheet workpiece, shortening the robot's motion path and improving welding efficiency.

[0081] The above-mentioned tubesheet workpiece welding method first determines the tube hole positions based on the global information of the tubesheet workpiece to achieve coarse positioning of each tube hole in the tubesheet workpiece. Then, based on the tube hole positions obtained by global positioning, a separate information collection pose is determined for each tube hole. Then, based on the tube hole information collected for each tube hole in the corresponding information collection pose, the desired welding trajectory of the tube hole is fitted to achieve precise positioning of each tube hole. Finally, according to the desired welding trajectory of each tube hole, the welding robot is controlled to weld the tubesheet workpiece. This is equivalent to determining the information collection pose based on the coarse positioning results and then achieving precise positioning of the tube hole based on the tube hole information collected in the information collection pose. This can complete the welding of the tubesheet workpiece without human intervention, which is beneficial to reducing labor costs. In addition, first determining the tube hole positions based on global information and then fitting the desired welding trajectory for each tube hole is equivalent to performing global coarse positioning first and then local precise positioning. This can ensure the accuracy of positioning and, therefore, the accuracy of the fitted desired welding trajectory, thereby improving the welding quality of the tubesheet workpiece.

[0082] In one embodiment, the global information is collected by a global collection device. In this embodiment, step S204 includes: obtaining a first extrinsic parameter of the global collection device; and determining the position information of each pore in the reference coordinate system based on the global information and the first extrinsic parameter.

[0083] Among them, the specific definition of the global acquisition device can be found above and will not be repeated here. The first external parameter of the global acquisition device is used to characterize the conversion relationship between the first coordinate system corresponding to the global acquisition device and the reference coordinate system. Taking the case where the global acquisition device is the first camera as an example, the first coordinate system may refer to the camera coordinate system of the first camera. The camera coordinate system is a three-dimensional rectangular coordinate system established with the focal center of the camera as the origin and the optical axis as the Z axis. The reference coordinate system refers to a coordinate system used as a reference. The reference coordinate system may, for example, refer to the robot base coordinate system or the world coordinate system.

[0084] Specifically, the controller can obtain a first extrinsic parameter from the global acquisition device, then perform pore identification based on the global information to determine the position information of each pore in the first coordinate system. Finally, the controller uses the first extrinsic parameter to transform the position information in the first coordinate system to obtain the position information of each pore in the reference coordinate system.

[0085] In this embodiment, setting a reference coordinate system can facilitate the conversion of position information in multiple coordinate systems into the same coordinate system, thereby ensuring the accuracy of pipe hole positioning and improving welding effects.

[0086] In one embodiment, the global information includes three-dimensional point cloud information and two-dimensional image information. In this embodiment, determining the position information of each hole in the reference coordinate system based on the global information and the first extrinsic parameter includes: determining the hole region in the tube sheet workpiece based on the two-dimensional image information; performing hole recognition on the three-dimensional point cloud information of the hole region to determine first position information of each hole in the first coordinate system; and performing coordinate transformation on each first position information based on the first extrinsic parameter to obtain the position information of each hole in the reference coordinate system.

[0087] Specifically, the controller can perform pore recognition on the 2D image information to determine the pore region containing the pores in the 2D image. Based on the mapping relationship between the 2D image information and the 3D point cloud information, the controller can determine the 3D point cloud information for the pore region. Next, the controller can perform pore recognition on the 3D point cloud information for the pore region to determine the first position information for each pore in the first coordinate system. Finally, the controller performs coordinate transformation on each piece of first position information based on the first extrinsic parameter to obtain the position information for each pore in the reference coordinate system.

[0088] In the above embodiment, the pore area is first determined based on the two-dimensional image information, and then the three-dimensional position information of the pore area is further detected, which can improve efficiency while ensuring the accuracy of the position information.

[0089] In one embodiment, based on the position information of the pore, determining the information collection posture for the pore includes: determining the coordinate system information corresponding to the local collection device in the reference coordinate system based on the position information of the pore; and determining the working posture matching the coordinate system information as the information collection posture of the local collection device for the pore.

[0090] Among them, the coordinate system information can be used to characterize the coordinate system, and the coordinate system information can include the position information of the coordinate origin, the direction information of the coordinate axis, and so on. The coordinate system information corresponding to the local acquisition device in the reference coordinate system includes the position of the coordinate origin and the coordinate axis direction of the coordinate system corresponding to the local acquisition device in the reference coordinate system. Taking the case where the local acquisition device is a second camera as an example, the coordinate system corresponding to the local acquisition device can refer to the camera coordinate system of the second camera, which is recorded as the second coordinate system; the coordinate system information corresponding to the local acquisition device can include the coordinate origin position and coordinate axis direction of the second coordinate system. Specifically, the controller can determine the coordinate system information corresponding to the local acquisition device in the reference coordinate system for each pipe hole based on the position information of the pipe hole, and determine the working posture matching the coordinate system information as the information acquisition posture of the local acquisition device for the pipe hole.

[0091] In this embodiment, the information collection posture in the reference coordinate system is determined based on the position information of the pipe hole in the reference coordinate system, which is equivalent to using an absolute positioning method to determine the information collection posture for the pipe hole, which can improve the accuracy of the information collection posture, thereby ensuring the smooth progress of the pipe hole information collection and improving the accuracy of the expected welding trajectory obtained based on the pipe hole information fitting.

[0092] In one embodiment, based on the position information of the pipe hole, the coordinate system information corresponding to the local acquisition device in the reference coordinate system is determined, including: determining the center of the pipe hole represented by the position information of the pipe hole, and the end face of the pipe hole to be welded; based on the device type of the local acquisition device, determining the characteristic point of the pipe hole from the center and the end face of the pipe hole, and determining the characteristic direction pointing to the characteristic point; determining the recommended viewing distance of the local acquisition device as the distance between the coordinate origin corresponding to the local acquisition device in the reference coordinate system and the characteristic point; and determining the characteristic direction as the direction of the first coordinate axis corresponding to the local acquisition device in the reference coordinate system.

[0093] The first coordinate axis direction coincides with the detection signal emission direction of the local acquisition device. The detection signal emission direction of the local acquisition device can refer to the optical axis of a 3D camera or a linear structured light acquisition device, or the electromagnetic wave transmission direction of a radar.

[0094] Specifically, obtaining the position information of each hole in the reference coordinate system effectively completes macro-positioning recognition of the hole in the tubesheet workpiece, enabling the approximate position and shape of the hole in the tubesheet workpiece to be determined. Based on this information, the controller can determine the hole center, as represented by the hole position information, and the hole end face to be welded. As shown in Figure 4, using the circular hole as an example, the controller can determine the hole center 411 and the hole end face 412 to be welded for each hole based on the hole position information.

[0095] Furthermore, the device type can be used to characterize the principle by which the local acquisition device acquires image information. This device type can, for example, be a 3D camera, a line structured light acquisition device, or a point structured light acquisition device. It is understood that different information acquisition principles inevitably lead to different information acquisition poses for the same pipe hole. For example, a 3D camera can acquire information about the pipe hole by shooting from above; whereas a line structured light acquisition device requires acquiring information from multiple lateral positions to fit the desired weld trajectory. Based on this, the controller can determine the characteristic point of the pipe hole from the center and end face of the pipe hole, and determine the characteristic direction pointing toward the characteristic point, based on the device type of the local acquisition device. The controller then determines the recommended viewing distance for the local acquisition device as the distance between the coordinate origin corresponding to the local acquisition device in the reference coordinate system and the characteristic point. This recommended viewing distance can be found in the recommended operating parameters of the local acquisition device; for example, it can be the optimal viewing distance value for a 3D camera. Finally, the controller determines the characteristic direction as the direction of the first coordinate axis corresponding to the local acquisition device in the reference coordinate system. On this basis, the second and third coordinate axis directions can be further determined in a coordinate plane perpendicular to the first coordinate axis direction. For ease of understanding, this application refers to the first coordinate axis direction as the Z direction, and the second and third coordinate axis directions as the X direction and Y direction, respectively.

[0096] In the above embodiment, combining the position information of the pipe hole in the reference coordinate system and the device type of the local acquisition device to determine the coordinate system information corresponding to the local acquisition device in the reference coordinate system can improve the accuracy of the coordinate system information and thereby improve the accuracy of the information acquisition posture represented by the coordinate system information.

[0097] In a specific embodiment, based on the device type of the local acquisition device, the characteristic point of the pore is determined from the center and end face of the pore, and the characteristic direction pointing to the characteristic point is determined, including: when the local acquisition device is a three-dimensional camera, the center of the pore is determined as the characteristic point of the pore; and the direction perpendicular to the end face of the pore and pointing to the characteristic point is determined as the characteristic direction.

[0098] Specifically, as shown in FIG5 , when the local acquisition device is a three-dimensional camera, the center of the tube hole O i Determine the characteristic point of the tube hole and make the line perpendicular to the end face of the tube hole and pointing to the characteristic point O i The direction of the second coordinate system is determined as the characteristic direction. C2 and the center of the tube hole O i The line connecting the two is perpendicular to the end face of the hole i, and the coordinate origin O C2 and the center of the tube hole O i The distance is h, where h is the recommended viewing distance of the 3D camera.

[0099] In another specific embodiment, based on the device type of the local acquisition device, the characteristic points of the tube hole are determined from the center and the end face of the tube hole, and the characteristic direction pointing to the characteristic point is determined, including: when the local acquisition device is a line structured light acquisition device, multiple edge points on the end face of the tube hole are determined as multiple characteristic points; and the characteristic direction corresponding to each characteristic point is determined respectively.

[0100] The characteristic line corresponding to the characteristic direction of any feature point is perpendicular to the tangent line of the pore edge passing through the feature point, and the angle between the line connecting the feature point and the pore center and the characteristic line is acute. Specifically, if the local acquisition device is a line structured light acquisition device, multiple shooting positions are required for each pore. Based on this, the controller can identify multiple edge points on the pore end face as multiple feature points and determine the characteristic direction corresponding to each feature point, thereby obtaining multiple information acquisition positions.

[0101] For example, as shown in FIG5 , when the local acquisition device is a line structured light acquisition device, any edge point P of the end face of the tube hole i can be determined as a feature point, and the P point and the tube hole center O i The connecting line is recorded as l1. Draw the tangent line l2 of the tube hole edge C through point P, and then determine the characteristic line l3 based on l1 and l2, and then determine the characteristic direction z corresponding to the characteristic point P on the characteristic line l3. Among them, the coordinate origin O C2 and the center of the tube hole O i The distance between the two points is h, where h is the recommended viewing distance of the line structured light acquisition device. L3 passes through point P and is perpendicular to L2. The angle between L3 and L1 is α, where α is less than 90°. For example, α can be 45° or 50°, etc., and can be determined based on the actual imaging performance of the line structured light acquisition device to ensure that the laser stripes on the weld are fully exposed to the field of view and are not blocked by adjacent pipe holes.

[0102] In the above embodiment, different characteristic point and characteristic direction determination methods are provided for two different types of local acquisition devices, which can match the characteristics of the local acquisition devices and are conducive to improving the flexibility of the tube-sheet workpiece welding method.

[0103] In one specific application, the tube-sheet workpiece welding method further includes determining the end face shape of each tube hole based on global information. In this embodiment, multiple edge points on the tube hole end face are determined as multiple feature points, including: determining a number of feature points that match the end face shape; and determining multiple feature points evenly distributed on the tube hole end face based on the number of feature points.

[0104] The end face shape can be, for example, circular, elliptical, or square. The number of feature points that match the end face shape refers to the number of feature points required to fit the end face shape. For example, a circle requires at least three feature points, and an ellipse requires at least four feature points. Specifically, the controller can determine the end face shape of each pore based on global information. Then, the number of feature points that match the end face shape is determined, and based on the number of feature points, a plurality of evenly distributed feature points are determined on the pore end face. For example, in the case where the pore end face shape is circular, four evenly distributed feature points can be determined on the pore end face, thereby determining four evenly distributed information collection positions.

[0105] In this embodiment, based on the number of feature points matching the end face shape, multiple feature points evenly distributed on the end face of the pipe hole are determined, which can ensure that the desired welding trajectory is fitted while improving the efficiency of pipe hole information collection and trajectory fitting.

[0106] In one embodiment, the tube-sheet workpiece welding method further includes determining the end face shape of each tube hole based on global information. In this embodiment, step S208 includes acquiring tube hole point cloud information acquired by a local acquisition device in an information acquisition position; and performing position fitting on the tube hole point cloud information based on a fitting algorithm that matches the end face shape of the tube hole to obtain a desired welding trajectory for the tube hole.

[0107] Specifically, the controller can determine the end face shape of each tube hole based on the global information of the tube sheet workpiece. Then, for each tube hole, the controller can obtain the point cloud information collected by the local acquisition device in the information acquisition posture. Using a fitting algorithm that matches the end face shape of the tube hole, the controller performs position fitting on the point cloud information to obtain the desired welding trajectory for the tube hole. For example, if the end face shape of the tube hole is circular, the controller can calculate the desired welding trajectory using a spatial circle fitting algorithm.

[0108] In this embodiment, based on a fitting algorithm that matches the end face shape of the tube hole, the tube hole point cloud information is positionally fitted to obtain the desired welding trajectory of the tube hole, which can match the trajectory fitting requirements of tube holes of different shapes. When the tube hole workpiece includes tube holes of various shapes, automatic welding of the tube sheet workpiece can be completed, which is beneficial to improving the flexibility of the tube sheet workpiece welding method.

[0109] In one embodiment, the reference coordinate system is the robot base coordinate system of the welding robot, and the local acquisition device is fixed to the tool flange at the end of the welding robot. In this embodiment, the working posture that matches the coordinate system information is determined as the information acquisition posture of the local acquisition device for the pipe hole, including: obtaining a second external parameter of the local acquisition device; determining a third external parameter of the local acquisition device based on the coordinate system information; combining the second and third external parameters to determine the robot coordinate transformation relationship between the robot tool coordinate system and the robot base coordinate system; and determining the working posture represented by the robot coordinate transformation relationship as the information acquisition posture of the local acquisition device for the pipe hole.

[0110] The robot base coordinate system is a rectangular coordinate system based on the robot mounting base and used to describe the motion of the robot body. The robot tool coordinate system refers to a coordinate system fixed to the flange and remains unchanged relative to the tool flange center of the robot arm. The tool coordinate system origin (TCP), also known as the center point of manipulator motion, is the working point of the tool mounted on the robot. The second external parameter is used to characterize the conversion relationship between the second coordinate system corresponding to the local acquisition device and the robot tool coordinate system of the welding robot; the third external parameter is used to characterize the conversion relationship between the second coordinate system and the robot base coordinate system; and the robot coordinate transformation relationship is used to characterize the conversion relationship between the robot tool coordinate system and the robot base coordinate system. For the same welding system, the second external parameter, the third external parameter, and the robot coordinate transformation relationship satisfy the following condition: second external parameter * robot coordinate transformation relationship = third external parameter. Therefore, the other transformation relationship can be determined based on any two of the above three transformation relationships.

[0111] Specifically, the second external parameter is determined by the installation position of the local acquisition device on the tool flange at the end of the robot arm. That is, after the local acquisition device is installed on the tool flange, as long as the relative position of the local acquisition device and the tool flange remains unchanged, the second external parameter of the local acquisition device will not change. Based on this, the controller can, on the one hand, obtain the second external parameter of the local acquisition device based on the relative position of the local acquisition device and the tool flange. On the other hand, the controller can determine the third external parameter of the local acquisition device based on the coordinate system information of the second coordinate system in the reference coordinate system. The controller then combines the second and third external parameters to determine the robot coordinate transformation relationship between the robot tool coordinate system and the robot base coordinate system. It can be understood that this robot coordinate transformation relationship can represent the posture of the end of the robot arm relative to the robot mounting base, that is, the posture of the end of the robot arm. After installation, the local acquisition device maintains a constant position relative to the end of the robot arm flange and completes information collection while following the movement of the robot arm. Therefore, the working posture represented by the robot coordinate transformation relationship can also represent the information collection posture of the local acquisition device. Based on this, the controller can determine the working posture represented by the robot coordinate transformation relationship as the information collection posture of the local collection device for the pipe hole.

[0112] In the above embodiment, the local acquisition device is fixed to the tool flange at the end of the welding robot, and the information acquisition posture of the local acquisition device is represented by the posture of the robot end. The information acquisition posture can be controlled by controlling the welding robot, thereby realizing weld tracking during the welding process, which is beneficial to further improve the welding effect.

[0113] In one embodiment, as shown in FIG6 , a tube sheet workpiece welding method is provided, the method comprising the following steps:

[0114] Step S601, obtaining the field of view of the global acquisition device;

[0115] Step S602: Based on the position of the tube sheet workpiece in the field of view, adjusting the fixed position of the global acquisition device so that the tube sheet workpiece is included in the field of view of the global acquisition device;

[0116] Step S603, obtaining global information collected by the global collection device for the tube sheet workpiece and a first external parameter of the global collection device;

[0117] The global information includes three-dimensional point cloud information and two-dimensional image information; the first external parameter is used to characterize the conversion relationship between the first coordinate system corresponding to the global acquisition device and the reference coordinate system; the reference coordinate system is the robot base coordinate system of the welding robot;

[0118] Step S604: determining the tube hole area in the tube sheet workpiece based on the two-dimensional image information;

[0119] Step S605: performing pore recognition on the three-dimensional point cloud information of the pore area to determine the first position information of each pore in the first coordinate system and the end face shape of each pore;

[0120] Step S606: performing coordinate transformation on each first position information based on the first external parameter to obtain position information of each pore in the reference coordinate system;

[0121] Step S607: for each tube hole, determining the tube hole center represented by the position information of the tube hole and the tube hole end face to be welded;

[0122] Step S608: When the local acquisition device is a 3D camera, the center of the pore is determined as the characteristic point of the pore; and the direction perpendicular to the end face of the pore and pointing to the characteristic point is determined as the characteristic direction;

[0123] Step S609: when the local acquisition device is a line structured light acquisition device, determine the number of feature points that match the end face shape of the tube hole;

[0124] Step S610: determining a plurality of evenly distributed characteristic points on the end face of the tube hole based on the number of characteristic points, and determining a characteristic direction corresponding to each characteristic point;

[0125] The characteristic line corresponding to the characteristic direction of any characteristic point is perpendicular to the tangent line of the pore edge passing through the characteristic point; the angle between the line connecting the characteristic point and the pore center and the characteristic line is an acute angle;

[0126] Step S611, determining the recommended viewing distance of the local acquisition device as the distance between the coordinate origin and the feature point corresponding to the local acquisition device in the reference coordinate system;

[0127] Step S612: determining the characteristic direction as the first coordinate axis direction corresponding to the local acquisition device in the reference coordinate system;

[0128] Wherein, the coordinate line in the direction of the first coordinate axis coincides with the optical axis of the local acquisition device;

[0129] Step S613, obtaining a second extrinsic parameter of the local acquisition device, and determining a third extrinsic parameter of the local acquisition device based on the coordinate system information;

[0130] The local acquisition device is fixed to the tool flange at the end of the welding robot; the second external parameter is used to characterize the conversion relationship between the second coordinate system corresponding to the local acquisition device and the robot tool coordinate system of the welding robot; the third external parameter is used to characterize the conversion relationship between the second coordinate system and the robot base coordinate system;

[0131] Step S614, combining the second external parameter and the third external parameter to determine the robot coordinate transformation relationship between the robot tool coordinate system and the robot base coordinate system;

[0132] Step S615, determining the working posture represented by the robot coordinate transformation relationship as the information collection posture of the local collection device for the pipe hole;

[0133] Step S616, obtaining pore information collected by the local collection device in the information collection posture for the pores;

[0134] Step S617: performing position fitting on the hole information based on a fitting algorithm that matches the end face shape of the hole to obtain a desired welding trajectory for the hole.

[0135] Step S618 , according to the desired welding trajectory of each pipe hole, the welding robot is controlled to weld each pipe hole in sequence according to the set welding sequence.

[0136] In the above embodiment, the hole positions are first determined based on the global information of the tubesheet workpiece to achieve coarse positioning of each hole in the tubesheet workpiece. Then, based on the hole positions obtained from global positioning, a separate information collection pose is determined for each hole. Then, based on the hole information collected for each hole at the corresponding information collection pose, a desired welding trajectory for the hole is fitted to achieve precise positioning of each hole. Finally, the welding robot is controlled to weld the tubesheet workpiece according to the desired welding trajectory for each hole. This is equivalent to determining the information collection pose based on the coarse positioning results and then achieving precise positioning of the hole based on the hole information collected at the information collection pose. This allows welding of the tubesheet workpiece to be completed without human intervention, which is beneficial for reducing labor costs. Furthermore, first determining the hole positions based on global information and then fitting the desired welding trajectory for each hole is equivalent to performing global coarse positioning first and then local precise positioning. This ensures positioning accuracy and, in turn, the accuracy of the fitted desired welding trajectory, thereby improving the welding quality of the tubesheet workpiece.

[0137] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the above-mentioned embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0138] Based on the same inventive concept, embodiments of the present application also provide a tube-sheet workpiece welding system for implementing the aforementioned tube-sheet workpiece welding method. The solution provided by this system is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more tube-sheet workpiece welding system embodiments provided below can be found in the aforementioned limitations of the tube-sheet workpiece welding method and will not be further elaborated here.

[0139] In one embodiment, as shown in FIG7 , a tube sheet workpiece welding system is provided, comprising a controller 701, a global acquisition device 702, a local acquisition device 703, and a welding robot 704 connected to the controller 701. The global acquisition device 702 is used to collect global information about the tube sheet workpiece, obtaining global information about the tube sheet workpiece; the local acquisition device 703 is used to collect information about each tube hole, obtaining information about each tube hole; the welding robot 704 is used to weld the tube sheet workpiece; and the controller 701 is used to implement the aforementioned tube sheet workpiece welding method.

[0140] The tube sheet workpiece includes multiple tube holes. A global acquisition device is a device capable of collecting information and may include at least one of a camera, radar, or laser. A local acquisition device is a device capable of collecting information and may include a camera, a line structured light acquisition device, a point structured light acquisition device, or the like. The recommended viewing distance for a global acquisition device is greater than that for a local acquisition device, and the positioning accuracy of a local acquisition device is greater than that of a global acquisition device.

[0141] Specifically, as shown in Figure 3, the global data acquisition device 1 can be fixed directly above the tube sheet workpiece 4, allowing global data collection of the entire tube sheet end face to be completed in a single acquisition. The local data acquisition device 3 can be fixed to the end flange of the welding robot's robotic arm 2 and can follow the movement of the robotic arm 2, thereby collecting data on each tube hole in different positions.

[0142] During tube-sheet welding, controller 701 acquires global information collected by global acquisition device 702 and determines the positional information of each tube hole on the tube-sheet workpiece based on this global information. Controller 701 then determines the information collection pose for each tube hole based on the positional information. Based on the tube hole information collected by local acquisition device 703 at this information collection pose, controller 701 fits the desired welding trajectory for each tube hole. After obtaining the desired welding trajectory for each tube hole, controller 701 controls welding robot 704 to weld the tube-sheet workpiece according to the desired welding trajectory for each tube hole.

[0143] The aforementioned tubesheet welding system first determines the hole positions based on the tubesheet's global information, achieving coarse positioning of each hole in the tubesheet. Then, based on the hole positions obtained from global positioning, it determines a specific information collection pose for each hole. Based on the hole information collected at the corresponding information collection pose, it fits the desired welding trajectory for each hole, achieving precise positioning of each hole. Finally, it controls the welding robot to weld the tubesheet according to the desired welding trajectory for each hole. This is equivalent to determining the information collection pose based on the coarse positioning results and then achieving precise positioning of the holes based on the hole information collected at the information collection pose. This system can complete tubesheet welding without human intervention, which helps reduce labor costs. Furthermore, first determining the hole positions based on global information and then fitting the desired welding trajectory for each hole is equivalent to performing global coarse positioning followed by local precise positioning. This ensures positioning accuracy, and thus the accuracy of the fitted desired welding trajectory, thereby improving the welding quality of the tubesheet.

[0144] In a specific embodiment, taking the case where the global acquisition device is a first camera, the local acquisition device is a second camera, and the shape of the end face of the tube hole is circular as an example, the tube sheet workpiece welding method and system are described in detail.

[0145] Specifically, a macro-positioning vision system can be used to identify the number of tube holes, measure their outer diameters, and perform coarse positioning of the center of each tube hole. The coordinates of the tube hole centers obtained by the macro-positioning vision system are then used to calculate the camera pose for the fine-positioning vision system. The fine-positioning vision system then accurately measures the characteristic positions of the tube holes, obtaining hole information. This information is then used to fit the precise trajectory of the tube holes. This precise trajectory becomes the desired welding trajectory for the tube holes.

[0146] Among them, the macro positioning vision system may include a first camera and a controller. As shown in Figure 3, the global acquisition device 1 (i.e., the first camera) can be installed directly above the tube sheet workpiece 4, and its relative position to the robot base coordinate system is fixed, and the camera field of view must be able to completely cover all the tube holes to be welded on the tube sheet workpiece 4, so that the three-dimensional point cloud data and two-dimensional image data of the tube sheet end face can be collected by taking a single photo. The controller can be deployed in the control cabinet 9 to obtain three-dimensional point cloud data and two-dimensional image data, and combine the three-dimensional point cloud data and two-dimensional image data to identify and fit the circular trajectory of the outer edge of the tube hole on the tube sheet. As shown in Figure 5, based on the fitted circular trajectory of the tube hole, the controller can calculate the position of the center of the circle in the camera coordinate system of the first camera (i.e., the first coordinate system). Cam O i =(x c_i ,y c_i ,z c_i ) and the outer diameter D of the tube holei , where i = 1, 2, 3…n, and n is the total number of tube holes on the tube sheet.

[0147] In a specific application, the specific process of pore identification includes the following steps:

[0148] 1. 2D image circle detection:

[0149] A parameter-free edge detection algorithm generates a continuous set of edge segments. Taking into account the influence of various welding noises, non-edge pixels are eliminated through non-maximum suppression, hysteresis thresholding, and erosion operations. This effectively mitigates the impact of arc light, smoke, and other noise on image quality. The least-squares linear equation of these edge segments is then calculated, along with the fitting error. If the fitting error is less than a given threshold, the edge segment is considered a straight line segment. Otherwise, it is split into two sub-segments, and each sub-segment is recursively fitted and split until all edge segments are fitted as straight lines or reach a predetermined minimum length. Based on the angular change rate of adjacent straight lines, segments with an angular change rate less than a threshold are converted into circular arcs. An adaptive polygonal approximation algorithm is then used to detect candidate circles and near-circular ellipses. Based on the Helmholtz principle, a reverse verification step is used to verify the candidate points, eliminating any false positives.

[0150] After performing image circle detection, circles within a certain radius are filtered and merged. After obtaining the 2D circle detection results, the circle center coordinates and radius are converted into candidate 2D regions for detection in the point cloud. Alternatively, object detection and image segmentation methods can be used to perform circle detection on 2D images.

[0151] 2. Convert the 2D area into a 3D point cloud area:

[0152] According to the mapping relationship between the two-dimensional image and the three-dimensional point cloud, the point cloud information corresponding to the pore area in the two-dimensional image can be determined, and then the three-dimensional coordinate value of the corresponding point in the camera coordinate system can be obtained by identifying the position of the point cloud information. For example, the first position can be confirmed as the upper left corner of the corresponding two-dimensional area, and the second position can be confirmed as the lower left corner of the corresponding two-dimensional area. The coordinates of the upper left corner and the lower left corner can be converted into a point cloud to obtain the coordinate value of the corresponding point in the camera coordinate system, thereby obtaining the three-dimensional point cloud area corresponding to the pore area. Assuming that the coordinates of a pixel in the depth map are (u, v) and the corresponding depth value is d, then the coordinates in the three-dimensional space corresponding to the pixel are (x, y, z). Assuming that the intrinsic parameter matrix of the camera is K, it is expressed as:

[0153] Among them, f x and f yis the focal length along the x-axis and y-axis, s is the skew coefficient, c x and c y is the coordinate of the principal point. The formula for converting an image point (u, v) to a point cloud (x, y, z) is:

[0154] In order to prevent the conversion process, the single pixel point of the conversion from being affected by sparks and arcs, resulting in abnormal ranges, the depth map can be filtered by median, and the median depth information of the pixel point is taken as the depth information to be converted. For a two-dimensional area, according to the upper left corner coordinate (u l ,v l ) and the lower right corner coordinates (u r ,v r ) to obtain the corresponding three-dimensional coordinates (x l ,y l ,z l ) and (x r ,y r ,z r ), so that the three-dimensional region satisfies the following relationship:

[0155] Among them, μ is used to expand the conversion area for the redundant area, μ is the expansion coefficient of the high probability area, and x min and x max Indicates the minimum and maximum x-axis coordinates of the transformed three-dimensional area, y min and y max Indicates the minimum and maximum y-axis coordinates of the transformed three-dimensional area, z min and z max Indicates the minimum and maximum z-axis coordinates of the transformed 3D region.

[0156] 3. Preprocess the acquired 3D point cloud data:

[0157] After performing straight-through filtering on the point cloud, circle detection was performed. Considering the potential for outliers and missing values ​​in welding scenarios due to arc light, metal spatter, and other influences, the point cloud was first calculated to remove outliers. Moving least squares was then used to complete the point cloud. This processed point cloud eliminates the influence of outliers such as arc light and fills in small missing values ​​caused by spatter.

[0158] 4. Perform point cloud circle detection on the pre-processed point cloud data:

[0159] A random consistency sampling algorithm can be used to implement point cloud circle detection. After obtaining the curve equation of the fitted circle, all points within the equation range are classified as points on the circle, and the remaining points continue to be used as the input point cloud for the next stage of point cloud detection. Specifically, after obtaining the results of the two-dimensional circle detection, the high-probability areas marked as point cloud circle detection are prioritized, and the point clouds in these areas are subjected to the three-dimensional circle detection algorithm. Then, the same three-dimensional circle detection algorithm is performed on the remaining point cloud areas to improve the efficiency and accuracy of circle detection. In this way, the information of the two-dimensional circle detection can be fully utilized to determine in advance the areas with a high probability of circular structures appearing, narrow the detection range, and save computing resources. At the same time, the shortcomings of two-dimensional circle detection can be avoided. For situations where the two-dimensional circle detection cannot be identified or missed, the three-dimensional circle detection algorithm can still be used to find the circular structure from the remaining point cloud. Optionally, a point cloud segmentation method can also be used for three-dimensional point cloud circle detection.

[0160] 5. Candidate circle filtering:

[0161] For the results obtained by the point cloud circle detection algorithm, we can filter circles within a specified radius and then merge the circles whose centers fall within the range. The merged circle result is the final circle result, as shown in Figure 4. The entire image is the input point cloud, and the bold circular curve is the final detected circle result.

[0162] After obtaining the position of the tube hole in the first coordinate system, the controller can combine the camera hand-eye matrix (i.e., the first external parameter of the first camera) to calculate the position information of the center of all tube holes in the robot base coordinate system. The camera hand-eye calibration process is completed in advance before the measurement begins. The calibration software calculates the pose matrix H of the camera coordinate system and the robot base coordinate system. hand-eye Then the position of the center of the hole in the robot base coordinate system can be obtained as:

[0163] O i =H hand-eye · Cam Oi

[0164] The above is the completion of the rough positioning of the tube sheet workpiece. The following is the precise positioning of each tube hole.

[0165] Among them, the fine positioning vision system may include a second camera and a controller, and the second camera is mounted on the tool flange at the end of the robot. The second camera can be a weld tracking sensor (such as a line structured light acquisition device) or a small field of view three-dimensional camera. First, the second camera needs to be calibrated offline to determine the position conversion matrix between the camera coordinate system of the second camera (i.e., the second coordinate system) and the robot tool coordinate system (i.e., the second external parameter of the second camera). Then, the controller calculates the coordinate system information of the second camera based on the center position and the outer diameter of the pipe hole identified by the macro positioning vision system, and then determines the shooting posture of the second camera.

[0166] The calculation method of the coordinate system information of the second camera is shown in Figure 5. The center of the hole is O i , the edge trajectory of the pore is C, l1 is the auxiliary line passing through the center of the circle and intersecting C at point P, l2 is the tangent line on C passing through point P, l3 passes through point P and is perpendicular to l2. The angle between l3 and l1 is α. c2 is the second camera coordinate system.

[0167] When the second camera is a small field of view 3D camera, the second phase coordinate system O c2 The z-axis of the robot is perpendicular to the plane of C and maintains a certain height h, where h is the optimal viewing distance of the camera. These two constraints are used to control the robot's position. A photo of the circular trajectory C is taken at a position that satisfies these conditions. A single shot is all it takes to obtain a high-precision point cloud of the pipe hole. A spatial circle fitting algorithm, such as the least squares method, RANSAC, conjugate gradient method, or Hough transform, is then used to calculate the precise trajectory of the pipe hole in the robot coordinate system.

[0168] When the second camera is a weld tracking sensor, at least four camera positions are required for each pipe hole. First, set the first camera position: As shown in Figure 3, make the second camera coordinate system O c2 The z-axis points to the direction of l3, and the typical value of the angle α between the z-axis and l1 is 45°, which can be adjusted according to the actual camera imaging effect to ensure that the laser stripes on the weld can be fully exposed to the camera field of view and are not blocked by adjacent pipe holes. c2 The distance between the origin and point P is h, and the optimal viewing distance of the camera is taken. After the first photo position is determined, the remaining photo positions are evenly and symmetrically distributed around the trajectory C at 360°. The symmetry axis is parallel to the normal vector of the plane where C is located and passes through O i , with the distribution's rotation radius equal to l1. Taking four camera positions as an example, the remaining three positions are obtained by rotating 90° along the axis of symmetry, starting from the first position. After calculating the camera pose for each camera position, the robot can be automatically controlled by a program to move to the corresponding position and activate the 2D camera of the weld tracking sensor to take pictures. An image recognition algorithm can be used to identify the locations of weld feature points from the laser stripes captured by the 2D camera. Finally, the hand-eye conversion matrix is ​​used to calculate the four locations on trajectory C. The coordinates of these four locations are fitted to obtain the precise center, radius, and spatial expression of trajectory C. It should be noted that although the macro-positioning vision system can identify the location of the hole center in the robot's base coordinate system and serve as the basis for setting the camera positions for fine positioning, the camera's viewing distance is much greater than that of the fine positioning camera, so the center coordinates obtained by macro-positioning are relatively large. Therefore, the desired weld trajectory is fitted based on the measurement results of the fine positioning vision system.

[0169] Finally, based on the desired welding trajectory for each tube hole, the welding robot is controlled to perform arc welding on each tube hole in sequence. This welding sequence can be determined as needed, for example, from outer ring to inner ring or from inner ring to outer ring, clockwise or counterclockwise, radially or circumferentially.

[0170] The above-described tube-sheet workpiece welding method and system utilizes macro- and fine-vision positioning strategies. The macro-positioning vision system automatically identifies the number of tube holes and the location of their center points. It also autonomously calculates the camera's image pose for each tube hole. It then precisely positions each tube hole at the corresponding image pose, fitting the desired welding trajectory and controlling the robot to automatically weld according to the desired welding trajectory. This method automatically identifies the location of the tube hole weld and performs welding. This fully automated process requires no teaching or manual operation, eliminating at least one welding technician or robot operator per workstation, thus reducing labor costs. Furthermore, because absolute positioning is used to calculate the weld seam's position in the robot's base coordinate system, visual positioning can eliminate errors in workpiece material, assembly, and positioning, thereby ensuring weld quality. Furthermore, for tube sheets from different batches, as long as the macro-positioning camera's field of view covers the tube sheet end face, it can fully adapt to changes in the number and size of tube holes. This eliminates the need for additional setup or teaching, resulting in high flexibility and convenient, hassle-free production changeovers.

[0171] Based on the same inventive concept, embodiments of the present application also provide a tube-sheet workpiece welding device for implementing the aforementioned tube-sheet workpiece welding method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more tube-sheet workpiece welding device embodiments provided below can be found in the aforementioned limitations of the tube-sheet workpiece welding method and will not be further elaborated here.

[0172] In one embodiment, as shown in FIG8 , a tube-sheet workpiece welding device is provided, comprising: a global information acquisition module 801 , a position information determination module 802 , an information acquisition posture determination module 803 , a trajectory fitting module 904 , and a welding module 805 , wherein:

[0173] The global information acquisition module 801 is used to acquire global information collected for a tube sheet workpiece; the tube sheet workpiece includes a plurality of tube holes;

[0174] A position information determination module 802 is used to determine the position information of each pore based on the global information;

[0175] An information collection posture determination module 803 is configured to determine, for each pore, an information collection posture for the pore based on the pore's position information;

[0176] A trajectory fitting module 804 is used to fit the desired welding trajectory of the pipe hole based on the pipe hole information collected in the information collection posture;

[0177] The welding module 805 is used to control the welding robot to weld the tube sheet workpiece according to the desired welding trajectory of each tube hole.

[0178] In one embodiment, the tubesheet workpiece welding apparatus further includes a position adjustment module configured to determine the field of view of a global acquisition device and, based on the position of the tubesheet workpiece within the field of view, adjust the fixed position of the global acquisition device to ensure that the tubesheet workpiece is within the field of view of the global acquisition device. In this embodiment, the global information acquisition module 801 is specifically configured to obtain global information collected by the global acquisition device regarding the tubesheet workpiece.

[0179] In one embodiment, the global information is collected by a global acquisition device. In this embodiment, the position information determination module 802 includes a first extrinsic parameter acquisition submodule for acquiring a first extrinsic parameter from the global acquisition device; and a position information determination submodule for determining the position information of each pore in a reference coordinate system based on the global information and the first extrinsic parameter. The first extrinsic parameter represents the conversion relationship between the first coordinate system corresponding to the global acquisition device and the reference coordinate system.

[0180] In one embodiment, the global information includes three-dimensional point cloud information and two-dimensional image information. In this embodiment, the position information determination submodule is specifically configured to: determine the tube hole region in the tube sheet workpiece based on the two-dimensional image information; perform tube hole recognition on the three-dimensional point cloud information of the tube hole region to determine first position information of each tube hole in a first coordinate system; and perform coordinate transformation on each first position information based on a first extrinsic parameter to obtain position information of each tube hole in a reference coordinate system.

[0181] In one embodiment, the information acquisition posture determination module 803 includes: a coordinate system information determination submodule, which is used to determine the coordinate system information corresponding to the local acquisition device in the reference coordinate system based on the position information of the pipe hole; and an information acquisition posture determination submodule, which is used to determine the working posture matching the coordinate system information as the information acquisition posture of the local acquisition device for the pipe hole.

[0182] In one embodiment, the coordinate system information determination submodule includes: a tube hole determination unit, which is used to determine the tube hole center represented by the tube hole position information and the tube hole end face to be welded; a feature analysis unit, which is used to determine the feature point of the tube hole from the tube hole center and the tube hole end face based on the device type of the local acquisition device, and determine the feature direction pointing to the feature point; a coordinate origin determination unit, which is used to determine the recommended viewing distance of the local acquisition device as the distance between the coordinate origin corresponding to the local acquisition device in the reference coordinate system and the feature point; a coordinate axis direction determination unit, which is used to determine the feature direction as the first coordinate axis direction corresponding to the local acquisition device in the reference coordinate system; the first coordinate axis direction coincides with the detection signal emission direction of the local acquisition device.

[0183] In one embodiment, the feature analysis unit is specifically used to: when the local acquisition device is a three-dimensional camera, determine the center of the tube hole as the feature point of the tube hole; and determine the direction perpendicular to the end face of the tube hole and pointing to the feature point as the feature direction.

[0184] In one embodiment, the tube-sheet workpiece welding apparatus further includes an end-face shape determination unit configured to determine the end-face shape of each tube hole based on global information. In this embodiment, the trajectory fitting module 804 is specifically configured to: obtain tube hole point cloud information acquired by the local acquisition device in the information acquisition position; perform position fitting on the tube hole point cloud information based on a fitting algorithm that matches the tube hole end-face shape, and obtain a desired welding trajectory for the tube hole.

[0185] In one embodiment, the feature analysis unit is specifically used to: when the local acquisition device is a line structured light acquisition device, determine multiple edge points on the end face of the tube hole as multiple feature points; determine the feature direction corresponding to each feature point respectively; the feature line where the feature direction corresponding to any feature point is located is perpendicular to the tangent of the tube hole edge passing through the feature point, and the angle between the line connecting the feature point and the center of the tube hole and the feature line is an acute angle.

[0186] In one embodiment, the tube-sheet workpiece welding apparatus further includes an end face shape determination unit configured to determine the end face shape of each tube hole based on global information. In this embodiment, the feature analysis unit is specifically configured to determine the number of feature points that match the end face shape; and based on the number of feature points, determine multiple feature points evenly distributed on the tube hole end face.

[0187] In one embodiment, the reference coordinate system is the robot base coordinate system of the welding robot, and the local acquisition device is fixed to the tool flange at the end of the welding robot. In this embodiment, the information acquisition posture determination submodule is specifically used to: obtain a second external parameter of the local acquisition device; the second external parameter is used to represent the conversion relationship between the second coordinate system corresponding to the local acquisition device and the robot tool coordinate system of the welding robot; determine a third external parameter of the local acquisition device based on the coordinate system information; the third external parameter is used to represent the conversion relationship between the second coordinate system and the robot base coordinate system; combine the second external parameter and the third external parameter to determine the robot coordinate conversion relationship between the robot tool coordinate system and the robot base coordinate system; and determine the working posture represented by the robot coordinate conversion relationship as the information acquisition posture of the local acquisition device for the pipe hole.

[0188] In one embodiment, the welding module 805 is specifically used to: obtain the welding sequence of each pipe hole; and control the welding robot to weld each pipe hole in sequence according to the desired welding trajectory of each pipe hole.

[0189] Each module in the aforementioned tube-to-sheet workpiece welding apparatus may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor within a computer device in the form of hardware, or may be stored in a computer device memory in the form of software, allowing the processor to call and execute the corresponding operations of each module.

[0190] In one embodiment, a computer device is provided, which may be a terminal. Its internal structure diagram may be as shown in FIG9 . The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer-readable instructions. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the non-volatile storage medium. The input / output interface of the computer device is configured to exchange information between the processor and an external device. The communication interface of the computer device is configured to communicate with an external terminal via wired or wireless communication, where the wireless communication may be achieved via Wi-Fi, a mobile cellular network, NFC (near-field communication), or other technologies. When executed by the processor, the computer-readable instructions implement a tube-to-sheet workpiece welding method or an object classification method. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0191] Those skilled in the art will understand that the structure shown in Figure 9 is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0192] In one embodiment, a computer device is further provided, including a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps in the above-mentioned method embodiments when executing the computer-readable instructions.

[0193] In one embodiment, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0194] In one embodiment, a computer program product is provided, comprising computer-readable instructions, which implement the steps in the above-mentioned method embodiments when executed by a processor.

[0195] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a non-volatile computer-readable storage medium. When the computer-readable instructions are executed, they can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0196] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0197] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A welding method for a tube sheet workpiece, characterized in that, the method includes: obtaining global information collected for the tube sheet workpiece; the tube sheet workpiece includes a plurality of tube holes; based on the global information, determining the position information of each of the tube holes; for each of the tube holes, based on the position information of the tube hole, determining the information acquisition pose for the tube hole; based on the tube hole information collected for the tube hole in the information acquisition pose, fitting to obtain the desired welding trajectory of the tube hole; and controlling a welding robot to weld the tube sheet workpiece according to the desired welding trajectory of each of the tube holes.

2. The method according to claim 1, characterized in that, the method further includes: obtaining the field of view range of the global acquisition device; based on the position of the tube sheet workpiece within the field of view range, adjusting the fixed position of the global acquisition device so that the tube sheet workpiece is included within the field of view of the global acquisition device; and the obtaining the global information collected for the tube sheet workpiece includes: obtaining the global information collected by the global acquisition device for the tube sheet workpiece.

3. The method according to claim 1, characterized in that, the global information is collected by a global acquisition device; the based on the global information, determining the position information of each of the tube holes includes: obtaining the first extrinsic parameters of the global acquisition device; the first extrinsic parameters are used to characterize the conversion relationship between the first coordinate system corresponding to the global acquisition device and the reference coordinate system; and based on the global information and the first extrinsic parameters, determining the position information of each of the tube holes in the reference coordinate system.

4. The method according to claim 3, characterized in that, the global information includes three-dimensional point cloud information and two-dimensional image information; the based on the global information and the first extrinsic parameters, determining the position information of each of the tube holes in the reference coordinate system includes: based on the two-dimensional image information, determining the tube hole regions in the tube sheet workpiece; performing tube hole recognition on the three-dimensional point cloud information of the tube hole regions to determine the first position information of each of the tube holes in the first coordinate system; and performing coordinate transformation on each of the first position information based on the first extrinsic parameters to obtain the position information of each of the tube holes in the reference coordinate system.

5. The method according to claim 3, characterized in that, the based on the position information of the tube hole, determining the information acquisition pose for the tube hole includes: based on the position information of the tube hole, determining the coordinate system information corresponding to the local acquisition device in the reference coordinate system; and determining the working pose matching the coordinate system information as the information acquisition pose of the local acquisition device for the tube hole.

6. The method according to claim 5, characterized in that, the based on the position information of the tube hole, determining the coordinate system information corresponding to the local acquisition device in the reference coordinate system includes: determining the center of the tube hole represented by the position information of the tube hole and the end face of the tube hole to be welded; Based on the device type of the local acquisition device, determine the feature points of the pipe hole from the center of the pipe hole and the end face of the pipe hole, and determine the feature direction pointing to the feature points; Determine the recommended viewing distance of the local acquisition device as the distance between the coordinate origin corresponding to the local acquisition device in the reference coordinate system and the feature points; and Determine the feature direction as the direction of the first coordinate axis corresponding to the local acquisition device in the reference coordinate system; the direction of the first coordinate axis coincides with the detection signal emission direction of the local acquisition device.

7. The method according to claim 6, wherein, The step of determining the feature points of the pipe hole from the center of the pipe hole and the end face of the pipe hole based on the device type of the local acquisition device and determining the feature direction pointing to the feature points includes: When the local acquisition device is a 3D camera, determine the center of the pipe hole as the feature point of the pipe hole; and Determine the direction perpendicular to the end face of the pipe hole and pointing to the feature point as the feature direction.

8. The method according to claim 7, wherein, The method further includes: Based on the global information, determine the end face shape of each pipe hole; The step of fitting the expected welding trajectory of the pipe hole based on the pipe hole information collected for the pipe hole in the information acquisition pose includes: Obtain the pipe hole point cloud information collected for the pipe hole by the local acquisition device in the information acquisition pose; and Based on a fitting algorithm matching the end face shape of the pipe hole, perform position fitting on the pipe hole point cloud information to obtain the expected welding trajectory of the pipe hole.

9. The method according to claim 6, wherein, The step of determining the feature points of the pipe hole from the center of the pipe hole and the end face of the pipe hole based on the device type of the local acquisition device and determining the feature direction pointing to the feature points includes: When the local acquisition device is a line structured light acquisition device, determine multiple edge points on the end face of the pipe hole as multiple feature points; and Respectively determine the feature direction corresponding to each feature point; the feature line where the feature direction corresponding to any one feature point is located is perpendicular to the tangent of the pipe hole edge passing through the feature point, and the angle between the line connecting the feature point and the center of the pipe hole and the feature line is an acute angle.

10. The method according to claim 9, wherein, The method further includes: Based on the global information, determine the end face shape of each pipe hole; The step of determining multiple edge points on the end face of the pipe hole as multiple feature points includes: Determine the number of feature points matching the end face shape; and Based on the number of feature points, determine multiple evenly distributed feature points on the end face of the pipe hole.

11. The method according to claim 5, wherein, The reference coordinate system is the robot base coordinate system of the welding robot, and the local acquisition device is fixed to the tool flange at the end of the welding robot; The step of determining the information acquisition pose of the local acquisition device for the pipe hole by determining the working pose matching the coordinate system information includes: Obtain the second extrinsic parameters of the local acquisition device; the second extrinsic parameters are used to characterize the conversion relationship between the second coordinate system corresponding to the local acquisition device and the robot tool coordinate system of the welding robot; Determine the third extrinsic parameters of the local acquisition device based on the coordinate system information; the third extrinsic parameters are used to characterize the conversion relationship between the second coordinate system and the robot base coordinate system; Combine the second extrinsic parameters and the third extrinsic parameters to determine the robot coordinate conversion relationship between the robot tool coordinate system and the robot base coordinate system; and Determine the working pose represented by the robot coordinate conversion relationship as the information acquisition pose of the local acquisition device for the tube hole.

12. The method according to any one of claims 1 to 11, characterized in that, The controlling the welding robot to weld the tube sheet workpiece according to the expected welding trajectory of each tube hole includes: Obtain the welding sequence of each tube hole; and According to the expected welding trajectory of each tube hole, control the welding robot to weld each tube hole in sequence according to the welding sequence.

13. A tube sheet workpiece welding device, characterized in that, The device includes: A global information acquisition module, configured to acquire global information collected for a tube sheet workpiece; the tube sheet workpiece includes a plurality of tube holes; A position information determination module, configured to determine the position information of each tube hole based on the global information; An information acquisition pose determination module, configured to, for each tube hole, determine the information acquisition pose for the tube hole based on the position information of the tube hole; A trajectory fitting module, configured to fit the expected welding trajectory of the tube hole based on the tube hole information collected for the tube hole in the information acquisition pose; and A welding module, configured to control the welding robot to weld the tube sheet workpiece according to the expected welding trajectory of each tube hole.

14. The device according to claim 13, characterized in that, The global information is collected by a global acquisition device; the position information determination module includes: A first extrinsic parameter acquisition sub-module, configured to acquire the first extrinsic parameters of the global acquisition device; the first extrinsic parameters are used to characterize the conversion relationship between the first coordinate system corresponding to the global acquisition device and the reference coordinate system; and A position information determination sub-module, configured to determine the position information of each tube hole in the reference coordinate system based on the global information and the first extrinsic parameters.

15. The device according to claim 14, characterized in that, The information acquisition pose determination module includes: A coordinate system information determination sub-module, configured to determine the coordinate system information corresponding to the local acquisition device in the reference coordinate system based on the position information of the tube hole; and An information acquisition pose determination sub-module, configured to determine the working pose matching the coordinate system information as the information acquisition pose of the local acquisition device for the tube hole.

16. The device according to claim 15, characterized in that, The coordinate system information determination sub-module includes: A tube hole determination unit for determining the center of the tube hole characterized by the position information of the tube hole and the end face of the tube hole to be welded; A feature analysis unit for determining feature points of the tube hole from the center of the tube hole and the end face of the tube hole based on the device type of the local acquisition device, and determining a feature direction pointing to the feature points; A coordinate origin determination unit for determining the recommended viewing distance of the local acquisition device as the distance between the coordinate origin corresponding to the local acquisition device in the reference coordinate system and the feature points; and An axis direction determination unit for determining the feature direction as the first axis direction corresponding to the local acquisition device in the reference coordinate system; the first axis direction coincides with the detection signal emission direction of the local acquisition device.

17. A tube-sheet workpiece welding system Characterized in that The system includes a controller, and a global acquisition device, a local acquisition device and a welding robot connected to the controller; The global acquisition device is used for globally acquiring information of the tube-sheet workpiece to obtain the global information of the tube-sheet workpiece; the tube-sheet workpiece includes a plurality of tube holes; The local acquisition device is used for respectively acquiring information for each of the tube holes to obtain the tube hole information of each tube hole; The welding robot is used for welding the tube-sheet workpiece; and The controller is used to implement the method according to any one of claims 1 to 12.

18. A computer device comprising a memory and a processor, the memory storing computer-readable instructions Characterized in that When the processor executes the computer-readable instructions, the steps of the method according to any one of claims 1 to 12 are implemented.

19. A computer-readable storage medium having computer-readable instructions stored thereon Characterized in that When the computer-readable instructions are executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

20. A computer program product comprising computer-readable instructions Characterized in that When the computer-readable instructions are executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images