Method and system for controlling automatic welding of metal products

Abstract

The application relates to a metal product automatic welding equipment control method and system. For a metal product to be welded, a target image is collected, a defect detection model is used to detect defects of the metal product to be welded according to the target image, and then welding control of the automatic welding equipment is performed according to the defect information. Since the control of the automatic welding equipment considers the defect problem of the metal product, the problem of poor welding effect of the metal product with defects can be solved. Moreover, since the pre-trained defect detection model is trained according to real images and simulation images, the model has strong generalization ability and can be applied to defect detection in various scenes, improving the accuracy of defect detection. Therefore, the technical scheme controls the automatic welding equipment to weld the metal product through defect information with high accuracy, improving the welding effect of the metal product.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automation, in particular, to a metal product automatic welding equipment control method and system. BACKGROUND

[0002] The automatic welding equipment is a device that realizes the welding process through automation technology, which is widely used in industrial production, can improve the welding efficiency and quality, and reduce manual intervention. The automatic welding equipment is widely used in automobile manufacturing, mechanical processing, petrochemical industry, aerospace, electronic and electrical industries, and can realize the welding of metal products involved in these industries.

[0003] At present, the automatic welding equipment for metal products usually adopts a fixed control strategy, for example: the welding points of the automatic welding equipment are planned in advance, and the automatic welding equipment is controlled to weld according to the welding points.

[0004] This control method can be applied to the scene of metal products without defects, but cannot be applied to the scene of metal products with defects, so there is a problem of poor welding effect for metal products with defects. SUMMARY

[0005] The purpose of the present application is to provide a metal product automatic welding equipment control method and system, which controls the automatic welding equipment to weld the metal product through high-accuracy defect information, and improves the welding effect of the metal product.

[0006] In a first aspect, the present application provides a metal product automatic welding equipment control method, comprising: acquiring a target image collected for a metal product to be welded; performing defect detection on the metal product to be welded according to the target image through a pre-trained defect detection model, to obtain defect information of the metal product to be welded, wherein the training data of the pre-trained defect detection model includes real images and simulation images, the real images are images of real metal products with real defects, and the simulation images are images of real metal products with simulation defects; and controlling an automatic welding equipment to weld the metal product to be welded according to at least the defect information.

[0007] Optionally, the metal product automatic welding equipment control method further comprises: obtaining a first real image obtained by image acquisition on a real metal product with a real defect; obtaining a second real image obtained by image acquisition on a real metal product without a real defect; determining simulation defect information for representing generation requirements of a simulation defect; generating a simulation image of a real metal product with a simulation defect through a pre-trained image generation model according to the first real image, the second real image, and the simulation defect information; and training a defect detection model to be trained according to the first real image and the simulation image to obtain a pre-trained defect detection model.

[0008] Optionally, the simulation defect information for representing generation requirements of a simulation defect comprises: obtaining a description text corresponding to the real defect; obtaining a preset description text corresponding to the simulation defect; generating a description text corresponding to the simulation defect according to the description text corresponding to the real defect and the preset description text; obtaining a shape mask corresponding to the simulation defect; obtaining a label box mask corresponding to the simulation defect; and determining the description text corresponding to the simulation defect, the shape mask, and the label box mask as the simulation defect information.

[0009] Optionally, the simulation defect information comprises a label box mask corresponding to the simulation defect, and the simulation image of the real metal product with the simulation defect is generated through the pre-trained image generation model according to the first real image, the second real image, and the simulation defect information, which comprises: covering the real defect in the first real image according to the label box mask to obtain a first covered image; covering a region to be generated with the simulation defect in the second real image according to the label box mask to obtain a second covered image; determining a target covered image according to the first covered image and the second covered image; and generating the simulation image of the real metal product with the simulation defect through the pre-trained image generation model according to the target covered image and the simulation defect information.

[0010] Optionally, the simulation defect information further comprises a description text and a shape mask corresponding to the simulation defect, the pre-trained image generation model comprises an image encoder, a shape encoder, a text encoder, an autoregressive model and an image decoder, and the generating of the simulation image of the real metal product with the simulation defect by the pre-trained image generation model according to the target cover image and the simulation defect information comprises: generating an initial image vector by the image encoder; encoding the target cover image by the image encoder to obtain a cover image condition vector; encoding the description text by the text encoder to obtain a description text condition vector; encoding the shape mask by the shape encoder to obtain a shape condition vector; generating a target image vector according to the cover image condition vector, the description text condition vector, the shape condition vector and the initial image vector by the autoregressive model; and decoding the target image vector by the image decoder to obtain the simulation image of the real metal product with the simulation defect.

[0011] Optionally, the metal product automatic welding equipment control method further comprises: obtaining an image sample, the image sample being an image of a real metal product sample with a real defect sample; determining real defect sample information according to the image sample, the real defect sample information comprising a label box mask sample corresponding to the real defect sample; covering the real defect sample in the image sample according to the label box mask sample to obtain a cover image sample; and training an image generation model to be trained according to the real defect sample information, the cover image sample and the image sample to obtain a pre-trained image generation model.

[0012] Optionally, the defect information comprises a defect position, a defect size and a defect type, and the controlling of the automatic welding equipment to weld the metal product to be welded according to at least the defect information comprises: obtaining a preset control parameter used for controlling the automatic welding equipment to weld the metal product to be welded; determining whether the metal product to be welded is a qualified metal product according to the defect size and the defect type; determining a target control parameter according to the defect position, the defect size, the defect type and the preset control parameter in a case where the metal product to be welded is a qualified metal product; determining a target control parameter according to the defect position, the defect size and the defect type in a case where the metal product to be welded is an unqualified metal product; and controlling the automatic welding equipment to weld the metal product to be welded according to the target control parameter.

[0013] Optionally, the automatic welding device is a laser welding machine, the preset control parameters include the plurality of laser welding points arranged in time sequence, the plurality of laser welding points respectively corresponding to the residence time and the plurality of laser welding points respectively corresponding to the laser parameters, and the determining the target control parameters according to the defect position, the defect size, the defect type and the preset control parameters includes: determining whether there is a target laser welding point matching the defect position in the plurality of laser welding points; in the case that there is a target laser welding point matching the defect position in the plurality of laser welding points, adjusting the residence time and the laser parameters corresponding to the target laser welding point according to the defect size and the defect type to obtain the target control parameters; in the case that there is no target laser welding point matching the defect position in the plurality of laser welding points, generating a new laser welding point according to the defect position, determining the residence time and the laser parameters corresponding to the new laser welding point according to the defect size and the defect type, and adjusting the preset control parameters according to the new laser welding point, the residence time and the laser parameters corresponding to the new laser welding point to obtain the target control parameters.

[0014] Optionally, the determining the target control parameters according to the defect position, the defect size and the defect type includes: generating a target laser welding point according to the defect position; determining the residence time and the laser parameters corresponding to the target laser welding point according to the defect size and the defect type; and determining the target control parameters according to the target laser welding point, the residence time and the laser parameters corresponding to the target laser welding point.

[0015] In a second aspect, the present application provides a metal product automatic welding device control system, comprising: an image acquisition device configured to acquire a target image of a metal product to be welded; an automatic welding device configured to weld the metal product to be welded; and a control device configured to execute the metal product automatic welding device control method according to the first aspect of the present application.

[0016] According to the above technical solution, a target image is acquired for a metal product to be welded, a defect detection model is used to detect defects of the metal product to be welded according to the target image, and then the automatic welding device is controlled according to the defect information. Since the control of the automatic welding device takes into account the defect problem of the metal product, the problem of poor welding effect of the metal product with defects can be solved; and since the pre-trained defect detection model is trained according to real images and simulated images, the generalization ability of the model is strong, and the model can be applied to defect detection in various scenarios to improve the accuracy of defect detection. Thus, the technical solution controls the automatic welding device to weld the metal product through defect information with high accuracy, thereby improving the welding effect of the metal product.

[0017] Other features and advantages of the present application will be made clear in the following detailed description of the application. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings:

[0019] Figure 1 is a block diagram of a metal product automatic welding equipment control system according to an exemplary embodiment.

[0020] Figure 2 is a flowchart of a metal product automatic welding equipment control method according to an exemplary embodiment.

[0021] Figure 3A is an example diagram of a bounding box mask according to an exemplary embodiment.

[0022] Figure 3B is an example diagram of a shape mask according to an exemplary embodiment.

[0023] Figure 3C is an example diagram of a masked image according to an exemplary embodiment.

[0024] Figure 4 is a block diagram of an image generation model according to an exemplary embodiment.

[0025] Figure 5 is a block diagram of a metal product automatic welding equipment control device according to an exemplary embodiment.

[0026] Figure 6 is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0027] The specific embodiments of the present application described herein are illustrative of specific embodiments which are intended to provide examples of the preferred embodiments of the present application and numerous modifications thereof. Thus, it will be apparent to those skilled in the art that changes in form and detail can be made to the specific embodiments described without departing from the spirit and scope of the application.

[0028] Automatic welding equipment is a device that realizes welding process through automation technology, which is widely used in industrial production, can improve welding efficiency and quality, and reduce manual intervention. Automatic welding equipment is widely used in automobile manufacturing, mechanical processing, petroleum chemical industry, aerospace, electronic and electrical industries, and can realize the welding of metal products involved in these industries.

[0029] Currently, automatic welding equipment used for metal products typically employs fixed control strategies, such as pre-planning the welding points of the automatic welding equipment and controlling the automatic welding equipment to weld according to the welding points.

[0030] This control method is applicable to scenarios where the metal products are defect-free, but not to scenarios where the metal products are defective. As a result, there is a problem of poor welding effect for defective metal products.

[0031] Based on this, the present application provides a technical solution that involves acquiring a target image of a metal product to be welded, using a defect detection model to perform defect detection on the metal product to be welded based on the target image, and then controlling the welding of an automatic welding equipment based on the defect information.

[0032] Because the control of automatic welding equipment takes into account the defects of metal products, it can solve the problem of poor welding effect for metal products with defects; and because the pre-trained defect detection model is trained on real images and simulated images, the model has strong generalization ability and can be applied to defect detection in various scenarios, thereby improving the accuracy of defect detection.

[0033] Therefore, this technical solution uses highly accurate defect information to control automatic welding equipment to weld metal products, thereby improving the welding effect of metal products.

[0034] In the embodiments of this application, the automatic welding equipment may be a laser welding equipment or a welding equipment of the same type as a laser welding equipment.

[0035] Figure 1 This is a block diagram illustrating an automatic metal welding equipment control system according to an exemplary embodiment, such as... Figure 1 As shown, the system includes: image acquisition equipment, automatic welding equipment, and control equipment.

[0036] The image acquisition device can be set up at a corresponding position on the welding process table to acquire images of the metal parts to be welded, thereby obtaining the target image. This image acquisition device can be an industrial camera, an infrared sensor camera, etc.

[0037] Automatic welding equipment can be set at a corresponding position on the welding process table to weld metal items to be welded on the welding process table.

[0038] For example, a laser welding device uses a high-energy density laser beam to weld metal materials, with the advantages of fast welding speed, high weld quality, small heat-affected zone, and non-contact processing. Its working principle includes: the laser welding device generates a high-energy density laser beam through a laser, which is focused on the metal surface, causing the metal to absorb the laser energy and rapidly heat up and melt, forming a molten metal pool, which then rapidly cools and solidifies, forming a weld. Depending on the laser power density, laser welding can be divided into heat conduction welding and laser deep penetration welding. Heat conduction welding is suitable for thin plate welding, while laser deep penetration welding is suitable for thick plate welding.

[0039] The control device is connected to the image acquisition device and the automatic welding device, for example, through Internet of Things communication connection. The control device can be a host computer.

[0040] The control device can obtain images from the image acquisition device and optimize the control of the automatic welding device based on the images collected by the image acquisition device.

[0041] Figure 2 is a flow chart of a metal product automatic welding device control method according to an exemplary embodiment, which includes the following steps:

[0042] Step S21, obtaining a target image collected for the metal product to be welded.

[0043] Step S22, detecting defects in the metal product to be welded based on the target image through a pre-trained defect detection model, and obtaining defect information of the metal product to be welded, wherein the training data of the pre-trained defect detection model includes real images and simulated images, the real images are images of real metal products with real defects, and the simulated images are images of real metal products with simulated defects.

[0044] Step S23, controlling the automatic welding device to weld the metal product to be welded based on at least the defect information.

[0045] Wherein, the metal product to be welded can be the metal product that needs to be welded at present. In the automatic industrial process, when the metal product reaches the designated welding position, the image acquisition device can collect images of the metal product to obtain the target image.

[0046] The metal product to be welded may or may not have defects. If there are defects, it will affect the welding strategy of the automatic welding device. If there are no defects, it will not affect the welding strategy of the automatic welding device, so defect detection can be performed based on the target image.

[0047] In step S22, the target image is input into the pre-trained defect detection model to obtain the defect information output by the defect detection model.

[0048] In some embodiments, the pre-trained defect detection model can be a deep convolutional network model, and by performing defect recognition on the image, corresponding defect information such as defect type, defect size, and defect shape can be obtained.

[0049] In some embodiments, the training data of the pre-trained defect detection model can include real images and simulation images, the real images being images of real metal products with real defects, and the simulation images being images of real metal products with simulation defects.

[0050] It can be understood that for some real defects, it is difficult to collect them, for example, some defects located at special positions of the metal product, some defects with special shapes, and some smaller defects. Since these defects occur less frequently, it is difficult to collect corresponding real images. However, in order to improve the detection accuracy of the model, images corresponding to these defect conditions are also needed. Therefore, using simulation images corresponding to simulation defects can make up for the lack of corresponding metal product images for these special defect conditions.

[0051] Furthermore, by combining real images and simulation images to train the defect detection model, the robustness of the defect detection model can be improved, and the detection accuracy can be improved.

[0052] As an optional implementation, the training process of the pre-trained defect model includes: obtaining a first real image obtained by image collection on a real metal product with a real defect; obtaining a second real image obtained by image collection on a real metal product without a real defect; determining simulation defect information for representing generation requirements of simulation defects; generating a simulation image of a real metal product with simulation defects by a pre-trained image generation model according to the first real image, the second real image, and the simulation defect information; and training the defect detection model to be trained according to the first real image and the simulation image, to obtain the pre-trained defect detection model.

[0053] In this implementation, the defect information label can be set for the real defect in the first real image, and the defect information label can be set for the simulation defect in the simulation image according to the simulation defect information. Furthermore, based on the first real image and the simulation image with the corresponding labels, the defect detection model to be trained is trained to obtain the pre-trained defect detection model.

[0054] In some embodiments, the simulation image can be generated by a pre-trained image generation model according to the first real image, the second real image, and the simulation defect information for representing the generation requirements of the simulation defects.

[0055] As an optional implementation, the simulation defect information used to characterize the generation requirement of the simulation defect comprises: obtaining a description text corresponding to the real defect; obtaining a preset description text corresponding to the simulation defect; generating a description text corresponding to the simulation defect according to the description text corresponding to the real defect and the preset description text; obtaining a shape mask corresponding to the simulation defect; obtaining a bounding box mask corresponding to the simulation defect; and determining the description text corresponding to the simulation defect, the shape mask and the bounding box mask as the simulation defect information.

[0056] In some embodiments, the description text corresponding to the real defect can be used to describe the type of the real defect, for example: crack, air hole or un-melted metal material, etc.

[0057] In some embodiments, the description text corresponding to the real defect can be determined by a user, or can be determined by detecting the real image through a large language model; or can be determined by a multi-modal model, etc., which is not limited herein.

[0058] In some embodiments, the preset description text corresponding to the simulation defect can be a description text uploaded by a user and related to the simulation defect required, which can also be used to describe the type of the simulation defect required.

[0059] In some embodiments, the description text corresponding to the simulation defect can be generated according to the description text corresponding to the real defect and the preset description text.

[0060] For example, in the preset description text, the texts existing in the description text corresponding to the real defect can be deleted, or can be expanded, adjusted, etc., to obtain the description text corresponding to the simulation defect. In addition, the description text corresponding to the simulation defect is mainly based on the preset description text, but the repeated description texts involved in the description text corresponding to the real defect need to be removed, so that the description text corresponding to the simulation defect can make up for the deficiency of the real defect.

[0061] In some embodiments, the shape mask corresponding to the simulation defect can be a pre-configured shape mask, or can be a real shape mask extracted based on other images.

[0062] In some embodiments, the bounding box mask corresponding to the simulation defect can be generated based on the shape mask corresponding to the simulation defect, for example: taking the bounding box mask of the shape mask as the bounding box mask.

[0063] Further, the description text corresponding to the simulation defect, the shape mask and the bounding box mask can be determined as the simulation defect information.

[0064] In some embodiments, generating, by the pre-trained image generation model, the simulated image of the real metal product with the simulated defect according to the first real image, the second real image and the simulated defect information can comprise: covering the real defect in the first real image according to the label box mask to obtain a first covered image; covering the region in the second real image where the simulated defect is to be generated according to the label box mask to obtain a second covered image; determining a target covered image according to the first covered image and the second covered image; and generating, by the pre-trained image generation model, the simulated image of the real metal product with the simulated defect according to the target covered image and the simulated defect information.

[0065] Figure 3A is an example diagram of a label box mask according to an example embodiment, as shown in Figure 3A The label box mask can be used to label the region of the defect in the image.

[0066] Figure 3B is an example diagram of a shape mask according to an example embodiment, as shown in Figure 3B The shape mask can be used to indicate the shape of the defect in the image.

[0067] Figure 3C is an example diagram of a covered image according to an example embodiment, as shown in Figure 3C In the covered image, the region where the simulated defect is to be generated is covered, and the other regions are retained.

[0068] Then, for the first real image, the first covered image can be obtained by covering the region where the real defect is located. And for the second real image, the second covered image can be obtained by covering the region where the simulated defect is to be generated.

[0069] Further, the image similarity between the first covered image and the second covered image can be determined, and the covered image with an image similarity lower than a preset similarity is determined as the target covered image. Therefore, the number of target covered images is greater than or equal to 1.

[0070] Further, the simulated defect information and the target covered image are input into the pre-trained image generation model to obtain the simulated image output by the pre-trained image generation model, wherein the simulated image includes the simulated defect.

[0071] Figure 4 is a block diagram of an image generation model according to an example embodiment, as shown in Figure 4 The pre-trained image generation model comprises an image encoder, a shape encoder, a text encoder, an autoregressive model and an image decoder.

[0072] The image encoder and the image decoder can also be implemented by an integrated network structure, for example, an adversarial network model. The image encoder can be used for image encoding, and the image decoder can be used for decoding to obtain an image.

[0073] The shape encoder can be used for encoding a shape mask, the text encoder can be used for encoding text, and the autoregressive model can generate an image through autoregressive prediction.

[0074] Therefore, as an optional implementation, a simulation image of a real metal product with a simulated defect is generated by a pre-trained image generation model according to the target cover image and the simulated defect information, including: generating an initial image vector by an image encoder; encoding the target cover image by the image encoder to obtain a cover image condition vector; encoding the description text by a text encoder to obtain a description text condition vector; encoding the shape mask by a shape encoder to obtain a shape condition vector; generating a target image vector according to the cover image condition vector, the description text condition vector, the shape condition vector and the initial image vector by an autoregressive model; and decoding the target image vector by an image decoder to obtain the simulation image of the real metal product with the simulated defect.

[0075] In some embodiments, the image encoder can generate the initial image vector through normal distribution random initialization.

[0076] In some embodiments, the image encoder can encode the target cover image into an image vector of the same size as the initial image vector to obtain the cover image condition vector, which can be used as a control condition for subsequent image generation.

[0077] In some embodiments, the text encoder can encode the description text into a vector of the same type as the initial image vector to obtain the description text condition vector, which can be used as a control condition for subsequent image generation.

[0078] In some embodiments, the shape encoder can encode the shape mask into an image vector of the same size as the initial image vector to obtain the shape condition vector.

[0079] In some embodiments, the autoregressive model generates the target image vector through autoregressive prediction based on the cover image condition vector, the description text condition vector, the shape condition vector and the initial image vector.

[0080] In some embodiments, the autoregressive model can also be replaced by other models, for example, a variational autoencoder, an adversarial network, etc.

[0081] In some embodiments, the image decoder decodes the target image vector to obtain a simulated image of a real metal object with simulated defects.

[0082] In some embodiments, the training process of the pre-trained image generation model may include: acquiring image samples, wherein the image samples are images of real metal samples containing real defects; determining real defect sample information based on the image samples, wherein the real defect sample information includes the bounding box mask samples corresponding to the real defect samples; masking the real defect samples in the image samples based on the bounding box mask samples to obtain masked image samples; and training the image generation model to be trained based on the real defect sample information, the masked image samples, and the image samples to obtain the pre-trained image generation model.

[0083] In some embodiments, image samples can be obtained in various ways, such as user uploads, big data acquisition, etc.

[0084] In some embodiments, shape mask samples, bounding box mask samples, and text description samples of the real defect samples can be extracted from image samples to obtain the real defect sample information. The extraction methods for these samples can be referred to the methods for determining various information in the foregoing embodiments, and will not be described in detail here.

[0085] In some embodiments, the image encoder, shape encoder, text encoder, and image decoder in the image generation model to be trained can be pre-trained encoders or decoders.

[0086] For example, an image encoder can be pre-trained to encode images; a shape encoder can be pre-trained to encode shapes; a text encoder can be pre-trained to encode text; and an image decoder can be pre-trained to decode. For specific training methods for encoders and decoders, refer to mature technologies in the field.

[0087] In some embodiments, image encoders, shape encoders, and text encoders can be used to encode the corresponding information. This encoded information, along with occluded image samples, can then be used as training samples, and the image samples as labels, to train an autoregressive model, thereby enabling the autoregressive model to generate images. For example, the autoregressive model can be used to make predictions based on the training samples to obtain prediction results. The loss between the prediction results and the image sample labels can then be determined, and the autoregressive model can be trained based on this loss.

[0088] Furthermore, the pre-trained image generation model can generate simulated images of real metal products with simulated defects based on the input masked image and simulated defect information.

[0089] Further, in combination with the foregoing embodiments introducing the training of the defect detection model, the pre-trained defect detection model can output defect information based on the input target image.

[0090] As an optional implementation, the defect information includes: defect position, defect size, and defect type.

[0091] The defect position can be the position of the defect relative to the metal product, the defect size can be represented by a size parameter, and the defect type can be represented by a defect type identifier.

[0092] In step S23, at least according to the defect information, the automatic welding device is controlled to weld the metal product to be welded.

[0093] In some embodiments, in addition to combining defect information, other information that has an impact on welding can also be combined for welding control, for example: positioning information can also be combined to determine whether there is a positioning deviation, and in the case of a positioning deviation, the control scheme needs to be adjusted, etc. In the embodiments of the present application, how to perform welding control based on defect information is mainly introduced, and it is not limited to welding control only according to defect information.

[0094] As an optional implementation, step S23 includes: obtaining a preset control parameter for controlling the automatic welding device to weld the metal to be welded; determining whether the metal product to be welded is a qualified metal product according to the defect size and the defect type; in the case that the metal product to be welded is a qualified metal product, determining a target control parameter according to the defect position, the defect size, the defect type, and the preset control parameter; in the case that the metal product to be welded is an unqualified metal product, determining a target control parameter according to the defect position, the defect size, and the defect type; and controlling the automatic welding device to weld the metal product to be welded according to the target control parameter.

[0095] In this implementation, the automatic welding device is configured with a preset control parameter, and the defect size and the defect type are used to first determine whether the metal product to be welded is qualified, and different control parameter updating methods are adopted according to the determination result.

[0096] In some embodiments, in the case that the defect size is not within a preset defect size range and / or the defect type does not belong to a preset defect type, the metal product to be welded is determined to be an unqualified metal product; otherwise, the metal product to be welded is determined to be a qualified metal product.

[0097] The preset defect size range and the preset defect type can be configured according to the defects that can be tolerated by the metal product in different scenarios.

[0098] In some embodiments, in the case that the metal product to be welded is a qualified metal product, the explanation can continue to be welded based on the preset control parameters, but the welding of the defective part needs to be considered.

[0099] In the case that the metal product to be welded is an unqualified metal product, the explanation cannot continue to be welded based on the preset control parameters, and the defect can be processed first.

[0100] In some embodiments, the automatic welding device is a laser welding machine, and the preset control parameters include a plurality of laser welding points arranged in a time sequence, a plurality of laser welding point corresponding residence times, and a plurality of laser welding point corresponding laser parameters.

[0101] In some embodiments, the laser welding point can be represented by a welding point coordinate, which can be a three-dimensional coordinate.

[0102] In some embodiments, the residence time and the laser parameter can correspond to different welding effects. The laser parameter can include the emission power of the laser beam, the irradiation speed of the laser beam, etc.

[0103] Further, according to the defect position, the defect size, the defect type, and the preset control parameters, the target control parameters can be determined, which can include: determining whether there is a target laser welding point matching the defect position in the plurality of laser welding points; in the case that there is a target laser welding point matching the defect position in the plurality of laser welding points, adjusting the residence time and the laser parameter corresponding to the target laser welding point according to the defect size and the defect type to obtain the target control parameters; in the case that there is no target laser welding point matching the defect position in the plurality of laser welding points, generating a new laser welding point according to the defect position, and determining the residence time and the laser parameter corresponding to the new laser welding point according to the defect size and the defect type, and adjusting the preset control parameters according to the new laser welding point, the residence time and the laser parameter corresponding to the new laser welding point to obtain the target control parameters.

[0104] In some embodiments, the defect position detected by the model can be the position of the same coordinate system as the laser welding point through the conversion of the camera coordinate system and the world coordinate system compared to the position of the image.

[0105] In some embodiments, the target laser welding point matching the defect position can be a laser welding point with the same or similar (e.g., very close) coordinates as the defect position.

[0106] In some embodiments, adjusting the residence time and the laser parameter corresponding to the target laser welding point according to the defect size and the defect type can include: increasing the residence time according to the defect size; and adjusting the emission power and speed of the laser beam according to the defect type.

[0107] In some embodiments, the laser parameters and dwell time suitable for metal products with different defect sizes and different defect types can be tested in advance, including offline testing or simulation testing, etc. Prior data is generated according to the test, so that the corresponding adjustment scheme can be determined by querying the prior data. For example, if the prior data is queried and it is found that the current emission power is lower than the corresponding emission power, the emission power is increased.

[0108] In some embodiments, in combination with the prior data, the dwell time and laser parameters corresponding to the newly added laser welding points can also be determined.

[0109] In some embodiments, according to the defect position, defect size and defect type, the target control parameter is determined, including: generating a target laser welding point according to the defect position; determining the dwell time and laser parameters corresponding to the target laser welding point according to the defect size and defect type; and determining the target control parameter according to the target laser welding point, the dwell time and laser parameters corresponding to the target laser welding point.

[0110] In this implementation, the defect position needs to be welded, so the target laser welding point can be generated based on the defect position, and the dwell time and laser parameters corresponding to the target laser welding point are further determined to obtain the target control parameter.

[0111] Wherein, based on the defect size and the defect type, the dwell time and the laser parameters corresponding to the target laser welding point can also be determined based on the prior data, which will not be repeated here.

[0112] Further, no matter what kind of defect the metal product has, the corresponding control of the automatic welding equipment can be realized, and the welding effect of the metal product is improved.

[0113] Figure 5 is a block diagram of a metal product automatic welding equipment control device 500 according to an exemplary embodiment, as shown in Figure 5 The device comprises:

[0114] The acquisition module 501 is configured to acquire a target image collected from a metal product to be welded.

[0115] The detection module 502 is configured to perform defect detection on the metal product to be welded according to the target image by using a pre-trained defect detection model, to obtain defect information of the metal product to be welded, wherein the training data of the pre-trained defect detection model includes real images and simulation images, the real images are images of real metal products with real defects, and the simulation images are images of real metal products with simulation defects.

[0116] The control module 503 is configured to control the automatic welding device to weld the metal product to be welded according to the defect information.

[0117] With regard to the apparatus in the above-described embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0118] Figure 6 is a block diagram of an electronic device 600 according to an example embodiment. As shown, the electronic device 600 can include a processor 601, a memory 602. The electronic device 600 can also include one or more of a multimedia component 603, an input / output (I / O) interface 604, and a communication component 605. Figure 6

[0119] ​The processor 601 is configured to control overall operations of the electronic device 600 to complete all or part of the steps of the metal product automatic welding device control method described above. The memory 602 is configured to store various types of data to support operations of the electronic device 600, which can include, for example, instructions for any application or method operating on the electronic device 600, and application-related data, such as contact data, sent and received messages, pictures, audio, video, and the like. The memory 602 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The multimedia component 603 can include a screen and an audio component. The screen can be, for example, a touch screen, and the audio component is configured to output and / or input audio signals. For example, the audio component can include a microphone configured to receive external audio signals. The received audio signals can be further stored in the memory 602 or transmitted through the communication component 605. The audio component also includes at least one speaker configured to output audio signals. The I / O interface 604 provides an interface between the processor 601 and other interface modules, which can be a keyboard, a mouse, a button, and the like. The buttons can be virtual buttons or physical buttons. The communication component 605 is configured to perform wired or wireless communication between the electronic device 600 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, near field communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 605 can include a Wi-Fi module, a Bluetooth module, an NFC module.

[0120] In an exemplary embodiment, the electronic device 600 can be implemented by one or more Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor or other electronic elements for executing the metal product automatic welding device control method described above.

[0121] In another exemplary embodiment, a computer readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the metal product automatic welding device control method described above. For example, the computer readable storage medium can be the memory 602 described above including program instructions, which can be executed by the processor 601 of the electronic device 600 to complete the metal product automatic welding device control method described above.

[0122] In another exemplary embodiment, a computer program product is also provided, which contains a computer program capable of being executed by a processor, which, when executed by the processor, implement the steps of the metal product automatic welding device control method described above.

[0123] The preferred embodiments of the present application are described in detail above with reference to the accompanying drawings, but the present application is not limited to the specific details in the above-described embodiments. Within the technical concept of the present application, various simple modifications can be made to the technical solution of the present application, and these simple modifications all belong to the protection scope of the present application.

[0124] In addition, it should be noted that each specific technical feature described in the above-described specific embodiments can be combined in any appropriate manner without contradiction, and in order to avoid unnecessary repetition, various possible combinations are not described again in the present application.

[0125] In addition, any combination of various different embodiments of the present application can also be made, as long as it does not deviate from the idea of the present application, and it should also be considered as disclosed content of the present application.

Claims

1. A control method for an automatic metal welding equipment, characterized in that, include: Acquire target images of the metal objects to be welded; Using a pre-trained defect detection model, defects are detected in the metal product to be welded based on the target image to obtain defect information of the metal product to be welded. The training data of the pre-trained defect detection model includes real images and simulated images. The real images are images of real metal products with real defects, and the simulated images are images of real metal products with simulated defects. Based at least on the defect information, control the automatic welding equipment to weld the metal item to be welded; The control method for the automatic metal welding equipment also includes: Acquire the first real image by image acquisition of a real metal product with real defects; Acquire a second real image by image acquisition of a real metal product that does not have real defects; Determine the simulation defect information used to characterize the generation requirements of the simulation defects; Using a pre-trained image generation model, a simulated image of a real metal product with simulated defects is generated based on the first real image, the second real image, and the simulated defect information. Based on the first real image and the simulated image, the defect detection model to be trained is trained to obtain a pre-trained defect detection model. The simulated defect information includes: the bounding box mask corresponding to the simulated defect; and the generation of a simulated image of a real metal object with simulated defects by a pre-trained image generation model based on the first real image, the second real image, and the simulated defect information, including: Based on the labeled box mask, the real defects in the first real image are masked to obtain the first masked image; Based on the labeled box mask, the region of the simulated defect to be generated in the second real image is masked to obtain the second masked image; The target masking image is determined based on the first masking image and the second masking image; Using a pre-trained image generation model, a simulated image of a real metal product with simulated defects is generated based on the target occlusion image and the simulated defect information. The simulated defect information further includes: descriptive text and shape mask corresponding to the simulated defect; the pre-trained image generation model includes: an image encoder, a shape encoder, a text encoder, an autoregressive model, and an image decoder; the generation of a simulated image of a real metal object with simulated defects using the pre-trained image generation model based on the target occlusion image and the simulated defect information includes: The image encoder generates an initial image vector. The target occlusion image is encoded by the image encoder to obtain an occlusion image condition vector; The text encoder is used to encode the descriptive text to obtain a descriptive text condition vector; The shape encoder encodes the shape mask to obtain a shape condition vector; The target image vector is generated using the autoregressive model based on the occlusion image condition vector, the description text condition vector, the shape condition vector, and the initial image vector. The target image vector is decoded by the image decoder to obtain a simulated image of a real metal product with simulated defects.

2. The control method for automatic metal welding equipment according to claim 1, characterized in that, The determination of the simulation defect information used to characterize the generation requirements of the simulation defect includes: Obtain the description text corresponding to the actual defect; Obtain the preset description text corresponding to the simulated defect; Based on the description text corresponding to the real defect and the preset description text, generate the description text corresponding to the simulated defect; Obtain the shape mask corresponding to the simulated defect; Obtain the bounding box mask corresponding to the simulated defect; The description text corresponding to the simulated defect, the shape mask, and the annotation box mask are determined as the simulated defect information.

3. The control method for automatic metal welding equipment according to claim 1, characterized in that, The control method for the automatic metal welding equipment also includes: Acquire image samples, wherein the image samples are images of real metal samples containing real defects; Based on the image samples, determine the real defect sample information, which includes the annotation box mask sample corresponding to the real defect sample; Based on the labeled bounding box mask sample, the real defect sample in the image sample is masked to obtain the masked image sample; Based on the real defect sample information, the occluded image sample, and the image sample, the image generation model to be trained is trained to obtain a pre-trained image generation model.

4. The control method for automatic metal welding equipment according to claim 1, characterized in that, The defect information includes: defect location, defect size, and defect type. Controlling the automatic welding equipment to weld the metal object to be welded, based at least on the defect information, includes: Obtain preset control parameters for controlling the automatic welding equipment to weld the metal to be welded; Based on the size and type of the defect, determine whether the metal product to be welded is a qualified metal product; If the metal product to be welded is a qualified metal product, the target control parameters are determined based on the defect location, the defect size, the defect type, and the preset control parameters. When the metal product to be welded is a defective metal product, the target control parameters are determined based on the location, size, and type of the defect. Based on the target control parameters, the automatic welding equipment is controlled to weld the metal product to be welded.

5. The control method for automatic metal welding equipment according to claim 4, characterized in that, The automatic welding equipment is a laser welding machine. The preset control parameters include multiple laser welding points arranged in a time sequence, the dwell time corresponding to each of the multiple laser welding points, and the laser parameters corresponding to each of the multiple laser welding points. Determining the target control parameters based on the defect location, the defect size, the defect type, and the preset control parameters includes: Determine whether there is a target laser welding point among the plurality of laser welding points that matches the defect location; If a target laser welding point matching the defect location exists among the plurality of laser welding points, the dwell time and laser parameters corresponding to the target laser welding point are adjusted according to the defect size and the defect type to obtain target control parameters; If no target laser welding point matching the defect location exists among the multiple laser welding points, a new laser welding point is generated based on the defect location. The dwell time and laser parameters corresponding to the new laser welding point are determined based on the defect size and defect type. The preset control parameters are adjusted based on the new laser welding point, the dwell time corresponding to the new laser welding point, and the laser parameters to obtain the target control parameters.

6. The control method for automatic metal welding equipment according to claim 4, characterized in that, The step of determining the target control parameters based on the defect location, the defect size, and the defect type includes: Based on the location of the defect, a target laser welding point is generated; Based on the defect size and the defect type, determine the dwell time and laser parameters corresponding to the target laser welding point; The target control parameters are determined based on the target laser welding point, the dwell time corresponding to the target laser welding point, and the laser parameters.

7. A control system for an automatic metal welding equipment, characterized in that, include: Image acquisition equipment is used to acquire target images of metal objects to be welded; Automatic welding equipment is used for welding metal parts. A control device for performing the automatic metal welding equipment control method as described in any one of claims 1 to 6.

Images