Interaction method and device applied to robot welding scene and electronic equipment

Abstract

The invention provides an interaction method and device applied to a robot welding scene and electronic equipment. The method comprises the following steps: acquiring a scanning image of a target workpiece and point cloud data corresponding to the scanning image based on handheld sensor equipment; establishing a three-dimensional model of the target workpiece based on the scanned image and the corresponding point cloud data; the scanning image and the corresponding point cloud data are input into a pre-trained neural network model, weld bead prediction information is obtained, and the weld bead prediction information comprises three-dimensional position information of a predicted weld bead; marking in the three-dimensional model of the target workpiece based on the weld bead prediction information to obtain a three-dimensional model with weld bead marks; and function configuration of the self-positioning function of the handheld sensor equipment is determined, and a target welding position is determined according to the three-dimensional model with the weld bead marks in a welding position determining mode corresponding to the function configuration, so that the welding robot conducts welding based on the target welding position. The method is high in generalization and universality and can be popularized to non-standard scenes.

Description

Technical Field

[0001] This application relates to the technical field of robotic automatic welding and deep learning technology in the field of artificial intelligence, etc., and particularly relates to an interaction method, device and electronic device applied to a robotic welding scenario. Background Art

[0002] With the continuous progress of technology, welding technology has developed rapidly. Researching and promoting intelligent welding equipment is to realize the automation of workpiece welding operations and ensure the stability and consistency of welding operations. How to enable a welding robot to obtain the welding position is the most important link in realizing automatic welding. In related technologies, traditional teaching-free robot welding mainly determines the welding position through manual marking methods. For example, one method is to manually mark the weld bead position on the three-dimensional model of the target workpiece, and the robot scans the workpiece in real time and then registers with the model to obtain the final welding position; another method is that the robot scans the target workpiece in real time, obtains the three-dimensional model of the target workpiece, and then manually standardizes the weld bead position according to the scanning result, and finally hands it over to the robot for welding.

[0003] However, traditional teaching-free robot welding has a strong dependence on manual marking and weak generalization ability, and it is difficult to be promoted to non-standard scenarios. Summary of the Invention

[0004] Embodiments of this application provide an interaction method, device and electronic device applied to a robotic welding scenario.

[0005] According to the first aspect of the embodiments of this application, there is provided an interaction method applied to a robotic welding scenario, including: Obtaining a scanned image of a target workpiece and point cloud data corresponding to the scanned image based on a handheld sensor device; Establishing a three-dimensional model of the target workpiece based on the scanned image and the corresponding point cloud data; Inputting the scanned image and the corresponding point cloud data into a pre-trained neural network model to obtain weld bead prediction information, where the weld bead prediction information includes three-dimensional position information of the predicted weld bead; Performing marking in the three-dimensional model of the target workpiece based on the weld bead prediction information to obtain a three-dimensional model with weld bead markings; Determining the function configuration of the self-positioning function of the handheld sensor device, and determining a target welding position according to the three-dimensional model with weld bead markings by using a welding position determination method corresponding to the function configuration, so that a welding robot performs welding based on the target welding position.

[0006] According to the second aspect of the embodiments of this application, there is provided an interaction device applied to a robotic welding scenario, including: An acquisition module, configured to acquire a scanned image of a target workpiece and point cloud data corresponding to the scanned image based on a handheld sensor device; A building module, configured to build a three-dimensional model of the target workpiece based on the scanned image and the corresponding point cloud data; A prediction module, configured to input the scanned image and the corresponding point cloud data into a pre-trained neural network model to obtain bead prediction information, where the bead prediction information includes three-dimensional position information of a predicted bead; A marking module, configured to perform marking in the three-dimensional model of the target workpiece based on the bead prediction information to obtain a three-dimensional model with bead markings; A determination module, configured to determine a function configuration of the self-positioning function of the handheld sensor device, and determine a target welding position according to the three-dimensional model with bead markings by using a welding position determination method corresponding to the function configuration, so that a welding robot performs welding based on the target welding position.

[0007] According to a third aspect of the embodiments of the present application, there is provided an interaction system applied to a robot welding scenario, including: A handheld sensor device, configured to scan a target workpiece to obtain a scanned image of the target workpiece and point cloud data corresponding to the scanned image; An interaction device, configured to build a three-dimensional model of the target workpiece based on the scanned image and the corresponding point cloud data, and input the scanned image and the corresponding point cloud data into a pre-trained neural network model to obtain bead prediction information, where the bead prediction information includes three-dimensional position information of a predicted bead; The interaction device is further configured to perform marking in the three-dimensional model of the target workpiece based on the bead prediction information to obtain a three-dimensional model with bead markings; The interaction device is further configured to determine a function configuration of the self-positioning function of the handheld sensor device, and determine a target welding position according to the three-dimensional model with bead markings by using a welding position determination method corresponding to the function configuration; A welding robot, configured to perform welding based on the target welding position.

[0008] According to a fourth aspect of the embodiments of the present application, there is provided an electronic device, including: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the foregoing first aspect.

[0009] According to a fifth aspect of the embodiments of the present application, a storage medium is provided. The storage medium stores instructions, and when the instructions run on an electronic device, the electronic device is caused to execute the method described in the foregoing first aspect.

[0010] According to a sixth aspect of the embodiments of the present application, a program product is provided. The program product includes at least one of a program and instructions. When at least one of the program and instructions is executed by a processor, the steps of the method described in the foregoing first aspect are implemented.

[0011] According to the technical solution of the present application, a three-dimensional model of a target workpiece can be constructed through a scanned image and point cloud data of the target workpiece scanned by a handheld sensor, and the three-dimensional position information of the welding bead can be predicted by using a neural network model based on the scanned image and the point cloud data. Marking the predicted three-dimensional position information of the welding bead in the three-dimensional model of the target workpiece can achieve a unique interaction mode of "scanning, identifying, and modeling simultaneously". The input of the neural network model in the present application is a two-dimensional image + point cloud data. Through the semantic information of the two-dimensional image and the spatial feature information of the point cloud data, the predicted information of the welding bead with three-dimensional position can be directly and accurately obtained, and the mapping between two dimensions and three dimensions is no longer required during the whole process, which can simplify the process and improve the efficiency. In addition, the present application can, through the function configuration of the self-positioning function of the handheld sensor device, determine the target welding position based on the three-dimensional model with welding bead markings by using a welding position determination method corresponding to the function configuration, thereby further solving the problem that it is difficult for a welding robot to obtain the welding position. It can be seen that the welding robot in the embodiments of the present application does not rely on manual marking, can greatly reduce the degree of manual intervention, improve the welding efficiency of the robot, and has strong generalization and versatility, and can be extended to more complex non-standard scenarios.

[0012] Additional aspects and advantages of the present application will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present application. Description of the Drawings

[0013] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description of the embodiments in conjunction with the drawings, where: Figure 1 is a flowchart of an interaction method applied to a robot welding scenario provided by an embodiment of the present application; Figure 2 is a flowchart of an interaction method applied to a robot welding scenario provided by an embodiment of the present application; Figure 3 is an example diagram of the internal processing logic for a neural network model to predict a welding bead provided by an embodiment of the present application; Figure 4Flowchart of an interaction method applied to a robot welding scenario provided by an embodiment of the present application; Figure 5 Block diagram of an interaction device applied to a robot welding scenario provided by an embodiment of the present application; Figure 6 Block diagram of an interaction system applied to a robot welding scenario provided by an embodiment of the present application; Figure 7 Block diagram of an electronic device according to an embodiment of the present application. Detailed implementation manners

[0014] The embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, and should not be construed as limiting the present application.

[0015] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms "a", "the", and "said" used in one or more embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of the present application refers to and includes any or all possible combinations of one or more of the associated listed items.

[0016] The interaction method, device, and electronic device applied to a robot welding scenario according to an embodiment of the present application will be described below with reference to the accompanying drawings.

[0017] It should be noted that the execution subject of the interaction method applied to a robot welding scenario according to an embodiment of the present application may be an interaction device applied to a robot welding scenario. This device can be implemented in software and / or hardware, and this device can be configured in an electronic device. Exemplarily, the electronic device may include, but is not limited to, a terminal, a server, etc.

[0018] Figure 1 Is the flowchart of the interaction method applied to a robot welding scenario provided by an embodiment of the present application. As Figure 1 shown, the interaction method applied to a robot welding scenario may include, but is not limited to, the following steps.

[0019] In step 101, a scanned image of a target workpiece and point cloud data corresponding to the scanned image are obtained based on a handheld sensor device.

[0020] In some embodiments, the handheld sensor device may include, but is not limited to, an RGB camera module and a depth sensor module capable of acquiring depth information. The depth sensor module may be, for example, a laser sensor, and the form of the acquired depth information may be, for example, 3D point cloud. Exemplarily, the handheld sensor device may be used to scan a target workpiece to obtain a scanned image of the target workpiece and the point cloud data corresponding to the scanned image.

[0021] Among them, in the embodiments of the present application, the scanned image may be an RGB color image. For example, the image of the target workpiece scanned by the RGB camera module in the handheld sensor device may be used as the scanned image. Or, in the embodiments of the present application, the scanned image may be a grayscale image. For example, the image of the target workpiece is scanned by the RGB camera module in the handheld sensor device, and the scanned image is an RGB image. The RGB image is converted into a grayscale image as the final scanned image of the target workpiece.

[0022] In some embodiments, coordinate calibration is performed between the RGB camera module and the depth sensor module in the handheld sensor device. That is to say, there is a mapping relationship between the pixel points of the image scanned by the RGB camera module and the pixel points in the point cloud data scanned by the depth sensor module. For example, the same pixel point has corresponding image information and point cloud information.

[0023] In step 102, a three-dimensional model of the target workpiece is established based on the scanned image and its corresponding point cloud data.

[0024] In the embodiments of the present application, based on the scanned image of the target workpiece and its corresponding point cloud data, a three-dimensional reconstruction library may be used to construct the three-dimensional model of the target workpiece. Exemplarily, the three-dimensional reconstruction library may be, but is not limited to, PCL (Point Cloud Library), Open3D, etc. The PCL library or Open3D library may be used to fuse the scanned image of the target workpiece and its point cloud data in real time to construct the three-dimensional model of the target workpiece.

[0025] In step 103, the scanned image and its corresponding point cloud data are input into a pre-trained neural network model to obtain bead prediction information, and the bead prediction information includes the three-dimensional position information of the predicted bead.

[0026] In an embodiment of the present application, the above neural network model has learned the mapping relationship between the fusion features of the image and the point cloud data and the weld bead position. The input of the neural network model is the image and the point cloud data, and the output of the neural network model may include the three-dimensional position of the weld bead. Exemplarily, the neural network model may be trained based on a semantic segmentation model such as U-Net or Mask R-CNN, with the input being the image + point cloud data and the output being the weld bead mask. In the post-processing stage of the neural network model, the 2D result can be mapped back to the 3D model to generate the spatial coordinates of the weld bead.

[0027] In step 104, based on the weld bead prediction information, annotation is performed in the three-dimensional model of the target workpiece to obtain a three-dimensional model with weld bead annotation.

[0028] In an embodiment of the present application, the weld bead prediction information (including the three-dimensional position information of the predicted weld bead) can be written into the three-dimensional model of the target workpiece to achieve the annotation of the weld bead position, and a three-dimensional model with weld bead annotation is obtained. It can be understood that based on the pixel points associated with the predicted weld bead included in the weld bead prediction information, the three-dimensional position information of the predicted weld bead is written into the three-dimensional model of the target workpiece, so that a three-dimensional model with weld bead annotation can be obtained.

[0029] In step 105, the function configuration of the self-positioning function of the handheld sensor device is determined, and the target welding position is determined according to the three-dimensional model with weld bead annotation by using a welding position determination method corresponding to the function configuration, so that the welding robot performs welding based on the target welding position.

[0030] In an embodiment of the present application, the function configuration of the self-positioning function of the above handheld sensor device may refer to whether the self-positioning function of the handheld sensor is enabled. Exemplarily, if the handheld sensor supports the self-positioning technology (or self-positioning function) and the self-positioning function is enabled, the handheld sensor can be aligned with the coordinate system of the welding robot in advance, so that the target welding position can be directly determined from the three-dimensional model with weld bead annotation. If the handheld sensor does not support the self-positioning technology or supports the self-positioning technology but the self-positioning function is not enabled, the target workpiece can be scanned locally by the welding robot first, and the final target welding position can be obtained after registration with the previously established three-dimensional model with weld bead annotation.

[0031] It should be noted that in some embodiments, the execution subject of the above steps 101 - 105 may be the handheld sensor; or, in some embodiments, the execution subject of the above steps 101 - 105 may be an electronic device (such as a host computer, etc.) communicatively connected to the handheld sensor. The present application does not make a specific limitation on this.

[0032] In the above embodiments, a three-dimensional model of the target workpiece can be constructed through the scanned image and point cloud data of the target workpiece scanned by the handheld sensor, and the three-dimensional position information of the weld bead can be predicted using a neural network model based on the scanned image and point cloud data. Marking the predicted three-dimensional position information of the weld bead in the three-dimensional model of the target workpiece can achieve a unique interaction method of "scanning, identifying, and modeling simultaneously". The input of the neural network model in this application is a two-dimensional image + point cloud data. Through the semantic information of the two-dimensional image and the spatial feature information of the point cloud data, the weld bead prediction information with three-dimensional positions can be directly and accurately obtained. The entire process does not require mapping between two dimensions and three dimensions, which can simplify the process and improve efficiency. In addition, this application can, through the functional configuration of the self-positioning function of the handheld sensor device, determine the target welding position based on the three-dimensional model with weld bead markings using a welding position determination method corresponding to this functional configuration, thereby further solving the problem that it is difficult for a welding robot to obtain the welding position. It can be seen that the welding robot in the embodiments of this application does not rely on manual marking, can greatly reduce the degree of manual intervention, improve the welding efficiency of the robot, and has strong generalization and versatility, and can be extended to more complex non-standard scenarios.

[0033] It should be noted that the prediction of the weld bead of the target workpiece can be achieved through a neural network model. Optionally, in some embodiments, as Figure 2 and Figure 3 shown, the optional implementation methods of inputting the scanned image and its corresponding point cloud data into the pre-trained neural network model to obtain the weld bead prediction information include but are not limited to the following steps.

[0034] In step 201, depth information (depth) is obtained based on the point cloud data.

[0035] Exemplarily, the point cloud data can be converted into a depth map (depthmap), and the depth information of each pixel point can be obtained from this depth map.

[0036] In step 202, based on the RGB information and depth information of the scanned image, a color image with depth information is obtained, and based on the grayscale information and depth information of the scanned image, a grayscale image with depth information is obtained.

[0037] In step 203, feature extraction is performed on the color image with depth information and the grayscale image with depth information based on the encoder in the neural network model to obtain multi-modal fusion features (Features).

[0038] In step 204, decoding processing is performed on the multi-modal fusion features based on the decoder in the neural network model to obtain the 2D coordinate points of the weld seam.

[0039] Exemplarily, the number of the welding seams can be N, where N can be a positive integer. That is to say, the decoder in the neural network model can be used to decode the multi-modal fusion features to predict the 2D coordinate points of N welding seams (N * welding seam(2D points)) that the target workpiece may need to be welded.

[0040] In step 205, the 2D coordinate points of the welding seams are mapped into the 3D space to obtain the 3D coordinate points of the welding seams, and the weld bead prediction information is determined based on the 3D coordinate points of the welding seams.

[0041] Exemplarily, the 2D coordinate points of N welding seams can be mapped into the 3D space, so that the 3D coordinate points of N welding seams (N * welding seam(3D points)) can be obtained. Based on the 3D coordinate points of the welding seams and combined with the depth map, the weld bead prediction information, such as the three-dimensional position information of the predicted weld bead, can be determined.

[0042] In the above embodiment, the neural network model can be used to process the scanned image and point cloud data of the target workpiece to predict the three-dimensional position of the weld bead of the target workpiece, which is convenient for marking in the three-dimensional model of the target workpiece based on the three-dimensional position of the weld bead, so that an accurate three-dimensional model with weld bead markings can be obtained. The embodiment of the present application realizes weld bead prediction and marking through the neural network model, which can greatly reduce the manual intervention process and improve the robot welding efficiency.

[0043] Figure 4 This is a flowchart of the interaction method applied to the robot welding scenario provided by the embodiment of the present application. As Figure 4 shown, the interaction method applied to the robot welding scenario may include but is not limited to the following steps.

[0044] In step 401, a scanned image of the target workpiece and the point cloud data corresponding to the scanned image are obtained based on the handheld sensor device.

[0045] Optionally, step 401 can be implemented by any one of the implementation manners in the embodiments of the present application. The embodiments of the present application do not make any limitations on this and will not be elaborated further.

[0046] In step 402, a three-dimensional model of the target workpiece is established based on the scanned image and the corresponding point cloud data.

[0047] Optionally, step 402 can be implemented by any one of the implementation manners in the embodiments of the present application. The embodiments of the present application do not make any limitations on this and will not be elaborated further.

[0048] In step 403, the scanned image and its corresponding point cloud data are input into a pre-trained neural network model to obtain bead prediction information, where the bead prediction information includes the three-dimensional position information of the predicted bead.

[0049] Optionally, step 403 can be implemented by any implementation manner in the embodiments of the present application. The embodiments of the present application do not limit this and will not elaborate further.

[0050] In step 404, based on the bead prediction information, annotation is performed on the three-dimensional model of the target workpiece to obtain a three-dimensional model with bead annotation.

[0051] Optionally, step 404 can be implemented by any implementation manner in the embodiments of the present application. The embodiments of the present application do not limit this and will not elaborate further.

[0052] In step 405, the beads to be welded are selected from the three-dimensional model with bead annotation based on deep learning technology or manual selection.

[0053] In some embodiments, when obtaining the bead prediction information through the neural network model, the bead prediction information can be visually displayed for the user to select the beads to be welded from the scanned beads. Thus, welding can be performed according to the user's requirements, ensuring the accuracy of the welding position.

[0054] In some embodiments, the optional implementation manner of selecting the beads to be welded from the three-dimensional model with bead annotation based on deep learning technology includes: obtaining process parameters and inputting the process parameters and the three-dimensional model with bead annotation into a pre-trained deep learning model for bead recommendation; the deep learning model has learned the mapping relationship between the process parameters and the bead features; in the case where the recommended beads output by the deep learning model belong to the beads in the three-dimensional model of the target workpiece, the recommended beads output by the deep learning model are determined as the selected beads to be welded. This can further reduce manual intervention and improve the welding efficiency.

[0055] Exemplarily, the process parameters may include, but are not limited to, welding method codes (such as MIG / TIG / laser welding, etc.) and / or material combination features. Among them, the welding method code can be a One-hot vector; the material combination features may include, but are not limited to: the yield strength ratio of the base material + welding material, the difference in thermal expansion coefficient, etc.

[0056] Exemplarily, the input of the deep learning model may include process parameters and a three-dimensional model with bead annotations, and the output may include a bead probability map. The internal processing logic architecture of the deep learning model may include a feature extraction layer, a spatial-process attention module, multi-scale feature fusion, and a constraint satisfaction layer. Among them, the feature extraction layer may perform geometric feature extraction on the three-dimensional model with bead annotations and encode the process parameters to obtain process features. The cross-modal attention mechanism is used through the spatial-process attention module and multi-scale feature fusion to achieve dynamic weighted fusion of geometric features and process parameters, and a gated recurrent unit is used to process the temporal dependence of the welding sequence. The projection gradient method is introduced in the decoding stage to ensure that the output meets the minimum weld length requirement, and the inaccessible area is excluded during inference through an accessibility mask. Optionally, a region proposal network (RPN) may be used to preferentially process high-curvature regions (i.e., high-incidence areas of welds); incremental prediction is implemented, and after the complex workpiece is processed in blocks, global optimization is performed.

[0057] In step 406, the function configuration of the self-positioning function of the handheld sensor device is determined, and based on the three-dimensional model with bead annotations and the selected bead to be welded, the target welding position is determined by using the welding position determination method corresponding to the function configuration, so that the welding robot performs welding based on the target welding position.

[0058] In an embodiment of the present application, the function configuration of the self-positioning function of the handheld sensor device may refer to whether the self-positioning function of the handheld sensor is enabled. Exemplarily, if the handheld sensor supports the self-positioning technology (or self-positioning function) and the self-positioning function is enabled, the handheld sensor can be aligned with the coordinate system of the welding robot in advance, so that the target welding position can be directly determined from the three-dimensional model with bead annotations. If the handheld sensor does not support the self-positioning technology or supports the self-positioning technology but does not enable the self-positioning function, the target workpiece can be scanned locally by the welding robot first, and after registration with the previously established three-dimensional model with bead annotations, the final target welding position can be obtained.

[0059] In some embodiments, in the case where the function configuration is that the self-positioning function of the handheld sensor device is enabled, the position information of the selected bead to be welded can be directly determined from the three-dimensional model with bead annotations, and the position information of the selected bead to be welded is determined as the target welding position. Exemplarily, if the handheld sensor supports the self-positioning technology (or self-positioning function) and the self-positioning function is enabled, it means that the handheld sensor is aligned with the coordinate system of the welding robot in advance. After the bead to be welded is selected from the three-dimensional model with bead annotations based on deep learning technology or manual selection, the position information of the selected bead to be welded can be directly determined as the target welding position, that is, the welding robot can directly perform welding based on the position information of the selected bead.

[0060] In some embodiments, when the self-positioning function of the handheld sensor device is not enabled in the above function configuration, a three-dimensional point cloud model of the target workpiece scanned by the welding robot body is obtained, and the three-dimensional point cloud model scanned by the welding robot body is registered with the three-dimensional model with weld bead markings to obtain a registration result, and the target welding position is determined based on the registration result in combination with the selected weld beads to be welded. Exemplarily, when the self-positioning function of the handheld sensor device is not enabled, after selecting the weld beads to be welded from the three-dimensional model with weld bead markings based on deep learning technology or manual selection, the welding robot body can scan the target workpiece, and register the coarse-grained three-dimensional point cloud model scanned by the welding robot with the previous three-dimensional model with weld bead markings (for example, a registration algorithm such as ICP can be used, but not limited to this), and the final target welding position of the welding robot can be obtained.

[0061] To further improve the accuracy of the welding position, optionally, in some embodiments, the target welding position and the welding robot path planning result can be superimposed and displayed, and the welding path can be corrected based on the analysis and comparison result of the target welding position and the welding robot path planning result. As an example, the target welding position and the welding robot path planning result can be superimposed and displayed through an AR display (or AR glasses), and gesture interaction is supported to correct the welding path. For example, the user can perform gesture operations on the superimposed effect of the displayed target welding position and the welding robot path planning result to correct the welding path in real time, thereby further improving the accuracy of the welding position.

[0062] Figure 5 is a block diagram of an interaction device applied to a robot welding scenario provided by an embodiment of the present application. As Figure 5 shown, the interaction device applied to the robot welding scenario may include: an acquisition module 501, a building module 502, a prediction module 503, a marking module 504, and a determination module 505.

[0063] Among them, the acquisition module 501 is used to obtain a scanned image of the target workpiece and point cloud data corresponding to the scanned image based on the handheld sensor device.

[0064] The building module 502 is used to build a three-dimensional model of the target workpiece based on the scanned image and the corresponding point cloud data.

[0065] The prediction module 503 is used to input the scanned image and the corresponding point cloud data into a pre-trained neural network model to obtain weld bead prediction information, and the weld bead prediction information includes three-dimensional position information of the predicted weld beads.

[0066] A labeling module 504, configured to perform labeling in a three-dimensional model of a target workpiece based on the bead prediction information, so as to obtain a three-dimensional model with bead labeling.

[0067] A determination module 505, configured to determine the functional configuration of the self-positioning function of the handheld sensor device, and determine a target welding position according to the three-dimensional model with bead labeling by using a welding position determination method corresponding to the functional configuration, so that a welding robot performs welding based on the target welding position.

[0068] In some embodiments, the determination module 505 is configured to: select a bead to be welded from the three-dimensional model with bead labeling based on deep learning technology or manual selection, and determine the target welding position according to the three-dimensional model with bead labeling and the selected bead to be welded by using a welding position determination method corresponding to the functional configuration.

[0069] In some embodiments, the determination module 505 is configured to: obtain process parameters, and input the process parameters and the three-dimensional model with bead labeling into a pre-trained deep learning model for bead recommendation; the deep learning model has learned the mapping relationship between the process parameters and the bead features; in the case that the recommended bead output by the deep learning model belongs to the bead in the three-dimensional model, determine the recommended bead output by the deep learning model as the selected bead to be welded.

[0070] In some embodiments, the determination module 505 is configured to: in the case that the functional configuration is that the self-positioning function of the handheld sensor device is enabled, determine the position information of the selected bead to be welded from the three-dimensional model with bead labeling, and determine the position information of the selected bead to be welded as the target welding position; or, in the case that the functional configuration is that the self-positioning function of the handheld sensor device is not enabled, obtain the three-dimensional point cloud model of the target workpiece scanned by the welding robot body, register the three-dimensional point cloud model scanned by the welding robot body with the three-dimensional model with bead labeling to obtain a registration result, and determine the target welding position based on the registration result and the selected bead to be welded.

[0071] In some embodiments, the prediction module 503 is configured to: obtain depth information based on point cloud data; obtain a color image with depth information based on the RGB information and the depth information of the scanned image, and obtain a grayscale image with depth information based on the grayscale information and the depth information of the scanned image; extract features from the color image with depth information and the grayscale image with depth information by using an encoder in a neural network model to obtain multi-modal fusion features; perform decoding processing on the multi-modal fusion features by using a decoder in the neural network model to obtain 2D coordinate points of the weld seam; map the 2D coordinate points of the weld seam into 3D space to obtain 3D coordinate points of the weld seam, and determine bead prediction information based on the 3D coordinate points of the weld seam.

[0072] In some embodiments, the interaction device applied to the robot welding scenario may further include a correction module. The correction module is configured to: superimpose and display the target welding position and the welding robot path planning result, and correct the welding path based on the analysis and comparison result of the target welding position and the welding robot path planning result.

[0073] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0074] Figure 6 It is a block diagram of an interaction system applied to the robot welding scenario provided by an embodiment of the present application. As Figure 6 shown, the interaction system applied to the robot welding scenario may include: a handheld sensor device 601, an interaction device 602, and a welding robot 603.

[0075] Among them, the handheld sensor device 601 is configured to scan the target workpiece to obtain a scanned image of the target workpiece and point cloud data corresponding to the scanned image.

[0076] The interaction device 602 is configured to establish a three-dimensional model of the target workpiece based on the scanned image and the corresponding point cloud data, and input the scanned image and the corresponding point cloud data into a pre-trained neural network model to obtain weld bead prediction information, where the weld bead prediction information includes three-dimensional position information of the predicted weld bead.

[0077] The interaction device 602 is further configured to perform annotation in the three-dimensional model of the target workpiece based on the weld bead prediction information to obtain a three-dimensional model with weld bead annotation.

[0078] The interaction device 602 is further configured to determine the function configuration of the self-positioning function of the handheld sensor device, and determine the target welding position according to the welding position determination method corresponding to the function configuration based on the three-dimensional model with weld bead annotation.

[0079] The welding robot 603 is configured to perform welding based on the target welding position.

[0080] Optionally, in some embodiments, the above interaction device 602 may be integrated in the handheld sensor device 601. Alternatively, the interaction device 602 may be configured in a host computer communicatively connected to the handheld sensor device 601, but not limited thereto. For example, the interaction device 602 may also be configured in the welding robot 603.

[0081] Regarding the system in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0082] According to an embodiment of the present application, the present application further provides an electronic device and a readable storage medium.

[0083] As Figure 7 shown, it is a block diagram of an electronic device according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0084] As Figure 7 shown, the electronic device includes: one or more processors 701, a memory 702, and interfaces for connecting the components, including a high-speed interface and a low-speed interface. The various components are interconnected using different buses and can be installed on a common motherboard or in other ways as needed. The processor can process instructions executed within the electronic device, including instructions stored in the memory or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories if needed. Similarly, multiple electronic devices can be connected, each device providing some necessary operations (such as, as a server array, a set of blade servers, or a multi-processor system). Figure 7 One processor 701 is taken as an example herein.

[0085] The memory 702 is the non-transitory computer-readable storage medium provided by the present application. Wherein, the memory stores instructions executable by at least one processor, so that the at least one processor executes the interaction method applied to the robot welding scenario provided by the present application. The non-transitory computer-readable storage medium of the present application stores computer instructions, and the computer instructions are used to cause a computer to execute the interaction method applied to the robot welding scenario provided by the present application.

[0086] The memory 702, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions / modules corresponding to the interaction method applied to the robot welding scenario in the embodiment of the present application (for example, attached Figure 5The acquisition module 501, establishment module 502, prediction module 503, annotation module 504, and determination module 505 shown). The processor 701 executes various functional applications and data processing of the server by running non-transitory software programs, instructions, and modules stored in the memory 702, that is, implements the interaction method applied to the robot welding scenario in the above method embodiments.

[0087] The memory 702 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 702 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory 702 may optionally include a memory remotely set relative to the processor 701, and these remote memories may be connected to the electronic device through a network. Examples of the above network include but are not limited to the Internet, enterprise intranet, local area network, mobile communication network, and combinations thereof.

[0088] The electronic device may further include: an input device 703 and an output device 704. The processor 701, memory 702, input device 703, and output device 704 may be connected through a bus or other means, Figure 7 taking the connection through the bus as an example.

[0089] The input device 703 can receive input digital or character information, and generate key signal inputs related to the user settings and function control of the electronic device, such as input devices like touch screens, keypads, mice, trackpads, touchpads, joysticks, one or more mouse buttons, trackballs, and joysticks. The output device 704 may include a display device, auxiliary lighting devices (such as LEDs), and tactile feedback devices (such as vibration motors), etc. The display device may include but is not limited to liquid crystal displays (LCDs), light-emitting diode (LED) displays, and plasma displays. In some embodiments, the display device may be a touch screen.

[0090] The various embodiments of the systems and techniques described herein can be implemented in digital electronic circuitry, integrated circuit systems, ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be a special-purpose or general-purpose programmable processor that receives data and instructions from, and transmits data and instructions to, a storage system, at least one input device, and at least one output device.

[0091] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor, and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, apparatus, and / or device (e.g., a disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0092] For providing interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic, speech, or tactile input).

[0093] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected with each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), the Internet, and blockchain networks.

[0094] A computer system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The client-server relationship is created by computer programs running on respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services ("Virtual Private Server", or simply "VPS"). The server can also be a server of a distributed system, or a server combined with blockchain.

[0095] It should be understood that various forms of the processes shown above can be used, steps can be reordered, added, or deleted. For example, the steps recited in this application can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this application can be achieved, and this is not limited herein.

[0096] The above specific embodiments do not constitute a limitation on the protection scope of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims

1. An interaction method applied to a robot welding scenario, characterized in that, Including: Obtaining a scanned image of a target workpiece and point cloud data corresponding to the scanned image based on a handheld sensor device; Establishing a three-dimensional model of the target workpiece based on the scanned image and the corresponding point cloud data; Inputting the scanned image and the corresponding point cloud data into a pre-trained neural network model to obtain bead prediction information, where the bead prediction information includes three-dimensional position information of the predicted bead; Performing annotation in the three-dimensional model of the target workpiece based on the bead prediction information to obtain a three-dimensional model with bead annotation; Determining a function configuration of the self-positioning function of the handheld sensor device, and determining a target welding position according to a welding position determination method corresponding to the function configuration based on the three-dimensional model with bead annotation, so that a welding robot performs welding based on the target welding position.

2. The method according to claim 1, wherein The method further includes: Selecting beads to be welded from the three-dimensional model with bead annotation based on deep learning technology or manual selection; Wherein, the determining the target welding position according to the welding position determination method corresponding to the function configuration based on the three-dimensional model with bead annotation includes: Determining the target welding position according to the three-dimensional model with bead annotation and the selected beads to be welded by using a welding position determination method corresponding to the function configuration.

3. The method according to claim 2, wherein Selecting beads to be welded from the three-dimensional model with bead annotation based on the deep learning technology includes: Obtaining process parameters, and inputting the process parameters and the three-dimensional model with bead annotation into a pre-trained deep learning model for bead recommendation; the deep learning model has learned the mapping relationship between process parameters and bead features; In the case that the recommended beads output by the deep learning model belong to the beads in the three-dimensional model, determining the recommended beads output by the deep learning model as the selected beads to be welded.

4. The method according to claim 2 or 3, characterized in that, The determining the target welding position according to the three-dimensional model with bead annotation and the selected beads to be welded by using a welding position determination method corresponding to the function configuration includes: In the case that the function configuration is that the self-positioning function of the handheld sensor device is enabled, determining the position information of the selected beads to be welded from the three-dimensional model with bead annotation, and determining the position information of the selected beads to be welded as the target welding position; or, In the case that the function configuration is that the self-positioning function of the handheld sensor device is not enabled, obtaining a three-dimensional point cloud model of the target workpiece scanned by the welding robot body, registering the three-dimensional point cloud model scanned by the welding robot body with the three-dimensional model with bead annotation to obtain a registration result, and determining the target welding position based on the registration result in combination with the selected beads to be welded.

5. The method according to claim 1, wherein The inputting the scanned image and the corresponding point cloud data into a pre-trained neural network model to obtain bead prediction information includes: Obtaining depth information based on the point cloud data; Based on the RGB information and the depth information of the scanned image, a color image with depth information is obtained, and based on the grayscale information and the depth information of the scanned image, a grayscale image with depth information is obtained; Based on the encoder in the neural network model, feature extraction is performed on the color image with depth information and the grayscale image with depth information to obtain multi-modal fusion features; Based on the decoder in the neural network model, decoding processing is performed on the multi-modal fusion features to obtain the 2D coordinate points of the weld; Map the 2D coordinate points of the weld to the 3D space to obtain the 3D coordinate points of the weld, and determine the weld bead prediction information based on the 3D coordinate points of the weld.

6. The method according to claim 1, wherein The method further includes: Overlay and display the target welding position and the welding robot path planning result, and correct the welding path based on the analysis and comparison result of the target welding position and the welding robot path planning result.

7. An interaction device applied to a robot welding scenario, characterized in that, Includes: An acquisition module, configured to acquire a scanned image of a target workpiece and the point cloud data corresponding to the scanned image based on a handheld sensor device; A building module, configured to build a three-dimensional model of the target workpiece based on the scanned image and the corresponding point cloud data; A prediction module, configured to input the scanned image and the corresponding point cloud data into a pre-trained neural network model to obtain weld bead prediction information, where the weld bead prediction information includes three-dimensional position information of the predicted weld bead; A marking module, configured to perform marking in the three-dimensional model of the target workpiece based on the weld bead prediction information to obtain a three-dimensional model with weld bead markings; A determination module, configured to determine the functional configuration of the self-positioning function of the handheld sensor device, and determine the target welding position according to the welding position determination method corresponding to the functional configuration based on the three-dimensional model with weld bead markings, so that the welding robot performs welding based on the target welding position.

8. An interaction system applied to a robot welding scenario, characterized in that, Includes: A handheld sensor device, configured to scan a target workpiece to obtain a scanned image of the target workpiece and the point cloud data corresponding to the scanned image; An interaction device, configured to build a three-dimensional model of the target workpiece based on the scanned image and the corresponding point cloud data, and input the scanned image and the corresponding point cloud data into a pre-trained neural network model to obtain weld bead prediction information, where the weld bead prediction information includes three-dimensional position information of the predicted weld bead; The interaction device is further configured to perform marking in the three-dimensional model of the target workpiece based on the weld bead prediction information to obtain a three-dimensional model with weld bead markings; The interaction device is further configured to determine the functional configuration of the self-positioning function of the handheld sensor device, and determine the target welding position according to the welding position determination method corresponding to the functional configuration based on the three-dimensional model with weld bead markings; A welding robot, configured to perform welding based on the target welding position.

9. An electronic device, characterized in that, Includes: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method according to any one of claims 1-6.

10. A storage medium, characterized in that, The storage medium stores instructions that, when run on an electronic device, cause the electronic device to execute the method according to any one of claims 1-6.

Images