Interaction method, device and electronic equipment applied to robot welding scene

Abstract

The present application proposes an interactive method, device and electronic device for use in robot welding scenarios. The method includes: obtaining a scanned image of a target workpiece and point cloud data corresponding to the scanned image based on a handheld sensor device; establishing a three-dimensional model of the target workpiece based on the scanned image and its corresponding point cloud data; inputting the scanned image and its corresponding point cloud data into a pre-trained neural network model to obtain weld bead prediction information, the weld bead prediction information including three-dimensional position information of the predicted weld bead; annotating the three-dimensional model of the target workpiece based on the weld bead prediction information to obtain a three-dimensional model with weld bead annotations; determining the functional configuration of the self-positioning function of the handheld sensor device, and determining the target welding position based on the three-dimensional model with weld bead annotations using a welding position determination method corresponding to the functional configuration, so that the welding robot performs welding based on the target welding position. The present application has strong generalization and versatility and can be extended to non-standard scenarios.

Description

Technical Field

[0001] The present application relates to the field of robot automatic welding technology and deep learning technology in the field of artificial intelligence, and in particular to an interaction method, device and electronic equipment applied to robot welding scenarios. Background Art

[0002] With the continuous advancement of science and technology, welding technology has developed rapidly. The research, development and promotion of intelligent welding equipment is to realize the automation of workpiece welding operations and ensure the stability and consistency of welding operations. How to enable the welding robot to obtain the welding position is the most important link in realizing automatic welding. In related technologies, traditional teaching-free robot welding mainly determines the welding position through manual marking. For example, one method is: manually mark the weld position on the three-dimensional model of the target workpiece, the robot scans the workpiece in real time, and then aligns with the model to obtain the final welding position; another method is: the robot scans the target workpiece in real time to obtain the three-dimensional model of the target workpiece, and then manually standardizes the weld position according to the scanning results, and finally gives it to the robot for welding.

[0003] However, traditional teaching-free robot welding relies heavily on manual labeling, has weak generalization, and is difficult to extend to non-standard scenarios. Summary of the Invention

[0004] The embodiments of the present application provide an interactive method, device, and electronic device for use in a robot welding scenario.

[0005] According to a first aspect of an embodiment of the present application, there is provided an interaction method applied to a robot welding scenario, comprising:

[0006] Acquire a scanned image of a target workpiece and point cloud data corresponding to the scanned image based on a handheld sensor device;

[0007] Establishing a three-dimensional model of the target workpiece based on the scanned image and its corresponding point cloud data;

[0008] Inputting the scanned image and its corresponding point cloud data into a pre-trained neural network model to obtain weld bead prediction information, wherein the weld bead prediction information includes three-dimensional position information of the predicted weld bead;

[0009] Annotating the three-dimensional model of the target workpiece based on the weld bead prediction information to obtain a three-dimensional model with weld bead annotations;

[0010] Determine the functional configuration of the self-positioning function of the handheld sensor device, and determine the target welding position according to the three-dimensional model with weld bead annotations using a welding position determination method corresponding to the functional configuration, so that the welding robot performs welding based on the target welding position.

[0011] According to a second aspect of an embodiment of the present application, there is provided an interactive device for use in a robot welding scenario, comprising:

[0012] An acquisition module, configured to acquire a scanned image of a target workpiece and point cloud data corresponding to the scanned image based on a handheld sensor device;

[0013] A building module, configured to build a three-dimensional model of the target workpiece based on the scanned image and its corresponding point cloud data;

[0014] A prediction module, configured to input the scanned image and its corresponding point cloud data into a pre-trained neural network model to obtain weld bead prediction information, wherein the weld bead prediction information includes three-dimensional position information of the predicted weld bead;

[0015] a marking module, configured to mark the three-dimensional model of the target workpiece based on the weld bead prediction information to obtain a three-dimensional model with weld bead markings;

[0016] A determination module is used to determine the functional configuration of the self-positioning function of the handheld sensor device, and determine the target welding position according to the three-dimensional model with weld bead annotations using a welding position determination method corresponding to the functional configuration, so that the welding robot performs welding based on the target welding position.

[0017] According to a third aspect of an embodiment of the present application, an interactive system for a robot welding scenario is provided, comprising:

[0018] A handheld sensor device is used to scan a target workpiece to obtain a scanned image of the target workpiece and point cloud data corresponding to the scanned image;

[0019] An interactive device for establishing a three-dimensional model of the target workpiece based on the scanned image and its corresponding point cloud data, and inputting the scanned image and its corresponding point cloud data into a pre-trained neural network model to obtain weld bead prediction information, wherein the weld bead prediction information includes three-dimensional position information of the predicted weld bead;

[0020] The interactive device is further configured to mark the three-dimensional model of the target workpiece based on the weld bead prediction information to obtain a three-dimensional model with weld bead markings;

[0021] The interactive device is further configured to determine a functional configuration of the self-positioning function of the handheld sensor device, and determine a target welding position based on the three-dimensional model with weld bead annotations using a welding position determination method corresponding to the functional configuration;

[0022] A welding robot is used for performing welding based on the target welding position.

[0023] According to a fourth aspect of the embodiments of the present application, there is provided an electronic device, including:

[0024] at least one processor; and

[0025] a memory communicatively connected to the at least one processor; wherein,

[0026] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0027] According to a fifth aspect of an embodiment of the present application, a storage medium is provided, which stores instructions. When the instructions are executed on an electronic device, the electronic device executes the method described in the first aspect above.

[0028] According to a sixth aspect of an embodiment of the present application, a program product is provided, which includes at least one of a program and an instruction, and when the at least one of the program and the instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0029] According to the technical solution of the present application, a three-dimensional model of the target workpiece can be constructed using the scanned image and point cloud data of the target workpiece scanned by a handheld sensor. Based on the scanned image and point cloud data, a neural network model is used to predict the three-dimensional position information of the weld bead. The predicted three-dimensional position information of the weld bead is annotated in the three-dimensional model of the target workpiece, realizing a unique interactive method of "scanning, recognizing, and modeling simultaneously." The input of the neural network model in the present application is a two-dimensional image + point cloud data. By using the semantic information of the two-dimensional image and the spatial feature information of the point cloud data, the weld bead prediction information with three-dimensional position can be directly and accurately obtained. The entire process does not require mapping between two-dimensional and three-dimensional, which can simplify the process and improve efficiency. In addition, the present application can determine the target welding position based on the three-dimensional model with weld bead annotations using a welding position determination method corresponding to the functional configuration of the handheld sensor device through the self-positioning function, thereby further solving the problem of welding robots having difficulty in obtaining welding positions. As can be seen, the welding robot in the embodiment of the present application does not rely on manual annotation, which can significantly reduce the degree of human intervention and improve the efficiency of robot welding. It has strong generalization and versatility and can be extended to more complex non-standard scenarios.

[0030] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0032] Figure 1 This is a flow chart of an interactive method applied to a robot welding scenario provided by an embodiment of the present application;

[0033] Figure 2 This is a flow chart of an interactive method applied to a robot welding scenario provided by an embodiment of the present application;

[0034] Figure 3 This is an example diagram of the internal processing logic of the neural network model for predicting welds provided in an embodiment of the present application;

[0035] Figure 4 A flowchart of an interactive method applied to a robot welding scenario provided in an embodiment of the present application;

[0036] Figure 5 This is a block diagram of an interactive device applied to a robot welding scenario provided by an embodiment of the present application;

[0037] Figure 6 This is a block diagram of an interactive system for robot welding scenarios provided by an embodiment of the present application;

[0038] Figure 7 is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0040] The terms used in one or more embodiments of the present application are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of the present application. The singular forms "a", "the" and "the" used in one or more embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of the present application refers to and includes any or all possible combinations of one or more associated listed items.

[0041] The following describes the interactive method, device, and electronic device applied to a robot welding scenario according to an embodiment of the present application with reference to the accompanying drawings.

[0042] It should be noted that the execution subject of the interactive method applied to the robotic welding scenario in the embodiment of the present application can be an interactive device applied to the robotic welding scenario, which can be implemented by software and / or hardware and configured in an electronic device. For example, the electronic device can include but is not limited to a terminal, a server, etc.

[0043] Figure 1 This is a flow chart of the interactive method for robot welding scenarios provided by the embodiment of the present application. Figure 1 As shown, the interactive method applied to the robot welding scenario may include but is not limited to the following steps.

[0044] In step 101 , a scanned image of a target workpiece and point cloud data corresponding to the scanned image are acquired based on a handheld sensor device.

[0045] In some embodiments, the handheld sensor device may include, but is not limited to, an RGB camera module and a depth sensor module capable of acquiring depth information. The depth sensor module may, for example, be a laser sensor, and the acquired depth information may be in the form of, for example, a 3D point cloud. For example, the handheld sensor device may be used to scan a target workpiece to acquire a scanned image of the target workpiece and point cloud data corresponding to the scanned image.

[0046] In the embodiment of the present application, the scanned image may be an RGB color image, for example, an image of the target workpiece scanned by an RGB camera module in a handheld sensor device may be used as the scanned image. Alternatively, in the embodiment of the present application, the scanned image may be a grayscale image, for example, an image of the target workpiece scanned by an RGB camera module in a handheld sensor device, the scanned image being an RGB image, which is converted into a grayscale image as the final scanned image of the target workpiece.

[0047] In some embodiments, a coordinate system calibration is performed between the RGB camera module and the depth sensor module in the handheld sensor device. That is, there is a mapping relationship between the pixel points of the image scanned by the RGB camera module and the pixel points in the point cloud data scanned by the depth sensor module. For example, the same pixel point has corresponding image information and point cloud information.

[0048] In step 102 , a three-dimensional model of the target workpiece is created based on the scanned image and its corresponding point cloud data.

[0049] In an embodiment of the present application, a 3D reconstruction library can be used to construct a 3D model of the target workpiece based on the scanned image of the target workpiece and its corresponding point cloud data. For example, the 3D reconstruction library can be, but is not limited to, PCL (Point Cloud Library), Open3D, etc. The PCL library or Open3D library can be used to fuse the scanned image of the target workpiece and its point cloud data in real time to construct the 3D model of the target workpiece.

[0050] In step 103, the scanned image and its corresponding point cloud data are input into a pre-trained neural network model to obtain weld bead prediction information, which includes three-dimensional position information of the predicted weld bead.

[0051] In an embodiment of the present application, the neural network model has learned the mapping relationship between the fusion features of image and point cloud data and the weld bead location. The neural network model takes image and point cloud data as input, and its output can include the 3D position of the weld bead. Exemplarily, the neural network model can be trained based on a semantic segmentation model such as U-Net or Mask R-CNN, taking image and point cloud data as input and outputting a weld bead mask. The post-processing stage of the neural network model can map the 2D results back to a 3D model to generate the weld bead spatial coordinates.

[0052] In step 104 , the three-dimensional model of the target workpiece is annotated based on the weld bead prediction information to obtain a three-dimensional model with weld bead annotations.

[0053] In embodiments of the present application, weld bead prediction information (including the three-dimensional position information of the predicted weld bead) can be written into a three-dimensional model of a target workpiece to annotate the weld bead position and obtain a three-dimensional model with the weld bead annotations. It will be appreciated that, based on the pixels associated with the predicted weld bead included in the weld bead prediction information, the three-dimensional position information of the predicted weld bead is written into the three-dimensional model of the target workpiece, thereby obtaining a three-dimensional model with the weld bead annotations.

[0054] In step 105, the functional configuration of the self-positioning function of the handheld sensor device is determined, and the target welding position is determined according to the welding position determination method corresponding to the functional configuration based on the three-dimensional model with weld bead annotations, so that the welding robot performs welding based on the target welding position.

[0055] In embodiments of the present application, the functional configuration of the aforementioned self-positioning function of the handheld sensor device may refer to whether the handheld sensor has the self-positioning function enabled. For example, if the handheld sensor supports self-positioning technology (or the self-positioning function) and the self-positioning function is enabled, the handheld sensor can be pre-aligned with the welding robot's coordinate system. This allows the target welding position to be determined directly from the 3D model with weld bead annotations. If the handheld sensor does not support self-positioning technology, or supports self-positioning technology but the self-positioning function is not enabled, the welding robot can first scan the target workpiece locally and align it with the previously established 3D model with weld bead annotations to determine the final target welding position.

[0056] It should be noted that, in some embodiments, the execution entity of the above steps 101 to 105 may be a handheld sensor; or, in some embodiments, the execution entity of the above steps 101 to 105 may be an electronic device (such as a host computer, etc.) that is communicatively connected to the handheld sensor. This application does not make any specific limitations on this.

[0057] In the above-described embodiment, a three-dimensional model of the target workpiece can be constructed using the scanned image and point cloud data of the target workpiece scanned by the handheld sensor. Based on the scanned image and point cloud data, a neural network model is used to predict the three-dimensional position information of the weld bead. The predicted three-dimensional position information of the weld bead is then annotated in the three-dimensional model of the target workpiece, achieving a unique interactive mode of "scanning, recognizing, and modeling simultaneously." The neural network model in this application uses a two-dimensional image and point cloud data as input. By using the semantic information of the two-dimensional image and the spatial feature information of the point cloud data, weld bead prediction information with three-dimensional position can be directly and accurately obtained. The entire process eliminates the need for mapping between two-dimensional and three-dimensional data, simplifying the process and improving efficiency. Furthermore, this application can utilize the self-positioning function of the handheld sensor device to determine the target welding position based on the three-dimensional model with weld bead annotations using a welding position determination method corresponding to this function configuration, thereby further solving the problem of welding robots having difficulty in obtaining welding positions. Thus, the welding robot in the embodiment of this application does not rely on manual annotation, significantly reducing the degree of human intervention and improving robot welding efficiency. Its high generalization and versatility allow it to be extended to more complex, non-standard scenarios.

[0058] It should be noted that the prediction of the target workpiece weld can be achieved through a neural network model. Figure 2 and Figure 3 As shown, the optional implementation method of inputting the scanned image and its corresponding point cloud data into the pre-trained neural network model to obtain the weld bead prediction information includes but is not limited to the following steps.

[0059] In step 201 , depth information (depth) is acquired based on point cloud data.

[0060] For example, the point cloud data may be converted into a depth map, and the depth information of each pixel may be obtained from the depth map.

[0061] In step 202, a color image with depth information is obtained based on the RGB information and depth information of the scanned image, and a grayscale image with depth information is obtained based on the grayscale information and depth information of the scanned image.

[0062] In step 203 , feature extraction is performed on the color image with depth information and the grayscale image with depth information based on the encoder in the neural network model to obtain multimodal fusion features.

[0063] In step 204, the multimodal fusion features are decoded based on the decoder in the neural network model to obtain the 2D coordinate points of the weld.

[0064] Exemplarily, the number of welds may be N, where N may be a positive integer. That is, the decoder in the neural network model may be used to decode the multimodal fusion features to predict the 2D coordinate points (N*welding seam (2D points)) of the N welds that may need to be welded on the target workpiece.

[0065] In step 205 , the 2D coordinate points of the weld are mapped into a 3D space to obtain the 3D coordinate points of the weld, and weld bead prediction information is determined based on the 3D coordinate points of the weld.

[0066] For example, the 2D coordinate points of N welds can be mapped into 3D space, thereby obtaining the 3D coordinate points of N welds (N*welding seam(3D points)). Based on the 3D coordinate points of the welds and combined with the depth map, the weld prediction information, such as the predicted 3D position information of the weld, can be determined.

[0067] In the above embodiment, the scanned image and point cloud data of the target workpiece can be processed by a neural network model to predict the three-dimensional position of the weld bead of the target workpiece, so as to facilitate annotation in the three-dimensional model of the target workpiece based on the three-dimensional position of the weld bead, thereby obtaining an accurate three-dimensional model with weld bead annotations. The embodiment of the present application realizes weld bead prediction and annotation through a neural network model, which can greatly reduce manual intervention procedures and improve the efficiency of robot welding.

[0068] Figure 4 This is a flow chart of an interactive method for robot welding scenarios provided in an embodiment of the present application. Figure 4As shown, the interactive method applied to the robot welding scenario may include but is not limited to the following steps.

[0069] In step 401 , a scanned image of a target workpiece and point cloud data corresponding to the scanned image are acquired based on a handheld sensor device.

[0070] Optionally, step 401 may be implemented using any of the implementation methods in the embodiments of the present application. The embodiments of the present application do not limit this and will not be described in detail.

[0071] In step 402 , a three-dimensional model of the target workpiece is created based on the scanned image and its corresponding point cloud data.

[0072] Optionally, step 402 may be implemented using any of the implementation methods in the embodiments of the present application. The embodiments of the present application do not limit this and will not be described in detail.

[0073] In step 403, the scanned image and its corresponding point cloud data are input into a pre-trained neural network model to obtain weld bead prediction information, which includes three-dimensional position information of the predicted weld bead.

[0074] Optionally, step 403 may be implemented using any implementation method in each embodiment of the present application. The embodiments of the present application do not limit this and will not be described in detail.

[0075] In step 404 , the three-dimensional model of the target workpiece is annotated based on the weld bead prediction information to obtain a three-dimensional model with weld bead annotations.

[0076] Optionally, step 404 may be implemented using any of the implementation methods in the embodiments of the present application. The embodiments of the present application do not limit this and will not be described in detail.

[0077] In step 405 , the weld bead to be welded is selected from the three-dimensional model with weld bead annotations based on deep learning technology or manual selection.

[0078] In some embodiments, when weld bead prediction information is obtained through a neural network model, the weld bead prediction information can be visualized to allow the user to select the weld bead to be welded from the scanned weld bead. This allows welding to be performed according to the user's needs, thereby ensuring the accuracy of the welding position.

[0079] In some embodiments, an optional implementation of selecting a weld bead to be welded from a 3D model with weld bead annotations based on deep learning technology includes: obtaining process parameters and inputting the process parameters and the 3D model with weld bead annotations into a pre-trained deep learning model for weld bead recommendation; the deep learning model has learned the mapping relationship between process parameters and weld bead characteristics; and if the recommended weld bead output by the deep learning model corresponds to a weld bead in the 3D model of the target workpiece, determining the recommended weld bead output by the deep learning model as the selected weld bead to be welded. This can further reduce manual intervention and improve welding efficiency.

[0080] For example, the process parameters may include, but are not limited to, welding method codes (e.g., MIG / TIG / laser welding) and / or material combination characteristics. The welding method code may be a one-hot vector; the material combination characteristics may include, but are not limited to, the yield strength ratio of the base material and the weld material, the difference in thermal expansion coefficient, etc.

[0081] Exemplarily, the input of this deep learning model may include process parameters and a 3D model with weld bead annotations, and the output may include a weld bead probability map. The internal processing logic architecture of this deep learning model includes a feature extraction layer, a spatial-process attention module, multi-scale feature fusion, and a constraint satisfaction layer. The feature extraction layer extracts geometric features from the 3D model with weld bead annotations and encodes the process parameters to obtain process features. The spatial-process attention module and multi-scale feature fusion employ a cross-modal attention mechanism to dynamically weightedly fuse geometric features and process parameters. A gated recurrent unit (GRU) is used to handle the temporal dependencies of the welding sequence. A projected gradient method is introduced in the decoding stage to ensure that the output meets the minimum weld length requirement, and a reachability mask is used to exclude unweldable areas during inference. Optionally, a region proposal network (RPN) can be used to prioritize high-curvature regions (i.e., areas with a high weld risk). This enables incremental prediction, allowing complex workpieces to be segmented and then globally optimized.

[0082] In step 406, the functional configuration of the self-positioning function of the handheld sensor device is determined, and based on the three-dimensional model with weld bead annotations and the selected weld bead to be welded, the target welding position is determined using a welding position determination method corresponding to the functional configuration, so that the welding robot performs welding based on the target welding position.

[0083] In embodiments of the present application, the functional configuration of the aforementioned self-positioning function of the handheld sensor device may refer to whether the handheld sensor has the self-positioning function enabled. For example, if the handheld sensor supports self-positioning technology (or the self-positioning function) and the self-positioning function is enabled, the handheld sensor can be pre-aligned with the welding robot's coordinate system. This allows the target welding position to be determined directly from the 3D model with weld bead annotations. If the handheld sensor does not support self-positioning technology, or supports self-positioning technology but the self-positioning function is not enabled, the welding robot can first scan the target workpiece locally and align it with the previously established 3D model with weld bead annotations to determine the final target welding position.

[0084] In some embodiments, when the aforementioned functional configuration enables a self-positioning function for the handheld sensor device, the position information of the selected weld bead to be welded can be directly determined from the three-dimensional model with weld bead annotations, and the position information of the selected weld bead to be welded can be determined as the target welding position. For example, if the handheld sensor supports self-positioning technology (or self-positioning function) and the self-positioning function is enabled, it indicates that the handheld sensor has been pre-aligned with the coordinate system of the welding robot. In this way, after the weld bead to be welded is selected from the three-dimensional model with weld bead annotations based on deep learning technology or manual selection, the position information of the selected weld bead to be welded can be directly determined as the target welding position, that is, the welding robot can directly perform welding based on the position information of the selected weld bead.

[0085] In some embodiments, when the above-mentioned function is configured such that the handheld sensor device does not enable the self-positioning function, a three-dimensional point cloud model of the target workpiece scanned by the welding robot body is obtained, and the three-dimensional point cloud model scanned by the welding robot body is registered with the three-dimensional model with weld bead annotations to obtain a registration result. The target welding position is then determined based on the registration result and the selected weld bead to be welded. Exemplarily, when the handheld sensor device does not enable the self-positioning function, after selecting the weld bead to be welded from the three-dimensional model with weld bead annotations based on deep learning technology or manual selection, the welding robot body can scan the target workpiece, and register the coarse-grained three-dimensional point cloud model scanned by the welding robot with the three-dimensional model previously with weld bead annotations (for example, a registration algorithm such as, but not limited to, ICP can be used) to obtain the welding robot's final target welding position.

[0086] To further improve the accuracy of the welding position, in some embodiments, the target welding position and the welding robot path planning results can be displayed in a superimposed manner, and the welding path can be corrected based on the analysis and comparison of the target welding position and the welding robot path planning results. As an example, the target welding position and the welding robot path planning results can be superimposed on an AR display (or AR glasses), and gesture interaction can be supported to correct the welding path. For example, a user can perform gestures on the superimposed effect of the displayed target welding position and the welding robot path planning results to correct the welding path in real time, thereby further improving the accuracy of the welding position.

[0087] Figure 5 This is a block diagram of an interactive device for robot welding scenarios provided by an embodiment of the present application. Figure 5 As shown, the interactive device applied to the robot welding scene may include: an acquisition module 501, an establishment module 502, a prediction module 503, a labeling module 504 and a determination module 505.

[0088] The acquisition module 501 is configured to acquire a scanned image of a target workpiece and point cloud data corresponding to the scanned image based on a handheld sensor device.

[0089] The building module 502 is used to build a three-dimensional model of the target workpiece based on the scanned image and its corresponding point cloud data.

[0090] The prediction module 503 is used to input the scanned image and its corresponding point cloud data into a pre-trained neural network model to obtain weld bead prediction information, which includes the three-dimensional position information of the predicted weld bead.

[0091] The marking module 504 is used to mark the three-dimensional model of the target workpiece based on the weld bead prediction information to obtain a three-dimensional model with weld bead markings.

[0092] The determination module 505 is used to determine the functional configuration of the self-positioning function of the handheld sensor device, and determine the target welding position according to the three-dimensional model with weld bead annotations using a welding position determination method corresponding to the functional configuration, so that the welding robot performs welding based on the target welding position.

[0093] In some embodiments, the determination module 505 is used to: select the weld to be welded from the three-dimensional model with weld bead annotations based on deep learning technology or manual selection, and determine the target welding position using a welding position determination method corresponding to the functional configuration according to the three-dimensional model with weld bead annotations and the selected weld to be welded.

[0094] In some embodiments, the determination module 505 is used to: obtain process parameters, and input the process parameters and the three-dimensional model with weld annotations into a pre-trained deep learning model for weld recommendation; the deep learning model has learned the mapping relationship between the process parameters and the weld characteristics; when the recommended weld output by the deep learning model belongs to the weld in the three-dimensional model, the recommended weld output by the deep learning model is determined as the selected weld that needs to be welded.

[0095] In some embodiments, the determination module 505 is used to: when the function configuration is that the handheld sensor device enables the self-positioning function, determine the position information of the selected weld to be welded from the three-dimensional model with the weld bead annotation, and determine the position information of the selected weld to be welded as the target welding position; or, when the function configuration is that the handheld sensor device does not enable the self-positioning function, obtain the three-dimensional point cloud model of the target workpiece scanned by the welding robot body, and align the three-dimensional point cloud model scanned by the welding robot body with the three-dimensional model with the weld bead annotation to obtain the alignment result, and determine the target welding position based on the alignment result in combination with the selected weld to be welded.

[0096] In some embodiments, the prediction module 503 is used to: obtain depth information based on point cloud data; obtain a color image with depth information based on the RGB information and depth information of the scanned image, and obtain a grayscale image with depth information based on the grayscale information and depth information of the scanned image; perform feature extraction on the color image with depth information and the grayscale image with depth information based on the encoder in the neural network model to obtain multimodal fusion features; perform decoding processing on the multimodal fusion features based on the decoder in the neural network model to obtain the 2D coordinate points of the weld; map the 2D coordinate points of the weld to the 3D space to obtain the 3D coordinate points of the weld, and determine the weld prediction information based on the 3D coordinate points of the weld.

[0097] In some embodiments, the interactive device for robotic welding scenarios may further include a correction module, wherein the correction module is configured to overlay and display the target welding position and the welding robot path planning result, and to correct the welding path based on the analysis and comparison results of the target welding position and the welding robot path planning result.

[0098] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0099] Figure 6 This is a block diagram of an interactive system for robot welding scenarios provided by an embodiment of the present application. Figure 6As shown, the interactive system applied to the robot welding scenario may include: a handheld sensor device 601, an interactive device 602 and a welding robot 603.

[0100] The handheld sensor device 601 is used to scan the target workpiece to obtain a scanned image of the target workpiece and point cloud data corresponding to the scanned image.

[0101] The interactive device 602 is used to establish a three-dimensional model of the target workpiece based on the scanned image and its corresponding point cloud data, and input the scanned image and its corresponding point cloud data into a pre-trained neural network model to obtain weld bead prediction information, which includes the three-dimensional position information of the predicted weld bead.

[0102] The interactive device 602 is further configured to mark the three-dimensional model of the target workpiece based on the weld bead prediction information, thereby obtaining a three-dimensional model with weld bead markings.

[0103] The interactive device 602 is further configured to determine the functional configuration of the self-positioning function of the handheld sensor device, and determine the target welding position according to the three-dimensional model with weld bead annotations using a welding position determination method corresponding to the functional configuration.

[0104] The welding robot 603 is used to perform welding based on a target welding position.

[0105] Optionally, in some embodiments, the interaction device 602 may be integrated into the handheld sensor device 601. Alternatively, the interaction device 602 may be configured in a host computer that is communicatively connected to the handheld sensor device 601, but is not limited thereto. For example, the interaction device 602 may also be configured in the welding robot 603.

[0106] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0107] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

[0108] like Figure 7 , is a block diagram of an electronic device according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0109] like Figure 7 As shown, the electronic device includes: one or more processors 701, a memory 702, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 701 is taken as an example.

[0110] Memory 702 is the non-transitory computer-readable storage medium provided in this application. The memory stores instructions executable by at least one processor, causing the at least one processor to execute the interactive method for robotic welding scenarios provided in this application. The non-transitory computer-readable storage medium of this application stores computer instructions for causing a computer to execute the interactive method for robotic welding scenarios provided in this application.

[0111] The memory 702 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the interactive method applied to the robot welding scene in the embodiment of the present application (for example, the attached Figure 5 The processor 701 executes the non-transient software programs, instructions, and modules stored in the memory 702 to execute various functional applications and data processing of the server, thereby implementing the interactive method applied to the robot welding scenario in the above method embodiment.

[0112] The memory 702 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 702 may optionally include a memory remotely located relative to the processor 701, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0113] The electronic device may further include: an input device 703 and an output device 704. The processor 701, the memory 702, the input device 703 and the output device 704 may be connected via a bus or other means. Figure 7 The bus connection is taken as an example.

[0114] Input device 703 can receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. Examples include a touch screen, keypad, mouse, trackpad, touchpad, pointing stick, one or more mouse buttons, trackball, joystick, and other input devices. Output device 704 may include a display device, auxiliary lighting devices (e.g., LEDs), and tactile feedback devices (e.g., vibration motors). Display devices may include, but are not limited to, liquid crystal displays (LCDs), light-emitting diode (LED) displays, and plasma displays. In some embodiments, the display device may be a touch screen.

[0115] Various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0116] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., a magnetic disk, an optical disk, a memory, a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0117] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0118] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0119] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical servers and VPS services ("Virtual Private Servers" or "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0120] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this application can be achieved. This is not a limitation herein.

[0121] The above specific embodiments do not constitute a limitation on the scope of protection of this application. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application shall be included within the scope of protection of this application.

Claims

1. An interactive method applied to a robot welding scenario, characterized in that: include: Acquire a scanned image of a target workpiece and point cloud data corresponding to the scanned image based on a handheld sensor device; Establishing a three-dimensional model of the target workpiece based on the scanned image and its corresponding point cloud data; Inputting the scanned image and its corresponding point cloud data into a pre-trained neural network model to obtain weld bead prediction information, wherein the weld bead prediction information includes three-dimensional position information of the predicted weld bead; wherein the input of the neural network model includes a scanned image of a target workpiece and its corresponding point cloud data, the neural network model obtains a color image with depth information and a grayscale image with depth information based on the depth information in the scanned image and its corresponding point cloud data, performs feature extraction on the color image with depth information and the grayscale image with depth information to obtain multimodal fusion features, and decodes the multimodal fusion features to output three-dimensional position information of N predicted weld beads, where N is a positive integer; Annotating the three-dimensional model of the target workpiece based on the weld bead prediction information to obtain a three-dimensional model with weld bead annotations; Selecting welds to be welded from the three-dimensional model with weld bead annotations based on deep learning technology or manual selection; Determine the functional configuration of the self-positioning function of the handheld sensor device, and determine the target welding position using a welding position determination method corresponding to the functional configuration based on the three-dimensional model with weld bead annotations and the selected weld bead to be welded, so that the welding robot performs welding based on the target welding position.

2. The method according to claim 1, wherein Selecting welds to be welded from the three-dimensional model with weld bead annotations based on the deep learning technology includes: Obtaining process parameters, and inputting the process parameters and the three-dimensional model with weld bead annotations into a pre-trained deep learning model to perform weld bead recommendation; the deep learning model has learned a mapping relationship between process parameters and weld bead characteristics; In a case where the recommended weld bead output by the deep learning model belongs to a weld bead in the three-dimensional model, the recommended weld bead output by the deep learning model is determined as the selected weld bead that needs to be welded.

3. The method according to claim 1 or 2, wherein: The method of determining a target welding position according to the three-dimensional model with weld bead annotations and the selected weld bead to be welded using a welding position determination method corresponding to the functional configuration includes: In the case where the function configuration enables the self-positioning function of the handheld sensor device, the position information of the selected weld bead to be welded is determined from the three-dimensional model with weld bead annotations, and the position information of the selected weld bead to be welded is determined as the target welding position; or When the function is configured such that the self-positioning function of the handheld sensor device is not enabled, a three-dimensional point cloud model of the target workpiece scanned by the welding robot body is obtained, and the three-dimensional point cloud model scanned by the welding robot body is aligned with the three-dimensional model with weld bead annotations to obtain an alignment result, and the target welding position is determined based on the alignment result in combination with the selected weld bead to be welded.

4. The method according to claim 1, wherein The step of inputting the scanned image and its corresponding point cloud data into a pre-trained neural network model to obtain weld bead prediction information includes: Acquiring depth information based on the point cloud data; Obtaining a color image with depth information based on the RGB information of the scanned image and the depth information, and obtaining a grayscale image with depth information based on the grayscale information of the scanned image and the depth information; Performing feature extraction on the color image with depth information and the grayscale image with depth information based on the encoder in the neural network model to obtain multimodal fusion features; Decoding the multimodal fusion features based on the decoder in the neural network model to obtain 2D coordinate points of the weld; The 2D coordinate points of the weld are mapped into a 3D space to obtain the 3D coordinate points of the weld, and the weld bead prediction information is determined based on the 3D coordinate points of the weld.

5. The method according to claim 1, wherein The method further comprises: The target welding position and the welding robot path planning result are superimposed and displayed, and the welding path is corrected based on the analysis and comparison results of the target welding position and the welding robot path planning result.

6. An interactive device applied to robot welding scenes, characterized in that: include: An acquisition module, configured to acquire a scanned image of a target workpiece and point cloud data corresponding to the scanned image based on a handheld sensor device; A building module, configured to build a three-dimensional model of the target workpiece based on the scanned image and its corresponding point cloud data; a prediction module, configured to input the scanned image and its corresponding point cloud data into a pre-trained neural network model to obtain weld bead prediction information, wherein the weld bead prediction information includes three-dimensional position information of the predicted weld bead; wherein the input of the neural network model includes a scanned image of a target workpiece and its corresponding point cloud data; the neural network model obtains a color image with depth information and a grayscale image with depth information based on the depth information in the scanned image and its corresponding point cloud data; performs feature extraction on the color image with depth information and the grayscale image with depth information to obtain multimodal fusion features; and decodes the multimodal fusion features to output three-dimensional position information of N predicted weld beads, where N is a positive integer; a marking module, configured to mark the three-dimensional model of the target workpiece based on the weld bead prediction information to obtain a three-dimensional model with weld bead markings; A determination module is used to select a weld bead to be welded from a three-dimensional model with weld bead annotations based on deep learning technology or manual selection, determine a functional configuration of the self-positioning function of the handheld sensor device, and determine a target welding position based on the three-dimensional model with weld bead annotations and the selected weld bead to be welded using a welding position determination method corresponding to the functional configuration, so that the welding robot performs welding based on the target welding position.

7. An interactive system applied to robot welding scenarios, characterized in that: include: A handheld sensor device is used to scan a target workpiece to obtain a scanned image of the target workpiece and point cloud data corresponding to the scanned image; An interactive device is configured to establish a three-dimensional model of the target workpiece based on the scanned image and its corresponding point cloud data, and input the scanned image and its corresponding point cloud data into a pre-trained neural network model to obtain weld bead prediction information, wherein the weld bead prediction information includes three-dimensional position information of the predicted weld bead; wherein the input of the neural network model includes the scanned image of the target workpiece and its corresponding point cloud data, the neural network model obtains a color image with depth information and a grayscale image with depth information based on the depth information in the scanned image and its corresponding point cloud data, performs feature extraction on the color image with depth information and the grayscale image with depth information to obtain multimodal fusion features, and decodes the multimodal fusion features to output three-dimensional position information of N predicted weld beads, where N is a positive integer; The interactive device is further configured to mark the three-dimensional model of the target workpiece based on the weld bead prediction information to obtain a three-dimensional model with weld bead markings; The interactive device is further configured to select a weld bead to be welded from the three-dimensional model with weld bead annotations based on deep learning technology or manual selection, determine a functional configuration of the self-positioning function of the handheld sensor device, and determine a target welding position using a welding position determination method corresponding to the functional configuration based on the three-dimensional model with weld bead annotations and the selected weld bead to be welded; A welding robot is used for performing welding based on the target welding position.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

9. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 5.

Images