Intelligent recommendation method for process parameters of manufacturing process, model training method and device

Abstract

The present disclosure is about a method, a model training method and a device for intelligently recommending process parameters of a manufacturing process. The recommendation method includes: obtaining multimodal information, the multimodal information including temperature information of a manufacturing area, humidity information of a manufacturing area, RGB image information containing a workpiece, three-dimensional point cloud information containing a workpiece, and at least two of the workpiece-related parameter information input by a user; identifying a target object based on at least one of the RGB image information and the three-dimensional point cloud information to obtain geometric parameter information of the target object; predicting process parameters based on at least one of the temperature information, humidity information, and workpiece-related parameter information in combination with the geometric parameter information of the target object to obtain candidate process parameter information; adjusting the candidate process parameter information based on user intervention feedback to obtain process parameter information to be used when manufacturing a workpiece by a manufacturing process. The present disclosure can provide customized process parameter recommendations for a manufacturing process.

Description

Technical Field

[0001] The present disclosure relates to the field of manufacturing technology, deep learning technology in the field of artificial intelligence, and more particularly to a method for intelligently recommending process parameters for a manufacturing process, a model training method, and a device. Background Art

[0002] In related technologies, the selection of manufacturing process parameters mainly relies on manual experience and some simple rule systems. These methods usually require manufacturing process engineers to manually debug based on past experience to determine the best process parameters, or rely on a limited process parameter database for selection and knowledge reasoning.

[0003] Therefore, there is currently a lack of effective means for comprehensively and accurately recommending process parameters. Summary of the invention

[0004] The embodiments of the present disclosure provide a method for intelligently recommending process parameters of a manufacturing process, a method for model training, and a device.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for intelligently recommending process parameters of a manufacturing process is provided, comprising:

[0006] Acquiring multimodal information, the multimodal information including at least two of temperature information of a manufacturing area, humidity information of the manufacturing area, RGB image information containing a workpiece, three-dimensional point cloud information containing the workpiece, and workpiece-related parameter information input by a user; the workpiece may form a target object after being manufactured and processed by the manufacturing process;

[0007] According to at least one of the RGB image information and the three-dimensional point cloud information, the target object is identified to obtain geometric parameter information of the target object;

[0008] Predicting process parameters according to at least one of the temperature information, the humidity information, and the workpiece-related parameter information in combination with the geometric parameter information of the target object to obtain candidate process parameter information;

[0009] The candidate process parameter information is adjusted according to user intervention feedback to obtain process parameter information to be adopted when the manufacturing process manufactures the workpiece.

[0010] According to a second aspect of an embodiment of the present disclosure, a training method for a process parameter intelligent recommendation model is provided, wherein the process parameter intelligent recommendation model includes a target object recognition network and a process parameter recommendation network, and the training method includes:

[0011] Acquire training data, the training data including temperature data of a manufacturing area, humidity data of the manufacturing area, RGB image data of a workpiece, three-dimensional point cloud data of the workpiece, workpiece-related parameter data, geometric parameter label data of the target object, and process parameter label data of a manufacturing process;

[0012] Inputting the RGB image data and the three-dimensional point cloud data into the target object recognition network to recognize the target object, and obtaining geometric parameter prediction data of the target object;

[0013] Inputting the geometric parameter prediction data, the temperature data, the humidity data, and the workpiece related parameter data into the process parameter recommendation network to predict the process parameters, and obtaining candidate process parameter prediction data;

[0014] generating a first loss value according to the candidate process parameter prediction data and the process parameter label data, and generating a second loss value according to the geometric parameter prediction data and the geometric parameter label data;

[0015] The process parameter intelligent recommendation model is trained based on the first loss value and the second loss value.

[0016] According to a third aspect of an embodiment of the present disclosure, a method for intelligently recommending process parameters of a manufacturing process is provided, comprising:

[0017] Acquiring multimodal information, the multimodal information including at least two of temperature information of a manufacturing area, humidity information of the manufacturing area, RGB image information containing a workpiece, three-dimensional point cloud information containing the workpiece, and workpiece-related parameter information input by a user; the workpiece may form a target object after being manufactured and processed by the manufacturing process;

[0018] Inputting the multimodal information into a pre-trained process parameter intelligent recommendation model to obtain candidate process parameter information output by the process parameter intelligent recommendation model; wherein the process parameter intelligent recommendation model is a model trained based on the method described in the second aspect above;

[0019] The candidate process parameter information is adjusted according to user intervention feedback to obtain process parameter information to be adopted when the manufacturing process manufactures the workpiece.

[0020] According to a fourth aspect of an embodiment of the present disclosure, there is provided a device for intelligently recommending process parameters of a manufacturing process, comprising:

[0021] an acquisition module, used for acquiring multimodal information, wherein the multimodal information includes at least two items of temperature information of a manufacturing area, humidity information of the manufacturing area, RGB image information containing a workpiece, three-dimensional point cloud information containing the workpiece, and workpiece-related parameter information input by a user; the workpiece may form a target object after being manufactured and processed by the manufacturing process;

[0022] an identification module, used to identify the target object according to at least one of the RGB image information and the three-dimensional point cloud information, and obtain geometric parameter information of the target object;

[0023] A process recommendation module, configured to predict process parameters according to at least one of the temperature information, the humidity information, and the workpiece-related parameter information in combination with the geometric parameter information of the target object, to obtain candidate process parameter information;

[0024] The adjustment module is used to adjust the candidate process parameter information according to user intervention feedback to obtain the process parameter information to be adopted when the manufacturing process manufactures the workpiece.

[0025] According to a fifth aspect of an embodiment of the present disclosure, a training device for a process parameter intelligent recommendation model is provided, wherein the process parameter intelligent recommendation model includes a target object recognition network and a process parameter recommendation network, and the training device includes:

[0026] an acquisition module, used to acquire training data, the training data including temperature data of a manufacturing area, humidity data of the manufacturing area, RGB image data containing a workpiece, three-dimensional point cloud data containing the workpiece, workpiece-related parameter data, geometric parameter label data of the target object, and process parameter label data of a manufacturing process;

[0027] A recognition module, used for inputting the RGB image data and the three-dimensional point cloud data into the target object recognition network to recognize the target object and obtain geometric parameter prediction data of the target object;

[0028] A prediction module, used for inputting the geometric parameter prediction data, the temperature data, the humidity data, and the workpiece related parameter data into the process parameter recommendation network to predict the process parameters and obtain candidate process parameter prediction data;

[0029] A generating module, configured to generate a first loss value according to the candidate process parameter prediction data and the process parameter label data, and to generate a second loss value according to the geometric parameter prediction data and the geometric parameter label data;

[0030] A training module is used to train the process parameter intelligent recommendation model based on the first loss value and the second loss value.

[0031] According to a sixth aspect of an embodiment of the present disclosure, there is provided a device for intelligently recommending process parameters of a manufacturing process, comprising:

[0032] A first acquisition module is used to acquire multimodal information, wherein the multimodal information includes at least two of temperature information of a manufacturing area, humidity information of the manufacturing area, RGB image information containing a workpiece, three-dimensional point cloud information containing the workpiece, and workpiece-related parameter information input by a user; the workpiece can be formed into a target object after being manufactured and processed by the manufacturing process;

[0033] A second acquisition module, used for inputting the multimodal information into a pre-trained process parameter intelligent recommendation model to obtain candidate process parameter information output by the process parameter intelligent recommendation model; wherein the process parameter intelligent recommendation model is a model trained based on the method described in the second aspect above;

[0034] The adjustment module is used to adjust the candidate process parameter information according to user intervention feedback to obtain the process parameter information to be adopted when the manufacturing process manufactures the workpiece.

[0035] According to a seventh aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0036] at least one processor; and

[0037] a memory communicatively connected to the at least one processor; wherein,

[0038] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the methods described in the first, second and third aspects.

[0039] According to an eighth aspect of an embodiment of the present disclosure, a storage medium is provided, wherein the storage medium stores instructions, and when the instructions are executed on an electronic device, the electronic device executes the methods described in the first, second and third aspects above.

[0040] According to a ninth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method described in the first, second and third aspects are implemented.

[0041] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: using the workpiece-related parameter information input by the user as prior information, combining the prior information input by the user and the real-time data collection of the manufacturing area environment by multiple sensors, and predicting the process parameters based on user interaction and real-time data, it is possible to comprehensively and accurately predict and recommend the process parameter information of the manufacturing process, which can not only improve the quality of the manufacturing process, but also significantly improve production efficiency, while reducing the reliance on manual experience. In addition, the present disclosure takes into account the differences in the requirements for manufacturing processes in different manufacturing scenarios (such as welding scenarios), and the present disclosure has innovated in the user interaction method, allowing users to make fine adjustments based on the recommended manufacturing process parameters to adapt to the requirements of different manufacturers.

[0042] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0044] Figure 1 The figure is a flow chart of a method for intelligently recommending process parameters of a manufacturing process according to an exemplary embodiment.

[0045] Figure 2 The figure is a flow chart of a method for training a process parameter intelligent recommendation model according to an exemplary embodiment.

[0046] Figure 3 The figure is a flow chart of a method for intelligently recommending process parameters of a manufacturing process according to an exemplary embodiment.

[0047] Figure 4 It is a schematic diagram of the structure of a weld recognition network according to an exemplary embodiment.

[0048] Figure 5 is a schematic structural diagram of a process parameter recommendation network according to an exemplary embodiment.

[0049] Figure 6 The figure is a flowchart showing a process from data collection to parameter recommendation according to an exemplary embodiment.

[0050] Figure 7 It is a structural diagram of a process parameter intelligent recommendation model according to an exemplary embodiment.

[0051] Figure 8 The figure is a block diagram of a device for intelligently recommending process parameters of a manufacturing process according to an exemplary embodiment.

[0052] Fig. 9 It is a block diagram of a training device for a process parameter intelligent recommendation model according to an exemplary embodiment.

[0053] Fig.10 The figure is a block diagram of a device for intelligently recommending process parameters of a manufacturing process according to an exemplary embodiment.

[0054] Fig.11 is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0055] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0056] The embodiments of the present disclosure are not exhaustive, but are only illustrative of some embodiments, and are not intended to be a specific limitation on the scope of protection of the present disclosure. In the absence of contradiction, each step in a certain embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a certain embodiment can also be implemented as an independent embodiment, and the order of the steps in a certain embodiment can be arbitrarily exchanged. In addition, the optional implementation methods in a certain embodiment can be arbitrarily combined; in addition, the embodiments can be arbitrarily combined, for example, some or all of the steps of different embodiments can be arbitrarily combined, and a certain embodiment can be arbitrarily combined with the optional implementation methods of other embodiments.

[0057] The prefixes such as "first" and "second" in the embodiments of the present disclosure are only used to distinguish different description objects, and do not constitute restrictions on the position, order, priority, quantity or content of the description objects. The statement of the description object refers to the description in the context of the claims or embodiments, and should not constitute unnecessary restrictions due to the use of prefixes. For example, if the description object is a "field", the ordinal number before the "field" in the "first field" and the "second field" does not limit the position or order between the "fields", and the "first" and "second" do not limit whether the "fields" they modify are in the same message, nor do they limit the order of the "first field" and the "second field". For another example, if the description object is a "level", the ordinal number before the "level" in the "first level" and the "second level" does not limit the priority between the "levels". For another example, the number of description objects is not limited by the ordinal number, and can be one or more. Taking the "first device" as an example, the number of "devices" can be one or more. In addition, the objects modified by different prefixes may be the same or different. For example, if the description object is "device", then the "first device" and the "second device" may be the same device or different devices, and their types may be the same or different. For another example, if the description object is "information", then the "first information" and the "second information" may be the same information or different information, and their contents may be the same or different.

[0058] In each embodiment of the present disclosure, if there is no special explanation and logical conflict, the terms and / or descriptions between the embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form a new embodiment according to their inherent logical relationship. The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit the present disclosure.

[0059] It should be noted that in the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0060] It should also be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0061] It is worth noting that in the embodiments of the present disclosure, certain software, components, models, etc. that already exist in the industry may be mentioned. They should be considered as exemplary and their purpose is only to illustrate the feasibility of implementing the technical solution of the present disclosure, but it does not mean that the applicant has or will necessarily use the solution.

[0062] In the related art, the selection of process parameters for manufacturing processes (such as welding processes) mainly relies on manual experience and some simple rule systems. These methods usually require manufacturing process engineers to manually debug based on past experience to determine the best process parameters, or rely on a limited process parameter database for selection and knowledge reasoning. Therefore, there is currently a lack of effective means for how to comprehensively and accurately recommend process parameters.

[0063] Based on this, the core goal of the present disclosure is to solve a key technical problem in traditional manufacturing processes: the formulation of manufacturing process parameters. Taking the manufacturing process as a welding process as an example, in existing welding practices, in order to ensure the quality of the weld, the welding process parameters of the specific weld must be pre-evaluated. This process is not only time-consuming, but also requires extremely high professional skills from welders or welding robot operators, and also leads to instability in welding quality and low production efficiency. In addition, this process is overly dependent on manual experience and lacks flexibility and adaptability. In order to overcome these limitations, the embodiment of the present disclosure provides an intelligent recommendation solution for process parameters of a manufacturing process. Through intelligent means, it can automatically analyze and recommend the best manufacturing process parameters, thereby improving the accuracy and efficiency of the selection of manufacturing process parameters and reducing dependence on manual experience.

[0064] The following describes the process parameter intelligent recommendation method, model training method and device of the manufacturing process of the embodiment of the present disclosure with reference to the accompanying drawings.

[0065] Figure 1 FIG. 1 is a flow chart of a method for intelligently recommending process parameters of a manufacturing process according to an exemplary embodiment. Figure 1 As shown, the process parameter intelligent recommendation method may include but is not limited to the following steps.

[0066] In step 101, multimodal information is acquired, and the multimodal information includes at least two items of temperature information of a manufacturing area, humidity information of a manufacturing area, RGB image information containing a workpiece, three-dimensional point cloud information containing a workpiece, and workpiece-related parameter information input by a user.

[0067] In some embodiments, the multimodal information may include, but is not limited to, temperature information of the manufacturing area, humidity information of the manufacturing area, RGB image information of the workpiece, three-dimensional point cloud information of the workpiece, and workpiece-related parameter information input by a user. In some embodiments, the workpiece may be formed into a target object after being processed by a manufacturing process. Exemplarily, taking welding as an example, the manufacturing process may be a workpiece to be welded, and the target object may be a weld, which is a joint where the workpieces to be welded are connected together by a welding process.

[0068] In some embodiments, the temperature information of the manufacturing area can be obtained by temperature sensing. For example, the temperature of the manufacturing area can be collected by a temperature sensor in the manufacturing area, so that the temperature information of the manufacturing area collected by the temperature sensor can be obtained in real time.

[0069] In some embodiments, the humidity information of the manufacturing area can be collected by a humidity sensor. For example, the humidity of the manufacturing area can be collected by a humidity sensor in the manufacturing area, so that the humidity information of the manufacturing area collected by the humidity sensor can be obtained in real time.

[0070] In some embodiments, the above-mentioned RGB image information containing the workpiece can be obtained by photographing the workpiece through a visual sensor module to obtain an RGB image. The above-mentioned three-dimensional point cloud information containing the workpiece can be obtained by photographing the workpiece through a visual sensor module to obtain a 3D point cloud. Exemplarily, taking the manufacturing process as a welding process as an example, the workpiece to be welded can be photographed by a visual sensor module to obtain 2D RGB image information and 3D point cloud information of the workpiece to be welded. Exemplarily, the 2D RGB image information and 3D point cloud information can be used to predict the precise position of the target object and obtain the three-dimensional geometric features near the target object. For example, taking the target object as a weld as an example, the 2D RGB image information and 3D point cloud information of the workpiece to be welded photographed by the visual sensor module can be used to predict the precise position of the weld and obtain the three-dimensional geometric features near the weld.

[0071] In some embodiments, the workpiece-related parameter information input by the user can be used as prior information. Exemplarily, a configuration input interface can be provided, through which the user can input the relevant parameters of the workpiece. In some embodiments, taking the manufacturing process as a welding process as an example, the workpiece-related parameter information may include but is not limited to at least one of the material (or plate type) of the workpiece to be welded, the plate thickness, the type of shielding gas, the weld joint form, the weld leg size, and the weld length. It should be noted that in the actual welding operation process, for the same batch of workpieces, the relevant parameter information of the workpiece is often deterministic, so the input of the relevant parameter information once can be reused until the next batch of workpieces is replaced.

[0072] In step 102, the target object is identified based on at least one of the RGB image information and the three-dimensional point cloud information to obtain geometric parameter information of the target object.

[0073] In some embodiments, the target object can be identified based on the RGB image information and the three-dimensional point cloud information to obtain the geometric parameter information of the target object. For example, taking the manufacturing process as a welding process and the target object as a weld as an example, the weld can be identified based on the RGB image information and the three-dimensional point cloud information of the workpiece to be welded to obtain the geometric parameter information of the weld. In some embodiments, the terms "geometric parameter information", "geometric features", "geometric parameters" and the like can be interchangeable.

[0074] In some embodiments, the RGB image information and the three-dimensional point cloud information may be aligned. Feature extraction is performed on the aligned RGB image information to obtain multi-scale features fused with depth information, and feature extraction is performed on the aligned three-dimensional point cloud information to obtain point cloud features fused with RGB information. A decoding operation is performed based on the multi-scale features fused with depth information and the point cloud features fused with RGB information to obtain geometric parameter information of the target object.

[0075] In some embodiments, based on the pre-trained multimodal encoder in the target object recognition network, features of the aligned RGB image information and the three-dimensional point cloud information can be extracted respectively to obtain multi-scale features of the fused depth information and point cloud features of the fused RGB information. Based on the decoder in the target object recognition network, the multi-scale features of the fused depth information and the point cloud features of the fused RGB information are decoded to obtain geometric parameter information of the target object. The input of the target object recognition network includes the aligned RGB image information and the aligned three-dimensional point cloud information, the output of the target object recognition network includes the geometric parameter information of the target object, and the target object recognition network has learned the mapping relationship between the aligned RGB image information and the three-dimensional point cloud information and the geometric parameter information of the target object.

[0076] Exemplarily, the target object recognition network can be a pre-trained model. After the RGB image information and the three-dimensional point cloud information are aligned, the aligned RGB image information and the aligned three-dimensional point cloud information can be input into the target object recognition network. The multimodal encoder in the target object recognition network can perform feature extraction on the input RGB image information and the three-dimensional point cloud information respectively to obtain multi-scale features (integrated with depth information) and point cloud features (integrated with RGB information). The multi-scale features and point cloud features are output through the decoder in the target object recognition network to output the prediction results of multiple branches to obtain the geometric parameter information of the target object. The number of the multiple branches is related to the type of information included in the geometric parameter information. For example, if the geometric parameter information includes 5 parameter information, the decoder will output the prediction results of 5 branches, and the prediction results of the 5 branches correspond to the corresponding parameter information respectively.

[0077] In some embodiments, taking the manufacturing process as a welding process and the target object as a weld as an example, the geometric parameter information of the weld may include but is not limited to: at least one of: a 2D mask, a category, a 3D trajectory of a welding line composed of multiple key points, a weld angle, a welding posture, a groove depth, a groove angle, and a groove gap.

[0078] In step 103, the process parameters are predicted based on at least one of the temperature information, the humidity information, and the workpiece-related parameter information in combination with the geometric parameter information of the target object to obtain candidate process parameter information.

[0079] In some embodiments, the process parameters can be predicted based on the temperature information, humidity information, workpiece-related parameter information and the geometric parameter information of the target object to obtain candidate process parameter information. In some embodiments, the temperature information, humidity information, workpiece-related parameter information and the geometric parameter information of the target object can be encoded respectively, and the encoding results of the temperature information, humidity information, workpiece-related parameter information and the geometric parameter information of the target object can be spliced ​​to obtain multimodal features, and the process parameters are predicted based on the multimodal features to obtain candidate process parameter information. Exemplarily, the temperature information, humidity information, workpiece-related parameter information and the geometric parameter information of the target object have numerical parameter information and category parameter information, for example, the numerical parameter information includes: plate thickness (in mm), weld leg size (in mm), weld length (in mm), weld angle (in degrees), groove depth (in mm), groove angle (in degrees), groove interval (in mm), temperature information (in degrees C), humidity information (in %). Category parameter information includes: workpiece plate type, shielding gas type, weld joint form, welding posture. For numerical parameter information, standardization or normalization processing is performed to obtain the encoding results of each numerical parameter information; for categorical parameter information, one-hot encoding can be used to convert it into a binary sequence with a length equal to the total number of categories, so as to obtain the encoding results of each categorical parameter information.

[0080] In a possible implementation, a deep learning model can be used to predict process parameters. In some embodiments, multimodal features can be input into a pre-trained process parameter recommendation network to predict process parameters and obtain candidate process parameter information. The input of the process parameter recommendation network includes multimodal features, the output of the multimodal features includes candidate process parameter information, and the multimodal features have learned the mapping relationship between the multimodal features and the candidate process parameter information.

[0081] For example, the process parameters can be predicted by combining the temperature information, humidity information, workpiece-related parameter information and the geometric parameter information of the target object through a pre-trained process parameter recommendation network to obtain candidate process parameter information output by the process parameter recommendation network. In some embodiments, the process parameter recommendation network can adopt a multilayer perceptron (MLP) architecture, but is not limited thereto.

[0082] In some embodiments, the process parameter recommendation network can be pre-obtained in the following manner: obtaining training data, which may include geometric parameter data of the target object, temperature data and humidity data of the manufacturing area, workpiece-related parameter data, and process parameter label data of the manufacturing process; encoding the temperature data, humidity data, workpiece-related parameter data, and geometric parameter data of the target object respectively; splicing the encoding results of the temperature data, humidity data, workpiece-related parameter data, and geometric parameter data of the target object to obtain multimodal features; inputting the multimodal features into the process parameter recommendation network to predict the process parameters to obtain candidate process parameter prediction data; generating a first loss value based on the candidate process parameter prediction data and the process parameter label data, and training the process parameter recommendation network based on the first loss value.

[0083] In some embodiments, the process parameter label data of the manufacturing process is derived from at least one of the process library of the manufacturing process and the candidate process parameter information adjusted based on user intervention feedback. Exemplarily, a data acquisition device can be built using a visual sensor module and a temperature and humidity sensor, and a batch of welding data can be collected in advance as an initial data set for initialization training of the model. During the acquisition process of each weld, the temperature and humidity are obtained, the plate type, plate thickness, and type of shielding gas are recorded, the geometric parameters of the groove are measured, and the workpiece-related parameter data are obtained. Query the existing welding process library to obtain the welding process parameters required for welding this weld as label data. Integrate the above information into the data set as a sample. During use, collect multimodal inputs and the final output after user fine-tuning as new samples to expand the data set, and continuously iterate the process parameter recommendation network.

[0084] In step 104, the candidate process parameter information is adjusted according to the user intervention feedback to obtain the process parameter information to be used when the manufacturing process manufactures the workpiece.

[0085] In some embodiments, user intervention feedback information on candidate process parameter information can be obtained, and the candidate process parameter information can be adjusted according to the user intervention feedback information to obtain the process parameter information to be used when the manufacturing process manufactures the workpiece, and the process parameter information is the best process parameter suitable for the workpiece. Exemplarily, a process parameter fine-tuning interface can be provided, through which the user can adjust the recommended candidate process parameter information, through which the intervention feedback information output by the user can be obtained, and the process parameter information adjusted by the user intervention feedback is output as the final process parameter, that is, the process parameter information predicted by the user fine-tuning is finally determined as the process parameter used for this manufacturing. Exemplarily, taking the manufacturing process as welding as an example, the process parameter information used for this welding that is finally determined can be provided to the mechanical arm in the welding machine and the welding robot, so that the mechanical arm in the welding machine and the welding robot can weld the corresponding workpiece based on the process parameters.

[0086] In the above embodiment, the workpiece-related parameter information input by the user is used as prior information, combined with the prior information input by the user and the real-time data collection of the manufacturing area environment by multiple sensors, and the process parameters are predicted based on user interaction and real-time data, which can comprehensively and accurately predict and recommend the process parameter information of the manufacturing process, which can not only improve the quality of the manufacturing process, but also significantly improve production efficiency, while reducing the dependence on manual experience. In addition, the present disclosure takes into account the differences in the requirements for manufacturing processes in different manufacturing scenarios (such as welding scenarios), and the present disclosure has innovated the user interaction method, allowing users to make fine adjustments based on the recommended manufacturing process parameters to adapt to the requirements of different manufacturers.

[0087] Figure 2 FIG. 1 is a flow chart of a method for training a process parameter intelligent recommendation model according to an exemplary embodiment. It should be noted that, in some embodiments, the process parameter intelligent recommendation model may include but is not limited to a target object recognition network and a process parameter recommendation network. Figure 2 As shown, the training method of the process parameter intelligent recommendation model may include but is not limited to the following steps.

[0088] In step 201, training data is obtained, and the training data may include but is not limited to temperature data of the manufacturing area, humidity data of the manufacturing area, RGB image data containing the workpiece, three-dimensional point cloud data containing the workpiece, workpiece-related parameter data, geometric parameter label data of the target object, and process parameter label data of the manufacturing process. In some embodiments, the above-mentioned workpiece can form a target object after being processed by the manufacturing process. Exemplarily, taking the manufacturing process as welding, the workpiece can be a workpiece to be welded, and the target object can be a weld, which is a joint where the workpieces to be welded are connected together by the welding process.

[0089] In some embodiments, the geometric parameter label data of the target object may be derived from the geometric parameter data obtained by the user from the actual measurement of the target object in the workpiece. For example, taking welding as an example, the user may actually measure the weld in the workpiece to obtain the geometric parameters of the weld, and use these measured geometric parameters as the geometric parameter label data of the weld for model training.

[0090] In some embodiments, the process parameter label data of the manufacturing process is derived from at least one of the process library of the manufacturing process and the candidate process parameter information adjusted based on user intervention feedback. Exemplarily, a data acquisition device can be built using a visual sensor module and a temperature and humidity sensor, and a batch of welding data can be collected in advance as an initial data set for initialization training of the model. During the acquisition process of each weld, the temperature and humidity are obtained, the plate type, plate thickness, and type of shielding gas are recorded, the geometric parameters of the groove are measured, and the workpiece-related parameter data are obtained. Query the existing welding process library to obtain the welding process parameters required for welding this weld as label data. Integrate the above information into the data set as a sample. During use, collect multimodal input and the final output after user fine-tuning as new samples to expand the data set, and continuously iterate the model.

[0091] In some embodiments, the temperature data of the manufacturing area, the humidity data of the manufacturing area, the RGB image data containing the workpiece, the three-dimensional point cloud data containing the workpiece, and the workpiece-related parameter data can be obtained by referring to the above Figure 1 The description of the relevant part in step 101 is not repeated here.

[0092] In step 202, the RGB image data and the three-dimensional point cloud data are input into a target object recognition network to recognize the target object, and obtain geometric parameter prediction data of the target object.

[0093] In some embodiments, the RGB image data and the three-dimensional point cloud data can be aligned. The aligned RGB image data and the three-dimensional point cloud data are input into the target object recognition network. The multimodal encoder in the target object recognition network extracts features from the aligned RGB image data and the three-dimensional point cloud data, respectively, to obtain multi-scale features that fuse depth information and point cloud features that fuse RGB information. The decoder in the target object recognition network decodes the multi-scale features that fuse depth information and the point cloud features that fuse RGB information to obtain geometric parameter data of the target object. In some embodiments, the terms "parameter information", "parameter data", "parameter" and the like can be interchangeable. In some embodiments, the terms "information" and "data" and the like can be interchangeable.

[0094] In step 203, the geometric parameter prediction data, temperature data, humidity data, and workpiece related parameter data are input into the process parameter recommendation network to predict the process parameters and obtain candidate process parameter prediction data.

[0095] In some embodiments, the geometric parameter prediction data, temperature data, humidity data, and workpiece-related parameter data can be encoded separately, and the encoding results of the geometric parameter prediction data, temperature data, humidity data, and workpiece-related parameter data can be spliced ​​to obtain multimodal features. The process parameter recommendation network predicts the process parameters based on the multimodal features, and the prediction results of the candidate process parameter prediction data are used as candidate process parameter prediction data.

[0096] In step 204, a first loss value is generated based on the candidate process parameter prediction data and the process parameter label data, and a second loss value is generated based on the geometric parameter prediction data and the geometric parameter label data.

[0097] In some embodiments, a first loss function may be used to calculate a first loss value based on candidate process parameter prediction data and process parameter label data. Exemplarily, the first loss function may be a cross entropy loss function, but is not limited thereto. Exemplarily, the process parameters may include multiple parameters, each parameter corresponds to a loss value, and the loss values ​​of all types of parameters are weighted and summed to obtain a value as the first loss value.

[0098] In some embodiments, a second loss value may be calculated using a second loss function based on the geometric parameter prediction data and the geometric parameter label data. Exemplarily, the second loss value may be a cross entropy loss function, but is not limited thereto. Exemplarily, the geometric parameter may include multiple parameters, each parameter corresponds to a loss value, and the loss values ​​of all types of parameters are weighted and summed to obtain a value as the second loss value.

[0099] In step 205, a process parameter intelligent recommendation model is trained based on the first loss value and the second loss value.

[0100] Exemplarily, the first loss value can be fed back to the process parameter recommendation network, and the second loss value can be fed back to the target object recognition network. During the training process, the process parameter recommendation network and the target object recognition network are jointly trained until the model training end condition is met, thereby obtaining a trained process parameter intelligent recommendation model. Exemplarily, the process parameter recommendation network and the target object recognition network can be trained together. For example, in a round of model training, the first loss value can be fed back to the process parameter recommendation network to adjust the network parameters of the process parameter recommendation network, and the second loss value can be fed back to the target object recognition network to adjust the network parameters of the target object recognition network. Exemplarily, the process parameter recommendation network and the target object recognition network can be trained separately. For example, the target object recognition network can be trained first. After the training of the target object recognition network is completed, the network parameters of the target object recognition network are fixed, and then the process parameter recommendation network is trained. When the training of the process parameter recommendation network is completed, the training of the process parameter intelligent recommendation model can be completed.

[0101] Figure 3 FIG. 1 is a flow chart of a method for intelligently recommending process parameters of a manufacturing process according to an exemplary embodiment. Figure 3 As shown, the method for intelligently recommending process parameters of the manufacturing process may include but is not limited to the following steps.

[0102] In step 301, multimodal information is obtained, and the multimodal information includes at least two of temperature information of a manufacturing area, humidity information of a manufacturing area, RGB image information of a workpiece, three-dimensional point cloud information of a workpiece, and workpiece-related parameter information input by a user. Exemplarily, the workpiece may be formed into a target object after being processed by a manufacturing process.

[0103] In step 302, the multimodal information is input into a pre-trained process parameter intelligent recommendation model to obtain candidate process parameter information output by the process parameter intelligent recommendation model.

[0104] In some embodiments, the above-mentioned process parameter intelligent recommendation model is a model trained by the training method of the process parameter intelligent recommendation model based on any of the above-mentioned embodiments, which will not be repeated here.

[0105] In step 303, the candidate process parameter information is adjusted according to the user intervention feedback to obtain the process parameter information to be used when the manufacturing process manufactures the workpiece.

[0106] In the above embodiment, an end-to-end architecture design is adopted to dynamically adjust parameters according to different materials and environments. Most parameters are automatically generated through sensor data, and only a small amount of prior information that cannot be collected needs to be manually input. This can reduce manual debugging time, quickly complete parameter settings, greatly reduce the operating threshold, and improve welding operation efficiency. It is suitable for non-standard welding scenarios.

[0107] In order to facilitate those skilled in the art to understand the present disclosure, the manufacturing process will be described in detail below by taking welding as an example.

[0108] It is worth noting that the present disclosure relates to an end-to-end intelligent welding process recommendation method based on user interaction and real-time data. The core of the method is a welding process parameter recommendation module based on a deep learning model. The main process includes the collection and calculation of the output parameters of the recommendation module, the prediction of welding process parameters, user intervention fine-tuning and confirmation, and the final process parameter output. In each welding process, the 2D RGB image and 3D point cloud near the weld are collected by the visual sensor module, and the geometric parameters of the weld are identified by the weld recognition module. The user-specified prior information and the temperature and humidity information collected by the sensor in real time are provided as input to the welding process parameter recommendation module, which predicts the process parameters. At this time, the user further fine-tunes the predicted process parameters, and finally determines the process parameters used for this welding and provides them to the welding machine and robotic arm. The innovations of the present disclosure are mainly reflected in the following aspects:

[0109] 1. Multimodal data collection and preprocessing: Different from the solutions in related technologies, the present disclosure uses multimodal input parameters as the model input source of the recommendation system, specifically including:

[0110] (a) Visual sensor module: The visual sensor module may include but is not limited to an RGB camera module and a depth sensor module capable of acquiring depth information. The depth sensor module may be, for example, a structured light or laser sensor, and the depth information acquired may be in the form of a 2D depth map or a 3D point cloud. Exemplarily, the welding workpiece may be collected by the visual sensor module to obtain a 2D RGB image and a 3D point cloud of the welding workpiece, which are used to predict the precise position of the weld and obtain the three-dimensional geometric features near the weld as one of the input variables of the welding process parameters. Exemplarily, the 2D RGB image and the 3D point cloud may be preprocessed, for example, the holes that may exist in the 3D point cloud may be identified and completed, and outliers may be filtered; the RGB image and the 3D point cloud may be aligned; the pixel values ​​of the RGB image may be normalized and resized according to the input size of the target object recognition network (also called the weld recognition network).

[0111] (b) Temperature and humidity sensors: Real-time acquisition of temperature and humidity near the welding area as one of the input variables of welding process parameters. For example, the temperature and humidity data can be pre-processed, such as filtering the temperature and humidity curves to eliminate noise interference.

[0112] (c) User input: used to determine some prior information that is difficult for the system to identify, such as the material, plate thickness, and type of shielding gas of the welding workpiece, as one of the input variables of the welding process parameters. In the actual welding process, for the same batch of workpieces, these parameters are often deterministic, so one input can be reused until the next batch of workpieces is replaced.

[0113] 2. Automatic extraction of weld geometry features: The significant difference between the present invention and other solutions is that, during the welding process, there is no need to build a high-precision three-dimensional model of the workpiece to be welded in advance, or to manually measure the angles of the baffles on both sides of the weld, the groove and other geometric structure feature dimensions in advance. Instead, the built-in target object recognition network (also called weld recognition network) is used to automatically extract these dimensional features as input to the welding parameter recommendation algorithm. For example, Figure 4 As shown, the RGB image and the 3D point cloud are provided as input to the weld recognition network, and the multi-scale features and point cloud features are extracted by the multimodal encoder (Encoder) in the weld recognition network. The prediction results of multiple branches (headers) are output by the decoder (Decoder) in the weld recognition network. The prediction results of the multiple branches may include but are not limited to: two-dimensional masks, categories, 3D trajectories of welding lines composed of multiple key points, weld angles, welding postures, groove depths and groove angles.

[0114] 3. Welding process parameter prediction: This part uses a deep learning model (the process parameter recommendation network in this article) to predict process parameters. It uses a multi-layer perceptron architecture. The process parameter recommendation network can include three parts: input layer, hidden layer and output layer. Among them:

[0115] Input layer (X): Encode the input parameters. The input parameters can include numerical parameters and categorical parameters. The numerical parameters include: plate thickness, weld leg size, weld length, weld angle, groove depth, groove angle, groove gap, temperature, and humidity. The categorical parameters include: workpiece plate type, shielding gas type, weld joint form, and welding posture. Figure 5 As shown in the figure, for numerical features, the encoder in the process parameter recommendation network standardizes or normalizes the numerical features to achieve encoding of the numerical features; for categorical features, the encoder uses One-Hot Encoding to convert the categorical features into a binary sequence with a length equal to the total number of categories to achieve encoding of the categorical features. The encoder concatenates the encoding results of all input parameters into a vector as the input of the hidden layer.

[0116] Hidden layer (H): Figure 5As shown, the hidden layer in the process parameter recommendation network can be composed of three fully connected layers, each of which is followed by a ReLU activation layer and a Dropout layer. For example, the Dropout rate can be set to 0.2.

[0117] Output layer (O): Figure 5 As shown, the output result of the hidden layer is processed by the activation function and then decoded by the decoder to obtain the predicted process parameters. Among them, a linear activation function can be used for numerical parameters, and a sigmoid activation function can be used for binary classification parameters. Exemplarily, the final process parameters may include but are not limited to: dry extension length (in mm), wire diameter (in mm), pulse duty cycle (in %), welding current (in A), welding voltage (in V), welding speed (in mm / s), whether to swing, swing width (in mm), swing frequency (in Hz), dwell time at both ends during swinging (in s), arc striking distance (in mm), arc striking speed (in mm / s), arc striking current (in A), arc striking voltage (in V), arc closing distance (in mm), arc closing speed (in mm / s), arc closing current (in A), welding gun tilt angle (in °), welding gun front and rear angle (in °).

[0118] 4. Construction of welding process data set and model training: Figure 6 As shown in the figure, the detailed process from data collection to parameter recommendation (including weld geometry parameter recommendation and welding process parameter recommendation) is shown, including the steps of data collection, parameter recommendation, model training, and feedback optimization. The data acquisition equipment is built using the visual sensor module and the temperature and humidity sensor, and a batch of welding data is collected in advance as the initial data set for the initialization training of the model. During the collection process of each weld, RGB images, point clouds, temperature and humidity are obtained, the plate type, plate thickness, and shielding gas type are recorded, and the geometric parameters of the groove are measured. Query the existing welding process library to obtain the welding process parameters required for welding this weld. Integrate the above information into the data set as a sample. During use, collect multimodal input and the final output after user fine-tuning as new samples to expand the data set, and continuously iterate the model.

[0119] In the process of training the process parameter intelligent recommendation model, the weld recognition network and the process parameter recommendation network can be jointly trained. Figure 7As shown, part of the geometric parameter information output by the weld recognition network can be used as the input of the process parameter recommendation network, and the process parameters can be predicted by combining the temperature, humidity and workpiece related parameter information to obtain the predicted process parameters. Among them, each geometric parameter can correspond to a loss value, and the loss values ​​of these geometric parameters can be weighted and summed to obtain the first loss value of the weld recognition network. Each process parameter can correspond to a loss value, and the loss values ​​of these process parameters can be weighted and summed to obtain the second loss value of the process parameter recommendation network. The first loss value and the second loss value are used to train the weld recognition network and the process parameter recommendation network.

[0120] In summary, the present disclosure can improve the quality and consistency of welds through comprehensive and accurate parameter recommendations. In addition, an end-to-end architecture design is adopted to dynamically adjust parameters according to different materials and environments. Most parameters are automatically generated through sensor data, and only a small amount of prior information that cannot be collected needs to be manually input. This can reduce manual debugging time, quickly complete parameter settings, greatly reduce the operating threshold, improve welding operation efficiency, and be suitable for non-standard welding scenarios, thereby improving user experience. The present disclosure quantifies the process used in each welding process into process parameters of multiple dimensions, reduces reliance on manual experience, reduces the risk of human error, and can be easily integrated into the robot automatic welding process. In addition, the welding process parameter database is updated by recording each welding process, and the prediction model is continuously iterated. As the database scale expands and the data distribution continues to improve, the accuracy and generalization of the entire recommendation system can be continuously improved.

[0121] Figure 8 FIG. 1 is a block diagram of a device for intelligently recommending process parameters of a manufacturing process according to an exemplary embodiment. Figure 8 As shown, the process parameter intelligent recommendation device of the manufacturing process may include: an acquisition module 801, an identification module 802, a process recommendation module 803 and an adjustment module 804.

[0122] Among them, the acquisition module 801 is used to acquire multimodal information, and the multimodal information includes at least two of the temperature information of the manufacturing area, the humidity information of the manufacturing area, the RGB image information containing the workpiece, the three-dimensional point cloud information containing the workpiece, and the workpiece-related parameter information input by the user; the workpiece can form a target object after being manufactured and processed by the manufacturing process.

[0123] The recognition module 802 is used to recognize the target object according to at least one of the RGB image information and the three-dimensional point cloud information, and obtain the geometric parameter information of the target object. In some embodiments, the recognition module 802 is used to: align the RGB image information and the three-dimensional point cloud information; extract features from the aligned RGB image information and the three-dimensional point cloud information respectively to obtain multi-scale features fused with depth information and point cloud features fused with RGB information; perform decoding operations based on the multi-scale features fused with depth information and the point cloud features fused with RGB information to obtain the geometric parameter information of the target object.

[0124] In some embodiments, the recognition module 802 is used to: extract features of the aligned RGB image information and the three-dimensional point cloud information respectively based on a pre-trained multimodal encoder in a target object recognition network, and obtain multi-scale features that fuse depth information and point cloud features that fuse RGB information; wherein the input of the target object recognition network includes the aligned RGB image information and the aligned three-dimensional point cloud information, the output of the target object recognition network includes the geometric parameter information of the target object, and the target object recognition network has learned the mapping relationship between the aligned RGB image information and the three-dimensional point cloud information and the geometric parameter information of the target object.

[0125] In some embodiments, the recognition module 802 is used to: based on a decoder in a target object recognition network, decode the multi-scale features fused with depth information and the point cloud features fused with RGB information to obtain geometric parameter information of the target object.

[0126] The process recommendation module 803 is used to predict the process parameters according to at least one of the temperature information, the humidity information and the workpiece related parameter information in combination with the geometric parameter information of the target object to obtain candidate process parameter information.

[0127] In some embodiments, the process recommendation module 803 is used to: respectively encode the temperature information, humidity information, workpiece-related parameter information, and the geometric parameter information of the target object; concatenate the encoding results of the temperature information, humidity information, workpiece-related parameter information, and the geometric parameter information of the target object to obtain multimodal features; predict the process parameters based on the multimodal features to obtain candidate process parameter information.

[0128] In some embodiments, the process recommendation module 803 is used to: input the multimodal features into a pre-trained process parameter recommendation network to predict the process parameters and obtain candidate process parameter information; wherein the input of the process parameter recommendation network includes the multimodal features, the output of the multimodal features includes the candidate process parameter information, and the multimodal features have learned the mapping relationship between the multimodal features and the candidate process parameter information.

[0129] In some embodiments, the process parameter recommendation network is pre-obtained in the following manner: obtaining training data, the training data including geometric parameter data of the target object, temperature data and humidity data of the manufacturing area, workpiece-related parameter data and process parameter label data of the manufacturing process; encoding the temperature data, humidity data, workpiece-related parameter data, and geometric parameter data of the target object respectively; concatenating the encoding results of the temperature data, humidity data, workpiece-related parameter data, and geometric parameter data of the target object to obtain multimodal features; inputting the multimodal features into the process parameter recommendation network to predict the process parameters to obtain candidate process parameter prediction data; generating a first loss value based on the candidate process parameter prediction data and the process parameter label data, and training the process parameter recommendation network based on the first loss value.

[0130] In some embodiments, the process parameter label data of the manufacturing process is derived from at least one of a process library of the manufacturing process and candidate process parameter information adjusted based on user intervention feedback.

[0131] The adjustment module 804 is used to adjust the candidate process parameter information according to the user intervention feedback to obtain the process parameter information to be used when the manufacturing process manufactures the workpiece.

[0132] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0133] Fig. 9 is a block diagram of a training device for a process parameter intelligent recommendation model according to an exemplary embodiment. The process parameter intelligent recommendation model includes a target object recognition network and a process parameter recommendation network, such as Fig. 9 As shown, the training device of the process parameter intelligent recommendation model includes: an acquisition module 901, an identification module 902, a prediction module 903, a generation module 904 and a training module 905.

[0134] The acquisition module 901 is used to acquire training data, and the training data includes temperature data of the manufacturing area, humidity data of the manufacturing area, RGB image data containing the workpiece, three-dimensional point cloud data containing the workpiece, workpiece-related parameter data, geometric parameter label data of the target object, and process parameter label data of the manufacturing process. In some embodiments, the process parameter label data of the manufacturing process is derived from at least one of the process library of the manufacturing process and the candidate process parameter information adjusted based on user intervention feedback.

[0135] The recognition module 902 is used to input the RGB image data and the three-dimensional point cloud data into the target object recognition network to recognize the target object and obtain the geometric parameter prediction data of the target object.

[0136] The prediction module 903 is used to input the geometric parameter prediction data, temperature data, humidity data, and workpiece related parameter data into the process parameter recommendation network to predict the process parameters and obtain candidate process parameter prediction data.

[0137] The generation module 904 is used to generate a first loss value according to the candidate process parameter prediction data and the process parameter label data, and to generate a second loss value according to the geometric parameter prediction data and the geometric parameter label data.

[0138] The training module 905 is used to train the process parameter intelligent recommendation model based on the first loss value and the second loss value.

[0139] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0140] Fig.10 FIG. 1 is a block diagram of a device for intelligently recommending process parameters of a manufacturing process according to an exemplary embodiment. Fig.10 As shown, the process parameter intelligent recommendation device of the manufacturing process may include: a first acquisition module 1001, a second acquisition module 1002 and an adjustment module 1003.

[0141] The first acquisition module 1001 is used to acquire multimodal information, wherein the multimodal information includes at least two of the temperature information of the manufacturing area, the humidity information of the manufacturing area, the RGB image information containing the workpiece, the three-dimensional point cloud information containing the workpiece, and the workpiece related parameter information input by the user; the workpiece can be formed into a target object after being processed by the manufacturing process;

[0142] The second acquisition module 1002 is used to input the multimodal information into a pre-trained process parameter intelligent recommendation model to obtain candidate process parameter information output by the process parameter intelligent recommendation model. The process parameter intelligent recommendation model is a model trained by the process parameter intelligent recommendation model training method based on any of the above embodiments.

[0143] The adjustment module 1003 is used to adjust the candidate process parameter information according to the user intervention feedback to obtain the process parameter information to be used when the manufacturing process manufactures the workpiece.

[0144] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0145] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0146] like Fig.11 , is a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers, intelligent robots (such as welding robots). The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0147] like Fig.11 As shown, the electronic device includes: one or more processors 1101, a memory 1102, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.11 A processor 1101 is taken as an example.

[0148] The memory 1102 is a non-transitory computer-readable storage medium provided by the present disclosure. The memory stores instructions executable by at least one processor to enable the at least one processor to perform the method described in any of the above embodiments provided by the present disclosure. The non-transitory computer-readable storage medium of the present disclosure stores computer instructions, which are used to enable a computer to perform the method described in any of the above embodiments provided by the present disclosure.

[0149] The memory 1102 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the method for intelligently recommending process parameters of a manufacturing process and the method for training a process parameter intelligent recommendation model in the embodiments of the present disclosure. The processor 1101 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1102, that is, to implement the method for intelligently recommending process parameters of a manufacturing process and the method for training a process parameter intelligent recommendation model in the above method embodiments.

[0150] The memory 1102 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1102 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1102 may optionally include a memory remotely arranged relative to the processor 1101, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0151] The electronic device may further include: an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103 and the output device 1104 may be connected via a bus or other means. Fig.11 The example of connecting through bus is taken in the following.

[0152] The input device 1103 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, and other input devices. The output device 1104 may include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor). The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0153] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0154] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0155] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0156] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0157] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0158] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in this application. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0159] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for intelligently recommending process parameters of a manufacturing process, characterized in that: include: Acquire multimodal information, the multimodal information including at least two of temperature information of a manufacturing area, humidity information of the manufacturing area, RGB image information containing a workpiece, three-dimensional point cloud information containing the workpiece, and workpiece-related parameter information input by a user; The workpiece can be formed into a target object after being manufactured by the manufacturing process, the manufacturing process is a welding process, and the target object is a weld; According to at least one of the RGB image information and the three-dimensional point cloud information, the target object is identified to obtain geometric parameter information of the target object; Predicting process parameters according to at least one of the temperature information, the humidity information, and the workpiece-related parameter information in combination with the geometric parameter information of the target object to obtain candidate process parameter information; The candidate process parameter information is adjusted according to user intervention feedback to obtain process parameter information to be used when the manufacturing process manufactures the workpiece, and the process parameter information is used for a mechanical arm in a welding machine and a welding robot to weld the workpiece; The step of identifying the target object according to at least one of the RGB image information and the three-dimensional point cloud information to obtain geometric parameter information of the target object includes: Aligning the RGB image information with the three-dimensional point cloud information; Extracting features from the aligned RGB image information and the three-dimensional point cloud information respectively to obtain multi-scale features fused with depth information and point cloud features fused with RGB information, including: extracting features from the aligned RGB image information and the three-dimensional point cloud information respectively based on a pre-trained multi-modal encoder in a target object recognition network to obtain multi-scale features fused with depth information and point cloud features fused with RGB information; A decoding operation is performed on the multi-scale features of the fused depth information and the point cloud features of the fused RGB information to obtain geometric parameter information of the target object, including: based on a decoder in the target object recognition network, a decoding operation is performed on the multi-scale features of the fused depth information and the point cloud features of the fused RGB information to obtain geometric parameter information of the target object.

2. The method according to claim 1, characterized in that The step of predicting the process parameters based on at least one of the temperature information, the humidity information, and the workpiece-related parameter information in combination with the geometric parameter information of the target object to obtain candidate process parameter information includes: Performing encoding operations on the temperature information, the humidity information, the workpiece-related parameter information, and the geometric parameter information of the target object respectively; Performing a splicing operation on the encoding results of the temperature information, the humidity information, the workpiece-related parameter information, and the geometric parameter information of the target object to obtain a multimodal feature; The process parameters are predicted according to the multimodal features to obtain the candidate process parameter information.

3. The method according to claim 2, characterized in that The predicting of process parameters according to the multimodal features to obtain the candidate process parameter information includes: Inputting the multimodal features into a pre-trained process parameter recommendation network to predict process parameters and obtain the candidate process parameter information; The input of the process parameter recommendation network includes multimodal features, the output of the multimodal features includes candidate process parameter information, and the multimodal features have learned a mapping relationship between the multimodal features and the candidate process parameter information.

4. The method according to claim 3, characterized in that The process parameter recommendation network is obtained in advance in the following way: Acquire training data, the training data including geometric parameter data of the target object, temperature data and humidity data of the manufacturing area, workpiece related parameter data and process parameter label data of the manufacturing process; Performing encoding operations on the temperature data, the humidity data, the workpiece-related parameter data, and the target object's geometric parameter data respectively; Performing a splicing operation on the encoding results of the temperature data, the humidity data, the workpiece-related parameter data, and the target object's geometric parameter data to obtain a multimodal feature; Inputting the multimodal features into a process parameter recommendation network to predict process parameters, and obtaining candidate process parameter prediction data; A first loss value is generated according to the candidate process parameter prediction data and the process parameter label data, and the process parameter recommendation network is trained according to the first loss value.

5. The method according to claim 4, characterized in that The process parameter label data of the manufacturing process is derived from at least one of a process library of the manufacturing process and candidate process parameter information adjusted based on user intervention feedback.

6. A training method for a process parameter intelligent recommendation model, characterized in that: The process parameter intelligent recommendation model includes a target object recognition network and a process parameter recommendation network, and the training method includes: Acquire training data, the training data including temperature data of a manufacturing area, humidity data of the manufacturing area, RGB image data of a workpiece, three-dimensional point cloud data of the workpiece, workpiece-related parameter data, geometric parameter label data of the target object, and process parameter label data of a manufacturing process, wherein the manufacturing process is a welding process, and the target object is a weld; Inputting the RGB image data and the three-dimensional point cloud data into the target object recognition network to recognize the target object, and obtaining geometric parameter prediction data of the target object; Inputting the geometric parameter prediction data, the temperature data, the humidity data, and the workpiece related parameter data into the process parameter recommendation network to predict the process parameters, and obtaining candidate process parameter prediction data; generating a first loss value according to the candidate process parameter prediction data and the process parameter label data, and generating a second loss value according to the geometric parameter prediction data and the geometric parameter label data; The process parameter intelligent recommendation model is trained based on the first loss value and the second loss value; the process parameters output by the process parameter intelligent recommendation model are used for the mechanical arm in the welding machine and the welding robot to weld the workpiece; The step of inputting the RGB image data and the three-dimensional point cloud data into the target object recognition network to recognize the target object and obtain the geometric parameter prediction data of the target object includes: Aligning the RGB image data with the three-dimensional point cloud data; Extracting features from the aligned RGB image data and the three-dimensional point cloud data respectively to obtain multi-scale features fused with depth information and point cloud features fused with RGB information, including: extracting features from the aligned RGB image data and the three-dimensional point cloud data respectively based on a pre-trained multi-modal encoder in a target object recognition network to obtain multi-scale features fused with depth information and point cloud features fused with RGB information; A decoding operation is performed on the multi-scale features of the fused depth information and the point cloud features of the fused RGB information to obtain geometric parameter prediction data of the target object, including: based on a decoder in the target object recognition network, a decoding operation is performed on the multi-scale features of the fused depth information and the point cloud features of the fused RGB information to obtain geometric parameter prediction data of the target object.

7. The method according to claim 6, characterized in that The process parameter label data of the manufacturing process is derived from at least one of a process library of the manufacturing process and candidate process parameter information adjusted based on user intervention feedback.

8. A method for intelligently recommending process parameters of a manufacturing process, characterized in that: include: Acquire multimodal information, the multimodal information including at least two of temperature information of a manufacturing area, humidity information of the manufacturing area, RGB image information containing a workpiece, three-dimensional point cloud information containing the workpiece, and workpiece-related parameter information input by a user; The workpiece can be formed into a target object after being manufactured by the manufacturing process, the manufacturing process is a welding process, and the target object is a weld; Inputting the multimodal information into a pre-trained process parameter intelligent recommendation model to obtain candidate process parameter information output by the process parameter intelligent recommendation model; wherein the process parameter intelligent recommendation model is a model trained based on the method according to any one of claims 6 to 7; The candidate process parameter information is adjusted according to user intervention feedback to obtain process parameter information to be used when the manufacturing process manufactures the workpiece. The process parameter information is used by a welding machine and a mechanical arm in a welding robot to weld the workpiece.

9. An intelligent process parameter recommendation device for a manufacturing process, characterized in that: include: an acquisition module, used for acquiring multimodal information, wherein the multimodal information includes at least two items of temperature information of a manufacturing area, humidity information of the manufacturing area, RGB image information of a workpiece, three-dimensional point cloud information of the workpiece, and workpiece-related parameter information input by a user; the workpiece can be formed into a target object after being manufactured by the manufacturing process, the manufacturing process is a welding process, and the target object is a weld; an identification module, configured to identify the target object according to at least one of the RGB image information and the three-dimensional point cloud information, and obtain geometric parameter information of the target object; A process recommendation module, configured to predict process parameters according to at least one of the temperature information, the humidity information, and the workpiece-related parameter information in combination with the geometric parameter information of the target object, to obtain candidate process parameter information; An adjustment module, used to adjust the candidate process parameter information according to user intervention feedback, to obtain the process parameter information to be used when the manufacturing process manufactures the workpiece, and the process parameter information is used for the mechanical arm in the welding machine and the welding robot to weld the workpiece; Among them, the recognition module is used to: align the RGB image information and the three-dimensional point cloud information; based on the pre-trained multimodal encoder in the target object recognition network, respectively extract features of the aligned RGB image information and the three-dimensional point cloud information to obtain multi-scale features of fused depth information and point cloud features of fused RGB information; based on the decoder in the target object recognition network, decode the multi-scale features of the fused depth information and the point cloud features of the fused RGB information to obtain geometric parameter information of the target object.

10. A training device for a process parameter intelligent recommendation model, characterized in that: The process parameter intelligent recommendation model includes a target object recognition network and a process parameter recommendation network, and the training device includes: an acquisition module, used to acquire training data, wherein the training data includes temperature data of a manufacturing area, humidity data of the manufacturing area, RGB image data of a workpiece, three-dimensional point cloud data of the workpiece, workpiece-related parameter data, geometric parameter label data of the target object, and process parameter label data of a manufacturing process, wherein the manufacturing process is a welding process, and the target object is a weld; A recognition module, used for inputting the RGB image data and the three-dimensional point cloud data into the target object recognition network to recognize the target object and obtain geometric parameter prediction data of the target object; A prediction module, used for inputting the geometric parameter prediction data, the temperature data, the humidity data, and the workpiece related parameter data into the process parameter recommendation network to predict the process parameters and obtain candidate process parameter prediction data; A generating module, configured to generate a first loss value according to the candidate process parameter prediction data and the process parameter label data, and to generate a second loss value according to the geometric parameter prediction data and the geometric parameter label data; A training module, used for training the process parameter intelligent recommendation model based on the first loss value and the second loss value; the process parameters output by the process parameter intelligent recommendation model are used for a mechanical arm in a welding machine and a welding robot to weld a workpiece; Among them, the recognition module is used to: align the RGB image data and the three-dimensional point cloud data; based on the pre-trained multimodal encoder in the target object recognition network, respectively extract features of the aligned RGB image data and the three-dimensional point cloud data to obtain multi-scale features of fused depth information and point cloud features of fused RGB information; based on the decoder in the target object recognition network, decode the multi-scale features of the fused depth information and the point cloud features of the fused RGB information to obtain the geometric parameter prediction data of the target object.

11. A device for intelligently recommending process parameters of a manufacturing process, characterized in that: include: a first acquisition module, configured to acquire multimodal information, wherein the multimodal information includes at least two items of temperature information of a manufacturing area, humidity information of the manufacturing area, RGB image information of a workpiece, three-dimensional point cloud information of the workpiece, and workpiece-related parameter information input by a user; the workpiece may be formed into a target object after being manufactured by the manufacturing process, the manufacturing process is a welding process, and the target object is welding; A second acquisition module is used to input the multimodal information into a pre-trained process parameter intelligent recommendation model to obtain candidate process parameter information output by the process parameter intelligent recommendation model; wherein the process parameter intelligent recommendation model is a model trained based on the method according to any one of claims 6 to 7; The adjustment module is used to adjust the candidate process parameter information according to user intervention feedback to obtain the process parameter information to be used when the manufacturing process manufactures the workpiece, and the process parameter information is used for the mechanical arm in the welding machine and the welding robot to weld the workpiece.

12. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

13. A storage medium storing instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device is caused to execute the method according to any one of claims 1 to 8.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 8.

Images