A method and an appartus for detecting weld area using a vision-language model

Abstract

The present disclosure relates to a method and apparatus for detecting a welding area using a vision-language model. According to one embodiment, a method for detecting a welding area using a vision-language model may include: inputting image data into a vision-language model; acquiring feature data of the image data; determining whether there is a welding-required part within the image data using the feature data; and detecting a welding area through a heat map of the welding-required part in response to the determination that the welding-required part exists.

Description

Technology Field

[0001] The present disclosure relates to a method and apparatus for detecting a weld area using a vision-language model. Background Technology

[0002] Welding processes in various industrial environments, including marine environments, are currently being carried out by welding robots. The welding process by a welding robot largely consists of the following steps: movement to the welding position, designation of the teaching point, touch sensing for detecting the welding position, and execution of the welding.

[0003] Technology for detecting precise welding locations on structures subject to welding is currently being developed. However, development of technology for detecting welding areas requiring welding in industrial environments with diverse structures is somewhat lacking.

[0004] Accordingly, there is a need for research and development on technology to efficiently detect welding areas in industrial environments.

[0005] The aforementioned background technology is technical information that the inventor possessed for the derivation of the present invention or acquired during the process of deriving the present invention, and it cannot be considered as prior art disclosed to the general public prior to the filing of the present invention. The problem to be solved

[0006] The present disclosure is to provide a method and apparatus for detecting a weld area using a vision-language model. The problems to be solved by the present invention are not limited to those mentioned above, and other problems and advantages of the present invention not mentioned can be understood from the following description and will be more clearly understood by embodiments of the present invention. Furthermore, it will be understood that the problems and advantages to be solved by the present invention can be realized by means and combinations thereof as set forth in the claims. means of solving the problem

[0007] As a technical means for achieving the aforementioned technical problem, the first aspect of the present disclosure is a method for detecting a welding area using a vision-language model, comprising: a step of inputting image data into a vision-language model; a step of acquiring feature data of the image data; a step of determining whether there is a welding-required part within the image data using the feature data; and a step of detecting a welding area through a heat map of the welding-required part in response to the determination that the welding-required part exists.

[0008] A second aspect of the present disclosure is an apparatus for detecting a welding area using a vision-language model, comprising at least one memory and at least one processor, wherein the at least one processor inputs image data and text data into a vision-language model, acquires feature data of the image data, determines whether a welding-required portion exists within the image data using the embedding results of the feature data and the text data, and detects a welding area through a heat map of the welding-required portion in response to the determination that the welding-required portion exists.

[0009] A third aspect of the present disclosure may provide a computer-readable recording medium having a program for executing a method according to a first aspect on a computer.

[0010] In addition to this, other methods for implementing the present invention, other systems, and computer-readable recording media storing a computer program for executing said methods may be further provided.

[0011] Other aspects, features, and advantages other than those described above will become clear from the following drawings, claims, and detailed description of the invention. Effects of the invention

[0012] According to the means for solving the problem of the present disclosure described above, the present disclosure enables the detection of a welding area in an industrial environment where a number of structures exist, thereby enabling the complete automation of the welding process.

[0013] In addition, the present disclosure can save time and cost spent on the dataset for training the model by detecting welding regions using a vision-language model that does not require additional training.

[0014] However, the effects obtainable through the present invention are not limited to those described above, and other unmentioned technical effects will be clearly understood by those skilled in the art from the description of the invention below. Brief explanation of the drawing

[0015] FIG. 1 is a conceptual diagram illustrating a method for detecting a welding area using a vision-language model according to one embodiment. FIG. 2 is a block diagram of a welding area detection device according to one embodiment. FIG. 3 is an exemplary configuration diagram of a system including a welding area detection device according to one embodiment. FIG. 4 is a flowchart illustrating a method for detecting a welding area using a vision-language model according to one embodiment. FIG. 5 is an exemplary drawing for explaining a system for detecting a welding area using a vision-language model according to one embodiment. FIG. 6 is an exemplary diagram illustrating a method for detecting a weld area in image data according to one embodiment. FIG. 7 is an exemplary drawing for explaining a method for calculating three-dimensional information of a welding area according to one embodiment. FIG. 8 is a flowchart illustrating a method for detecting a welding area by determining whether a welding-required part exists according to one embodiment. Specific details for implementing the invention

[0016] The advantages and features of the present invention, and the methods for achieving them, will become clear by referring to the embodiments described in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments presented below, but can be implemented in various different forms and should be understood to include all modifications, equivalents, and substitutions that fall within the spirit and scope of the present invention. The embodiments presented below are provided to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention. In describing the present invention, detailed descriptions of related known technologies are omitted if it is determined that such detailed descriptions may obscure the essence of the present invention.

[0017] The terms used in this application are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0018] Some embodiments of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented by various numbers of hardware and / or software configurations that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors or by circuit configurations for a specific function. Additionally, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented as algorithms executed on one or more processors. Furthermore, the present disclosure may employ prior art for electronic configuration, signal processing, and / or data processing, etc. Terms such as "mechanism," "element," "means," and "configuration" may be used broadly and are not limited to mechanical and physical configurations.

[0019] Furthermore, the connecting lines or connecting members between the components depicted in the drawings are merely illustrative of functional connections and / or physical or circuit connections. In the actual device, connections between components may be represented by various alternative or added functional connections, physical connections, or circuit connections.

[0020] The present disclosure will be described in detail below with reference to the attached drawings.

[0021] FIG. 1 is a conceptual diagram illustrating a method for detecting a welding area using a vision-language model according to one embodiment.

[0022] Referring to FIG. 1, a welding area detection device (not shown) can detect a welding area (40) that requires welding work in an industrial environment where various structures exist, particularly in a ship environment.

[0023] For example, a welding area detection device can detect a welding area (40) using a vision-language model (30), which is a type of neural network model.

[0024] In machine learning technology and cognitive science, a neural network model refers to a statistical learning algorithm implemented based on the structure of biological neural networks, or a structure that executes such an algorithm.

[0025] For example, a neural network model can represent a model capable of problem-solving by having nodes, which are artificial neurons forming a network through the combination of synapses as in biological neural networks, learn to reduce the error between the correct output corresponding to a specific input and the inferred output by repeatedly adjusting the weights of the synapses. For example, a neural network model may include random probability models and neural network models used in artificial intelligence learning methods such as machine learning and deep learning.

[0026] For example, a neural network model can be implemented as a multilayer perceptron (MLP) composed of multiple layers of nodes and connections between them. The neural network model according to the present embodiment can be implemented using one of various artificial neural network model structures including an MLP. For example, the neural network model may be composed of an input layer that receives an input signal or data from the outside, an output layer that outputs an output signal or data corresponding to the input data, and at least one hidden layer located between the input layer and the output layer, which receives a signal from the input layer, extracts a feature, and transmits it to the output layer. The output layer receives a signal or data from the hidden layer and outputs it to the outside.

[0027] In one embodiment, the neural network model may refer to a Large Vision Language Model (LVLM). The LVLM is a model that has advantages in terms of generalization by learning a vast amount of data and has excellent processing capabilities for new data that has not been previously learned. For example, even if the LVLM has not learned data regarding the detection of a welding area (40), it can recognize the welding surface included in the image data (10) by analyzing the image data (10) according to the input of image data (10) and text data (20) requiring the detection of the welding area (40), and the welding area detection device can obtain information about the welding area (40) based on the information about the welding surface recognized through the LVLM. Through this, the welding area detection device can save time and cost consumed in building a dataset for learning to detect the welding area (40).

[0028] The welding area detection device can input image data (10) and text data (20) into a vision-language model (30). Additionally, the welding area detection device can obtain information about the welding area (40) as output data based on the input of image data (10) and text data (20) to the vision-language model (30).

[0029] In one embodiment, when a welding area detection device intends to detect a welding area (40) in different image data (10), it can embed text data (20) requesting detection of the welding area (40) into a vision-language model (30) without having to input the same text data (20) every time. Through this, the welding area detection device can obtain information about the welding area (40) by inputting only the image data (10) into the vision-language model (30).

[0030] For example, the welding area detection device can determine whether there is a part requiring welding among the areas included in the image data (10). For example, the welding area detection device can determine whether there is a part requiring welding, i.e., a welding surface, within the image data (10) based on feature data obtained from the image data (10). Additionally, the welding area detection device can perform the detection process of the welding area (40) only when a part requiring welding exists.

[0031] For example, if there is a part that requires welding, the welding area detection device can generate a heat map of the image data (10). Here, the heat map may refer to a map indicating the degree of interest that a vision-language model (30) has in each area of ​​the image data (10).

[0032] For example, a welding area detection device can detect a welding area (40) based on a heat map. Specifically, the welding area detection device can detect an area as a welding area (40) where the degree of interest of the vision-language model (30) within the heat map, for example, the activation value within the heat map, exceeds a predetermined threshold value.

[0033] The welding area detection device can calculate three-dimensional information about the welding area (40) using information about the welding area (40) in the image data (10) and the PCD (Point Cloud Data) of the image data (10). In addition, the welding area detection device can transmit the calculated three-dimensional information to a welding device (50), for example, a welding robot, to control it to perform a welding operation.

[0034] The welding area detection device can generate and provide an interface to the user that displays the welding area (40) within the image data (10). Through this, the user can easily check information about the area requiring welding in the work environment.

[0035] FIG. 2 is a block diagram of a welding area detection device according to one embodiment.

[0036] Referring to FIG. 2, the welding area detection device (100) includes a processor (110), memory (120), an input / output interface (130), and a communication module (140). For convenience of explanation, FIG. 2 only shows components related to the present invention. Accordingly, other general-purpose components in addition to those shown in FIG. 2 may be included in the welding area detection device (100). Furthermore, it is obvious to those skilled in the art that the processor (110), memory (120), input / output interface (130), and communication module (140) shown in FIG. 2 may be implemented as independent devices.

[0037] The processor (110) can process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Here, the instructions may be provided from memory (120) or an external device. Additionally, the processor (110) can control the overall operation of other components included in the welding area detection device (100).

[0038] For example, the processor (110) can input image data into a vision-language model. Here, the image data may mean an image containing at least one structure within a working environment.

[0039] Additionally, the processor (110) can acquire feature data of the image data. For example, the vision-language model can embed the image data according to the input of the image data and output feature data in the form of a vector.

[0040] Additionally, the processor (110) can determine whether there is a welding required part in the image data using feature data.

[0041] Additionally, if there is a part requiring welding within the image data, the processor (110) can detect a welding area through a heat map of the part requiring welding. For example, the processor (110) can generate a heat map indicating the level of interest of the vision-language model within the image data and detect an area where the level of interest of the vision-language model exceeds a predetermined threshold as a welding area.

[0042] The process performed in FIGS. 4 to 8 below may correspond to a process performed by a processor (110).

[0043] The processor (110) may be implemented as an array of multiple logic gates, or as a combination of a general-purpose microprocessor and memory storing a program that can be executed on the microprocessor. For example, the processor (110) may include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some environments, the processor (110) may include an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. For example, the processor (110) may refer to a combination of processing devices such as a combination of a digital signal processor (DSP) and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors combined with a digital signal processor (DSP) core, or any other combination of such configurations.

[0044] The memory (120) may include any non-transient computer-readable recording medium. In one embodiment, the memory (120) may include a permanent mass storage device such as a random access memory (RAM), read-only memory (ROM), disk drive, solid state drive (SSD), or flash memory. In another example, a permanent mass storage device such as a ROM, SSD, flash memory, or disk drive may be a separate permanent storage device distinct from the memory. Additionally, the memory (120) may store an operating system (OS) and at least one program code (e.g., code for the processor (110) to perform operations described later with reference to FIGS. 4 through 8).

[0045] These software components may be loaded from a computer-readable recording medium separate from the memory (120). This separate computer-readable recording medium may be a recording medium that can be directly connected to the welding area detection device (100), and may include, for example, an input / output computer-readable recording medium such as a floppy drive, disk, tape, DVD / CD-ROM drive, or memory card. Alternatively, the software components may be loaded into the memory (120) via a communication module (140) rather than a computer-readable recording medium. For example, at least one program may be loaded into the memory (120) based on a computer program (e.g., a computer program for the processor (110) to perform the operation described later with reference to FIGS. 4 to 8) which is installed by files provided through the communication module (140) by developers or a file distribution system that distributes installation files for the application.

[0046] The input / output interface (130) may be a means for interfacing with a device for input and / or output (e.g., keyboard, mouse, etc.) that is connected to or included in the welding area detection device (100). In FIG. 2, the input / output interface (130) is shown as an element configured separately from the processor (110), but is not limited thereto, and the input / output interface (130) may be configured to be included in the processor (110).

[0047] The communication module (140) may provide a configuration or function for the welding area detection device (100) to communicate with an external device (not shown) via a network. Additionally, the communication module (140) may provide a configuration or function for the welding area detection device (100) to communicate with another external device. For example, control signals, commands, data, etc. provided under the control of the processor (110) may be transmitted to an external device via the communication module (140) and the network.

[0048] Additionally, although not shown in FIG. 2, the welding area detection device (100) may include a display module (not shown). For example, the display module may display an interface indicating a welding area within image data and provide it to the user.

[0049] Additionally, although not shown in FIG. 2, the welding area detection device (100) may include a shooting module (not shown). For example, the shooting module may refer to a camera, a LiDAR sensor, a depth camera, a thermal imaging camera, etc., capable of generating image data, but is not limited thereto.

[0050] FIG. 3 is an exemplary configuration diagram of a system including a welding area detection device according to one embodiment.

[0051] Referring to FIG. 3, the welding area detection device (310) may include any kind of server managing a web and / or app capable of providing artificial intelligence services. Additionally, the welding area detection device (310) of FIG. 3 may be the same device as the welding area detection device of FIG. 1 and / or the welding area detection device (100) of FIG. 2.

[0052] The external device (320) refers to the entity that provides the data necessary for the welding area detection device (310) to detect the welding area. The external device (320) may include any type of server that manages various information. The external device (320) may include a database and may include a server that manages a web service API capable of providing information, but is not limited thereto. For example, the external device (320) may correspond to a sensor and a shooting device that generate image data within the work environment. Additionally, the external device (320) may correspond to a user terminal that acquires text data entered by a user.

[0053] The welding area detection device (310) and the external device (320) can communicate with each other and / or with other devices via a network. The network is a comprehensive data communication network that enables different entities to communicate smoothly with each other, and may include wired internet, wireless internet, and mobile wireless communication networks. For example, the network may include a Local Area Network (LAN), a Wide Area Network (WAN), a Value Added Network (VAN), a Mobile Radio Communication Network, a satellite communication network, and combinations thereof. Additionally, wireless communication may include, for example, Wi-Fi, Bluetooth, Bluetooth Low Energy, ZigBee, Wi-Fi Direct (WFD), Ultrawideband (UWB), Infrared Data Association (IrDA), Near Field Communication (NFC), but is not limited thereto.

[0054] The welding area detection device (310) can communicate with an external device (320) via a network. By communicating via a network, the welding area detection device (310) can receive data from the external device (320) and provide a response based on the received data.

[0055] FIG. 4 is a flowchart illustrating a method for detecting a welding area using a vision-language model according to one embodiment.

[0056] Referring to FIG. 4, the method of detecting a weld area using a vision-language model consists of steps processed chronologically in the weld area detection device (100) and / or processor (110) shown in FIG. 2. Therefore, even if details are omitted below, the description above regarding the weld area detection device (100) or processor (110) shown in FIG. 2 can also be applied to the method of detecting a weld area using the vision-language model of FIG. 4.

[0057] In step 410, the welding area detection device can input image data into a vision-language model.

[0058] For example, a weld area detection device may additionally input text data along with image data into a vision-language model. Here, the image data may refer to an image containing at least one structure within the working environment. Additionally, the text data may refer to a sentence requesting the detection of a weld area.

[0059] In one embodiment, the welding area detection device can embed text data requesting the detection of a welding area in a vision-language model so as not to input the same text data requesting the detection of a welding area every time during the process of detecting different welding areas in different image data.

[0060] In one embodiment, the welding area detection device can acquire text data such as "Where is the welding area in the image?". Additionally, the welding area detection device can convert the above-mentioned text data into vector data and embedding it.

[0061] In one embodiment, the welding area detection device can acquire Point Cloud Data (PCD) rather than image data. The welding area detection device can preprocess the PCD into image data. For example, the welding area detection device can preprocess the PCD into two-dimensional image data by projecting it into two dimensions.

[0062] In step 420, the welding area detection device can acquire feature data of the image data.

[0063] For example, a welding area detection device can acquire feature data of image data using a Convolutional Neural Network (CNN) or a Transformer-based model. Here, feature data may refer to data representing the characteristics of the image data as vectors.

[0064] In step 430, the welding area detection device can determine whether there is a welding required part within the image data using feature data.

[0065] For example, a welding area detection device defines joints on a metal surface, areas where cracks have occurred, areas where weld lines do not exist, etc., as areas requiring welding, and if such areas exist, it can determine that areas requiring welding exist within the image data.

[0066] For example, a welding area detection device can determine the presence of a welding-requiring area by analyzing the pattern of feature data in image data. Specifically, feature data obtained through a CNN or Transformer-based model may contain feature data regarding welding-requiring areas and feature data regarding welding-unrequiring areas in different patterns. Additionally, if feature data having the pattern of a welding-requiring area is included in the feature data of the image data, the welding area detection device can determine that a welding-requiring area exists within the image data.

[0067] In step 440, the welding area detection device can detect the welding area through a heat map of the welding required part in response to the determination that the welding required part exists.

[0068] For example, if there are parts requiring welding within the image data, the welding area detection device can generate a heat map of the image data. Here, the heat map may correspond to a type of attention map that indicates the degree (Attention Score) that a vision-language model pays attention to each pixel of the image data or each pre-set area.

[0069] In addition, the welding area detection device can detect a welding area based on information about each pixel or each area of ​​image data included in a heat map. In one embodiment, the welding area detection device can detect two-dimensional information of the welding area within the image data.

[0070] The welding area detection device can calculate three-dimensional information about the welding area using information about the welding area within the image data and the PCD corresponding to the image data. The welding area detection device can transmit the three-dimensional information about the welding area to a welding device or a welding robot to control the welding operation to be performed on the welding area.

[0071] FIG. 5 is an exemplary drawing for explaining a system for detecting a welding area using a vision-language model according to one embodiment.

[0072] Referring to FIG. 5, the welding area detection system (500) may include a data transmission and reception unit (510), an encoder unit (520), a decoder unit (530), a heat map generation unit (540), a welding area detection unit (550), and a welding area output unit (560).

[0073] The data transmission and reception unit (510) can receive image data in which at least one structure within the work environment is captured from an image sensor installed in the work environment. Additionally, the data transmission and reception unit (510) can receive text data requesting the detection of a welding area from a user terminal, etc.

[0074] The encoder unit (520) can embed image data and text data. For example, the encoder unit (520) can embed image data into feature data in the form of vectors. Additionally, the encoder unit (520) can detect a welding area for the received image data by embedding previously stored text data, even if the data transmission / reception unit (510) does not receive text data requesting the detection of a separate welding area.

[0075] The decoder unit (530) can determine whether there is a part requiring welding within the image data by using feature data of the image data generated by the encoder unit (520). For example, the decoder unit (530) can determine whether there is a part requiring welding within the image data by analyzing the pattern of the feature data using a CNN or Transformer-based model.

[0076] Additionally, the decoder unit (530) can output text indicating whether there is a part requiring welding in the image data. Specifically, if there is no part requiring welding in the image data, the decoder unit (530) can output the text "There is no part requiring welding in the image data". Additionally, if there is a part requiring welding in the image data, the decoder unit (530) can output the text "The part where the vertical plate and the horizontal plate meet in the image data is the welding area".

[0077] The heat map generation unit (540) can generate a heat map of the image data when the decoder unit (530) determines that there is a part requiring welding within the image data. For example, the heat map generation unit (540) can generate a heat map by visualizing a specific layer of the encoder unit (520).

[0078] The welding area detection unit (550) can detect a welding area using information about each pixel or each area in the heat map generated by the heat map generation unit (540).

[0079] The welding area output unit (560) can project the welding area in three dimensions using information and PCD of the part detected as the welding area. In addition, the welding area output unit (560) can generate and output an interface that displays the welding area within the image data.

[0080] In one embodiment, when the data transmission / reception unit (510) receives image data, the welding area output unit (560) can back-project the image data into three-dimensional data. In another embodiment, when the data transmission / reception unit (510) receives PCD, the welding area output unit (560) can convert the PCD data into two-dimensional image data and then back-project it into three-dimensional data.

[0081] FIG. 6 is an exemplary diagram illustrating a method for detecting a weld area in image data according to one embodiment.

[0082] Referring to FIG. 6, the welding area detection device can acquire image data including at least one structure (611, 612) installed in the work environment. Additionally, the welding area detection device can acquire feature data of the image data by inputting the image data into a vision-language model. Here, the feature data may refer to data representing each pixel of the image data or data of each designated area in a vector form.

[0083] A welding area detection device can determine the presence of areas requiring welding within image data by utilizing feature data of the image data. For example, the welding area detection device defines joints on a metal surface, areas where cracks have occurred, and areas where weld lines do not exist as areas requiring welding, and can determine whether such areas requiring welding exist.

[0084] Referring to FIG. 6, the welding area detection device can determine that a welding-required area exists if there is a seam between the first structure (611) and the second structure (612).

[0085] If it is determined that there is a part requiring welding within the image data (600), the welding area detection device can generate a heat map of the image data (600). Here, the heat map may correspond to a type of attention map that indicates the degree (Attention Score) that a vision-language model pays attention to each pixel or each pre-set area of ​​the image data (600).

[0086] The welding area detection device can detect the accurate welding area (620) within the image data (600) using a heat map.

[0087] In one embodiment, the welding area detection device can obtain an activation value for each pixel of the image data (600) through a heat map. Here, the activation value may represent a value indicating the degree to which the vision-language model pays attention to each pixel. Specifically, the activation value may represent a value indicating the degree to which the vision-language model focuses on a pixel within the image data (600) when generating output data based on the text data and image data (600) input to the vision-language model. In one embodiment, the activation value may represent a weight assigned to a pixel when the vision-language model generates output data.

[0088] For example, a welding area detection device can obtain an activation value for each pixel of image data (600) from a predetermined layer among a plurality of layers of an encoder included in a vision-language model. Here, the predetermined layer can be arbitrarily set by the user.

[0089] Additionally, the welding area detection device can detect an area containing pixels whose activation value exceeds a predetermined threshold as a welding area (620). Here, the predetermined threshold can be arbitrarily set by the user. Additionally, the predetermined threshold can be set based on various welding type information including information on the work environment, material information of the structure, work information, information on the welding device, etc.

[0090] In another embodiment, the welding area detection device can obtain an activation value for each preset area of ​​the image data (600) through a heat map. Additionally, the welding area detection device can detect areas where the activation value exceeds a predetermined threshold as welding areas (620).

[0091] FIG. 7 is an exemplary drawing for explaining a method for calculating three-dimensional information of a welding area according to one embodiment.

[0092] Referring to FIG. 7, the welding area detection device can detect the welding area (720) of the structure (710) included in the image data (700). Additionally, the welding area detection device can calculate three-dimensional information of the welding area (720).

[0093] For example, the welding area detection device can acquire image data (700) and a PCD corresponding to the image data (700). Additionally, the welding area detection device can calculate three-dimensional information of the welding area (720) using the image data (700) and the PCD. Specifically, the welding area detection device can calculate three-dimensional information of the welding area (720) by matching the image data (700) with the PCD and back-projecting it.

[0094] Additionally, the welding area detection device can detect the welding start point (730) and welding end point (740) of the welding device in the welding area (720).

[0095] For example, the welding area detection device can calculate welding type information of the welding area (720). Here, the welding type information may include various information related to welding, such as information on the work environment, welding type information, material information of the structure (710), work information, and information on the welding device. In one embodiment, the welding area detection device can obtain welding type information using a neural network model. For example, the welding area detection device can obtain welding type information by inputting image data (700) into a neural network model that has learned welding type information from various images.

[0096] Additionally, the welding area detection device can calculate the welding start point (730) and welding end point (740) of the welding device within the welding area (720) using welding type information. Specifically, the welding area detection device can calculate three-dimensional information corresponding to the welding start point (730) and three-dimensional information corresponding to the welding end point (740) among the three-dimensional information of the welding area (720).

[0097] The welding area detection device can transmit three-dimensional information of the welding area (720), three-dimensional information of the welding start point (730), and three-dimensional information of the welding end point (740) to the welding device so that the welding device can perform a welding operation.

[0098] FIG. 8 is a flowchart illustrating a method for detecting a welding area by determining whether a welding-required part exists according to one embodiment.

[0099] A method for detecting a welding area by determining whether a part requiring welding exists consists of steps processed sequentially in the welding area detection device (100) and / or processor (110) shown in FIG. 2. Therefore, even if details are omitted below, the description above regarding the welding area detection device (100) or processor (110) shown in FIG. 2 can also be applied to the method for detecting a welding area by determining whether a part requiring welding exists in FIG. 8.

[0100] In step 810, the welding area detection device can input image data and text data into a vision-language model.

[0101] Here, image data may refer to an image containing at least one structure within the working environment. Additionally, text data may refer to a sentence requesting the detection of a weld area.

[0102] In one embodiment, the welding area detection device can embed text data requesting the detection of a welding area in a vision-language model so as not to input the same text data requesting the detection of a welding area every time during the process of detecting different welding areas in different image data.

[0103] In step 820, the welding area detection device can acquire feature data of the image data.

[0104] For example, a welding area detection device can acquire feature data of image data using a CNN or Transformer-based model. Here, feature data may refer to data representing the characteristics of the image data as vectors.

[0105] In step 830, the welding area detection device can determine whether there is a part that needs welding.

[0106] For example, a welding area detection device defines joints on a metal surface, areas where cracks have occurred, areas where weld lines do not exist, etc., as parts requiring welding, and if such areas exist, it can determine that parts requiring welding exist within the image data.

[0107] For example, a welding area detection device can determine the presence of a welding-requiring area by analyzing the patterns of feature data in image data. Specifically, feature data obtained through CNN or Transformer-based models may contain feature data regarding welding-requiring areas and feature data regarding welding-unrequiring areas in different patterns. Additionally, the welding area detection device can determine that a welding-requiring area exists within the image data if feature data having the pattern of a welding-requiring area is included in the feature data of the image data.

[0108] If there is no part requiring welding, the welding area detection device can terminate the welding area detection process without detecting a welding area because there is no part requiring welding within the image data.

[0109] If there is a part that needs welding, in step 840, the welding area detection device can generate a heat map using image data.

[0110] In step 850, the weld area detection device can detect the weld area using a heat map.

[0111] Here, the heat map may correspond to a type of attention map that indicates the degree (Attention Score) that a vision-language model pays attention to each pixel of image data or each pre-set region.

[0112] For example, a welding area detection device can obtain activation values ​​for each pixel of image data or for each pre-designated area through a heat map. Here, the activation value may refer to a value indicating the degree to which a vision-language model pays attention to each pixel or each area. Additionally, the welding area detection device can detect an area as a welding area that includes pixels or areas where the activation value exceeds a predetermined threshold. Here, the predetermined threshold can be arbitrarily set by the user. Furthermore, the predetermined threshold can be set based on various welding type information, including information on the work environment, material information of the structure, work information, and information on the welding device.

[0113] In step 860, the weld area detection device can generate three-dimensional information about the welded part.

[0114] For example, a weld area detection device can acquire image data and a PCD corresponding to the image data. In addition, the weld area detection device can calculate three-dimensional information of the weld area using the image data and the PCD. Specifically, the weld area detection device can calculate three-dimensional information of the weld area by matching the image data with the PCD and back-projecting it.

[0115] In step 870, the welding area detection device can transmit three-dimensional information about the welding area to the welding robot.

[0116] In addition, the welding area detection device can control the welding device to perform welding operations on the welding area. The welding area detection device can achieve complete automation of the welding operation without manually specifying the welding area.

[0117] An embodiment according to the present invention may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded on a computer-readable medium. In this case, the medium may include a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and a hardware device specifically configured to store and execute program instructions, such as a ROM, RAM, or flash memory.

[0118] Meanwhile, the above-mentioned computer program may be one specifically designed and configured for the present invention or one known and available to those skilled in the art of computer software. Examples of computer programs may include not only machine code, such as that generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc.

[0119] According to one embodiment, the method according to various embodiments of the present disclosure may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two users and devices. In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0120] Unless explicitly stated or contrary to the order of the steps constituting the method according to the present invention, said steps may be performed in a suitable order. The present invention is not necessarily limited by the order in which said steps are described. The use of all examples or exemplary terms (e.g., etc.) in the present invention is merely for the purpose of describing the present invention in detail, and the scope of the present invention is not limited by said examples or exemplary terms unless limited by the claims. Furthermore, those skilled in the art will understand that various modifications, combinations, and changes may be made according to design conditions and factors within the scope of the claims or equivalents to which they are added.

[0121] Accordingly, the scope of the present invention should not be limited to the embodiments described above, and all scopes equivalent to or equivalently modified from the claims set forth below, as well as the claims set forth below, shall be considered to fall within the scope of the concept of the present invention. Explanation of the symbols

[0122] 10: Image data 20: Text data 30: Vision-Language Model 40: Welding area 50: Welding device

Claims

Claim 1 A method comprising: a step of inputting image data into a Vision-Language Model; a step of acquiring feature data of the image data; a step of determining whether there is a part requiring welding within the image data using the feature data; and a step of detecting a welding area through a heat map of the part requiring welding in response to the determination that the part requiring welding exists. Claim 2 The method of claim 1 further comprises the step of embedding a first text for detecting the weld area into the vision-language model. Claim 3 A method according to claim 1, wherein the step of inputting image data comprises: a step of acquiring Point Cloud Data (PCD); a step of converting the PCD into image data; and a step of inputting the image data into the vision-language model. Claim 4 A method according to claim 1, wherein the step of determining whether the welding-required part exists comprises the step of obtaining a second text indicating whether the welding-required part exists within the image data. Claim 5 The method according to claim 1, wherein the step of detecting the welding area comprises: generating the heat map representing the activation value of each of the pixels included in the image data; and detecting the area including pixels in which the activation value exceeds a predetermined threshold as the welding area. Claim 6 In claim 5, the step of generating the heat map comprises the step of obtaining the activation value of each of the pixels through a predetermined layer of an encoder included in the vision-language model; a method. Claim 7 The method according to claim 1, wherein the step of detecting the welding area comprises the step of calculating three-dimensional information of the welding area using the image data and the PCD corresponding to the image data. Claim 8 A method according to claim 1, wherein the step of detecting the welding area comprises: a step of calculating welding type information of the welding area; and a step of detecting a welding start point and a welding end point within the welding area based on the welding type information. Claim 9 The method of claim 1 further comprises the step of generating an interface representing the welding area among the areas included in the image data. Claim 10 The method of claim 1 further comprises the step of controlling a welding robot to perform a welding operation on the welding area. Claim 11 A computing device comprising at least one memory; and at least one processor; wherein the at least one processor inputs image data and text data into a Vision-Language Model, acquires feature data of the image data, determines whether a welding-required part exists within the image data using the embedding results of the feature data and the text data, and detects a welding area through a heat map of the welding-required part in response to the determination that the welding-required part exists. Claim 12 In claim 11, the input of the image data and the text data is a computing device that acquires Point Cloud Data (PCD), converts the PCD into image data, and inputs the image data into the vision-language model. Claim 13 A computing device according to claim 11, wherein detecting the welding area comprises generating the heat map representing the activation value of each pixel included in the image data, and detecting the area including pixels whose activation value exceeds a predetermined threshold as the welding area. Claim 14 A computing device according to claim 11, wherein detecting the welding area involves calculating three-dimensional information of the welding area using the image data and a PCD corresponding to the image data. Claim 15 A computer-readable recording medium having a program for executing the method according to claim 1 on a computer.

Images