Method, system for generating a phage map based on object segmentation and, learning method, system
Patent Information
- Application Number
- KR1020240191306
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2024-12-19
- Publication Date
- 2026-09-16
- Estimated Expiration
- 2044-12-19
Smart Images
Figure R1020240191306_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to a method and system for generating a retention map based on object segmentation, and a learning method and system. Background Technology
[0002] The technology for estimating gripping maps is a technique that enables robots to effectively grasp various objects in real-world environments, and interest in it is growing in an increasing number of fields. Conventionally, gripping maps were determined using shape information of target objects based on geometric analysis, and for this purpose, a 3D model of the target object was required. Therefore, sufficient consideration of environmental variables such as surface irregularities, positional changes, and lighting changes of the target object is required.
[0003] Meanwhile, regarding techniques for estimating phage maps, methods have been proposed to predict phage maps more accurately by utilizing deep learning models based on deep neural networks, such as Convolutional Neural Networks (CNNs), to learn the shape and surface characteristics of objects. While these deep learning-based phage map estimation techniques were previously limited to estimating phage maps specific to a single object after training, methods capable of processing multiple objects simultaneously are increasingly being researched. In other words, this technology is executed by identifying and classifying target objects through a separate network and estimating phage maps for the classified objects. The problem to be solved
[0004] The present invention relates to a method and system for generating a retention map based on object segmentation, capable of generating a more accurate retention map for an unlearned target object, and a learning method and system.
[0005] In addition, the present invention relates to a method and system for generating a retention map based on object segmentation, and a learning method and system, which can perform effective retention map estimation by omitting a learning process that is complex and requires a large amount of resources and performing an efficient adaptation process. means of solving the problem
[0006] To solve the problem described above, a method for generating a gripping map according to the present invention may include: receiving a prompt generated to specify the target object along with an image in which the target object is captured; extracting a feature map corresponding to the image using a pre-prepared image encoder; inputting the prompt into a pre-prepared prompt encoder to generate a token for the target object determined to be gripped in the image; and inputting the feature map and the token into a pre-prepared mask decoder to generate a mask representing an area corresponding to the target object in the image and a gripping map representing an area where the target object can be gripped.
[0007] Additionally, the grip map generation system according to the present invention includes: a storage unit that stores an image in which a target object is captured and a prompt generated to specify the target object; and a control unit that generates a grip map corresponding to the image and the prompt using an image encoder, a prompt encoder, and a mask decoder. The control unit may extract a feature map corresponding to the image using the image encoder, input the prompt to the prompt encoder to generate a token for the target object determined to be gripped in the image, and input the feature map and the token to the mask decoder to generate a mask representing an area corresponding to the target object in the image and a grip map representing an area where the target object can be gripped.
[0008] Additionally, a program stored on a computer-readable recording medium according to the present invention is executed by one or more processes in an electronic device and is a program stored on a computer-readable recording medium, wherein the program may include instructions for performing the steps of: receiving a prompt generated to specify the target object along with an image in which the target object is captured; extracting a feature map corresponding to the image using a pre-prepared image encoder; inputting the prompt into a pre-prepared prompt encoder to generate a token for the target object determined to be grasped in the image; and inputting the feature map and the token into a pre-prepared mask decoder to generate a mask representing an area corresponding to the target object in the image and a grasping map representing an area where the target object can be grasped.
[0009] Additionally, the learning method according to the present invention may include the steps of: receiving a learning image, a learning prompt, and correct answer data; extracting a learning feature map corresponding to the learning image using a pre-prepared image encoder and inputting the learning prompt into a pre-prepared prompt encoder to generate a learning token for a target object determined to be grasped in the learning image; inputting the learning feature map and the learning token into a pre-prepared mask decoder to generate a learning mask representing an area corresponding to the target object in the learning image and a learning grasping map representing an area where the target object can be grasped; and calculating a loss function by comparing the correct answer data with the learning mask and the learning grasping map, and training at least one of the image encoder, the prompt encoder, and the mask decoder based on the loss function.
[0010] Additionally, the learning system according to the present invention includes: a storage unit in which a learning image, a learning prompt, and correct answer data are stored; and a control unit that trains at least one of an image encoder, a prompt encoder, and a mask decoder using the learning image, the learning prompt, and the correct answer data. The control unit extracts a learning feature map corresponding to the learning image using the image encoder, inputs the learning prompt to the prompt encoder to generate a learning token for a target object determined to be grasped in the learning image, inputs the learning feature map and the learning token to the mask decoder to generate a learning mask representing an area corresponding to the target object in the learning image and a learning grasping map representing an area where the target object can be grasped, calculates a loss function by comparing the correct answer data with the learning mask and the learning grasping map, and trains at least one of the image encoder, the prompt encoder, and the mask decoder based on the loss function.
[0011] Additionally, a program stored on a computer-readable recording medium according to the present invention is executed by one or more processes in an electronic device and is a program stored on a computer-readable recording medium, wherein the program may include instructions for performing the steps of: receiving a training image, a training prompt, and correct answer data; extracting a training feature map corresponding to the training image using a pre-prepared image encoder and inputting the training prompt into a pre-prepared prompt encoder to generate a training token for a target object determined to be grasped in the training image; inputting the training feature map and the training token into a pre-prepared mask decoder to generate a training mask representing an area corresponding to the target object in the training image and a training grasping map representing an area capable of grasping the target object; and calculating a loss function by comparing the correct answer data with the training mask and the training grasping map, and training at least one of the image encoder, the prompt encoder, and the mask decoder based on the loss function. Effects of the invention
[0012] According to various embodiments of the present invention, a method and system for generating a retention map based on object segmentation and a learning method and system can generate a more accurate retention map for an unlearned target object by generating a retention map for a target object based on an image and a prompt specifying a target object on the image.
[0013] In addition, according to various embodiments of the present invention, a method and system for generating a phage map based on object segmentation and a learning method and system can perform effective phage map estimation by omitting a learning process that requires complex and large amounts of resources and performing an efficient adaptation process by equipping modules such as a trainable adapter and a multi-perceptron layer to an encoder and a decoder that have been pre-trained based on large-scale training data and training only said modules. Brief explanation of the drawing
[0014] FIG. 1 illustrates an embodiment of a phage map generation system according to the present invention. FIG. 2 illustrates a learning system according to the present invention. FIG. 3 illustrates a phage map generation system according to the present invention. FIG. 4 is a flowchart illustrating a learning method according to the present invention. FIG. 5 illustrates an example of training an image encoder and a prompt decoder. FIG. 6 is a flowchart illustrating a method for generating a phage map according to the present invention. FIG. 7 illustrates an example of generating a feature map. FIG. 8 illustrates an example of generating a token. FIG. 9 illustrates an example of generating a mask and a phage map. FIG. 10 is a block diagram illustrating the structure of a computing device that performs the learning method and the retention map generation method of the present invention. Specific details for implementing the invention
[0015] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components are assigned the same reference number regardless of the drawing symbols, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not have distinct meanings or roles in themselves. Furthermore, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.
[0016] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.
[0017] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.
[0018] Singular expressions include plural expressions unless the context clearly indicates otherwise.
[0019] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.
[0020] FIG. 1 illustrates an embodiment of a retention map generation system according to the present invention. FIG. 2 illustrates a learning system according to the present invention. FIG. 3 illustrates a retention map generation system according to the present invention.
[0021] Referring to FIG. 1, the grazing map generation system (200) according to the present invention receives a prompt (33) generated to specify the target object along with an image (31) of the target object, inputs the image (31) into an image encoder to generate a feature map, inputs the prompt (33) into a prompt encoder (4) to generate a token (34), and inputs the feature map and the token (34) into a mask decoder to generate a grazing map (37) for the target object.
[0022] Here, the target object may be an object that is grasped by a gripper provided in a control device such as a robot, and is configured to move its position or perform a specific interaction. Such target objects may include a variety of objects ranging from small to medium-sized objects used in daily life to large objects used in specific specialized fields such as industry or commerce.
[0023] That is, the target object may vary depending on the form of the robot, etc., provided to grasp the target object, but it should be understood that the type or shape of such target object is not limited in this specification.
[0024] Meanwhile, the image (31) may be a picture of a target object taken using a camera or an imaging element. The image (31) may be a picture of a single target object or a picture of two or more target objects. Additionally, the image (31) may include multiple pixels having RGB values, and the color value of each of the multiple pixels may be configured in a format other than RGB.
[0025] The prompt (33) may include information indicating a target object to be grasped in the image (31). This prompt (33) may be generated based on user input and, depending on the embodiment, may be generated in voice and text format, or may be generated to include information about one or more pixels selected in the image (31) based on user input. That is, the prompt (33) may include information in voice and text format indicating the type, color, shape, etc. of the target object, or may include information in the form of a bounding box or a polygon box according to polygon labeling set on the image (31) based on user input, or may include information in the form of a point set on the image (31) based on user input.
[0026] Meanwhile, the image encoder may be implemented (or trained) to analyze visual features from the image (31) and generate a feature map corresponding to the image (31). In one embodiment, the image encoder may be implemented based on a Vision Transformer (ViT). Such an image encoder may be pre-trained based on large-scale training data to extract a feature map from the image (31).
[0027] Accordingly, when an image (31) is input, the image encoder can divide the previously input image (31) into image patches of a predetermined size and convert each divided image patch into a feature vector to generate a feature map corresponding to the image (31). That is, the feature map generated by the image encoder may include multiple feature vectors corresponding to each of the multiple image patches divided from the image (31).
[0028] Additionally, the image encoder may be configured such that a plurality of encoder blocks (2) that generate multi-scale feature maps are connected according to the embodiment, and in this case, an adapter (3) may be provided between the plurality of encoder blocks (2) (or at the output terminal of each image encoder). Accordingly, each encoder block (2) may be implemented to receive a composite feature map formed by synthesizing a feature map generated from a previous encoder block and an adaptive feature map generated from an adapter (3), and to generate a new feature map corresponding to the input composite feature map.
[0029] At this time, the adapter (3) may be composed of learnable parameters, and the adapter (3) may also be implemented in the form of a Multi-Layer Perceptron (MLP). In one embodiment, the adapter (3) may be a Rein adapter.
[0030] Accordingly, the adapter (3) can be learned according to the learning method described below, and at this time, the adapter (3) can be learned by the learning system (100) according to the present invention. In addition, the encoder block (2) on which the adapter (3) is provided may have been learned in advance through a separate system, that is, it can be understood that the encoder block (2) is not learned separately during the process of learning the adapter (3). In other words, the learning system (100) according to the present invention can be understood as adapting a large artificial intelligence model to a grasping task.
[0031] In one embodiment, the image encoder can be represented as shown in Equation 1 below, and the adapter (3) can be represented as shown in Equation 2 below.
[0032]
[0033] Here, B i represents the i-th encoder block (2) of the image encoder, and f i represents a feature map generated in the i-th encoder block (2), and represents a feature map generated from an adapter (3) provided in the i-th encoder block (2), and f out It can represent the feature map output from the image encoder.
[0034] Additionally, PE represents an embedding block (1) (e.g., Patch Embad) implemented to divide an image (31) into image patches of a predetermined size, n represents the number of image patches, and c may represent the embedding dimension of the first feature map.
[0035]
[0036] Here, Ad represents the adapter (3), can represent the embedding dimension for the i-th feature map.
[0037] Additionally, the prompt encoder (4) may be implemented (or trained) to analyze an image (31) using a prompt (33) generated based on user input and to generate a token (34) according to the analysis result. At this time, the token (34) is a learnable token and may include information about a target object determined to be grasped from the image (31) according to user input (or prompt (33)), and, for example, may be generated in the form of a parameter corresponding to an area (or target object) selected in the image (31) based on the prompt (33).
[0038] To this end, the prompt encoder (4) can be trained to generate a bounding box (or polygonal box) for a target object based on the prompt (33) when the prompt (33) (and image (31)) is input, and to generate a feature vector for an image region corresponding to the previously generated bounding box as a token (34).
[0039] Such a prompt encoder (4) can be learned according to the learning method described below, and at this time, the prompt encoder (4) can be learned by the learning system (100) according to the present invention. In addition, according to an embodiment, the prompt encoder (4) may have been learned in advance, and in such case, the prompt encoder (4) can be understood as having been learned through a separate system.
[0040] Meanwhile, the mask decoder can generate a mask (35) and a gripping map (37) by analyzing the feature map generated from the image (31) and the token (34) generated from the prompt (33) together. At this time, the mask (35) includes information about the area where the target object exists in the image (31), and the gripping map (37) may include information about the angle and width of the gripper capable of gripping the target object according to the mask (35).
[0041] To this end, the mask decoder may include a plurality of decoder blocks (5), a feature map fusion block (7), a multi-perceptron layer (6), and a phage header (8). Accordingly, the mask decoder can be implemented such that when a feature map is generated from an image encoder and a token (34) is generated from a prompt encoder (4), the feature map and the token (34) are input to the plurality of decoder blocks (5), and the data output from the plurality of decoder blocks (5) is input to the multi-perceptron layer (6).
[0042] Additionally, the mask decoder can be implemented such that multi-scale feature maps generated based on a plurality of encoder blocks (5) are synthesized and input into a feature map fusion block (7), and a mask (35) corresponding to a target object is generated by synthesizing the data output from the feature map fusion block (7) and the data output from the multi-perceptron layer (6).
[0043] Additionally, the mask decoder can be implemented so that the previously generated mask (35) is input into the grip header (8) to generate a grip map (37) corresponding to the target object. At this time, the grip header (8) may include a mask header, an identification confidence header, a gripper angle header, and a gripper width header, and each header may include a learnable parameter.
[0044] Accordingly, the mask header can be trained to generate a mask (35) for an area corresponding to a target object based on an image (31) and a token (34), the identification confidence header can be trained to estimate the confidence of the mask (35) generated from the mask header, the gripper angle header can be trained to estimate the angle of the gripper capable of gripping the target object, and the gripper width header can be trained to estimate the width of the gripper capable of gripping the target object.
[0045] In this regard, multiple decoder blocks (5) may be pre-trained based on large-scale training data, and the first decoder block may be implemented to receive a feature map generated from the last encoder block of the image encoder and a token (34) generated from the prompt encoder (4), and the other decoder blocks may be implemented to receive data generated from the previous decoder block.
[0046] Meanwhile, the feature map fusion block (7), the multi-perceptron layer (6), and the phage header (8) can be learned according to the learning method described later, and at this time, the feature map fusion block (7), the multi-perceptron layer (6), and the phage header (8) can be learned by the learning system (100) according to the present invention. In addition, the decoder block (5) may have been learned in advance through a separate system, that is, it can be understood that the decoder block (5) is not learned separately during the process of learning the feature map fusion block (7), the multi-perceptron layer (6), and the phage header (8).
[0047] Additionally, the feature map fusion block (7), multi-perceptron layer (6), and phage header (8) included in the mask decoder and the adapter (3) included in the image encoder may be learned together, or the feature map fusion block (7), multi-perceptron layer (6), phage header (8), and adapter (3) may be learned independently of each other or sequentially according to a predetermined order.
[0048] For example, with reference to FIG. 2, a learning system (100) can learn an adapter included in an image encoder (21), a feature map fusion block included in a mask decoder (25), a multi-perceptron layer, and a phage header using learning data (10) consisting of a learning image (11), a learning prompt (13), and correct answer data (15).
[0049] Here, the training prompt (13) may be a prompt provided to correspond to the training image (11), and the correct answer data (15) may include a correct answer mask for a target object provided to correspond to the training image (11) and the training prompt (13), and a correct answer gripping map, wherein the correct answer gripping map may include a gripper angle and a gripper width for gripping the target object according to the correct answer mask. That is, the correct answer data (15) may include a correct answer mask labeled on the training image (11) and the training prompt (13), and a correct answer gripping map.
[0050] Accordingly, the learning system (100) can generate a feature map by inputting a training image (11) into an image encoder (21), generate a token by inputting a training prompt (13) into a prompt encoder (23), and generate a training mask and a training propagation map by inputting the feature map and the token into a mask decoder (25).
[0051] Through this, the learning system (100) can calculate a mask loss by comparing a learning mask with a correct answer mask, and calculate a phage loss by comparing a learning phage map with a correct answer phage map. Additionally, the learning system (100) can calculate (or define) a loss function by summing the previously calculated mask loss and phage loss, and can train a feature map fusion block, a multi-perceptron layer, a phage header, and an adapter based on this loss function.
[0052] In one embodiment, the learning system (100) may define a loss function according to the following mathematical formula 3.
[0053]
[0054] Here, L mask represents mask loss, and L grasprepresents the retention loss, where λ1 represents a predetermined hyperparameter or weight (e.g., 2) for the mask loss, and λ2 represents a predetermined hyperparameter or weight (e.g., 1) for the retention loss.
[0055] That is, the learning system (100) calculates a loss function by assigning weights to the mask loss and the retention loss, respectively, and may assign a greater weight to the mask loss than to the retention loss.
[0056] Meanwhile, the learning system (100) may calculate a retention loss based on a learning mask (or a correct answer mask). In this case, the learning system (100) may calculate the retention loss by assigning weights to the loss calculated for the area corresponding to the learning mask and the loss calculated for the area not corresponding to the learning mask in the retention map, respectively, and at this time, a greater weight may be assigned to the area corresponding to the learning mask than to the area not corresponding to the learning mask.
[0057] In one embodiment, the learning system (100) can calculate (or define) the retention loss according to the following mathematical formula 4.
[0058]
[0059] Here, L fore represents the loss calculated for the region corresponding to the training mask in the propagation map, and L back represents the loss calculated for regions in the propagation map that do not correspond to the training mask, λ3 represents a predetermined hyperparameter or weight (e.g., 1) for the loss of regions corresponding to the training mask, and λ4 represents a predetermined hyperparameter or weight (e.g., 0.01) for the loss of regions corresponding to the training mask.
[0060] In this regard, the learning system (100) according to the present invention may include an input unit (110), a storage unit (120), a control unit (130), and an output unit (140).
[0061] The input unit (110) may receive information necessary for the operation of the learning system (100) according to the present invention. To this end, the input unit (110) may be connected to a separate input device, server, or external storage device via a wireless or wired network.
[0062] Accordingly, the input unit (110) can receive learning data (10) (e.g., a learning image (11), a learning prompt (13), and correct answer data (15)) from a separate input device, a server, or an external storage device.
[0063] Additionally, the input unit (110) can receive user input required to train at least one of an image encoder (21), a prompt encoder (23), and a mask decoder (25) based on training data (10).
[0064] Additionally, the storage unit (120) may store commands and information necessary for the operation of the learning system (100) according to the present invention. For example, the storage unit (120) may store learning data (10) input through the input unit (110). Additionally, the storage unit (120) may store an image encoder (21), a prompt encoder (23), and a mask decoder (25).
[0065] Additionally, the storage unit (120) can store various data generated during the process of training at least one of the image encoder (21), prompt encoder (23), and mask decoder (25).
[0066] The control unit (130) can control the overall operation of the learning system (100) according to the present invention. That is, the control unit (130) can train an adapter included in an image encoder (21), a feature map fusion block included in a mask decoder (25), a multi-perceptron layer, and a phage header using learning data (10) composed of a learning image (11), a learning prompt (13), and correct answer data (15).
[0067] Specifically, the control unit (130) receives a training image (11), a training prompt (13), and correct answer data (15), extracts a training feature map corresponding to the training image (11) using a pre-prepared image encoder (21), and inputs the training prompt (13) into a pre-prepared prompt encoder (23) to generate a training token for a target object determined to be grasped from the training image (11).
[0068] To this end, the control unit (130) can extract a training feature map by inputting a training image (11) into an image encoder (21) which consists of a plurality of pre-trained encoder blocks and an adapter provided for each of the plurality of encoder blocks.
[0069] Additionally, the control unit (130) can input a training prompt (13) generated to identify a target object in a training image (11) into a pre-prepared prompt encoder (23) to generate a training token indicating the area where the target object is located in the training image (11).
[0070] Accordingly, the control unit (130) inputs a learning feature map and a learning token into a pre-prepared mask decoder (25) to generate a learning mask representing an area corresponding to a target object in a learning image (11) and a learning grasping map representing an area where the target object can be grasped.
[0071] That is, the control unit (130) can generate a learning mask and a learning phage map by inputting a learning feature map generated from an image encoder (21) and a learning token generated from a prompt encoder (23) into a mask decoder (25) which consists of a plurality of pre-learned decoder blocks, a feature map fusion block, a multi-perceptron layer, and a phage header.
[0072] Furthermore, the control unit (130) can calculate a loss function by comparing the correct answer data (15) with a learning mask and a learning retention map, and can train at least one of an image encoder (21), a prompt encoder (23), and a mask decoder (25) based on the loss function.
[0073] To this end, the control unit (130) calculates a mask loss by comparing a correct answer mask included in the correct answer data (15) with a learning mask, calculates a retention loss by comparing a correct answer retention map included in the correct answer data (15) with a learning retention map, calculates a loss function by summing the mask loss and the retention loss, and can train at least one of an image encoder (21), a prompt encoder (23), and a mask decoder (25) based on the calculated loss function.
[0074] At this time, the control unit (130) can fix the parameters of the multiple encoder blocks and train the adapter based on the previously calculated loss function for an image encoder (21) composed of multiple encoder blocks and an adapter provided for each of the multiple encoder blocks, and also fix the parameters of the multiple decoder blocks and train the feature map fusion block, the multi-perceptron layer, and the phage header for a mask decoder (25) composed of multiple decoder blocks, a feature map fusion block, a multi-perceptron layer, and a phage header based on the previously calculated loss function.
[0075] The output unit (140) can output information generated by the operation of the learning system (100) according to the present invention. To this end, the output unit (140) may be connected to a separate visual output device, server, or external storage device via a wireless or wired network.
[0076] Accordingly, the output unit (140) can output various data generated during the process of training at least one of the training data (10), image encoder (21), prompt encoder (23), and mask decoder (25) so that the user can visually check them through a separate output device, server, or external storage device, etc., and according to an embodiment, the various data generated during the process of training at least one of the training data (10), image encoder (21), prompt encoder (23), and mask decoder (25) may also be transmitted to another device.
[0077] Meanwhile, referring to FIG. 3, the grazing map generation system (200) according to the present invention may include an input unit (210), a storage unit (220), a control unit (230), and an output unit (240).
[0078] The input unit (210) may receive information necessary for the operation of the grazing map generation system (200) according to the present invention. To this end, the input unit (210) may be connected to a separate input device, server, or external storage device via a wireless or wired network.
[0079] Accordingly, the input unit (210) may receive an image (31) and a prompt (33) from a separate input device, server, or external storage device. At this time, the input unit (210) may receive user input for generating the prompt (33), and the input unit (210) may also receive user input required during the process of generating a grip map (37) for a target object based on the image (31) and the prompt (33).
[0080] Additionally, the storage unit (220) may store commands and information necessary for the operation of the phage map generation system (200) according to the present invention. For example, the storage unit (220) may store an image (31) and a prompt (33) input through the input unit (210). Additionally, the storage unit (220) may store an image encoder (40), a prompt encoder (50), and a mask decoder (60).
[0081] Additionally, the storage unit (220) can store various data generated during the process of generating a phage map (37) using an image encoder (40), a prompt encoder (50), and a mask decoder (60).
[0082] The control unit (230) can control the overall operation of the grip map generation system (200) according to the present invention. That is, the control unit (230) receives a prompt (33) generated to specify the target object along with an image (31) in which the target object is captured, inputs the image (31) into an image encoder (40) to generate a feature map, inputs the prompt (33) into a prompt encoder (50) to generate a token, and inputs the feature map and the token into a mask decoder (60) to generate a grip map (37) for the target object.
[0083] Specifically, the control unit (230) can receive a prompt (33) generated to identify the target object along with an image (31) in which the target object is captured. At this time, the control unit (230) can receive a prompt (33) generated to identify the target object in the image (31) according to user input along with the image (31).
[0084] Accordingly, the control unit (230) can extract a feature map corresponding to the image (31) using a pre-prepared image encoder (40). To do this, the control unit (230) can extract a feature map by inputting the image (31) into an image encoder (40) which consists of a plurality of encoder blocks and an adapter provided for each of the plurality of encoder blocks.
[0085] Additionally, the control unit (230) can input a prompt (33) into a pre-prepared prompt encoder (50) to generate a token for a target object determined to be grasped in the image (31). To this end, the control unit (230) can input a prompt (33) generated to specify the target object in the image (31) according to user input into the pre-prepared prompt encoder (50) to generate a token indicating the area where the target object is located in the image (31).
[0086] Furthermore, the control unit (230) can input a feature map and a token into a pre-prepared mask decoder (60) to generate a mask (35) representing an area corresponding to a target object in an image (31) and a grasping map (37) representing an area where the target object can be grasped.
[0087] That is, the control unit (230) can generate a mask (35) and a phage map (37) by inputting a feature map generated in an image encoder (40) and a token generated in a prompt encoder (50) into a mask decoder (60) which consists of a plurality of decoder blocks, a feature map fusion block, a multi-perceptron layer, and a phage header.
[0088] The output unit (240) can output information generated by the operation of the grazing map generation system (200) according to the present invention. To this end, the output unit (240) may be connected to a separate visual output device, server, or external storage device via a wireless or wired network.
[0089] Accordingly, the output unit (240) can output various data generated during the process of generating a mask (35) and a phage map (37) using an image encoder (40), a prompt encoder (50), and a mask decoder (60) so that the user can visually check the image (31) and the prompt (33) through a separate output device, server, or external storage device, and according to the embodiment, the various data generated during the process of generating a mask (35) and a phage map (37) using an image encoder (40), a prompt encoder (50), and a mask decoder (60) can be transmitted to another device.
[0090] Based on the configuration of the learning system (100) and the retention map generation system (200) examined above, the learning method and the retention map generation method will be explained in more detail below.
[0091] FIG. 4 is a flowchart illustrating a learning method according to the present invention. FIG. 5 illustrates an embodiment of training an image encoder and a prompt decoder. FIG. 6 is a flowchart illustrating a method for generating a phage map according to the present invention. FIG. 7 illustrates an embodiment of generating a feature map. FIG. 8 illustrates an embodiment of generating a token. FIG. 9 illustrates an embodiment of generating a mask and a phage map.
[0092] Referring to FIG. 4, the learning system (100) according to the present invention receives a learning image, a learning prompt, and correct answer data (S100), extracts a learning feature map corresponding to the learning image using a pre-prepared image encoder, and inputs a learning prompt into a pre-prepared prompt encoder to generate a learning token for a target object determined to be grasped from the learning image (S200).
[0093] Specifically, the learning system (100) can extract a training feature map by inputting a training image into an image encoder composed of a plurality of pre-trained encoder blocks and an adapter provided for each of the plurality of encoder blocks.
[0094] For example, the learning system (100) may divide a training image into image patches of a predetermined size and input the divided training images into a first encoder block to generate a first training feature map. Additionally, the learning system (100) may input the first training feature map into a first adapter provided in the first encoder block to generate a first training adapter feature map.
[0095] Accordingly, the learning system (100) can generate a first synthetic feature map for learning by synthesizing (or connecting) the first learning feature map and the first learning adapter feature map, and generate a second learning feature map by inputting the first synthetic feature map for learning into the second encoder block. Additionally, the learning system (100) can generate a second learning adapter feature map by inputting the second learning feature map into the second adapter provided in the second encoder block.
[0096] Accordingly, the learning system (100) can generate a second synthetic feature map for learning by synthesizing (or connecting) a second feature map for learning and a second adapter feature map for learning, and can generate a multi-scale feature map for learning by repeating the process of generating a feature map for learning through a plurality of encoder blocks provided in an image encoder and an adapter provided in each encoder block.
[0097] At this time, the multiple training feature maps corresponding to the multi-scale training feature maps may include training feature maps output from multiple encoder blocks, and the final training feature map output from the image encoder may be the training feature map output from the last encoder block among the multiple encoder blocks.
[0098] Furthermore, the learning system (100) can input a learning prompt generated to specify a target object in a learning image into a pre-prepared prompt encoder, thereby generating a learning token that indicates the area where the target object is located in the learning image.
[0099] For example, the learning system (100) can input a learning prompt generated in the form of a point corresponding to a learning image into a prompt encoder. In this case, the learning system (100) can generate a bounding box (or polygonal box) representing the area of a target object based on the location where the learning prompt is specified on the learning image through the prompt encoder, and can generate the previously generated bounding box (or parameter converted from the bounding box) as a learning token.
[0100] As another example, the learning system (100) may input a learning prompt generated in the form of voice or text corresponding to a learning image into a prompt encoder. In this case, the learning system (100) may generate a bounding box (or polygonal box) representing the area of a target object on the learning image based on the learning prompt through the prompt encoder, and may generate the previously generated bounding box (or parameter converted from the bounding box) as a learning token.
[0101] As another example, the learning system (100) may input a learning prompt generated in the form of a bounding box or a polygonal box corresponding to a learning image into a prompt encoder. In this case, the learning system (100) may generate a learning token by converting the learning prompt into a parameter (or, learnable token) form through the prompt encoder.
[0102] The learning system (100) according to the present invention can input a learning feature map and a learning token into a pre-prepared mask decoder to generate a learning mask representing an area corresponding to a target object in a learning image and a learning grasping map representing an area where the target object can be grasped (S300).
[0103] Specifically, the learning system (100) can generate a learning mask and a learning phage map by inputting a learning feature map generated from an image encoder and a learning token generated from a prompt encoder into a mask decoder consisting of a plurality of pre-learned decoder blocks, a feature map fusion block, a multi-perceptron layer, and a phage header.
[0104] For example, the learning system (100) can input a learning feature map generated in the last encoder block among a plurality of encoder blocks included in an image encoder and a learning token generated in a prompt encoder into a first decoder block, and input data output from the first decoder block into a second decoder block.
[0105] Accordingly, the learning system (100) can repeat the process of inputting data output from the previous decoder block into each of the plurality of decoder blocks, and input data output from the last decoder block among the plurality of decoder blocks into the multi-perceptron layer.
[0106] Additionally, the learning system (100) can input multiple learning feature maps generated at different scales in each of the multiple encoder blocks included in the image encoder into a feature map fusion block, and synthesize (or connect) the data output from the feature map fusion block and the data output from the multi-perceptron layer to generate a learning mask.
[0107] At this time, the learning system (100) can input a learning mask previously generated into the learning header to generate a learning memory map corresponding to the learning mask.
[0108] The learning system (100) according to the present invention can calculate a loss function by comparing the correct answer data with a learning mask and a learning retention map, and can train at least one of an image encoder, a prompt encoder, and a mask decoder based on the loss function (S400).
[0109] Specifically, as shown in FIG. 5, the learning system (100) calculates a mask loss by comparing a correct answer mask (17) included in the correct answer data (15) with a learning mask (27), calculates a retention loss by comparing a correct answer retention map (18) included in the correct answer data (15) with a learning retention map (28), calculates a loss function (29) by summing the mask loss and the retention loss, and can train at least one of an image encoder (21), a prompt encoder, and a mask decoder (23) based on the calculated loss function (29).
[0110] At this time, the learning system (100) can fix the parameters of the multiple encoder blocks and train the adapter based on the previously calculated loss function (29) for an image encoder (21) composed of multiple encoder blocks and an adapter provided for each of the multiple encoder blocks, and also fix the parameters of the multiple decoder blocks and train the feature map fusion block, the multi-perceptron layer, and the phage header for a mask decoder (25) composed of multiple decoder blocks, a feature map fusion block, a multi-perceptron layer, and a phage header based on the previously calculated loss function (29).
[0111] In this regard, the learning system (100) can calculate a loss function (29) by assigning weights to the mask loss and the retention loss, respectively, and the learning system (100) can assign a greater weight to the mask loss than to the retention loss.
[0112] Additionally, the learning system (100) can classify the retention loss into a mask area loss and a non-mask area loss based on the learning mask (27). At this time, the mask area loss may be a loss calculated for the area corresponding to the learning mask (27) in the learning retention map (28), and the non-mask area loss may be a loss calculated for the area not corresponding to the learning mask (27) in the learning retention map (28).
[0113] Accordingly, the learning system (100) can calculate the retention loss by assigning weights to the mask region loss and the non-mask region loss, respectively, and the learning system (100) can assign a greater weight to the mask region loss than to the non-mask region loss.
[0114] Referring to FIG. 6, the grazing map generation system (200) according to the present invention can receive a prompt generated to specify a target object along with an image of the target object being captured (S500).
[0115] Specifically, the shard map generation system (200) can receive a prompt generated to specify a target object in the image according to user input, along with the image.
[0116] For example, the shard map generation system (200) may receive a prompt generated in the form of voice or text along with an image. In this case, the prompt may include information describing a target object to be identified in the image.
[0117] As another example, the phage map generation system (200) may receive a prompt generated in the form of a point along with an image. In this case, the prompt may include information indicating a specific location (or a specific pixel) on the image.
[0118] As another example, the spool map generation system (200) may receive a prompt generated in the form of a bounding box or a polygonal box along with an image. In this case, the prompt may include information indicating the location (or pixel) of the bounding box or the polygonal box on the image.
[0119] The phage map generation system (200) according to the present invention can extract a feature map corresponding to an image using a pre-prepared image encoder (S600).
[0120] Specifically, the phage map generation system (200) can extract a feature map by inputting an image into an image encoder comprising a plurality of encoder blocks and an adapter provided in each of the plurality of encoder blocks, wherein the adapter may be learned by the learning system (100) according to the present invention.
[0121] For example, with reference to FIG. 7, the phage map generation system (200) can divide an image (31) into image patches of a predetermined size and input the divided images into a first encoder block (41) to generate a first feature map (71). Additionally, the phage map generation system (200) can input the first feature map (71) into a first adapter (46) provided in the first encoder block (41) to generate a first adapter feature map.
[0122] Accordingly, the phage map generation system (200) can generate a first composite feature map by synthesizing (or connecting) the first feature map (71) and the first adapter feature map, and generate a second feature map (72) by inputting the first composite feature map into the second encoder block (42). Additionally, the phage map generation system (200) can generate a second adapter feature map by inputting the second feature map (72) into the second adapter (47) provided in the second encoder block (42).
[0123] Accordingly, the phage map generation system (200) can generate a second composite feature map by synthesizing (or connecting) the second feature map (72) and the second adapter feature map, and can generate a multi-scale feature map (70) by repeating the process of generating feature maps through a plurality of encoder blocks provided in an image encoder and an adapter provided in each encoder block.
[0124] At this time, the multiple feature maps (71, 72, 73) corresponding to the multiple scale feature map (70) may include feature maps output from multiple encoder blocks, and the final feature map output from the image encoder may be the feature map (73) output from the last encoder block among the multiple encoder blocks.
[0125] Referring again to FIG. 6, the grip map generation system (200) according to the present invention can generate a token for a target object to be gripped in an image by inputting a prompt into a pre-prepared prompt encoder (S700).
[0126] Specifically, the shard map generation system (200) can generate a token (51) indicating the area where the target object is located in the image (31) by inputting a prompt (33) generated to specify the target object in the image (31) according to user input into a pre-prepared prompt encoder (50) as shown in FIG. 8.
[0127] For example, the rift map generation system (200) can input a prompt generated in the form of a point corresponding to an image into a prompt encoder. In this case, the rift map generation system (200) can generate a bounding box (or polygonal box) representing the area of a target object based on the location where the prompt is specified on the image through the prompt encoder, and can generate the previously generated bounding box (or parameter converted from the bounding box) as a token.
[0128] As another example, the phage map generation system (200) can input a prompt generated in the form of voice or text corresponding to an image into a prompt encoder. In this case, the phage map generation system (200) can generate a bounding box (or polygonal box) representing the area of a target object on the image based on the prompt through the prompt encoder, and can generate the previously generated bounding box (or parameter converted from the bounding box) as a token.
[0129] As another example, the perimap generation system (200) can input a prompt generated in the form of a bounding box or a polygonal box corresponding to an image into a prompt encoder. In this case, the perimap generation system (200) can generate a token by converting the prompt into a parameter form through the prompt encoder.
[0130] Referring again to FIG. 6, the gripping map generation system (200) according to the present invention inputs a feature map and a token into a pre-prepared mask decoder to generate a mask representing an area corresponding to a target object in an image and a gripping map representing an area where the target object can be gripped (S800).
[0131] Specifically, the phage map generation system (200) can generate a mask and a phage map by inputting a feature map generated in an image encoder and a token generated in a prompt encoder into a mask decoder composed of a plurality of decoder blocks, a feature map fusion block, a multi-perceptron layer, and a phage header, wherein the feature map fusion block, the multi-perceptron layer, and the phage header may be learned by the learning system (100) according to the present invention.
[0132] For example, with reference to FIG. 9, the phage map generation system (200) inputs a feature map (70) generated from the last encoder block among the plurality of encoder blocks included in the image encoder and a token (51) generated from the prompt encoder into the first decoder block (61), and inputs the data output from the first decoder block (61) into the second decoder block (62). Accordingly, the phage map generation system (200) can repeat the process of inputting the data output from the previous decoder block into each of the plurality of decoder blocks.
[0133] That is, the phage map generation system (200) can input a feature map (70) generated from an image encoder and a token (51) generated from a prompt encoder into the first decoder block among the plurality of decoder blocks, and input data output from the last decoder block among the plurality of decoder blocks into a multi-perceptron layer (65).
[0134] Additionally, the phage map generation system (200) inputs multiple feature maps generated at different scales from each of the multiple encoder blocks included in the image encoder into the feature map fusion block (63), and can generate a mask (35) by synthesizing (or connecting) the data output from the feature map fusion block (63) and the data output from the multi-perceptron layer (65).
[0135] At this time, the spool map generation system (200) can input the previously generated mask (35) into the spool header (67) to generate a spool map (37) corresponding to the mask (35).
[0136] Through the above configurations, the learning system (100) and the retention map generation system (200) according to the present invention can generate a more accurate retention map for an unlearned target object by generating a retention map for a target object based on an image and a prompt specifying a target object on the image.
[0137] In addition, the learning system (100) and the phage map generation system (200) according to the present invention can perform effective phage map estimation by omitting a learning process that requires complex and large amounts of resources and performing an efficient adaptation process by equipping modules such as a trainable adapter and a multi-perceptron layer to an encoder and decoder that have been pre-trained based on large-scale learning data and training only the said modules. That is, the learning system (100) according to the present invention can adapt a pre-trained large artificial intelligence model to the phage task.
[0138] Furthermore, the learning system (100) and the memory map generation system (200) according to the present invention are configured as a computing device and can perform at least one function related to the aforementioned learning method and memory map generation method.
[0139] FIG. 10 is a block diagram illustrating the structure of a computing device that performs the learning method and the retention map generation method of the present invention.
[0140] The computing device (1000) may include a user interface module (1001), a network communication module (1002), one or more processors (1003), data storage (1004), one or more camera(s) (1018), one or more sensors (1020) and a power system (1022), all of which may be connected to each other via a system bus, a network or other connection mechanism (1005).
[0141] The user interface module (1001) may be operable to transmit data to an external user input / output device and / or receive data from an external user input / output device.
[0142] For example, in the present invention, the learning system (100) receiving learning data or the retention map generation system (200) receiving an image and a prompt can be performed by external input using a user interface module. At this time, the user interface module (1001) may include a touch screen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a voice recognition module, or other similar devices.
[0143] Additionally, the user interface module (1001) may also be configured to provide output to one or more user display devices, such as a cathode ray tube (CRT), liquid crystal display, light-emitting diode (LED), display using digital light processing (DLP) technology, or printer.
[0144] The user interface module (1001) may also be configured to produce an audible output using a device such as a speaker, a speaker jack, an audio output port, an audio output device, an earphone and / or other similar device.
[0145] The user interface module (1001) may additionally be configured with one or more tactile devices capable of generating tactile outputs, such as vibrations and / or other outputs, detectable by touch and / or physical contact with the computing device (1000).
[0146] The network communication module (1002) may include one or more devices that provide one or more wireless interface(s) (1007) and / or one or more wired interface(s) (1608) that are configurable to communicate through a network.
[0147] Additionally, the network communication module (1002) may be configured to provide reliable security and / or authenticated communication.
[0148] One or more processors (1003) may include one or more general-purpose processors and / or one or more special-purpose processors (e.g., digital signal processor, tensor processing unit (TPU), graphics processing unit (GPU), neural network processing unit (NPU), application integrated circuit, application semiconductor (ASIC), etc.). One or more processors (1003) may be configured to execute computer-readable instructions (1006) contained in data storage (1004) and / or other instructions described herein.
[0149] As an example of this, the learning and inference described in this specification are performed in a neural network processing unit (NPU), and efficiency can be increased by processing data operations at high speed with low power.
[0150] The data storage (1004) may include one or more non-transient computer-readable storage media that can be read and / or accessed by at least one of one or more processors (1003).
[0151] One or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, or other memory or disk storage devices. In some examples, data storage (1004) may be implemented using a single physical device (e.g., one optical, magnetic, organic, or other memory or disk storage device), whereas in other examples, data storage (1004) may be implemented using two or more physical devices.
[0152] The data storage (1004) may include computer-readable instructions (1006) and additional data. The data storage (1004) may include a storage necessary to perform at least some of the methods, scenarios, and techniques described herein and / or at least some of the functions of the device and network.
[0153] In some examples, the data storage (1004) may include a storage for the learned neural network model (1010) described in the present invention (e.g., an image encoder, a prompt encoder, and a mask decoder).
[0154] Meanwhile, the computing device (1000) may include one or more camera(s) (1018), one or more sensors (1020) and / or a power system (1022).
[0155] The camera(s) (1018) can capture light and / or electromagnetic radiation emitted as visible light, infrared radiation, ultraviolet radiation and / or light of one or more other frequencies. The sensor (1020) may be configured to measure conditions within the computing device (1000) and / or conditions of the computing device (1000) environment and to provide data regarding these conditions. The power system (1022) may include one or more batteries (1024) and / or one or more external power interfaces (1026) for providing power to the computing device (1000).
[0156] Meanwhile, although the learning system (100) and the memory map generation system (200) of the present invention have been described above as being implemented as a computing device, the present invention is not limited thereto. For example, the functions of the neural network and / or computing device may be distributed among a plurality of computing clusters.
[0157] Furthermore, the present invention described above can be implemented as a program that is executed by one or more processes in an electronic device and stored on a computer-readable recording medium.
[0158] Accordingly, the present invention can be implemented as computer-readable code or instructions on a medium on which a program is recorded. That is, various control methods according to the present invention can be provided in the form of integrated or individual programs.
[0159] Meanwhile, computer-readable media include all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.
[0160] Furthermore, the computer-readable medium may be a server or cloud storage that includes a storage and is accessible to an electronic device via communication. In this case, the computer may download the program according to the present invention from the server or cloud storage via wired or wireless communication.
[0161] Furthermore, in the present invention, the computer described above is an electronic device equipped with a processor, namely a CPU (Central Processing Unit), and no special limitations are placed on its type.
[0162] Meanwhile, the above detailed description should not be interpreted restrictively in all respects but should be considered exemplary. The scope of the invention shall be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are included within the scope of the invention.
Claims
Claim 1 A method for generating a phasing map, comprising: receiving a prompt generated to specify the target object along with an image in which the target object is captured; extracting a feature map corresponding to the image using a pre-prepared image encoder; inputting the prompt into a pre-prepared prompt encoder to generate a token for the target object determined to be grasped in the image; and inputting the feature map and the token into a pre-prepared mask decoder to generate a mask representing an area corresponding to the target object in the image and a phasing map representing an area capable of grasping the target object, wherein the image encoder comprises a plurality of encoder blocks and an adapter provided in each of the plurality of encoder blocks, and the mask decoder comprises a plurality of decoder blocks, a feature map fusion block, a multi-perceptron layer, and a phasing header. Claim 2 delete Claim 3 A method for generating a phage map according to claim 1, wherein the step of generating a phage map comprises: inputting a feature map generated in the image encoder and a token generated in the prompt encoder into the first decoder block among the plurality of decoder blocks; inputting data output from the last decoder block among the plurality of decoder blocks into the multi-perceptron layer; inputting a plurality of feature maps generated at different scales in each of the plurality of encoder blocks included in the image encoder into the feature map fusion block; synthesizing the data output from the feature map fusion block and the data output from the multi-perceptron layer to generate the mask; and inputting the generated mask into the phage header to generate a phage map corresponding to the mask. Claim 4 A phasing map generation system comprising: a storage unit storing an image in which a target object is captured, along with a prompt generated to specify the target object; and a control unit generating a phasing map corresponding to the image and the prompt using an image encoder, a prompt encoder, and a mask decoder, wherein the control unit extracts a feature map corresponding to the image using the image encoder, inputs the prompt to the prompt encoder to generate a token for the target object determined to be phased in the image, inputs the feature map and the token to the mask decoder to generate a mask representing an area corresponding to the target object in the image and a phasing map representing an area capable of phasing the target object, wherein the image encoder is composed of a plurality of encoder blocks and an adapter provided in each of the plurality of encoder blocks, and the mask decoder is composed of a plurality of decoder blocks, a feature map fusion block, a multi-perceptron layer, and a phasing header. Claim 5 A program stored on a computer-readable recording medium that is executed by one or more processes in an electronic device, the program comprising instructions for performing steps such as: receiving a prompt generated to specify the target object along with an image in which the target object is captured; extracting a feature map corresponding to the image using a pre-prepared image encoder; inputting the prompt into a pre-prepared prompt encoder to generate a token for the target object determined to be grasped in the image; and inputting the feature map and the token into a pre-prepared mask decoder to generate a mask representing an area corresponding to the target object in the image and a grasping map representing an area capable of grasping the target object, wherein the image encoder comprises a plurality of encoder blocks and an adapter provided for each of the plurality of encoder blocks, and the mask decoder comprises a plurality of decoder blocks, a feature map fusion block, a multiperceptron layer, and a grasping header. Claim 6 A step of receiving a training image, a training prompt, and correct answer data; a step of extracting a training feature map corresponding to the training image using a pre-prepared image encoder and inputting the training prompt into a pre-prepared prompt encoder to generate a training token for a target object determined to be grasped in the training image; a step of inputting the training feature map and the training token into a pre-prepared mask decoder to generate a training mask representing an area corresponding to the target object in the training image and a training grasping map representing an area where the target object can be grasped; A learning method comprising the step of calculating a loss function by comparing the above correct answer data with the above training mask and the above training phage map, and training at least one of the image encoder, the above prompt encoder and the above mask decoder based on the above loss function, wherein the image encoder is composed of a plurality of encoder blocks and an adapter provided in each of the plurality of encoder blocks, and the mask decoder is composed of a plurality of decoder blocks, a feature map fusion block, a multiperceptron layer, and a phage header. Claim 7 A learning method according to claim 6, wherein the learning step comprises: a step of calculating a mask loss by comparing a correct answer mask included in the correct answer data with a learning mask; a step of calculating a retention loss by comparing a correct answer retention map included in the correct answer data with a learning retention map; a step of calculating a loss function by summing the mask loss and the retention loss; and a step of learning at least one of the image encoder, the prompt encoder, and the mask decoder based on the calculated loss function. Claim 8 In claim 6, the learning step comprises, for an image encoder comprising a plurality of encoder blocks and an adapter provided for each of the plurality of encoder blocks, fixing the parameters of the plurality of encoder blocks and learning the adapter based on the calculated loss function, and for a mask decoder comprising a plurality of decoder blocks, a feature map fusion block, a multi-perceptron layer, and a phage header, fixing the parameters of the plurality of decoder blocks and learning the feature map fusion block, the multi-perceptron layer, and the phage header based on the calculated loss function. Claim 9 A storage unit for storing training images, training prompts, and correct answer data; The control unit includes a training unit that trains at least one of an image encoder, a prompt encoder, and a mask decoder using the training image, the training prompt, and the correct answer data. The control unit extracts a training feature map corresponding to the training image using the image encoder, inputs the training prompt to the prompt encoder to generate a training token for a target object determined to be grasped in the training image, inputs the training feature map and the training token to the mask decoder to generate a training mask representing an area corresponding to the target object in the training image and a training grasping map representing an area where the target object can be grasped, calculates a loss function by comparing the correct answer data with the training mask and the training grasping map, and trains at least one of the image encoder, the prompt encoder, and the mask decoder based on the loss function. The image encoder is composed of a plurality of encoder blocks and an adapter provided in each of the plurality of encoder blocks, and the mask decoder is composed of a plurality of decoders A learning system comprising blocks, a feature map fusion block, a multiperceptron layer, and a phage header. Claim 10 A program that is executed by one or more processes in an electronic device and stored on a computer-readable recording medium, wherein the program comprises: a step of receiving a training image, a training prompt, and correct answer data; a step of extracting a training feature map corresponding to the training image using a pre-prepared image encoder and inputting the training prompt into a pre-prepared prompt encoder to generate a training token for a target object determined to be grasped in the training image; and a step of inputting the training feature map and the training token into a pre-prepared mask decoder to generate a training mask representing an area corresponding to the target object in the training image and a training grasping map representing an area capable of grasping the target object. A program stored on a computer-readable recording medium, comprising instructions for calculating a loss function by comparing the above correct answer data, the above learning mask, and the above learning phage map, and for training at least one of the image encoder, the above prompt encoder, and the above mask decoder based on the above loss function, wherein the image encoder is composed of a plurality of encoder blocks and an adapter provided in each of the plurality of encoder blocks, and the mask decoder is composed of a plurality of decoder blocks, a feature map fusion block, a multiperceptron layer, and a phage header.