Method for performing annotation on basis of prompt

The method generates prompts from user input to improve annotation efficiency and accuracy for rare or domain-specific data by using multimodal models for AI annotation, addressing the limitations of existing AI models with sparse data containing multiple objects.

WO2024237705A1PCT designated stage expired Publication Date: 2024-11-21WIZGENE INC
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2024/006659
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2024-05-16
Filing Date
2024-05-16
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Existing artificial intelligence models struggle with annotating data that is rare or does not belong to a specific class or domain, as they are often dependent on specific data classes and perform poorly with sparse data containing a large number of objects.

Method used

A method is developed to generate prompts based on user input information, including text, voice, image, and 3D data, using multimodal models that learn text, image, voice, and 3D models, allowing for annotation of target images using AI models for object detection, segmentation, and skeleton detection.

Benefits of technology

This approach improves the efficiency and accuracy of annotation for sparse data with many objects by providing a prompt-based annotation method that can handle diverse input types and adjust AI model parameters for variable results, enhancing annotation performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2024006659_21112024_PF_FP_ABST
    Figure KR2024006659_21112024_PF_FP_ABST
Patent Text Reader

Abstract

According to one embodiment of the present disclosure, disclosed is a method for performing annotation on the basis of a prompt, which is performed by a computing device. The method may include the steps of: obtaining a target image to be annotated; generating the prompt for performing the annotation on the basis of user input information; and performing annotation on the target image by utilizing an artificial intelligence model on the basis of the generated prompt.
Need to check novelty before this filing date? Find Prior Art

Description

How to perform prompt-based annotations

[0001] The present invention relates to a method for performing annotation, and more particularly, to a technology for performing annotation by generating a prompt based on user input information.

[0002]

[0003] Most existing tools that utilize AI models for annotation utilize image-based AI models. Furthermore, because these AI models are often dependent on data from specific classes or domains, they present a challenge in annotating data that is rare or does not belong to a specific class or domain.

[0004] Korean Patent No. 10-1867558 (June 7, 2018) discloses a computerized method and device for annotating at least one feature of a view image.

[0005]

[0006] The present disclosure aims to provide a method for generating prompts based on user input information for data that is rare or does not belong to a specific class or domain, and for annotating the data using an artificial intelligence model.

[0007] Meanwhile, the technical task to be achieved by the present disclosure is not limited to the technical task mentioned above, and may include various technical tasks within a scope obvious to a person skilled in the art from the contents described below.

[0008]

[0009] According to one embodiment of the present disclosure for achieving the aforementioned task, a method for performing prompt-based annotation by a computing device is disclosed. The method may include the steps of: acquiring a target image for performing the annotation; generating a prompt for performing the annotation based on user input information; and annotating the target image using an artificial intelligence model based on the generated prompt.

[0010] In one embodiment, the user input information for generating the prompt may include at least one of text input, voice input, image input, video input, or 3D data input.

[0011] In one embodiment, the step of generating the prompt for performing the annotation based on the user input information may include the step of extracting a word or sentence for the target image using an image-text model; the step of displaying the extracted word or sentence as a recommended word or recommended sentence; and the step of generating the prompt based on the user input information received in response to the recommended word or recommended sentence.

[0012] In one embodiment, the user input information for generating the prompt includes two or more types of inputs among text input, voice input, image input, video input, or 3D data input, and the generated prompt may include at least one of a text prompt or an image prompt.

[0013] In one embodiment, the user input information includes at least one of the text input or the voice input; and at least one of the image input, the video input, or the 3D data input, and the text prompt is generated based on at least one of the text input or the voice recognition text generated by performing voice recognition on the voice input, and the image prompt can be generated based on at least one of the image input, a frame image included in the video input, or data obtained by converting the 3D data into 2D data.

[0014] In one embodiment, the user input information includes both the text input and the voice input, and the text prompt can be generated by combining the text input and a voice recognition text generated by performing voice recognition on the voice input.

[0015] In one embodiment, the user input information includes two or more inputs of the image input, the video input, or the 3D data input, and the image prompt can be generated by combining at least two or more of the image input, the frame image included in the video input, or the data converted from the 3D data to 2D data.

[0016] In one embodiment, the generated prompt includes at least one of a first type of prompt information or a second type of prompt information, and the first type of prompt information and the second type of prompt information may have opposite tendencies with respect to annotations.

[0017] In one embodiment, the artificial intelligence model for performing the annotation may include at least one of an object detection model that performs object detection based on the generated prompt; a segmentation model that performs segmentation based on the generated prompt; or a skeleton detection model that performs skeleton detection based on the generated prompt.

[0018] In one embodiment, the step of annotating the target image using an artificial intelligence model based on the generated prompt may include a step of variably providing annotation results for the target image by adjusting the values ​​of confidence or output parameters of the artificial intelligence model.

[0019] In one embodiment, the step of variably providing the result of annotation for the target image may include the step of variably displaying text or numbers representing the result of annotation for the target image; or the step of variably displaying a shape representing the result of annotation for the target image.

[0020] In one embodiment, the step of variably displaying a shape representing the result of annotation for the target image may include the step of variably displaying a contrast of a bounding box representing the result of annotation for the target image.

[0021] According to one embodiment of the present disclosure for achieving the above-described task, a computer program stored in a computer-readable storage medium is disclosed. When the computer program is executed on one or more processors, the computer program causes the one or more processors to perform the following operations for performing annotation based on a prompt, wherein the operations may include: an operation of acquiring a target image for performing the annotation; an operation of generating the prompt for performing the annotation based on user input information; and an operation of performing annotation on the target image using an artificial intelligence model based on the generated prompt.

[0022] In one embodiment, the user input information for generating the prompt may include at least one of text input, voice input, image input, video input, or 3D data input.

[0023] In one embodiment, the operation of generating the prompt for performing the annotation based on the user input information may include the operation of extracting a word or sentence for the target image using an image-text model; the operation of displaying the extracted word or the extracted sentence as a recommended word or a recommended sentence; and the operation of generating the prompt based on the user input information received in response to the recommended word or the recommended sentence.

[0024] In one embodiment, the user input information for generating the prompt includes two or more types of inputs among text input, voice input, image input, video input, or 3D data input, and the generated prompt may include at least one of a text prompt or an image prompt.

[0025] In one embodiment, the generated prompt includes at least one of a first type of prompt information or a second type of prompt information, and the first type of prompt information and the second type of prompt information may have opposite tendencies with respect to annotations.

[0026] In one embodiment, the artificial intelligence model for performing the annotation may include at least one of an object detection model that performs object detection based on the generated prompt; a segmentation model that performs segmentation based on the generated prompt; or a skeleton detection model that performs skeleton detection based on the generated prompt.

[0027] In one embodiment, the operation of annotating the target image using an artificial intelligence model based on the generated prompt may include an operation of variably providing annotation results for the target image by adjusting the values ​​of confidence or output parameters of the artificial intelligence model.

[0028] A computing device according to one embodiment of the present disclosure for achieving the aforementioned task is disclosed. The device comprises at least one processor; and a memory, wherein the at least one processor is configured to acquire a target image to be annotated; generate a prompt for performing the annotation based on user input information; and annotate the target image using an artificial intelligence model based on the generated prompt.

[0029] In one embodiment, the user input information for generating the prompt may include at least one of text input, voice input, image input, video input, or 3D data input.

[0030] In one embodiment, the at least one processor may include: extracting a word or sentence for the target image using an image-text model; displaying the extracted word or sentence as a recommended word or sentence; and generating the prompt based on the user input information received in response to the recommended word or sentence.

[0031] In one embodiment, the user input information for generating the prompt includes two or more types of inputs among text input, voice input, image input, video input, or 3D data input, and the generated prompt may include at least one of a text prompt or an image prompt.

[0032] In one embodiment, the generated prompt includes at least one of a first type of prompt information or a second type of prompt information, and the first type of prompt information and the second type of prompt information may have opposite tendencies with respect to annotations.

[0033] In one embodiment, the artificial intelligence model for performing the annotation may include at least one of an object detection model that performs object detection based on the generated prompt; a segmentation model that performs segmentation based on the generated prompt; or a skeleton detection model that performs skeleton detection based on the generated prompt.

[0034] In one embodiment, the at least one processor may be configured to variably provide annotation results for the target image by adjusting values ​​of confidence or output parameters of the artificial intelligence model.

[0035]

[0036] The present disclosure generates a prompt for performing annotation based on user input information, and performs annotation using an artificial intelligence model based on the generated prompt, thereby improving the efficiency and accuracy of annotation of data that is sparse and has a large number of objects.

[0037] Meanwhile, the effects of the present disclosure are not limited to the effects mentioned above, and various effects may be included within a range apparent to those skilled in the art from the contents described below.

[0038]

[0039] FIG. 1 is a block diagram of a computing device for performing annotation based on a prompt according to one embodiment of the present disclosure.

[0040] FIG. 2 is a conceptual diagram illustrating a neural network according to one embodiment of the present disclosure.

[0041] FIG. 3 is a schematic flowchart for performing prompt-based annotation according to one embodiment of the present disclosure.

[0042] FIG. 4 is a diagram illustrating an embodiment of generating a prompt by extracting words or sentences for a target image using an image-text model according to one embodiment of the present disclosure.

[0043] FIG. 5 is a flowchart illustrating a method for performing prompt-based annotation according to one embodiment of the present disclosure.

[0044] FIG. 6 is a simplified, general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

[0045]

[0046] Various embodiments are now described with reference to the drawings. In this specification, various descriptions are provided to facilitate understanding of the present disclosure. However, it will be apparent that these embodiments may be practiced without these specific details.

[0047] As used herein, the terms "component," "module," "system," and the like refer to computer-related entities, hardware, firmware, software, a combination of software and hardware, or an execution of software. For example, a component may be, but is not limited to, a procedure running on a processor, a processor, an object, a thread of execution, a program, and / or a computer. For example, both an application running on a computing device and the computing device may be a component. One or more components may reside within a processor and / or a thread of execution. A component may be localized within a single computer. A component may be distributed between two or more computers. Furthermore, these components may execute from various computer-readable media having various data structures stored therein. Components may communicate via local and / or remote processes, for example, by signals comprising one or more data packets (e.g., data from one component interacting with another component in a local system, a distributed system, and / or data transmitted to another system via a network such as the Internet via signals).

[0048] Furthermore, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from context, "X employs A or B" is intended to mean either of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, "X employs A or B" can apply to any of these cases. Furthermore, the term "and / or" as used herein should be understood to refer to and include all possible combinations of one or more of the associated items listed.

[0049] Additionally, the terms "comprises" and / or "comprising" should be understood to imply the presence of the features and / or components in question. However, it should be understood that the terms "comprises" and / or "comprising" do not exclude the presence or addition of one or more other features, components, and / or groups thereof. Furthermore, unless otherwise specified or clear from the context to refer to the singular form, the singular in the specification and claims should generally be construed to mean "one or more."

[0050] And, the term "at least one of A or B" should be interpreted to mean "if it includes only A", "if it includes only B", or "if it is combined in the composition of A and B".

[0051] Those skilled in the art should further appreciate that the various illustrative logical blocks, configurations, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0052] The description of the disclosed embodiments is provided to enable a person skilled in the art to make or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Therefore, the present invention is not limited to the embodiments disclosed herein. The present invention is to be construed in the widest scope consistent with the principles and novel features disclosed herein.

[0053] In the present disclosure, network function, artificial neural network and neural network can be used interchangeably.

[0054]

[0055] FIG. 1 is a block diagram of a computing device for performing prompt-based annotation according to one embodiment of the present disclosure.

[0056] The configuration of the computing device (100) illustrated in FIG. 1 is merely a simplified example. In one embodiment of the present disclosure, the computing device (100) may include other configurations for performing the computing environment of the computing device (100), and only some of the disclosed configurations may constitute the computing device (100).

[0057] A computing device (100) may include a processor (110), memory (130), and network unit (150).

[0058] The processor (110) may be configured with one or more cores, and may include a processor for data analysis and deep learning, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device. The processor (110) may read a computer program stored in the memory (130) to perform data processing for machine learning according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the processor (110) may perform operations for learning a neural network. The processor (110) may perform calculations for learning a neural network, such as processing input data for learning in deep learning (DL), extracting features from input data, calculating errors, and updating weights of a neural network using backpropagation. At least one of the CPU, GPGPU, and TPU of the processor (110) may process learning of a network function. For example, a CPU and a GPGPU can jointly process network function learning and data classification using network functions. Furthermore, in one embodiment of the present disclosure, processors of multiple computing devices can be jointly used to process network function learning and data classification using network functions. Furthermore, a computer program executed on a computing device according to one embodiment of the present disclosure may be a CPU, GPGPU, or TPU executable program.

[0059] According to one embodiment of the present disclosure, the memory (130) can store any form of information generated or determined by the processor (110) and any form of information received by the network unit (150).

[0060] According to one embodiment of the present disclosure, the memory (130) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk. The computing device (100) may also operate in relation to web storage that performs the storage function of the memory (130) on the internet. The description of the above-described memory is merely an example, and the present disclosure is not limited thereto.

[0061] The network unit (150) according to one embodiment of the present disclosure can use various wired communication systems such as a public switched telephone network (PSTN), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed ​​DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and a local area network (LAN).

[0062] In addition, the network unit (150) presented in this specification can use various wireless communication systems such as CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division Multi Access), SC-FDMA (Single Carrier-FDMA) and other systems.

[0063] In the present disclosure, the network unit (150) may be configured regardless of the communication mode, such as wired or wireless, and may be configured as various communication networks, such as a local area network (LAN), a personal area network (PAN), and a wide area network (WAN). In addition, the network may be the well-known World Wide Web (WWW), and may also utilize a wireless transmission technology used for short-distance communication, such as infrared (IrDA: Infrared Data Association) or Bluetooth.

[0064] The techniques described in this specification can be used in other networks as well as the networks mentioned above.

[0065]

[0066] FIG. 2 illustrates an exemplary structure of an artificial intelligence-based model according to one embodiment of the present disclosure.

[0067] Throughout this specification, the terms artificial intelligence model, artificial intelligence-based model, computational model, neural network, network function, and neural network may be used interchangeably.

[0068] A neural network can be composed of a set of interconnected computational units, generally referred to as nodes. These nodes can also be referred to as neurons. A neural network consists of at least one node. The nodes (or neurons) that make up a neural network can be interconnected by one or more links.

[0069] Within a neural network, one or more nodes connected via links can form a relationship between input nodes and output nodes. The concept of input nodes and output nodes is relative, meaning that any node that is in an output node relationship with one node can also be in an input node relationship with another node, and vice versa. As described above, the relationship between input nodes and output nodes can be created based on links. One input node can be connected to one or more output nodes via links, and vice versa.

[0070] In a relationship between input nodes and output nodes connected through a single link, the data of the output node can have its value determined based on the data input to the input node. Here, the link interconnecting the input nodes and output nodes can have a weight. The weight can be variable and can be varied by the user or an algorithm so that the neural network can perform a desired function. For example, when one or more input nodes are interconnected to one output node through each link, the output node can determine the output node value based on the values ​​input to the input nodes connected to the output node and the weight set on the link corresponding to each input node.

[0071] As described above, a neural network is a network in which one or more nodes are interconnected through one or more links, forming input and output node relationships within the network. The characteristics of a neural network can be determined based on the number of nodes and links within the network, the relationships between the nodes and links, and the weights assigned to each link. For example, if two neural networks have the same number of nodes and links but different weight values ​​for the links, the two neural networks can be perceived as different from each other.

[0072] A neural network can be composed of a set of one or more nodes. A subset of the nodes comprising the neural network can form a layer. Some of the nodes comprising the neural network can form a layer based on their distances from the initial input node. For example, a set of nodes that are n distances from the initial input node can form n layers. The distance from the initial input node can be defined by the minimum number of links required to reach the node from the initial input node. However, this definition of a layer is arbitrary for illustrative purposes, and the degree of a layer within a neural network can be defined in a different way than described above. For example, a layer of nodes can be defined by its distance from the final output node.

[0073] In one embodiment of the present disclosure, a set of neurons or nodes may be defined as a layer.

[0074] An initial input node may refer to one or more nodes within a neural network into which data is directly input without going through links with other nodes. Alternatively, within a neural network, it may refer to nodes that do not have other input nodes connected by links in the relationship between nodes based on links. Similarly, a final output node may refer to one or more nodes within a neural network that do not have output nodes in their relationship with other nodes. Furthermore, a hidden node may refer to nodes that constitute a neural network other than the initial input node and the final output node.

[0075] A neural network according to one embodiment of the present disclosure may be a neural network in which the number of nodes in an input layer may be the same as the number of nodes in an output layer, and the number of nodes decreases and then increases as it progresses from the input layer to the hidden layer. In addition, a neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes in an input layer may be less than the number of nodes in an output layer, and the number of nodes decreases as it progresses from the input layer to the hidden layer. In addition, a neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes in an input layer may be greater than the number of nodes in an output layer, and the number of nodes increases as it progresses from the input layer to the hidden layer. A neural network according to another embodiment of the present disclosure may be a neural network in a combined form of the neural networks described above.

[0076] A deep neural network (DNN) can refer to a neural network that includes multiple hidden layers in addition to input and output layers. Using a DNN, one can identify latent structures in data. This can include images, text, videos, audio, protein sequence structures, gene sequence structures, peptide sequence structures, the latent structure of music (e.g., what objects are in the image, what the content and emotion of the text are, what the content and emotion of the audio are, etc.), and / or the binding affinity between peptides and MHC. A DNN can include a convolutional neural network (CNN), a recurrent neural network (RNN), an autoencoder, a restricted Boltzmann machine (RBM), a deep belief network (DBN), a Q-network, a U-network, a Siamese network, a generative adversarial network (GAN), a transformer, and more. The description of the deep neural network described above is only an example and the present disclosure is not limited thereto.

[0077] The artificial intelligence-based model of the present disclosure can be represented by a network structure of any structure described above, including an input layer, a hidden layer, and an output layer.

[0078] The neural network that can be used in the artificial intelligence-based model of the present disclosure may be trained using at least one of supervised learning, self-supervised learning, unsupervised learning, semi-supervised learning, transfer learning, active learning, or reinforcement learning. Training of the neural network may be a process of applying knowledge to the neural network to perform a specific action.

[0079] Neural networks can be trained to minimize output errors. This process involves repeatedly inputting training data into the neural network, calculating the neural network output and target error for the training data, and backpropagating the neural network error from the output layer to the input layer to update the weights of each node in the neural network to reduce the error. In supervised learning, training data with the correct answer for each training data is used (i.e., labeled training data). In unsupervised learning, the correct answer may not be labeled for each training data. For example, in supervised learning for data classification, the training data may be data with each category labeled. Labeled training data is input to the neural network, and the error can be calculated by comparing the output (category) of the neural network with the training data labels. Alternatively, in unsupervised learning for data classification, the error can be calculated by comparing the input training data with the neural network output. The calculated error is backpropagated in the neural network in the backward direction (i.e., from the output layer to the input layer), and the connection weights of each node in each layer of the neural network can be updated according to the backpropagation. The amount of change in the connection weights of each node to be updated can be determined by the learning rate. The neural network's calculation of the input data and the backpropagation of the error can constitute a learning cycle (epoch). The learning rate can be applied differently depending on the number of iterations of the neural network's learning cycle. For example, a high learning rate can be used in the early stages of neural network training to quickly achieve a certain level of performance, thereby increasing efficiency. A lower learning rate can be used in the later stages of training to increase accuracy.

[0080] In neural network training, training data can typically be a subset of real-world data (i.e., the data to be processed using the trained neural network). Therefore, there can be a learning cycle where errors on the training data decrease but errors on the real-world data increase. Overfitting is a phenomenon where excessive training on the training data leads to increased errors on the real-world data. For example, a neural network trained on yellow cats may fail to recognize cats when shown non-yellow colors, a type of overfitting. Overfitting can increase errors in machine learning algorithms. Various optimization methods can be used to prevent overfitting. These methods include increasing the training data, regularization, dropout, which disables some nodes in the network during the learning process, and the use of batch normalization layers.

[0081]

[0082] The present disclosure utilizes a multimodal model that simultaneously learns text, images, voice, motion, and 3D models to generate prompts based on user input information, and annotates target images using an artificial intelligence model based on the generated prompts. This improves the efficiency and accuracy of annotation for target images that are sparse and contain a large number of objects.

[0083] Meanwhile, annotation, as described below, can refer to the process of adding metadata, such as objects (e.g., polylines, bounding boxes, etc.) or image categories (e.g., "clear day," "outdoors," etc.), used to describe the original data in the form of tags to a dataset. In other words, annotation can be defined as the process of annotating source data (e.g., target images) to enable an AI model to understand the data's content.

[0084]

[0085] FIG. 3 is a schematic flowchart for performing prompt-based annotation according to one embodiment of the present disclosure.

[0086] According to one embodiment of the present disclosure, a computing device (100) may acquire a target image to be annotated (S10). For example, the target image to be annotated may include image data utilized for various purposes, such as artificial intelligence model learning, content search, data analysis, automatic generation of image-based captions, image editing tools, and medical image diagnosis tools. In addition, the target image to be annotated may be an image that clearly shows objects or features required for specific tasks, such as object detection, image segmentation, pose estimation, medical image analysis, or autonomous vehicles, and that an artificial intelligence model can effectively learn and generalize. In addition, the target image to be annotated may include a static image, a video image (dynamic image), a 3D image, and the like. However, the target image to be annotated is not limited thereto, and may include various formats of images utilized for specific purposes.

[0087]

[0088] According to one embodiment of the present disclosure, a computing device (100) may obtain user input information for generating a prompt (S20). The user input information for generating the prompt may include at least one of text input, voice input, image input, video input, or 3D data input. In addition, the user input information for generating the prompt may include two or more types of inputs among text input, voice input, image input, video input, or 3D data input. For example, the computing device (100) may obtain two or more types of inputs, namely text input and image input, as user input information for generating the prompt. In another example, the computing device (100) may obtain two or more types of inputs, namely voice input and video input, as user input information for generating the prompt. However, the above-described matters are merely examples, and the present disclosure is not limited thereto.

[0089] Alternatively, the computing device (100) may generate related input information of different types when only one type of user input information for generating a prompt is acquired. For example, the computing device (100) may generate related input information of an image type when only one type of input is acquired among text input or voice input. For example, when the user input information is acquired only as a text input, the computing device (100) may generate related input information of an image type for the acquired text input by utilizing a separate image generation model. In addition, when the user input information is acquired only as a voice input, the computing device (100) may perform voice recognition on the voice input to acquire voice recognition text, and may generate related input information of an image type for the voice recognition text by utilizing a separate image generation model. For example, a separate image generation model may include, but is not limited to, models that generate images from given text, such as Stable Diffusion, DALL-E, Imagen, and Midjourney, and any model that has been developed or will be developed in the future may be applied.

[0090] In addition, the computing device (100) may generate text-type related input information when only one type of input is acquired among image input, video input, and 3D data input as the user input information for generating a prompt. For example, when the user input information only acquires an image input, the computing device (100) may generate text-type related input information for the acquired image input by utilizing a separate text generation model. In addition, when the user input information only acquires a video input, the computing device (100) may generate text-type related input information for the video input by utilizing a separate text generation model for a frame image included in the video input. In addition, when the user input information only acquires 3D data, the computing device (100) may convert the 3D data into a 2D image, and generate text-type related input information for the 3D data by utilizing a separate text generation model for the converted 2D image. For example, a separate text generation model is a model that generates text from a given image, and may include, but is not limited to, CNN-LSTM-based models, Attention Mechanism-based models, Transformer-based models, Neural Image Captioning Models, DALL·E, and CLIP (Contrastive Language-Image Pre-Training), and models that have been developed or will be developed in the future may be applied.

[0091]

[0092] Alternatively, the computing device (100) may generate image-type related input information when both text input and voice input are acquired as user input information for generating a prompt. For example, the computing device (100) may perform voice recognition on text input or voice input, input the generated voice recognition text into a separate image generation model, and acquire image-type related input information. For example, the separate image generation model may include, but is not limited to, Stable Diffusion, DALL-E, Imagen, Midjourney, etc., which generate an image from a given text, and any model that has been developed or will be developed in the future may be applied. In addition, the computing device (100) may generate text-type related input information when two or more types of input are acquired as user input information for generating a prompt among image input, video input, and 3D data input. For example, the computing device (100) may input image input, a frame image included in a video input for a video input, or 2D data generated by converting 3D data into 2D data into a separate text generation model, and may obtain related input information of the text type. For example, the separate text generation model is a model that generates text from a given image, and may include a CNN-LSTM-based model, an Attention Mechanism-based model, a Transformer-based model, Neural Image Captioning Models, DALL·E, and CLIP (Contrastive Language-Image Pre-Training), but is not limited thereto, and a model that has been developed or will be developed in the future may be applied.

[0093] Meanwhile, the computing device (100) can generate various prompts by generating related input information different from the acquired user input information. This can improve the performance of the AI ​​model performing the annotation. Furthermore, by utilizing text prompts and image prompts, the target image can be annotated from various perspectives, effectively complementing any missing portions.

[0094]

[0095] According to one embodiment of the present disclosure, the computing device (100) may generate a prompt for performing annotation based on user input information (S30). At this time, the generated prompt may include at least one of a text prompt or an image prompt. According to one embodiment, when the user input information is at least one of a text input and a voice input, the computing device (100) may generate a text prompt. For example, the computing device (100) may generate a text prompt based on at least one of a text input and a voice recognition text generated by performing voice recognition on a voice input. Meanwhile, when the user input information includes both a text input and a voice input, the computing device (100) may also generate a text prompt by combining voice recognition text generated by performing voice recognition on the text input and the voice input.

[0096] In addition, when the user input information is at least one of an image input, a video input, or a 3D data input, the computing device (100) may generate an image prompt. For example, the computing device (100) may generate an image prompt based on at least one of an image input, a frame image included in a video input, or data obtained by converting 3D data into 2D data. Meanwhile, when the user input information includes two or more of an image input, a video input, or a 3D data input, the computing device (100) may generate an image prompt by combining at least two or more of an image input, a frame image included in a video input, or data obtained by converting 3D data into 2D data.

[0097] Alternatively, the computing device (100) may generate a prompt for performing annotation by combining text prompts and image prompts. For example, if the user input information is a voice input and a video input, the computing device (100) may perform voice recognition on the voice input and generate a text prompt based on the generated voice recognition text, and may generate an image prompt based on a frame image included in the video input. For example, the computing device (100) may generate “Please annotate the animal appearing in the video” as a prompt for performing annotation. Here, the frame image included in the video may be utilized as the image prompt, and “Please annotate the animal appearing in the video” may be the text prompt. Meanwhile, the computing device (100) may generate a prompt that includes both a text prompt and an image prompt, thereby enabling clear annotation even when the target image for performing annotation includes an object that cannot be specified (e.g., a rare animal) or an object that is difficult to describe in text.

[0098]

[0099] According to one embodiment of the present disclosure, the generated prompt may include at least one of a first type of prompt information or a second type of prompt information. For example, the first type of prompt information and the second type of prompt information may have opposite tendencies with respect to annotation. For example, the first type of prompt information may be information having a positive tendency. On the other hand, the second type of prompt information may be information having a negative tendency. For example, the computing device (100) may annotate a target image in response to the first type of prompt information, and may exclude content corresponding to the second type of prompt information from the annotation. For example, if the user input information is a text input and an image input, and the prompt generated based on the user input is “Please annotate only images (e.g., dogs),” the computing device (100) may determine the image (e.g., dogs) as the first prompt information and annotate the target image. As another example, the user input information may be a voice input and a video input, and the prompt generated based on the user input may be “Please annotate the animals included in the target image, excluding the animals in the video (e.g., dogs).” Here, the computing device (100) may determine the animal as the first prompt information, and determine the frame image included in the video (e.g., dogs) input as the second prompt, and may annotate the objects corresponding to the second prompt information (e.g., animals) while excluding the objects corresponding to dogs in the target image. For example, when the generated prompt only includes a text prompt, the text prompt may include at least one of the first type of prompt or the second type of prompt information.For another example, if the generated prompt only includes an image prompt, the image prompt may include at least one of the first type of prompt or the second type of prompt information. For another example, if the generated prompt includes a text prompt and an image prompt, the text prompt may include at least one of the first type of prompt or the second type of prompt information, and the image prompt may include at least one of the first type of prompt or the second type of prompt information. However, the above is merely an example and the present disclosure is not limited thereto.

[0100]

[0101] According to one embodiment of the present disclosure, the computing device (100) may perform annotation on an image using an artificial intelligence model based on the generated prompt (S40). For example, the artificial intelligence model for performing annotation may include at least one of an object detection model that performs object detection based on the generated prompt, a segmentation model that performs segmentation based on the generated prompt, or a skeleton detection model that performs skeleton detection based on the generated prompt. For example, the computing device (100) may perform annotation on a target image using at least one of a bounding box, a skeleton, an edge, a depth map, and a segmentation map using the artificial intelligence model based on the generated prompt. In other words, the computing device (100) can annotate an object included in a target image with at least one of a bounding box, a skeleton, an edge, a depth map, and a segmentation map based on a detection result of an object detection model, a segmentation model, or a skeleton detection model based on a generated prompt.

[0102]

[0103] According to one embodiment of the present disclosure, the computing device (100) can variably provide annotation results for a target image by adjusting the confidence or output parameter values ​​of the artificial intelligence model. For example, some of the outputs of the artificial intelligence model can be expressed as probability values. For example, if the artificial intelligence model for performing annotation is an object detection model that performs object detection based on a generated prompt, the probability for each object can be output. In this case, the computing device (100) can annotate the object only if the confidence for the object corresponding to the generated prompt for performing annotation based on user input information is greater than or equal to a preset threshold. As another example, if the artificial intelligence model predicts multiple classes or objects, the computing device (100) can variably provide annotation results for the target image by adjusting the confidence or output parameter values ​​of the artificial intelligence model so as to generate annotations for the corresponding classes or objects by considering only the top K predictions with the highest probability. As another example, the computing device (100) can variably provide annotation results for a target image by adjusting the confidence or output parameter values ​​of the artificial intelligence model to reduce the predicted probability for all classes at a constant rate or to increase the probability of a specific class to strengthen the annotation for that class. Meanwhile, the computing device (100) can provide accurate and highly reliable annotation results for the target image by adjusting the confidence or output parameter values ​​of the artificial intelligence model.

[0104]

[0105] According to one embodiment, the computing device (100) can variably display text or numbers indicating the results of annotation on a target image. For example, the computing device (100) can variably display text or numbers indicating the results of annotation on a target image by combining at least one of adjusting the size of the text or numbers, adjusting the style, changing the color, adding a background, adjusting the position, adjusting the transparency, and dynamic display. For example, the computing device (100) can adjust the size of the text or numbers indicating the results of annotation on a target image or emphasize the text or numbers by using a bold font. In addition, the computing device (100) can select a color that contrasts with the surrounding environment to display the text or numbers indicating the results of annotation on a target image so that the text or numbers stand out more. In addition, the computing device (100) can add a background so that the text or numbers indicating the results of annotation on a target image are clearly visible without clashing with the surrounding environment. Additionally, the computing device (100) may adjust the position of text or numbers representing the results of annotation for the target image so that they do not overlap with the main object. For example, the computing device (100) may display text only for annotations with a preset reliability level or higher, or may display the corresponding information by changing the content of the text.

[0106]

[0107] In addition, the computing device (100) can variably display a shape representing the result of annotation for the target image. For example, the shape can include a polyline, a bounding box, an edge, a line, etc. For example, the computing device (100) can variably display the contrast of the bounding box representing the result of annotation for the target image. For example, the computing device (100) can variably display the contrast of the bounding box by combining at least one of adjusting the thickness of the bounding box, changing the color, adjusting the transparency, changing the style (e.g., dotted line, dash, solid line, etc.), and enhancing the contrast with the surrounding environment. As an example, the computing device (100) can adjust the thickness of the bounding box representing the result of annotation for the target image. In addition, the computing device (100) can also change the color of the bounding box representing the result of annotation for the target image to a color that contrasts with the surrounding environment. Additionally, the computing device (100) may change the style of the bounding box representing the annotation result for the target image to various styles, such as dotted lines, dashes, and solid lines, to display the bounding box so that it contrasts with the surrounding environment. Furthermore, the computing device (100) may analyze the colors or patterns of the surrounding environment to change the style and color of the bounding box representing the annotation result for the target image.

[0108] However, the operation of the computing device (100) adjusting the reliability or output parameter values ​​of the artificial intelligence model to variably provide annotation results for the target image is not limited thereto, and various embodiments may exist. For example, the computing device (100) may output the reliability or output parameter values ​​of the artificial intelligence model to the user equipment, and variably provide annotation results for the target image based on user input information that changed the reliability or output parameter values ​​of the artificial intelligence model.

[0109]

[0110] According to one embodiment of the present disclosure, the computing device (100) may provide an annotated image to a user device. Furthermore, if user input, such as addition, modification, or deletion, is obtained from the user device regarding the result of annotating the target image, the computing device (100) may also change the result of annotating the target image based on the user input.

[0111]

[0112] FIG. 4 is a diagram illustrating an embodiment of generating a prompt by extracting words or sentences for a target image using an image-text model according to one embodiment of the present disclosure.

[0113] Referring to FIG. 4 as an example, the computing device (100) can obtain a target image (10) on which annotation is to be performed. In addition, the computing device (100) can extract words or sentences for the target image (10) by utilizing an image-text model. For example, the image-text model is a model that generates text from a given image, and may include, but is not limited to, a CNN-LSTM-based model, an Attention Mechanism-based model, a Transformer-based model, Neural Image Captioning Models, DALL·E, and CLIP (Contrastive Language-Image Pre-Training). A model that has been developed or will be developed in the future may be applied.

[0114] In addition, the computing device (100) can display the extracted words or sentences as recommended words or recommended sentences (30-a). For example, the computing device (100) can display specific words or classes as options in a menu window and acquire user input information received in response to specific words or classes in the form of tags. For example, referring to FIG. 4, the computing device (100) can display words or sentences for a target image (10) on the user device, such as [a cat and a dog sitting together in a meadow], [a dog], [a cat], and [a tree], by utilizing an image-text model.

[0115] Additionally, the computing device (100) may generate a prompt (40-a) based on user input information received in response to a recommended word or sentence. For example, if the user input information received in response to a recommended word or sentence is information for selecting an option corresponding to [dog] or [cat], the computing device (100) may generate a prompt such as "Please annotate dogs and cats."

[0116] In addition, the computing device (100) may perform annotation on the target image using an artificial intelligence model based on a prompt generated based on received user input information (50-a). For example, the artificial intelligence model for performing annotation may include at least one of an object detection model that performs object detection based on the generated prompt, a segmentation model that performs segmentation based on the generated prompt, or a skeleton detection model that performs skeleton detection based on the generated prompt. For example, the computing device (100) may perform annotation on the target image using at least one of a bounding box, a skeleton, an edge, a depth map, and a segmentation map using an artificial intelligence model based on the generated prompt. In other words, the computing device (100) may perform annotation on an object included in a target image using at least one of a bounding box, a skeleton, an edge, a depth map, and a segmentation map based on a detection result of an object detection model, a segmentation model, or a skeleton detection model based on a generated prompt. For example, the computing device (100) may utilize an artificial intelligence model to annotate each of a dog and a cat included in the target image based on a prompt generated based on user input information, such as “Please annotate dogs and cats.”

[0117] In addition, the computing device (100) can variably provide the results of annotation for the target image by adjusting the values ​​of the confidence or output parameters of the artificial intelligence model. For example, the computing device (100) can variably display text or numbers representing the results of annotation for the target image. In addition, the computing device (100) can variably display a shape representing the results of annotation for the target image. For example, the shape may include a polyline, a bounding box, an edge, a line, etc. As an example, the computing device (100) can variably display the contrast of the bounding box representing the results of annotation for the target image. Referring again to 50-a of FIG. 4, the computing device (100) can display a dotted bounding box for an object corresponding to a cat, and a solid bounding box for an object corresponding to a dog.

[0118] Meanwhile, the computing device (100) can extract words or sentences for a target image (10) by utilizing an image-text model, and generate a prompt based on user input information received in response to a recommended word or sentence from a user device, thereby maintaining consistency in words or sentences used among multiple users, thereby improving accuracy in an operation of annotating a target image.

[0119]

[0120] Below, we will briefly review the operating flow of the present invention based on the detailed description above.

[0121] FIG. 5 is a flowchart illustrating a method for performing prompt-based annotation according to one embodiment of the present disclosure.

[0122] The method for performing prompt-based annotation as illustrated in FIG. 5 can be performed by the computing device (100) described above. Therefore, even if omitted below, the description of the computing device (100) can be equally applied to the description of the method for performing prompt-based annotation.

[0123] Referring to FIG. 5, a method for performing annotation based on a prompt may include a step of obtaining a target image for performing the annotation (S110), a step of generating the prompt for performing the annotation based on user input information (S120), and a step of performing annotation on the target image using an artificial intelligence model based on the generated prompt.

[0124] Step S110 is a step for acquiring a target image for annotation. For example, the target image for annotation may include image data utilized for various purposes, such as artificial intelligence model training, content search, data analysis, automatic image-based caption generation, image editing tools, and medical image diagnostic tools.

[0125] Step S120 is a step of generating the prompt for performing the annotation based on user input information. For example, the user input information for generating the prompt may include at least one of a text input, a voice input, an image input, a video input, or a 3D data input. In addition, the user input information for generating the prompt may include two or more types of inputs from a text input, a voice input, an image input, a video input, or a 3D data input, and the generated prompt may include at least one of a text prompt or an image prompt. According to one embodiment, the step S120 may include a step of extracting a word or a sentence for the target image using an image-text model, a step of displaying the extracted word or the extracted sentence as a recommended word or a recommended sentence, and a step of generating the prompt based on the user input information received in response to the recommended word or the recommended sentence.

[0126] Step S130 is a step of annotating the target image using an artificial intelligence model based on the generated prompt. For example, the artificial intelligence model may include at least one of an object detection model that performs object detection based on the generated prompt, a segmentation model that performs segmentation based on the generated prompt, or a skeleton detection model that performs skeleton detection based on the generated prompt.

[0127] According to one embodiment of the present disclosure, a method for performing annotation based on a prompt may further include a step of providing the result of performing annotation on a target image to a user device in step S130. When a user input, such as addition, modification, or deletion, regarding the result of performing annotation on a target image is obtained from the user device, the computing device (100) may change the result of performing annotation on the target image based on the user input.

[0128]

[0129] The steps described in the above description may be further divided into additional steps or combined into fewer steps, depending on the implementation of the present disclosure. Furthermore, some steps may be omitted as needed, and the order of the steps may be changed.

[0130]

[0131] Meanwhile, a computer-readable medium storing a data structure according to an embodiment of the present disclosure is disclosed.

[0132] A data structure can refer to the organization, management, and storage of data to enable efficient access and modification. A data structure can refer to the organization of data to solve a specific problem (e.g., data retrieval, data storage, or data modification in the shortest possible time). A data structure can also be defined as the physical or logical relationships between data elements designed to support specific data processing functions. Logical relationships between data elements can include connections between user-defined data elements. Physical relationships between data elements can include actual relationships between data elements physically stored on a computer-readable storage medium (e.g., persistent storage). Specifically, a data structure can include a collection of data, relationships between data, and functions or commands applicable to the data. An effectively designed data structure allows a computing device to perform operations while minimizing the use of its resources. Specifically, a computing device can improve the efficiency of operations, reading, inserting, deleting, comparing, exchanging, and searching through an effectively designed data structure.

[0133] Data structures can be categorized as linear or nonlinear, depending on their form. A linear data structure can be a structure in which only one data item is linked to the next. Linear data structures can include lists, stacks, queues, and deques. A list can refer to a series of data sets with an internal order. Lists can also include linked lists. A linked list is a data structure in which data is linked in a single line, each item having a pointer. In a linked list, a pointer can contain information about the next or previous item. Linked lists can be expressed as singly linked lists, doubly linked lists, or circular linked lists, depending on their form. A stack can be a data listing structure with limited data access. A stack can be a linear data structure in which data operations (e.g., insertion or deletion) can only be performed at one end of the data structure. Data stored in a stack can be a Last-in-First-out (LIFO) data structure. A queue is a data structure with limited access to data. Unlike a stack, it can be a first-in, first-out (FIFO) data structure, with later data being retrieved later. A deck can be a data structure that can process data at both ends.

[0134] A nonlinear data structure can be a structure in which multiple pieces of data are connected behind a single piece of data. Nonlinear data structures can include graph data structures. A graph data structure can be defined by vertices and edges, and an edge can include a line connecting two different vertices. Graph data structures can include tree data structures. A tree data structure can be a data structure in which there is only one path connecting two different vertices among multiple vertices included in the tree. In other words, it can be a data structure that does not form a loop in a graph data structure.

[0135] Throughout this specification, the terms computational model, neural network, network function, and neural network may be used interchangeably. Hereinafter, they are collectively referred to as neural networks. The data structure may include a neural network. And the data structure including the neural network may be stored on a computer-readable medium. The data structure including the neural network may also include preprocessed data for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, loss functions for learning the neural network, etc. The data structure including the neural network may include any of the components disclosed above. That is, the data structure including the neural network may be configured to include all or any combination of preprocessed data for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, loss functions for learning the neural network, etc. In addition to the aforementioned configurations, a data structure including a neural network may include any other information that determines the characteristics of the neural network. Furthermore, the data structure may include any form of data used or generated in the computational process of the neural network, and is not limited to the aforementioned. The computer-readable medium may include a computer-readable recording medium and / or a computer-readable transmission medium. A neural network may be composed of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network is composed of at least one node.

[0136] The data structure may include data input to a neural network. The data structure including the data input to the neural network may be stored on a computer-readable medium. The data input to the neural network may include training data input during the neural network training process and / or input data input to the neural network after training has been completed. The data input to the neural network may include data that has undergone preprocessing and / or data that is the target of preprocessing. Preprocessing may include a data processing process for inputting data to the neural network. Accordingly, the data structure may include data that is the target of preprocessing and data generated by the preprocessing. The above-described data structure is merely an example, and the present disclosure is not limited thereto.

[0137] The data structure may include weights of the neural network. (In this specification, the terms "weight" and "parameter" may be used interchangeably.) The data structure including the weights of the neural network may be stored in a computer-readable medium. The neural network may include a plurality of weights. The weights may be variable and may be varied by a user or an algorithm so that the neural network can perform a desired function. For example, when one or more input nodes are interconnected to one output node by respective links, the output node may determine a data value output from the output node based on values ​​input to the input nodes connected to the output node and weights set for links corresponding to each input node. The above-described data structure is merely an example, and the present disclosure is not limited thereto.

[0138] By way of example and not limitation, the weights may include weights that vary during the neural network training process and / or weights that have completed neural network training. The weights that vary during the neural network training process may include weights at the start of the training cycle and / or weights that vary during the training cycle. The weights that have completed neural network training may include weights that have completed the training cycle. Accordingly, a data structure including the weights of a neural network may include a data structure including weights that vary during the neural network training process and / or weights that have completed neural network training. Therefore, the above-described weights and / or combinations of each weight are included in the data structure including the weights of a neural network. The above-described data structures are merely examples and the present disclosure is not limited thereto.

[0139] A data structure including neural network weights can be stored in a computer-readable storage medium (e.g., memory, hard disk) after going through a serialization process. Serialization can be a process of converting a data structure into a form that can be stored on the same or different computing devices and later reconstructed and used. A computing device can serialize the data structure to transmit and receive data over a network. The serialized data structure including neural network weights can be reconstructed on the same or different computing devices through deserialization. The data structure including neural network weights is not limited to serialization. Furthermore, the data structure including neural network weights can include a data structure that increases computational efficiency while minimizing the use of computing device resources (e.g., a B-Tree, a Trie, an m-way search tree, an AVL tree, a Red-Black Tree in nonlinear data structures). The foregoing is merely an example, and the present disclosure is not limited thereto.

[0140] The data structure may include hyperparameters of a neural network. Furthermore, the data structure including the hyperparameters of the neural network may be stored on a computer-readable medium. The hyperparameters may be variables that can be varied by the user. The hyperparameters may include, for example, a learning rate, a cost function, the number of learning cycle repetitions, weight initialization (e.g., setting a range of weight values ​​to be subject to weight initialization), and the number of hidden units (e.g., the number of hidden layers, the number of nodes in the hidden layer). The above-described data structure is merely an example, and the present disclosure is not limited thereto.

[0141]

[0142] FIG. 6 is a simplified, general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

[0143] Although the present disclosure has been described above as being generally implemented by a computing device, those skilled in the art will appreciate that the present disclosure may also be implemented in combination with computer-executable instructions and / or other program modules that may be executed on one or more computers and / or as a combination of hardware and software.

[0144] Generally, program modules include routines, programs, components, data structures, and the like that perform specific tasks or implement specific abstract data types. Furthermore, those skilled in the art will appreciate that the methods of the present disclosure can be implemented with other computer system configurations, including single-processor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, and the like, each of which may be operatively connected to one or more associated devices.

[0145] The described embodiments of the present disclosure can also be practiced in distributed computing environments, where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0146] Computers typically include a variety of computer-readable media. Computer-readable media can be any media that can be accessed by a computer, and includes both volatile and nonvolatile media, transitory and non-transitory media, removable and non-removable media. By way of example, and not limitation, computer-readable media can include computer-readable storage media and computer-readable transmission media. Computer-readable storage media includes both volatile and nonvolatile media, transitory and non-transitory media, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be accessed by a computer and used to store the desired information.

[0147] Computer-readable transmission media typically includes any information delivery media that embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism. The term modulated data signal means a signal that has one or more of its characteristics set or changed so as to encode information in the signal. By way of example, and not limitation, computer-readable transmission media includes wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, RF, infrared, or other wireless media. Combinations of any of the above are also intended to be included within the scope of computer-readable transmission media.

[0148] An exemplary environment for implementing various aspects of the present disclosure is illustrated, including a computer (1102), which includes a processing unit (1104), system memory (1106), and a system bus (1108). The system bus (1108) connects system components, including but not limited to the system memory (1106), to the processing unit (1104). The processing unit (1104) may be any of a variety of commercially available processors. Dual processors and other multiprocessor architectures may also be utilized as the processing unit (1104).

[0149] The system bus (1108) may be any of several types of bus structures that may be additionally interconnected to a memory bus, a peripheral bus, and a local bus using any of a variety of commercial bus architectures. The system memory (1106) includes read-only memory (ROM) (1110) and random access memory (RAM) (1112). A basic input / output system (BIOS) is stored in non-volatile memory (1110), such as ROM, EPROM, or EEPROM, and includes basic routines that help transfer information between components within the computer (1102), such as during start-up. The RAM (1112) may also include high-speed RAM, such as static RAM, for caching data.

[0150] The computer (1102) also includes an internal hard disk drive (HDD) (1114) (e.g., EIDE, SATA) - which may also be configured for external use within a suitable chassis (not shown), a magnetic floppy disk drive (FDD) (1116) (e.g., for reading from or writing to a removable diskette (1118)), and an optical disk drive (1120) (e.g., for reading from or writing to a CD-ROM disk (1122) or other high-capacity optical media such as a DVD). The hard disk drive (1114), the magnetic disk drive (1116), and the optical disk drive (1120) may be connected to the system bus (1108) by a hard disk drive interface (1124), a magnetic disk drive interface (1126), and an optical drive interface (1128), respectively. The interface (1124) for implementing an external drive includes at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

[0151] These drives and their associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, and the like. In the case of the computer (1102), the drives and media correspond to storing any data in a suitable digital format. While the description of computer-readable media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those of ordinary skill in the art will appreciate that other types of computer-readable media, such as zip drives, magnetic cassettes, flash memory cards, cartridges, and the like, may also be used in the exemplary operating environment, and that any such media may contain computer-executable instructions for performing the methods of the present disclosure.

[0152] A number of program modules, including an operating system (1130), one or more application programs (1132), other program modules (1134), and program data (1136), may be stored in the drive and RAM (1112). All or portions of the operating system, applications, modules, and / or data may also be cached in RAM (1112). It will be appreciated that the present disclosure may be implemented in various commercially available operating systems or combinations of operating systems.

[0153] A user may enter commands and information into the computer (1102) via one or more wired / wireless input devices, such as a keyboard (1138) and a pointing device such as a mouse (1140). Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, a touch screen, and the like. These and other input devices are often connected to the processing unit (1104) via an input device interface (1142) that is connected to the system bus (1108), but may be connected by other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, and the like.

[0154] A monitor (1144) or other type of display device is also connected to the system bus (1108) via an interface, such as a video adapter (1146). In addition to the monitor (1144), the computer typically includes other peripheral output devices (not shown), such as speakers, a printer, and so on.

[0155] The computer (1102) may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) (1148), via wired and / or wireless communications. The remote computer(s) (1148) may be a workstation, a computing device computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment device, a peer device, or other conventional network node, and generally include many or all of the components described for the computer (1102), although for simplicity, only the memory storage device (1150) is shown. The logical connections shown include wired / wireless connections to a local area network (LAN) (1152) and / or a larger network, such as a wide area network (WAN) (1154). Such LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which may be connected to a worldwide computer network, such as the Internet.

[0156] When used in a LAN networking environment, the computer (1102) is connected to a local network (1152) via a wired and / or wireless communication network interface or adapter (1156). The adapter (1156) may facilitate wired or wireless communications to the LAN (1152), which may also include a wireless access point installed therein for communicating with the wireless adapter (1156). When used in a WAN networking environment, the computer (1102) may include a modem (1158), be connected to a communications computing device on the WAN (1154), or have other means of establishing communications over the WAN (1154), such as via the Internet. The modem (1158), which may be internal or external and wired or wireless, is connected to the system bus (1108) via a serial port interface (1142). In a networked environment, program modules or portions thereof described for the computer (1102) may be stored in a remote memory / storage device (1150). It will be appreciated that the network connections depicted are exemplary and other means of establishing a communications link between the computers may be used.

[0157] The computer (1102) operates to communicate with any wireless device or object that is arranged and operates via wireless communication, such as a printer, a scanner, a desktop and / or portable computer, a portable data assistant (PDA), a communication satellite, any equipment or location associated with a radio-detectable tag, and a telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network, or may simply be an ad hoc communication between at least two devices.

[0158] Wi-Fi (Wireless Fidelity) enables connections to the Internet and other devices without wires. Wi-Fi is a wireless technology that allows devices, such as computers, to send and receive data anywhere within the coverage area of ​​a base station, both indoors and outdoors, similar to cell phones. Wi-Fi networks use wireless technologies called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in the unlicensed 2.4 and 5 GHz radio bands, at data rates of, for example, 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual-band).

[0159] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips referenced in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0160] Those skilled in the art will appreciate that the various illustrative logical blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, various forms of programs or design code (referred to herein, for convenience, as software), or a combination of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0161] The various embodiments presented herein can be implemented as a method, apparatus, or article of manufacture using standard programming and / or engineering techniques. The term article of manufacture includes a computer program, carrier, or media accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Furthermore, various storage media presented herein include one or more devices and / or other machine-readable media for storing information.

[0162] It should be understood that the specific order or hierarchy of steps in the presented processes is merely an example of exemplary approaches. It should be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of the present disclosure based on design priorities. The appended method claims provide elements of various steps in a sample order, but are not intended to be limited to the specific order or hierarchy presented.

[0163] The description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments disclosed herein, but is to be construed in the broadest scope consistent with the principles and novel features disclosed herein.

[0164]

[0165] As described above, the relevant contents have been described in the best form for carrying out the invention.

Claims

1. A method for performing annotation based on a prompt, performed by a computing device, A step of obtaining a target image on which the above annotation is to be performed; generating the prompt for performing the annotation based on user input information; and A step of annotating the target image using an artificial intelligence model based on the generated prompt; including, method.

2. In paragraph 1, The user input information for generating the prompt includes at least one of text input, voice input, image input, video input, or 3D data input. method.

3. In paragraph 2, The step of generating the prompt for performing the annotation based on the user input information comprises: A step of extracting words or sentences for the target image by utilizing an image-text model; A step of displaying the extracted word or the extracted sentence as a recommended word or a recommended sentence; and A step of generating the prompt based on the user input information received in response to the recommended word or the recommended sentence. including, method.

4. In paragraph 2, The user input information for generating the above prompt includes two or more types of input among text input, voice input, image input, video input, or 3D data input, The generated prompt includes at least one of a text prompt or an image prompt. method.

5. In paragraph 4, The above user input information is: At least one input of the text input or the voice input; and At least one input of the image input, the video input, or the 3D data input Including, The text prompt is generated based on at least one of the text input or the speech recognition text generated by performing speech recognition on the speech input, The image prompt is generated based on at least one of the image input, a frame image included in the video input, or data converted from the 3D data to 2D data. method.

6. In paragraph 4, The above user input information is: Including both the above text input and the above voice input, The above text prompt is generated by combining the text input and the speech recognition text generated by performing speech recognition on the speech input. method 7. In paragraph 4, The above user input information is: Including two or more inputs among the image input, the video input, or the 3D data input, The image prompt is generated by combining at least two of the image input, the frame image included in the video input, or the data converted from the 3D data to 2D data. method.

8. In paragraph 1, The generated prompt includes at least one of the first type of prompt information or the second type of prompt information, The first type of prompt information and the second type of prompt information have opposite tendencies with respect to annotations. method.

9. In paragraph 1, The artificial intelligence model for performing the above annotation is, An object detection model that performs object detection based on the generated prompt; A segmentation model that performs segmentation based on the generated prompt; or A skeleton detection model that performs skeleton detection based on the generated prompts above. Containing at least one of, method.

10. In paragraph 1, The step of annotating the target image using an artificial intelligence model based on the generated prompt is as follows: A step of variably providing the results of annotation for the target image by adjusting the values ​​of the confidence or output parameters of the artificial intelligence model. including, method.

11. In paragraph 10, The step of variably providing the results of annotation for the above target image is: A step of variably displaying text or numbers representing the results of annotation for the target image; or A step of variably displaying a shape representing the result of annotation for the above target image. including, method.

12. In paragraph 11, The step of variably displaying a shape representing the result of annotation for the above target image is as follows: A step of variably displaying the contrast of a bounding box representing the result of annotation for the above target image. including, method.

13. A computer program stored in a computer-readable storage medium, wherein when the computer program is executed on one or more processors, the computer program causes the one or more processors to perform the following operations for performing annotation based on a prompt, wherein the operations are: An action of obtaining a target image on which the above annotation is to be performed; An action to generate the prompt for performing the annotation based on user input information; and An action of annotating the target image using an artificial intelligence model based on the generated prompt; including, A computer program stored on a computer-readable storage medium.

14. In paragraph 13, The user input information for generating the prompt includes at least one of text input, voice input, image input, video input, or 3D data input. A computer program stored on a computer-readable storage medium.

15. In paragraph 14, The action of generating the prompt for performing the annotation based on the user input information is: An action of extracting words or sentences for the target image by utilizing an image-text model; An action of displaying the extracted word or sentence as a recommended word or recommended sentence; and An action of generating the prompt based on the user input information received in response to the recommended word or the recommended sentence. including, A computer program stored on a computer-readable storage medium.

16. In paragraph 14, The user input information for generating the above prompt includes two or more types of input among text input, voice input, image input, video input, or 3D data input, The generated prompt includes at least one of a text prompt or an image prompt. A computer program stored on a computer-readable storage medium.

17. In paragraph 13, The generated prompt includes at least one of the first type of prompt information or the second type of prompt information, The first type of prompt information and the second type of prompt information have opposite tendencies with respect to annotations. A computer program stored on a computer-readable storage medium.

18. In paragraph 13, The artificial intelligence model for performing the above annotation is, An object detection model that performs object detection based on the generated prompt; A segmentation model that performs segmentation based on the generated prompt; or A skeleton detection model that performs skeleton detection based on the generated prompts above. Containing at least one of, A computer program stored on a computer-readable storage medium.

19. In paragraph 13, The action of annotating the target image using an artificial intelligence model based on the generated prompt is as follows: An operation that provides variable annotation results for the target image by adjusting the confidence or output parameter values ​​of the artificial intelligence model. including, A computer program stored on a computer-readable storage medium.

20. As a computing device, at least one processor; and memory; Including, At least one processor, Obtain the target image to be annotated; Generate a prompt to perform the annotation based on user input information; and It is configured to perform annotation on the target image using an artificial intelligence model based on the generated prompt. device.

21. In paragraph 20, The user input information for generating the prompt includes at least one of text input, voice input, image input, video input, or 3D data input. device.

22. In paragraph 21, At least one processor, A step of extracting words or sentences for the target image by utilizing an image-text model; A step of displaying the extracted word or the extracted sentence as a recommended word or a recommended sentence; and A step of generating the prompt based on the user input information received in response to the recommended word or the recommended sentence. including, device.

23. In paragraph 21, The user input information for generating the above prompt includes two or more types of input among text input, voice input, image input, video input, or 3D data input, The generated prompt includes at least one of a text prompt or an image prompt. device.

24. In paragraph 20, The generated prompt includes at least one of the first type of prompt information or the second type of prompt information, The first type of prompt information and the second type of prompt information have opposite tendencies with respect to annotations. device.

25. In paragraph 20, The artificial intelligence model for performing the above annotation is, An object detection model that performs object detection based on the generated prompt; A segmentation model that performs segmentation based on the generated prompt; or A skeleton detection model that performs skeleton detection based on the generated prompts above. Containing at least one of, device.

26. In paragraph 20, At least one processor, By adjusting the confidence or output parameter values ​​of the artificial intelligence model, the results of annotation for the target image are variably provided. device.

Citation Information

Patent Citations

  • Computerized method and device for annotating at least one feature of an image of a view

    KR1020140038355A

  • Selecting next user prompt types

    KR102395988B1

  • Red Ginseng Dietary Fiber Multi-coating Method With Improved Acid Resistance, Bile Resistance, And Storage Stability Of Lactic Acid Bacteria And Preparing Method Of Multi-coated Lactic Acid Bacteria Using The Same

    KR102408923B1