AI service cartridge and AI service system using the same

Abstract

One feature of the present disclosure proposes an artificial intelligence service cartridge. The artificial intelligence service cartridge is an artificial intelligence service cartridge used in combination with a computer device, comprising: an inference accelerator that drives an artificial intelligence model and is connected to the computer device by being fastened through a first slot; and a cartridge board that includes at least one component connected to the computer device by being fastened through a second slot different from the first slot and supporting the accelerator to operate in conjunction with the computing device.

Description

Technology Field

[0001] The present invention relates to technology related to the provision of artificial intelligence (AI) services, and in particular, to an AI service cartridge that provides an AI model and an AI serving system using a hardware cartridge, and an AI service system using the same. Background Technology

[0002] Inference services using artificial intelligence are generally performed by deploying a pre-trained deep learning model in a cloud environment and accessing the model through remote calls (gRPC / HTTP interface).

[0003] In a conventional AI serving system using artificial intelligence, model loader, version manager, batch stream manager, and service interface functions are each implemented for a specific computing resource, and then AI services are provided by utilizing the entire computing resource.

[0004] These conventional AI serving systems using artificial intelligence have limitations, such as the fact that computing resource environments like the AI ​​model and the serving system must be fixed during system configuration, and that a large amount of resources must be invested to update or change the model later. The problem to be solved

[0005] The embodiments are intended to facilitate the updating of AI models and serving systems in order to solve the problems of the prior art described above.

[0006] In one embodiment of the present invention, we aim to provide an AI service cartridge that provides AI services without any separate work simply by inserting it into a computer system, and an AI service system using the same.

[0007] In one embodiment of the present invention, the AI ​​service cartridge includes a dedicated accelerator and storage for storing an AI model and a serving system for the same, so that the AI ​​service cartridge connected to the computer system can easily build an AI service environment without separate version control or batch processing.

[0008] In one embodiment of the present invention, the AI ​​service cartridge provides an AI model, an AI serving system, and a dedicated accelerator as a whole, so that the user can easily update the model and the serving system simply by replacing the cartridge hardware.

[0009] However, the technical problems that the present disclosure aims to solve are not limited to the technical problems described above, and other technical problems may exist. means of solving the problem

[0010] One feature of the present disclosure proposes an artificial intelligence service cartridge. The artificial intelligence service cartridge is an artificial intelligence service cartridge used in combination with a computer device, comprising: an inference accelerator that drives an artificial intelligence model and is connected to the computer device by being fastened through a first slot; and a cartridge board that includes at least one component connected to the computer device by being fastened through a second slot different from the first slot and supporting the accelerator to operate in conjunction with the computing device.

[0011] The above artificial intelligence service cartridge operates based on power provided by the computer device and operates in conjunction with the computer device to perform AI processing to enable the computer device to provide AI services.

[0012] The cartridge board may include at least one GPU (Graphics Processing Unit) or at least one NPU (Neural Processing Unit).

[0013] The inference accelerator is connected to the computer device via a PCI (Peripheral Component Interconnect) slot or a PCIe (Peripheral Component Interconnect-Express) slot, and the cartridge board can be connected to the computer device via any one of a USB (Universal Serial Bus), a serial port, or Thunderbolt.

[0014] The cartridge board may include: storage for storing application files for services using an artificial intelligence model; and a local memory electrically connected to the inference accelerator and storing intermediate data required for artificial intelligence inference by the inference accelerator.

[0015] The cartridge board may further include a high-dimensional vector database; and a vector DB accelerator that compares the similarity of a high-dimensional vector query generated by the inference accelerator with the high-dimensional vector database and extracts a high-dimensional vector having a similarity greater than or equal to a preset similarity with the high-dimensional vector query.

[0016] The above inference accelerator can receive a query received from a user by the computer device through an artificial intelligence service as input, and generate the high-dimensional query vector by applying an artificial intelligence model to the query.

[0017] One feature of the present disclosure proposes another example of an artificial intelligence service cartridge. Another example of an artificial intelligence service cartridge is an artificial intelligence service cartridge used in combination with a computer device, comprising: a cartridge board connected to the computer device by being fastened through a first slot; an inference accelerator located on the cartridge board and driving an artificial intelligence model; and at least one component located on the cartridge board and supporting the inference accelerator to operate in conjunction with the computing device.

[0018] The above inference accelerator may include at least one GPU (Graphics Processing Unit) or at least one NPU (Neural Processing Unit).

[0019] The cartridge board above can be connected to the computer device through any one of a PCI (Peripheral Component Interconnect) slot, a PCIe (Peripheral Component Interconnect-Express) slot, a USB (Universal Serial Bus), a serial port, or Thunderbolt.

[0020] Another feature of the present disclosure proposes an artificial intelligence service system. The artificial intelligence service system comprises: a computer device; and an artificial intelligence service cartridge coupled to and used with the computer device. The artificial intelligence service cartridge comprises: an inference accelerator that drives an artificial intelligence model and is connected to the computer device by being fastened through a first slot; and a cartridge board that is connected to the computer device by being fastened through a second slot different from the first slot and includes at least one component that supports the accelerator to operate in conjunction with the computing device. Effects of the invention

[0021] According to the embodiments, the effect of providing AI services without separate work is provided simply by plugging it into a computer system.

[0022] According to the embodiments, the AI ​​service cartridge includes a dedicated accelerator and storage for storing an AI model and a serving system for it, so that the AI ​​service cartridge connected to the computer system provides the effect of easily establishing an AI service environment without separate version control or batch processing.

[0023] According to the embodiments, the AI ​​service cartridge provides an AI model, an AI serving system, and a dedicated accelerator as a single unit, so that the user can easily replace or update the model and serving system simply by replacing the cartridge hardware.

[0024] According to the embodiments, the AI ​​service cartridge provides the effect of enhancing service stability by providing hardware optimized for the model along with the model, rather than a conventional hardware-dependent model selection.

[0025] According to the embodiments, the AI ​​service cartridge provides additional security features (model encryption) at the hardware and software levels, thereby facilitating the provision of security-sensitive services.

[0026] According to the embodiments, the AI ​​service cartridge provides the effect of maximizing the reusability of existing systems by using a standard slot interface in an existing computer environment, such as PCIe.

[0027] The effects according to the present disclosure are not limited to those exemplified above, and a wider variety of effects are included within the present disclosure. Brief explanation of the drawing

[0028] FIG. 1 is a configuration diagram of an artificial intelligence service system using an artificial intelligence service cartridge according to one embodiment of the present disclosure. FIG. 2 is a drawing illustrating an embodiment of an artificial intelligence service cartridge according to an embodiment of the present disclosure. FIG. 3 is a flowchart illustrating a method for providing artificial intelligence services through an artificial intelligence service system using an artificial intelligence service cartridge according to one embodiment of the present disclosure. FIGS. 4 to 6 are block diagrams illustrating examples of configurations of various types of artificial intelligence service cartridges according to embodiments of the present disclosure. FIG. 7 is a drawing illustrating an example of an artificial intelligence service cartridge according to one embodiment of the present disclosure. FIG. 8 is a flowchart illustrating a method for providing artificial intelligence services in one embodiment illustrated in FIG. 7. FIG. 9 is a drawing illustrating another example of an artificial intelligence service cartridge according to one embodiment of the present disclosure. FIG. 10 is a flowchart illustrating a method for providing artificial intelligence services in one embodiment illustrated in FIG. 9. FIG. 11 is a flowchart illustrating a method for providing AI services using an AI service cartridge according to an embodiment of the present invention. FIG. 12 is a drawing illustrating another embodiment of an artificial intelligence service cartridge according to one embodiment of the present disclosure. Specific details for implementing the invention

[0029] Specific structural or functional descriptions of embodiments according to the concept of the present invention disclosed herein are provided merely for the purpose of explaining embodiments according to the concept of the present invention, and embodiments according to the concept of the present invention may be implemented in various forms and are not limited to the embodiments described herein.

[0030] Embodiments according to the concept of the present invention may be subject to various modifications and may take various forms; therefore, embodiments are illustrated in the drawings and described in detail in this specification. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes modifications, equivalents, or substitutions that fall within the spirit and scope of the present invention.

[0031] Terms such as "first" or "second" may be used to describe various components, but said components should not be limited by said terms. For the sole purpose of distinguishing one component from another, for example, without departing from the scope of rights according to the concept of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0032] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. Conversely, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Expressions describing the relationships between components, such as "between," "exactly between," or "directly adjacent to," should be interpreted in the same way.

[0033] The terms used herein are used merely to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “comprising” or “having” are intended to specify the existence of the described features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0034] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification.

[0035] In this specification, the term "processor" may refer to hardware capable of performing functions and operations according to each name described in this specification, computer program code capable of performing specific functions and operations, or an electronic recording medium loaded with computer program code capable of performing specific functions and operations.

[0036] In other words, the term "processor" may refer to a functional and / or structural combination of hardware for carrying out the technical concept of the present invention and / or software for driving said hardware.

[0037] Hereinafter, embodiments will be described in detail with reference to the attached drawings. However, the scope of the patent application is not limited or restricted by these embodiments. Identical reference numerals in each drawing denote identical components.

[0039] FIG. 1 is a configuration diagram of an artificial intelligence service system using an artificial intelligence service cartridge according to one embodiment of the present disclosure.

[0040] The artificial intelligence service cartridge (hereinafter referred to as the ‘AI service cartridge’) (100) is a standalone hardware device and can be used in combination with an existing conventional computer device (200).

[0041] When the AI ​​service cartridge (100) is coupled to the computer device (200), the computer device (200) can be implemented as an AI model driving system without separate setup work by using the AI ​​model and serving system provided from the hardware-based accelerator for AI included in the AI ​​service cartridge (100)—e.g., a GPU (Graphics Processing Unit) or an NPU (Neural Processing Unit)—and the hardware-based accelerator for AI included in the AI ​​service cartridge (100).

[0042] By using the AI ​​service cartridge (100), the computer device (200) can be implemented as a host computer for the AI ​​service without any special system settings or implementation for the computer device (200).

[0043] In one embodiment, the AI ​​service cartridge (100) includes a dedicated accelerator and storage for storing an AI model and a serving system for the same, so that the AI ​​service cartridge connected to the computer system can easily build an AI service environment without separate version control or batch processing.

[0044] In one embodiment, the AI ​​service cartridge (100) provides an AI model, an AI serving system, and a dedicated accelerator as a single piece of hardware, so the user can easily replace or update the AI ​​model and serving system simply by replacing the AI ​​service cartridge hardware.

[0045] In one embodiment, the AI ​​service cartridge (100) can improve service stability by providing hardware optimized for the model along with the model, rather than a conventional hardware-dependent model selection.

[0046] In one embodiment, the AI ​​service cartridge (100) can provide additional security features, such as model encryption, at the hardware and / or software level, thereby providing security as the cartridge itself.

[0047] In one embodiment, the AI ​​service cartridge (100) can maximize the reusability of existing computer systems by using a standard slot interface in an existing computer environment, such as PCI.

[0049] Hereinafter, various embodiments of an AI service cartridge according to the present disclosure, a method for providing AI services using the same, and a system thereof will be described with reference to FIGS. 2 to 10.

[0051] FIG. 2 is a drawing illustrating an embodiment of an artificial intelligence service cartridge according to an embodiment of the present disclosure.

[0052] Figure 2 illustrates an AI service cartridge (100) connected to the mainboard (210) of a host computer.

[0053] The AI ​​service cartridge (100) may include an accelerator (120) and a cartridge board (110).

[0054] For example, the accelerator (120) may be a GPU or an NPU and may be connected to the main board (210) of a host computer through a PCI or PCIe slot (211).

[0055] Various components may be placed on the cartridge board (110). The cartridge board (110) may be connected to a host computer through a separate slot (212)—e.g., USB, serial port, Thunderbolt, etc.—and may receive power to supply power to the accelerator (120).

[0056] Although the connection structure between the accelerator (120) and the cartridge board (110) is not disclosed in FIG. 2, the two components are electrically connected and electrically coupled, such as by supplying power or providing control signals.

[0057] The AI ​​service cartridge (100) can be designed with various modified components depending on the AI ​​service provided, and the AI ​​service cartridge (100) can include at least one component from the list of components below.

[0058] Component List

[0059] Device Controller: Performs communication with the controller (CPU) of the host computer device (hereinafter abbreviated as 'host computer') and control of components within the AI ​​service cartridge (100).

[0060] Inference Accelerator: A component that accelerates the inference process of an AI model stored in memory using parallelization technology or a dedicated computational circuit, as a GPU or NPU included in the AI ​​service cartridge (100). Multiple inference accelerators may be installed depending on the need (required computational power).

[0061] Vector DB Accelerator: A hardware module that efficiently stores large volumes of high-dimensional vectors (embeddings) and indexes, searches at high speed for the vector most similar to the user's input among the stored vectors, and returns a set of associated indexes; it can be implemented using a dedicated chip or circuit board.

[0062] Local Memory: A component composed of volatile memory such as DDR (Double Data Rate), GDDR (Graphics Double Data Rate), HBM (High Bandwidth Memory), or SRAM (Static Random Access Memory), which temporarily stores AI model parameters and embedding model parameters to quickly provide data to the inference accelerator.

[0063] Storage: A storage component that stores AI models, high-dimensional vectors (embeddings), vector indexes, inference engines, service agents, and various databases.

[0064] External interface: As an interface with a host computer, it supports standards such as PCIe, USB, and Therderbolt, provides a communication channel between the AI ​​service cartridge (100) and the host computer, and can support multiple interfaces.

[0065] Additional Power Port: May include a separate additional power port—e.g., External Power Pin—for use when power supplied through the external interface is insufficient.

[0067] FIG. 3 is a flowchart illustrating a method for providing artificial intelligence services through an artificial intelligence service system using an artificial intelligence service cartridge according to one embodiment of the present disclosure.

[0068] FIG. 3 illustrates an example of a process in which an AI service cartridge (100) is connected to a host computer device (200) so that the computer device is implemented as a host of an AI service.

[0069] Referring to FIG. 3, the computer device (200) can identify an AI service cartridge (100) connected through a second connector—e.g., a connector other than PCI / PCIe to which an accelerator is connected, such as the USB, Serial, Thunderbolt, etc. described above (S301).

[0070] In one embodiment, the computer device (200) can copy the AI ​​model of the AI ​​service cartridge (100) to the storage of the main body (S302).

[0071] In one embodiment, the computer device (200) can compare an AI model stored in the storage of the computer device body with an AI model provided by the AI ​​service cartridge (100), and in the case of a new model, copy the AI ​​model of the AI ​​service cartridge (100) to the storage of the computer device body.

[0072] The computer device (200) can copy the serving system of the AI ​​service cartridge (100) to the storage of the main body (S303).

[0073] In one embodiment, the computer device (200) can compare the serving system stored in the main body's storage with the serving system provided by the AI ​​service cartridge (100) and, in the case of a new serving system, copy the serving system of the AI ​​service cartridge (100) to the main body's storage.

[0074] The computer device (200) can start an AI service by running a serving system corresponding to an AI service cartridge (100) (S304).

[0075] In one embodiment, a serving system operated by a computer device (200) requests a decryption key from an AI service cartridge (100) as needed, and can use this decryption process to use the service activation key or to decrypt an AI model.

[0076] In one embodiment, a serving system operated by a computer device (200) may request verification of fingerprint recognition performed by the computer device (200) from an AI service cartridge (100) as needed, and may use this for activation key of the service or for decoding of the AI ​​model.

[0077] The serving system, which has completed operation in the computer device (200), can provide AI services using a GPU or a dedicated accelerator installed in the AI ​​service cartridge (100) (S305).

[0078] In the example of FIG. 3, an example was illustrated in which an AI model and an AI serving system were copied from an AI service cartridge (100) to a computer device (200), but this is not limited thereto and various modifications can be implemented as described below.

[0080] FIGS. 4 to 6 are block diagrams illustrating examples of configurations of various forms of AI service cartridges according to embodiments of the present disclosure.

[0081] FIGS. 4 to 6 illustrate examples of five types of AI service cartridges (100) from figure (a) to figure (e), and it can be seen that each example of an AI service cartridge (100) has different components in at least some parts.

[0082] In an example according to figure (a) shown in FIG. 4, the AI ​​service cartridge (100) includes a GPU, local memory linked to the GPU, and storage. The local memory may use Massive Local Memory (MLM) or Low Power Double Data Rate (LPDDR), but is not limited thereto. The storage stores AI model data and serving system data for running the AI ​​model. Similar to the description above with reference to FIG. 3, the data stored in the storage can be copied to a computer device, and the computer device can run the GPU based on it to provide AI services.

[0083] In an example according to figure (b) shown in FIG. 4, the AI ​​service cartridge (100) includes an NPU, local memory and storage connected to the NPU. An example according to figure (b) illustrates an example in which an NPU is applied in place of the GPU in the example of figure (a), illustrating a configuration more optimized for an AI operating environment.

[0084] In an example according to figure (c) illustrated in FIG. 5, the AI ​​service cartridge (100) includes a device controller, an NPU, local memory, storage, and a vector DB accelerator. In the example according to figure (c), since a device controller is provided, the computer device can provide AI services by directly interacting with the AI ​​service cartridge (100), that is, by directly driving the AI ​​model through the device controller of the AI ​​service cartridge (100) without the need to install the AI ​​model and serving system on the computer device. The vector DB accelerator is a model that embeds an input query to extract a high-dimensional vector, that is, it may be a module that specializes in processing only vector DB processing. By separately providing such a vector DB accelerator, the input query is embedded into a multidimensional vector, and the AI ​​model is driven to utilize such an embedded vector in the NPU, thereby enabling faster processing. For example, when a user enters a query requesting a search for images similar to an input image, the NPU generates a multidimensional query vector by performing neural network-based inference based on the input image, and the vector DB accelerator can run an AI that performs a similarity-based search between the multidimensional query vector and multidimensional vectors for other images to output target images with a similarity level above a certain threshold as similar images.

[0085] In an example according to figure (d) illustrated in FIG. 5, the AI ​​service cartridge (100) includes an NPU, a first local memory linked to the NPU, a vector DB accelerator, a second local memory linked to the vector DB accelerator, and storage. Here, the first local memory may be an LPDDR capable of storing large-scale data or AI model parameters, and the second local memory may be an MLM capable of storing intermediate data required for vector similarity comparison and data used immediately.

[0086] In one example according to figure (e) shown in FIG. 6, the AI ​​service cartridge (100) illustrates a structure in which a pair of local memory (LPDDR) directly connected to and operating with the NPU is connected in a chain form, and this example can be utilized to use a Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), or Gated Recurrent Unit (GRU) model as an AI model.

[0088] FIG. 7 is a drawing illustrating an example of an AI service cartridge according to an embodiment of the present disclosure, and FIG. 8 is a flowchart illustrating a method for providing an artificial intelligence service in an embodiment illustrated in FIG. 7.

[0089] FIGS. 7 and 8 illustrate an example of an AI service cartridge that provides an AI service for performing video search based on natural language-based queries.

[0090] Referring to FIG. 7, the AI ​​service cartridge (100) may include an NPU, a first local memory (MBM), a second local memory (LPDDR), a vector DB accelerator, and storage. To provide a natural language-based video search service, the AI ​​service cartridge (100) may include the following models and service engines.

[0091] - A model that extracts (embeddings) high-dimensional vectors from images

[0092] - A model that extracts (embeds) high-dimensional vectors of the search target from user input (natural language query).

[0093] - Application file for natural language-based video search service

[0094] - DB system

[0095] Here, application files for the DB system and search service can be installed on the host and stored in storage for such installation. Each AI model can be implemented through an NPU, and the first local memory (MBM) is an ultra-high-speed local memory located near the NPU that can store and process intermediate data required for AI inference and data used immediately. The second local memory (LPDDR) is a low-power memory that can store large-scale data or AI model parameters.

[0096] Referring to FIG. 8, the method of how the AI ​​service cartridge (100) operates with the host computer (200) is described.

[0097] When the AI ​​service cartridge (100) is connected to the host computer (200) (S801), the host computer (200) can recognize the AI ​​service cartridge (100) and install the application file and DB system for the natural language-based video search service on the host (S802). According to the embodiment, the host computer (200) may have the application file and DB system for the natural language-based video search service pre-installed through a separate path—e.g., online download, etc.

[0098] When an application file for a natural language-based video search service is installed on the host computer (200) in this manner, the host computer (200) is immediately ready to provide the AI ​​service, so it can receive a natural language-based video search query, that is, a natural language query, from a user and provide it to the AI ​​service cartridge (100) (S803).

[0099] The search process can be performed in an AI service cartridge (100), and first, the AI ​​service cartridge (100) can build a vector DB by embedding the images to be searched into a high dimension and storing them as multidimensional vectors. To do this, a first AI model can be operated to map images of a video—e.g., thumbnail images—into a multidimensional vector space.

[0100] The first AI model may include a Convolutional Neural Network (CNN). For example, the first AI model may preprocess an input image to an appropriate size, extract features from the preprocessed image using a CNN, and generate a multidimensional embedding vector in the last layer or pooling layer of the CNN. For example, this multidimensional vector may be a high-dimensional vector such as 512 dimensions or 1024 dimensions that summarizes the content of the image.

[0101] Additionally, the AI ​​service cartridge (100) may include a second AI model, which is a text embedding model that embeds text into the aforementioned multidimensional space. The second AI model may perform the role of mapping natural language (text) into the same multidimensional vector space as the aforementioned image embedding model.

[0102] The second AI model can be a natural language processing (NLP) model, and for example, transformer-based models (e.g., BERT, GPT, CLIP, etc.) can be used. These transformer-based models can grasp the context and meaning of text and convert it into vectors.

[0103] The AI ​​service cartridge (100) can extract a high-dimensional query vector from a natural language query using a second AI model.

[0104] Accordingly, when a natural language search query for a user's video search is input, the AI ​​service cartridge (100) extracts a high-dimensional query vector corresponding to the natural language search query using a second AI model (S804), performs a similarity determination with the query vector DB for video images using a vector DB accelerator, and can extract an index containing information about images that are determined to have a similarity greater than a preset level—e.g., information about video images (S805). The AI ​​service cartridge (100) can provide the index information extracted in this way, that is, index information about the video having the image most similar to the user's natural language search query, to the host computer (200).

[0105] The host computer (200) can provide an image search AI service that responds to the user's interactive query by providing a list of images that respond to the user's query based on the received index (S806).

[0106] As previously explained, according to the embodiment, the NPU configuration of the AI ​​service cartridge (100) may be implemented with a GPU and a device controller, or may be changed to a plurality of NPUs, and the configuration of the memory may also be varied in type, number, storage capacity, etc.

[0108] FIG. 9 is a drawing illustrating another example of an artificial intelligence service cartridge according to one embodiment of the present disclosure, and FIG. 10 is a flowchart illustrating a method for providing an artificial intelligence service in one embodiment illustrated in FIG. 9.

[0109] FIGS. 9 and 10 illustrate an example of an AI service cartridge that provides an AI service to automatically generate a document of a specific format—e.g., a patent specification—based on data for creating a document entered by a user.

[0110] Referring to FIG. 9, the AI ​​service cartridge (100) may include an NPU, local memory (LPDDR), and storage. To provide a document creation service of a specific format, the AI ​​service cartridge (100) may include the following models and service engines.

[0111] - LLM model that generates specific documents from user input descriptions

[0112] - Specific document creation service application file

[0113] Here, a specific document creation application file can be installed on a host computer and stored in storage for such installation. The LLM AI model can be implemented through an NPU.

[0114] Referring to FIG. 10, the method of how the AI ​​service cartridge (100) operates with the host computer (200) is described.

[0115] When the AI ​​service cartridge (100) is connected to the host computer (200) (S1001), the host computer (200) can recognize the AI ​​service cartridge (100) and install a specific document creation service application file on the host (S1002). According to an embodiment, the host computer (200) may have previously installed the specific document creation service application file through a separate path—e.g., online download, etc.

[0116] In this way, when an application file for a specific document creation service is installed on the host computer (200), the host computer (200) is immediately ready to provide the AI ​​service, so it can receive a document creation query containing document content for creating a specific document from a user and provide it to the AI ​​service cartridge (100) (S1003).

[0117] The document creation process can be performed in an AI service cartridge (100), and first, the AI ​​service cartridge (100) can create a document that fits a specific format using LLM based on the description in the document creation query and provide it to a host computer (200) (S1004).

[0118] The host computer (200) can perform an AI service that automatically generates a specific type of document according to the user's request by returning a specific type of document generated from the AI ​​service cartridge (100) to the user.

[0120] FIG. 11 is a flowchart illustrating a method for providing AI services using an AI service cartridge according to an embodiment of the present invention.

[0121] Referring to FIG. 11, the host computer device (200) and the AI ​​service cartridge (100) are electrically connected, so when power is applied, both are operated (S1101).

[0122] A service launcher application for AI services is installed on the computer device (200), and when it is executed (S1102), the service launcher is executed.

[0123] A service launcher executed on a computer device (200) detects an AI service cartridge (100) (S1103) and can request an inference engine from the AI ​​service cartridge (100) (S1104).

[0124] The NPU (or device controller) of the AI ​​service cartridge (100) can run an inference engine by uploading a pre-trained model to local memory (S1105).

[0125] The AI ​​service cartridge (100) can provide an executable file of an inference engine to a computer device (200) (S1106), and a service launcher executed on the computer device (200) can execute the provided inference engine (S1107).

[0126] When a user (300) makes a service request for an AI service to a computer device (200) (S1109), the service launcher of the computer device (200) provides an inference acceleration request (S1109) related to the AI ​​service to the AI ​​service cartridge (100). That is, the inference accelerator of the AI ​​service cartridge (100) performs inference acceleration processing (S1110), and the result response data can be provided to the user through the computing device (S1111, 1112).

[0127] The example of FIG. 11 relates to an example in which an inference engine is executed through a service launcher of a computer device (200), and a processing unit (NPU, Device controller) of an AI service cartridge (100) has the main function of accelerating the operation of given data or managing the flow of data.

[0128] Meanwhile, according to another embodiment, the inference engine may be executed in the AI ​​service cartridge (100) itself, or the inference engine may be run in the cartridge. In this case, steps S1104, S1106 to S1107, etc. may be omitted.

[0130] FIG. 12 is a drawing illustrating another embodiment of an artificial intelligence service cartridge according to one embodiment of the present disclosure.

[0131] FIG. 12 illustrates an AI service cartridge (100) connected to the main board (210) of a host computer. The AI ​​service cartridge (100) illustrates an embodiment in which an accelerator (120) and at least one component (130) are implemented together on a single cartridge board (110).

[0132] For example, the accelerator (120) can be a GPU or an NPU.

[0133] The AI ​​service cartridge (100) can be inserted and electrically connected to the main board (210) of a host computer through various interfaces (210)—e.g., PCI or PCIe slot, USB, serial port, Thunderbolt, etc.

[0134] The AI ​​service cartridge (100) can be designed with various modified components depending on the AI ​​service provided, and the component list for the AI ​​service cartridge (100) corresponds to the one described above with reference to FIG. 2, etc., so the description is omitted here.

[0136] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0137] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0138] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0139] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0140] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 An artificial intelligence service cartridge used in combination with a computer device, comprising: an inference accelerator for driving an artificial intelligence model; and a cartridge board connected to the computer device through a slot and electrically connected to the inference accelerator; wherein the cartridge board comprises a storage for storing application files for a service using the artificial intelligence model. Claim 2 In claim 1, the storage further stores an artificial intelligence service cartridge that stores an artificial intelligence model and an inference engine of the artificial intelligence model. Claim 3 An artificial intelligence service cartridge according to claim 1, wherein the cartridge board further comprises a local memory in which parameters of an artificial intelligence model stored in the storage are uploaded and intermediate data required for artificial intelligence inference by the inference accelerator is stored. Claim 4 In claim 1, the artificial intelligence service cartridge is an artificial intelligence service cartridge that, when connected to the computer device, provides a service application file to the computer device so that it can be installed on the computer device. Claim 5 In claim 1, the cartridge board further comprises a device controller that communicates with the controller of the computer device and controls the components within the artificial intelligence service cartridge; and the artificial intelligence service cartridge drives the artificial intelligence model through the device controller. Claim 6 An artificial intelligence service cartridge according to claim 1, wherein the storage further stores an LLM model that generates a document from a user's input description, and the inference accelerator drives the LLM model based on a description in a document creation query received from the computer device to generate a document, and provides the generated document to the computer device. Claim 7 An artificial intelligence service cartridge used in combination with a computer device, comprising: an inference accelerator that drives an artificial intelligence model and is connected to the computer device through a first slot; and a cartridge board that is connected to the computer device through a second slot different from the first slot and includes at least one component that supports the inference accelerator to operate in conjunction with the computer device; wherein the cartridge board further comprises: a high-dimensional vector database; and a vector DB accelerator that compares the similarity of a high-dimensional vector query generated by the inference accelerator with the high-dimensional vector database and extracts a high-dimensional vector having a similarity greater than or equal to a preset similarity with the high-dimensional vector query. Claim 8 In claim 7, the inference accelerator is an artificial intelligence service cartridge that receives a query input from a user of the computer device through an artificial intelligence service as input, and generates the high-dimensional vector query by applying an artificial intelligence model to the query. Claim 9 In claim 7, the artificial intelligence service cartridge further comprises: a first local memory that stores artificial intelligence model parameters in conjunction with the inference accelerator; and a second local memory that is faster than the first local memory and stores intermediate data required for vector similarity comparison. Claim 10 In claim 7, the inference accelerator drives a first artificial intelligence model that maps an image to a multidimensional vector space and a second artificial intelligence model that maps text to the same multidimensional vector space as the first artificial intelligence model, and generates the high-dimensional vector query using the second artificial intelligence model, an artificial intelligence service cartridge.

Images