Apparatus and method for providing input to a multimodal language model

Abstract

The present invention relates to an input device and method for a multimodal language model. An input device for a multimodal language model according to the present invention includes a text processing unit that tokenizes input text to generate language tokens, and a video processing unit that detects an analysis area image for analysis for each frame of an input video sequence, selects the detected analysis area images, and tokenizes the selected analysis area image sequence to generate vision tokens. According to the present invention, language tokens are generated from input text and vision tokens are generated from an input high-resolution video sequence so that they can be used as inputs for a multimodal language model.

Description

Technology Field

[0001] The present invention relates to an input device and method for a multimodal language model, and more specifically, to an input device and method for a multimodal language model that enables text and high-resolution video sequences to be input and used in a multimodal language model. Background Technology

[0002] Since the announcement of OpenAI's GPT-3 in 2020, Large Language Models (LLMs) have demonstrated human-level intelligence capabilities in general knowledge, logical reasoning, scientific question answering, coding, and mathematical problem solving. However, they have limitations as Artificial General Intelligence (AGI) because they lack the ability to reason by integrating multi-sensory information like humans do.

[0003] To overcome these limitations, active research is currently underway to integrate various multi-modal information, such as vision (images), audio, and sensor data, into transactional language models. In particular, multi-modal language models like Vision-Language Models (VLMs) possess high-dimensional visual reasoning capabilities—such as scene interpretation, event analysis, and chart comprehension—going beyond simple image description.

[0004] A common method for integrating vision information into large language models is to convert the output of a pre-trained image encoder into a token format that the model can understand via a projector. In this process, the image encoder is used without separate retraining, and various image encoders with improved performance have recently emerged.

[0005] However, most image encoders are limited to inputs with a resolution of 224 to 512, and when processing high-resolution images, many tokens are generated, which may exceed the input limits of large language models or cause processing delay issues.

[0006] To address this problem, various approaches are being studied that combine global images (low-resolution full images) and partial images (high-resolution segmented images) to reduce the number of input tokens while maintaining the amount of information.

[0007] However, while multimodal language models such as vision language models demonstrate excellent performance in complex image-based reasoning and logical development, they fall short of human visual integration capabilities for understanding and reasoning about temporally continuous information.

[0008] Furthermore, video data generates thousands of vision tokens per image patch, which causes processing latency; therefore, there are limitations to processing long video sequences in vision language models.

[0009] Also, since video data changes over time, video processing of various lengths ranging from seconds to minutes is required, but general vision language models have a limited number of frames that can be processed, and small models can only support about 16 to 100 frames.

[0010] Therefore, there is a need for a method to overcome these limitations and enable the processing of long, high-resolution video sequences along with text in multimodal language models, such as vision language models. Prior art literature

[0011] X. Want, et.al., “LongLLaVA: Scaling multi-modal LLMs to 1000 images efficiently via hybrid architecture,” arXiv preprint arXiv:2409.02889v2, 2024. arXiv:2503.12559v1, 2025.W. Ren, et.al., “Vamba: Understanding hour-long videos with hybrid mamba-transformers,” arXiv preprint arXiv:2503.11579, 2025. The problem to be solved

[0012] Accordingly, the objective of the present invention is to provide an input device and method for a multimodal language model that enables text and high-resolution long video sequences to be input and used in the multimodal language model. means of solving the problem

[0013] An input device for a multimodal language model according to the present invention for achieving the above objective includes a text processing unit that tokenizes input text to generate language tokens, and a video processing unit that detects an analysis area image for analysis for each frame of an input video sequence, selects the detected analysis area image, and tokenizes the selected analysis area image sequence to generate vision tokens.

[0014] The video processing unit may include the analysis area image, an analysis area detection unit that generates image-related information related to the analysis area image, and a sampler unit that selects an analysis area image to perform tokenization using the image-related information in the analysis area image sequence, an image encoding unit that generates a feature vector describing image features for each of the selected analysis area images, and a tokenization unit that generates the vision token by projecting the feature vector.

[0015] Additionally, the video processing unit further includes an additional information extraction unit that generates additional information capable of configuring the spatial location of the analysis area image, and the sampler unit may further generate spatiotemporal location information of the selected analysis area image, and the additional information may include at least one of depth map information and camera pose information corresponding to the analysis area image.

[0016] The video processing unit may further include a position information generation unit that generates position feature information describing position features for each of the spatiotemporal position information of the selected analysis area image, and the tokenization unit may generate the vision token by projecting the feature vector and embedding the position feature information.

[0017] The above position feature information may include time information of each image frame within the selected analysis area image sequence, and the above image-related information may be at least one of the center coordinates and optical flow information of the analysis area image.

[0018] Meanwhile, the input method for a multimodal language model according to the present invention for achieving the above objective comprises the steps of: generating language tokens by tokenizing input text; detecting an analysis area image for analysis for each frame of an input video sequence, selecting the detected analysis area image, and generating vision tokens by tokenizing the sequence of selected analysis area images; and inputting the language tokens and the vision tokens into a multimodal language model.

[0019] The step of generating the vision token may include: generating at least one of an analysis area image for analysis and center coordinates and optical flow information for each frame of the video sequence; selecting an analysis area image to perform tokenization using at least one of the center coordinates and optical flow information in the analysis area image sequence; generating a feature vector describing image features for each of the selected analysis area images; and generating a vision token by projecting the feature vector.

[0020] In addition, to achieve the above objective, the present invention may provide a processor-readable recording medium having a program for executing the input method of the multimodal language model in a processor. Effects of the invention

[0021] According to the present invention, language tokens are generated from input text, and vision tokens are generated by detecting and selecting analysis area images from each frame of an input high-resolution video sequence, thereby enabling the text and high-resolution video sequence to be used as inputs to a multimodal language model.

[0022] In addition, the present invention enables the analysis of text and long high-resolution video sequences using a small-scale multimodal language model with a limited input token size, and also allows frames before and after a change in motion to be effectively selected in consideration of analysis efficiency, thereby providing high accuracy in text and video analysis.

[0023] In addition, the present invention can minimize the processing delay of a multimodal language model by performing sampling and tokenization such that the number of vision tokens input to the multimodal language model is significantly reduced.

[0024] In addition, by adding position information for the analysis area image according to the present invention, high-performance video analysis is made possible by enabling a multimodal language model to use spatiotemporal integrated information in the analysis of a video sequence accompanied by camera movement.

[0025] Accordingly, by using a small-scale multimodal language model that runs in the field without accessing a cloud with large computing power, it is possible to perform efficient actions and situational awareness in various devices and applications, such as mobile devices, robots, disaster situations, or military applications, and it is also possible to develop edge-based intelligent applications that execute predetermined commands based on the results. Brief explanation of the drawing

[0026] FIG. 1 is a drawing referenced in the description of a multimodal language model input device according to an embodiment of the present invention, FIG. 2 is a block diagram of the video processing unit in FIG. 1, FIG. 3 is a drawing referenced in the description of the analysis area detection unit in FIG. 2. FIG. 4 is a drawing referenced in the description of the additional information extraction unit in FIG. 2, FIG. 5 is a drawing referenced in the description of the sampler section in FIG. 2, FIG. 6 is a drawing referenced in the description of the position information generation unit in FIG. 2, FIG. 7 is a drawing referenced in the description of the image encoding unit in FIG. 2, and Figure 8 is a drawing referenced in the description of the tokenization section in Figure 2. Specific details for implementing the invention

[0027] In this specification, where it is stated that one component is "connected" or "connected" to another component, it should be understood that the component may be directly connected or connected to the other component, or that there may be other components in between. Other expressions describing the relationship between components, such as "between" or "neighboring to," and expressions such as one component "transmits" a signal to another component, should be interpreted in the same way.

[0028] The present invention will be described in more detail below with reference to the drawings.

[0029] FIG. 1 is a drawing referenced in the description of an input device for a multimodal language model according to an embodiment of the present invention.

[0030] Referring to FIG. 1, the input device (100) of the multimodal language model may include a text processing unit (150) and a video processing unit (200).

[0031] The text processing unit (150) tokenizes the input text to generate language tokens. The text processing unit (150) can generate language tokens by dividing the input text into small units, such as token sequences, mapping them to numeric IDs, and then converting them into vectors that can be processed by a language model.

[0032] The video processing unit (200) can detect an analysis area image for analysis for each frame of an input video sequence, select the detected analysis area image, and tokenize the selected analysis area image sequence to generate a vision token.

[0033] Vision tokens refer to token-unit representations that transform video data so that a language model can understand and process them. In other words, video data is tokenized like text to make it input into a language model.

[0034] The language tokens generated by the text processing unit (150) and the vision tokens generated by the video processing unit (2000) are input into the multimodal language model (300) and used.

[0035] A multi-modal language model (300) refers to an artificial intelligence language model that can process various input formats, such as images and videos, in addition to text, such as Vision-Language Models (VLM).

[0036] Figure 2 is a block diagram of the video processing unit in Figure 1.

[0037] Referring to FIG. 2, the video processing unit (200) may include an analysis area detection unit (210), an additional information extraction unit (220), a sampler unit (230), a location information generation unit (240), an image encoding unit (250), and a tokenization unit (260). When implemented in an actual application, these components may be configured such that two or more components are combined into one component as needed, or one component is subdivided into two or more components.

[0038] The analysis area detection unit (210) generates an analysis area image, which is an area containing an object to be analyzed, and image-related information related to the analysis area image for each frame of the input video sequence. The image-related information includes at least one of the center coordinates of the analysis area image and optical flow information.

[0039] The additional information extraction unit (220) generates additional information that can configure the spatial location of the analysis area image. The additional information includes depth map information for each analysis area image, camera pose information, etc.

[0040] The sampler unit (230) selects an analysis area image to be tokenized using image-related information and additional information from an analysis area image sequence, which is a set of analysis area images arranged continuously over time, and generates spatiotemporal location information of the selected analysis area image.

[0041] The location information generation unit (240) generates location feature information that describes the location features for each of the spatiotemporal location information of the selected analysis area image.

[0042] The image encoding unit (250) generates a feature vector describing image features for each selected analysis area image.

[0043] The tokenization unit (260) projects a feature vector and embeds location information to generate a vision token that can be used as input to a multimodal language model (300).

[0044] With this configuration, the video processing unit (200) converts the input video sequence to generate a vision token, which can then be input into the multimodal language model (300).

[0045] Figure 3 is a drawing referenced in the description of the analysis area detection unit illustrated in Figure 2.

[0046] Referring to FIG. 3, the analysis area detection unit (210) receives a long video sequence of high resolution, detects an analysis area image containing an object for each input frame to be analyzed, and outputs image-related information such as center coordinates and optical flow information for the detected analysis area image.

[0047] The input and output signals of the analysis area detection unit (210) can be expressed by the following formulas.

[0048]

[0049]

[0050] Here, , ,

[0051] In the above equation, the input signal V is the N+1 input frames F from t=n to t=n+N. t It is a set of input frames composed of, and in the output signal, R represents a set of detected analysis area images, C represents a set of center coordinates within the analysis area images, and O represents a set of optical flow information corresponding to pixel positions of the analysis area images.

[0052] Optical flow information is an estimate of the speed and direction of an object moving over time at the pixel level. In other words, it is a vector field indicating how much and where each pixel has moved between two consecutive frames.

[0053] Specific input frame F t and the analysis area image R corresponding to that time t (or O t Taking the resolution of ) as an example, F t When ga is a video frame input with 1,920x1,080 FHD resolution, the detected analysis region image R here t It can be 224x224 in size, and R t The center of is F t C to determine which location in t It will output as.

[0054] The analysis area detection unit (210) may use various methods to generate an output signal such as the above equation for an input video sequence. For example, it may use a method of performing video re-identification to determine the frame-by-frame location of an object and whether it is the same instance, and then generate an analysis area image and the center coordinates of each analysis area image. Video re-identification is a technique for re-identifying the same object (usually a person) in multiple videos or images captured by multiple cameras. That is, it involves finding an object seen in one video or camera again at a different point in time, in a different place, or with a different camera.

[0055] In addition, region-specific movement can be tracked through methods such as detecting optical flow information and performing regional merging of pixel-specific movements to generate an analysis region image and its center coordinates from the movement region, or an output can be generated by combining video re-recognition and optical flow information.

[0056] Various methods can be used for detecting optical flow information. For example, detection can be performed using traditional block motion estimation with two adjacent frames as input, or a neural network can be used to detect optical flow information by inputting two adjacent frames or the entire video.

[0057] The analysis area detection unit (210) may output only one analysis area image per input frame, or it may output multiple different analysis area images. When the analysis area detection unit (210) outputs multiple analysis area images, the video processing unit (200) may determine the analysis priority and process the analysis area image sequences one by one according to the determined priority.

[0058] In this case, the determination of analysis priority may apply pre-set object-specific priorities, or it may be configured to process movement areas where the type of object cannot be identified with a high priority.

[0059] Figure 4 is a drawing referenced in the description of the additional information extraction unit illustrated in Figure 2.

[0060] Referring to FIG. 4, the additional information extraction unit (220) takes a long high-resolution video sequence as input and extracts information that can construct a three-dimensional spatial position for an analysis area image detected by the analysis area detection unit (210).

[0061] The output of the additional information extraction unit (220) can be depth map information per input frame in its simplest form. Camera attitude information can be added to the depth map information in order to generate integrated information using a world coordinate system.

[0062] The input and output signals of the additional information extraction unit (220) can be expressed as follows in mathematical formulas.

[0063]

[0064]

[0065] Here,

[0066] In the above formula, the video sequence V input to the additional information extraction unit (220) is the same as the input from the analysis area detection unit (210).

[0067] D, which is output from the additional information extraction unit (220), is each input frame F t Depth map information D corresponding to each pixel position t It is a set of, and W is the input frame F t Camera pose information W that converts the camera coordinate system to the world coordinate systemt It is a set of. That is, the output of the additional information extraction unit (220) can be a depth map information sequence D, or {D, W} including a depth map information sequence and world coordinate system transformation parameters (camera pose information sequence).

[0068] The world coordinate system is an absolute reference coordinate system established based on the three-dimensional space of the real world, which serves as a common standard when representing all objects, cameras, movements, etc., and depth map information is information indicating how far each pixel in an image is from the camera.

[0069] Generating a depth map information sequence D from an input video sequence can be implemented in various ways. For example, an image-based depth map generation neural network that generates a depth map using each frame of the video as input can be used, or a video-based depth map generation neural network that uses adjacent frames of the video as input can be used.

[0070] In addition, the camera attitude information sequence W for the input video sequence There are various ways to implement the method for generating it. For example, a neural network that estimates the camera pose using adjacent video frames as input can be used, or a neural network that estimates both the depth map and the camera pose simultaneously using adjacent video frames as input can be used.

[0071] Figure 5 is a drawing referenced in the description of the sampler part illustrated in Figure 2.

[0072] Referring to FIG. 5, the sampler unit (230) receives the output signal of the analysis area detection unit (210) and the output signal of the additional information extraction unit (220), and generates a selected analysis area image sequence and spatiotemporal location information of an object within the selected analysis area image.

[0073] At this time, the purpose of selecting the analysis area images is to effectively remove redundant elements present in the image sequences representing the behavior and situation of the object within the sequence to be analyzed through the multimodal language model (300), thereby leaving only a very small number of analysis area images and the spatiotemporal location information of the object corresponding thereto.

[0074] The input and output signals of the sampler unit (230) can be expressed as follows:

[0075]

[0076]

[0077] In the input signal, O and W can be optionally input, and am.

[0078] In the above equation, the analysis area image sequence R input to the sampler unit (230), the center coordinate sequence C of each frame within the analysis area image sequence, the optical flow information sequence O of each pixel within the analysis area, the depth map information sequence D of each pixel within the analysis area image, and the camera pose information sequence W that performs world coordinate transformation of each image within the analysis area image sequence are as described above.

[0079] In the above equation, the selected analysis area image sequence R output by the sampler unit (230). s and the spatiotemporal location information sequence P of the selected analysis area image s is determined by a set of time indices I selected by the sampler unit (130), and a spatiotemporal location information sequence P S Each spacetime location information P t It can be generated by comprehensively utilizing the information input into the sampler unit (230).

[0080] Spatiotemporal location information P of the sampler unit (230) t It can be generated in various ways as follows.

[0081] First, if W is not input, the depth frame information D of the input depth map information sequence t From, the average distance Z of the front object t Calculate the calculated average distance and the center coordinate C having the same time information as the depth frame used in the average distance calculation within the input analysis area image sequence. t Combined with, spacetime location information P t = (x t , y t , z t It can generate , t).

[0082] When W is input, each depth frame D of the input depth map information sequence t From, the average distance Z of the front object t Calculate the calculated average distance and the area center coordinate C having the same time information as the depth frame used in the average distance calculation within the input analysis area image sequence. t Combined with, spatial position S t = (x t , y t , z t Generate ) and camera pose information W having the same time information as the depth map frame used in the average distance calculation within the input analysis area image sequence. t By transforming the spatial location, the spatiotemporal location information P t = (x t , y t , z t It can generate , t).

[0083] The task of the sampler unit (230) to construct a selected time index set I can be implemented in various ways. For example, by modeling the spatial movement of each frame within the analysis area image sequence and detecting the change in that movement, the time before and after the time index where the change occurs can be included in the selected index set. Additionally, by modeling the object movement between two adjacent times in the spatial movement change-based selected index set and detecting the change in that object movement, the time before and after the time index where the change occurs can be added.

[0084] In addition, it may be implemented to generate a set of time indices I selected through neural network learning from some input combinations of the sampler part (230).

[0085] At this time, spatial movement modeling of each frame within the analysis area image sequence can be implemented in various ways from the input of the sampler unit (230). For example, a change prediction device for the center coordinate sequence C of the frame within the analysis area image sequence can be configured to detect the time index at which the prediction error exceeds a reference value. Additionally, the spatiotemporal position P of the sampler unit (230) t Time index detection can also be performed by configuring a change prediction device based on .

[0086] Meanwhile, detection of changes in object movement within a specific time interval can be implemented in various ways using some input combinations of the sampler unit (230).

[0087] For example, depending on whether the optical flow information sequence O and the camera attitude information sequence W are input, if both O and W are input, O is corrected with W to obtain the optical flow information O in the world coordinate system. c Generates and corrected O cAn optical flow prediction device can be configured from to detect a time index based on the prediction error. Here, the prediction device may use a traditional optical flow predictor, and O t c It can also be implemented by training a neural network that models the flow of.

[0088] In cases where the optical flow information sequence O is input but the camera pose information sequence W is not input, the method can be implemented as in the case where O and W are input by generating camera pose information through a neural network that estimates W using R and C, or by using C to correct O to the viewpoint of the analysis area image sequence and then configuring an optical flow prediction device for the corrected O to detect the time index.

[0089] In cases where W is input but O is not, optical flow can be estimated between two adjacent frames of R, and after correcting the estimated optical flow using C, an implementation can be utilized for cases where both optical flow information O and camera pose information W are input, or a change detection neural network can be constructed that uses the corrected analysis target area frame as input after correcting R with W to detect the time index.

[0090] In addition, if W and O are not input, W can be estimated from R and C and O can be generated to utilize the implementation for cases where both optical flow information O and camera attitude information W are input, or a time index can be detected using a token compressor that selects key frames within a interval with R as input.

[0091] In the implementation of detecting changes in object movement within a specific time interval based on whether optical flow information sequence O and camera pose information sequence W are input, the implementation when specific information is input includes all implementations when such information is not input.

[0092] The generation of a set of time indices through the training of a neural network can be implemented in various ways. For example, a video token compression neural network that directly detects selected time indices can be implemented by inputting an analysis area image sequence R input into a sampler (230). R t E P t Selective time index detection can also be implemented through a video token compression neural network that combines the position encoding results of R t E P t It is also possible to implement a video token compression neural network that combines. And, R t and D t or R t and D t , and O t It is also possible to implement a video token compression neural network that combines all of them.

[0093] Figure 6 is a drawing referenced in the description of the location information generation unit illustrated in Figure 2.

[0094] Referring to FIG. 6, the position information generation unit (240) generates position feature information so that the vision token generated by the video processing unit (200) can include spatiotemporal position information. The input and output signals of such a position information generation unit (240) can be expressed mathematically as follows.

[0095]

[0096]

[0097] Here,

[0098] In the above equation, the output signal PF of the position information generating unit (240) at a specific time index t t It can be implemented in various ways.

[0099] For example, spatiotemporal location information P entered for each image token information. tIt can also be generated and inserted as a text token. For each token per image, it can also be implemented by extending the language model's trigonometric function-based position encoding to four dimensions to generate a position vector. In addition, P t It can also be implemented by configuring a neural network to learn the generation of position vectors for each vision token based on position information for each token.

[0100] In this case, an example implemented with a text token is “image space location: Image time location: t It can be like ”, and the position feature information generated as a position vector is expressed as an N-dimensional real value.

[0101] Figure 7 is a drawing referenced in the description of the image encoding unit illustrated in Figure 2.

[0102] Referring to FIG. 7, the image encoding unit (250) analyzes the target area sequence R selected by the sampler unit (230). s It performs the role of generating feature vectors that describe image features for each image frame.

[0103] The input and output of the image encoding unit (250) can be expressed as follows in mathematical formulas.

[0104]

[0105]

[0106] Here,

[0107] In the above equation, the output feature vector PF of the image encoding unit (250) at a specific time index t t It can be implemented by applying any one of the general vision processors, or it can be constructed by concatenating feature vectors generated by applying various implementations.

[0108] Figure 8 is a drawing referenced in the description of the tokenization unit illustrated in Figure 2.

[0109] Referring to FIG. 8, the tokenization unit (260) [receives] each image feature vector IF output from the image encoding unit (250). t Projecting and the location feature information PF generated by the location information generation unit (240). t A vision token that can be input into a multi-modal language model (300) is generated by embedding it.

[0110] Here, projection refers to the process of converting a single data representation into a representation space that a multimodal language model (300) can understand, and embedding refers to the process of representing the data as a vector that the multimodal language model (300) can understand.

[0111] The projection of the tokenization unit (260) can be implemented in various ways. For example, it can be implemented by learning a simple linear neural network layer, or by learning a neural network composed of 2 to 3 layers. In addition, it can be implemented by learning a convolution-based video token compression neural network.

[0112] The position feature information embedding of the tokenization unit (260) can be implemented differently depending on the output format of the position information generation unit (240). That is, if the output format of the position information generation unit (240) is a text token, it is implemented to connect a text token after each projected image token, and if the output format of the position information generation unit (240) is a position vector, a vision token can be generated by connecting a position vector to each projected image token.

[0113] Also, an additional token compression device capable of learning may be added to the tokenization unit (260).

[0114] Meanwhile, the input device and method for a multimodal language model according to the present invention are not limited to the configurations of the embodiments described above; rather, all or part of each embodiment may be selectively combined to allow for various modifications to the embodiments.

[0115] Furthermore, the present invention can be implemented as a computer program on a programmable computer. Such a computer may include a processor, a storage device, an input device, and an output device. To implement the contents described in the present invention, program code may be input via a mouse or keyboard input device. Such programs may be implemented in a high-level language or an object-oriented language. They may also be implemented as a computer system implemented in assembly or machine code.

[0116] The present invention may also be implemented as processor-readable code on a processor-readable recording medium. A processor-readable recording medium includes all types of recording devices in which data that can be read by a processor is stored. Examples of processor-readable recording media include ROM, RAM, CD-ROM, optical data storage devices, etc. Additionally, the processor-readable recording medium may be distributed across networked computer systems, allowing processor-readable code to be stored and executed in a distributed manner.

[0117] Furthermore, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. Various modifications are possible by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention. Explanation of the symbols

[0118] 100: Input device for a multimodal language model 150: Text processing unit 200: Video processing unit 300: Multimodal language model

Claims

Claim 1 An input device for a multimodal language model comprising: a text processing unit that tokenizes input text to generate language tokens; an analysis area detection unit that generates image-related information for each frame of an input video sequence, the analysis area image being an area containing an object to be analyzed, and at least one of the center coordinates and optical flow information of the analysis area image; a sampler unit that selects an analysis area image to be tokenized using the image-related information in the analysis area image sequence; an image encoding unit that generates a feature vector describing image features for each of the selected analysis area images; and a tokenization unit that generates a vision token by projecting the feature vector. Claim 2 delete Claim 3 delete Claim 4 The input device of a multimodal language model according to claim 1, wherein the video processing unit further comprises an additional information extraction unit that generates additional information capable of configuring the spatial location of the analysis area image, and the sampler unit further generates spatiotemporal location information of the selected analysis area image. Claim 5 In paragraph 4, the above additional information is an input device of a multimodal language model comprising at least one of depth map information and camera pose information corresponding to the analysis area image. Claim 6 An input device for a multimodal language model according to claim 4, further comprising a position information generation unit that generates position feature information describing position features for each of the spatiotemporal position information of the selected analysis area image, wherein the tokenization unit, when generating the position feature information, projects the feature vector and embeds the position feature information to generate the vision token. Claim 7 An input device for a multimodal language model, characterized in that, in claim 6, the position feature information includes time information of each image frame within the selected analysis area image sequence. Claim 8 delete Claim 9 A method for inputting a multimodal language model comprising: a step of generating language tokens by tokenizing text input in a text processing unit; a step of generating image-related information for each frame of a video sequence input in an analysis area detection unit, wherein the analysis area image is an area containing an object to be analyzed, and at least one of the center coordinates of the analysis area image and optical flow information; a step of selecting an analysis area image to be tokenized in the analysis area image sequence using the image-related information in a sampler unit; a step of generating a feature vector describing image features for each of the selected analysis area images in an image encoding unit; a step of generating a vision token by projecting the feature vector in a tokenization unit; and a step of inputting the language tokens and the vision tokens into a multimodal artificial intelligence language model. Claim 10 In claim 9, the input method of a multimodal language model further comprises: a step of generating additional information capable of configuring the spatial location of the analysis area image in an additional information extraction unit; a step of generating spatiotemporal location information of the selected analysis area image in a sampler unit; a step of generating location feature information describing a location feature for each of the spatiotemporal location information of the selected analysis area image in a location information generation unit; and a step of generating the vision token by projecting the feature vector and embedding the location feature information in the tokenization unit when generating the location feature information.

Images