Apparatus and method for providing vision input to a vision-language model

Abstract

The present invention relates to a vision input device and method. A vision input device according to the present invention comprises: an analysis area image for analysis for each frame of an input video sequence; an analysis area detection unit that generates image-related information related to the analysis area image; an additional information generation unit that generates additional information capable of constituting the spatial location of the analysis area image; a sampler unit that selects an analysis area image to perform tokenization using image-related information and additional information in an analysis area image sequence and generates spatiotemporal location information of the selected analysis area image; an image encoding unit that generates a feature vector describing image features for each selected analysis area image; a location information generation unit that generates location feature information describing location features for each spatiotemporal location information of the selected analysis area image; and a tokenization unit that projects the feature vector and embeds the location feature information to generate a vision token. According to the present invention, a vision token is generated from a high-resolution video sequence and can be used as input to a vision language model.

Description

Technology Field

[0001] The present invention relates to a vision input device and a method thereof, and more specifically, to a vision input device and a method thereof that enables the input of a high-resolution video sequence into a vision language model for use. Background Technology

[0002] Since the announcement of OpenAI's GPT-3 in 2020, Large Language Models (LLMs) have demonstrated human-level intelligence capabilities in general knowledge, logical reasoning, scientific question answering, coding, and mathematical problem solving. However, they lack the ability to reason by integrating multi-sensory information like humans, which limits their implementation in Artificial General Intelligence (AGI).

[0003] To overcome these limitations, active research is currently underway to integrate various multi-modal information, such as vision (images), audio, and sensor data, into transaction language models. In particular, Vision-Language Models (VLMs) possess high-dimensional visual reasoning capabilities that go beyond simple image description, including scene interpretation, event analysis, and chart comprehension.

[0004] Generally, the method of integrating vision information into a large language model involves converting the output of a pre-trained image encoder into a token format that the model can understand via a projector. In this process, the image encoder is used without separate retraining, and recently, various image encoders with improved performance have emerged.

[0005] However, most image encoders are limited to inputs with a resolution of 224 to 512, and when processing high-resolution images, many tokens are generated, which may exceed the input limits of large language models or cause processing delay issues.

[0006] To address this problem, various approaches are being studied that combine global images (low-resolution full images) and partial images (high-resolution segmented images) to reduce the number of input tokens while maintaining the amount of information.

[0007] However, while vision language models demonstrate excellent performance in complex image-based reasoning and logical development, they fall short of human visual integration capabilities for understanding and reasoning about temporally continuous information.

[0008] Furthermore, video data generates thousands of vision tokens per image patch, which causes processing latency; therefore, there are limitations to processing long video sequences in vision language models.

[0009] Also, since video data changes over time, video processing of various lengths ranging from seconds to minutes is required, but general vision language models have a limited number of frames that can be processed, and small models can only support about 16 to 100 frames.

[0010] Therefore, a method is needed to overcome these limitations and enable the processing of long, high-resolution video sequences within a vision language model. Prior art literature

[0011] K. Li, et.al., “VideoChat: Chat-centric video understanding,” arXiv preprint arXiv:2305.06355v2, 2024.E. Song, et.al., “MovieChat: From dense token to sparse memory for long video understanding,” arXiv preprint arXiv:2307.16449v4, 2024.X. Zhang, et.al., “inyLLaVA-Video: A simple framework of small-scale large multimodal models for video understanding,” arXiv preprint arXiv:2501.15513, 2025. The problem to be solved

[0012] Accordingly, the objective of the present invention is to provide a vision input device and a method thereof that enable the input and use of a high-resolution, long video sequence into a vision language model. means of solving the problem

[0013] A vision input device according to the present invention for achieving the above objective comprises: an analysis area image for analysis for each frame of an input video sequence; an analysis area detection unit that generates image-related information related to the analysis area image; an additional information generation unit that generates additional information capable of configuring the spatial position of the analysis area image; a sampler unit that selects an analysis area image to perform tokenization using the image-related information and the additional information in the analysis area image sequence and generates spatiotemporal position information of the selected analysis area image; an image encoding unit that generates a feature vector describing an image feature for each of the selected analysis area images; a position information generation unit that generates position feature information describing a position feature for each of the spatiotemporal position information of the selected analysis area image; and a tokenization unit that projects the feature vector and embeds the position feature information to generate a vision token.

[0014] The image-related information may be at least one of the center coordinates of the analysis area image and optical flow information, and the additional information may be at least one of depth map information, camera attitude information, and GPS information of the camera.

[0015] The sampler unit above can generate the spatiotemporal location information using the center pixel coordinates of the selected analysis area image, any one of an image coordinate system, a world coordinate system, and a geometric coordinate system.

[0016] The above position feature information can be generated as text tokens or as position vectors by extending trigonometric function-based position encoding, and time information of each image frame within the selected analysis area image sequence may be added to the above position feature information.

[0017] The above vision token can be configured by connecting the text token to each projected image token when the output format of the above position feature information is a text token, and by connecting the position vector to each projected image token when the output format of the above position feature information is a position vector.

[0018] In addition, the above additional information generating unit can receive and use external additional information from at least one of an IMS sensor and a GPS sensor installed on a mobile device or a robot.

[0019] Meanwhile, a vision input method according to the present invention for achieving the above objective comprises the steps of: generating an analysis area image for analysis and image-related information related to the analysis area image for each frame of an input video sequence; generating additional information capable of configuring the spatial position of the analysis area image; selecting an analysis area image to perform tokenization using the image-related information and the additional information in the analysis area image sequence and generating spatiotemporal position information of the selected analysis area image; generating position feature information explaining position features for each of the spatiotemporal position information of the selected analysis area image; generating a feature vector explaining image features for each of the selected analysis area images; and projecting the feature vector and embedding the position feature information to generate a vision token.

[0020] In addition, to achieve the above objective, the present invention may provide a processor-readable recording medium that records a program for executing the vision input method in a processor. Effects of the invention

[0021] According to the present invention, by detecting and selecting an analysis area image from each frame of a high-resolution video sequence to generate a vision token, a high-resolution video sequence can be used as input to a vision language model.

[0022] In addition, the present invention enables the analysis of long, high-resolution video sequences using a small-scale vision language model with a limited input token size, and also allows frames before and after a change in motion to be effectively selected in consideration of analysis efficiency, thereby providing high accuracy in video analysis of the vision language model.

[0023] In addition, the present invention can minimize the processing delay of a vision language model by performing sampling and talkization so that the number of video vision tokens input to the vision language model is significantly reduced.

[0024] In addition, the present invention enables high-performance video analysis by adding integrated location information to the analysis area image, thereby allowing the vision language model to use spatiotemporal integrated information in the analysis of video sequences accompanied by camera movement.

[0025] Accordingly, by using a small-scale vision language model that runs in the field without accessing a cloud with large computing power, it is possible to perform efficient actions and situational awareness in various devices and applications, such as mobile devices, robots, disaster situations, or military applications, and it is also possible to develop edge-based intelligent applications that execute predetermined commands based on the results. Brief explanation of the drawing

[0026] FIG. 1 is a block diagram of a vision input device according to an embodiment of the present invention, FIG. 2 is a drawing referenced in the description of the analysis area detection unit in FIG. 1. FIG. 3 is a drawing referenced in the description of the additional information generation unit in FIG. 1, FIG. 4 is a drawing referenced in the description of the sampler section in FIG. 1, FIG. 5 is a drawing referenced in the description of the position information generation unit in FIG. 1, FIG. 6 is a drawing referenced in the description of the image encoding unit in FIG. 1, FIG. 7 is a drawing referenced in the description of the tokenization unit in FIG. 1, and FIG. 8 is a flowchart provided to describe a vision input method according to an embodiment of the present invention. Specific details for implementing the invention

[0027] In this specification, where it is stated that one component is "connected" or "connected" to another component, it should be understood that the component may be directly connected or connected to the other component, or that there may be other components in between. Other expressions describing the relationship between components, such as "between" or "neighboring to," and expressions such as one component "transmits" a signal to another component, should be interpreted in the same way.

[0028] The present invention will be described in more detail below with reference to the drawings.

[0029] FIG. 1 is a block diagram of a vision input device according to an embodiment of the present invention.

[0030] Referring to FIG. 1, the vision input device (100) may include an analysis area detection unit (110), an additional information generation unit (120), a sampler unit (130), a location information generation unit (140), an image encoding unit (150), and a tokenization unit (160).

[0031] When these components are implemented in actual applications, two or more components may be combined into a single component as needed, or one component may be subdivided into two or more components.

[0032] The analysis area detection unit (110) generates an analysis area image, which is an area containing an object to be analyzed, and image-related information related to the analysis area image for each frame of the input video sequence. The image-related information includes at least one of the center coordinates of the analysis area image and optical flow information.

[0033] The additional information generation unit (120) generates additional information that can configure the spatial location of the analysis area image. The additional information includes depth map information for each analysis area image, camera attitude information, and GPS information of the camera, and the camera attitude information and the GPS information of the camera can be received as external additional information.

[0034] The sampler unit (130) selects an analysis area image to be tokenized using image-related information and additional information from an analysis area image sequence, which is a set of analysis area images arranged continuously over time, and generates spatiotemporal location information of the selected analysis area image.

[0035] The location information generation unit (140) generates location feature information that describes the location features for each of the spatiotemporal location information of the selected analysis area image.

[0036] The image encoding unit (150) generates a feature vector describing image features for each selected analysis area image.

[0037] The tokenization unit (160) projects feature vectors and embeds location information to generate vision tokens that can be used as input to Vision-Language Models (VLM). A vision token refers to a token-unit representation of video data converted so that the vision-language model can understand and process it. In other words, the video data is tokenized like text so that it can be input into the vision-language model.

[0038] With this configuration, the vision input device (100) converts the input video sequence to generate a vision token and enables the generated vision token to be used as an input to a vision language model.

[0039] Figure 2 is a drawing referenced in the description of the analysis area detection unit illustrated in Figure 1.

[0040] Referring to FIG. 2, the analysis area detection unit (110) receives a long video sequence of high resolution, detects an analysis area image containing an object for each input frame to be analyzed, and outputs image-related information such as center coordinates and optical flow information for the detected analysis area image.

[0041] The input and output signals of the analysis area detection unit (110) can be expressed as follows in mathematical formulas.

[0042]

[0043]

[0044] Here, , ,

[0045] In the above equation, the input signal V is the N+1 input frames F from t=n to t=n+N. t It is a set of input frames composed of, and in the output signal, R represents a set of detected analysis area images, C represents a set of center coordinates within the analysis area images, and O represents a set of optical flow information corresponding to pixel positions of the analysis area images.

[0046] Optical flow information is an estimate of the speed and direction of an object moving over time at the pixel level. In other words, it is a vector field indicating how much and where each pixel has moved between two consecutive frames.

[0047] Specific input frame F t and the analysis area image R corresponding to that time t (or O t Taking the resolution of ) as an example, F t When ga is a video frame input with 1,920x1,080 FHD resolution, the detected analysis region image R here t It can be 224x224 in size, and R t The center of is F t C to determine which location in t It will output as.

[0048] The analysis area detection unit (110) may use various methods to generate an output signal such as [Equation 1] for an input video sequence. For example, it may use a method of performing video re-identification to determine the frame-by-frame location of an object and whether it is the same instance, and then generate an analysis area image and the center coordinates of each analysis area image. Video re-identification is a technique for re-identifying the same object (usually a person) in multiple videos or images captured by multiple cameras. That is, it is to find an object seen in one video or camera again at a different point in time, in a different place, or with a different camera.

[0049] In addition, region-specific movement can be tracked through methods such as detecting optical flow information and performing regional merging of pixel-specific movements to generate an analysis region image and its center coordinates from the movement region, or an output can be generated by combining video re-recognition and optical flow information.

[0050] Various methods can be used to detect optical flow information. For example, detection can be performed using traditional block motion estimation with two adjacent frames as input, or a neural network can be used to detect optical flow information by inputting two adjacent frames or the entire video.

[0051] The analysis area detection unit (110) may output only one analysis area image per input frame, or it may output multiple different analysis area images. When the analysis area detection unit (110) outputs multiple analysis area images, the analysis priority can be determined within the vision input device (100), and the analysis area image sequences can be processed sequentially one by one according to the determined priority.

[0052] In this case, the determination of analysis priority may apply pre-set object-specific priorities, or it may be configured to process movement areas where the type of object cannot be identified with a high priority.

[0053] Figure 3 is a drawing referenced in the description of the additional information generation unit illustrated in Figure 1.

[0054] Referring to FIG. 3, the additional information generation unit (120) takes a long high-resolution video sequence as input and extracts information that can construct a three-dimensional spatial position for an analysis area image detected by the analysis area detection unit (110).

[0055] The output of the additional information generation unit (120) can be depth map information for each input frame in its simplest form. Camera attitude information can be added to the depth map information to generate integrated information using a world coordinate system. Additionally, the camera's GPS (Global Positioning System) information can be added to the output of the additional information generation unit (120) to construct a three-dimensional spatial position based on a geographic coordinate system.

[0056] The input and output signals of the additional information generation unit (120) can be expressed as follows in mathematical formulas.

[0057]

[0058] Here,

[0059]

[0060] Here,

[0061] In the above equation, the video sequence V input to the additional information generation unit (120) is the same as the input from the analysis area detection unit (110), and W is the input frame F t Camera pose information W that converts the camera coordinate system to the world coordinate system t It is a set of, and G is each input frame F t G, the camera's GPS information such as latitude, longitude, and altitude indicating the camera's location t It is a set of.

[0062] The world coordinate system is an absolute reference coordinate system established based on the three-dimensional space of the real world, and it is a coordinate system that serves as a common standard when representing all objects, cameras, movements, etc.

[0063] D, which is output from the additional information generation unit (120), is each input frame F t Depth map information D corresponding to each pixel position t It is a set of, and W and G are the same as in the input signal.

[0064] Depth map information indicates how far each pixel in an image is from the camera.

[0065] When the input of the additional information generation unit (120) is given only as a video sequence V, the output may be depth map information D, or {D, W} containing depth map information and world coordinate system transformation parameters (camera pose information).

[0066] Generating depth map information D for an input video sequence can be implemented in various ways. For example, an image-based depth map generation neural network that generates a depth map using each frame of the video as input can be used, or a video-based depth map generation neural network that uses adjacent frames of the video as input can be used.

[0067] In addition, camera attitude information W for the input video sequence There are various ways to implement the method for generating it. For example, a neural network that estimates the camera pose using adjacent video frames as input can be used, or a neural network that estimates both the depth map and the camera pose simultaneously using adjacent video frames as input can be used.

[0068] When the input to the additional information generation unit (120) is a video sequence and camera pose information {V, W}, the output is depth map information and camera pose information {D, W}. In this case, the camera pose information W is used as is, and the depth map information D can be generated using a method that generates depth map information based on the input video sequence.

[0069] When the input to the additional information generation unit (120) is a video sequence and camera GPS information {V, G}, the output is depth map information, camera attitude information, and camera GPS information {D, W, G}. In this case, the camera GPS information G is used as is, while the depth map information D and camera attitude information W are generated using a method that generates depth map information and camera attitude information based on the input video.

[0070] When the input of the additional information generation unit (120) includes a video sequence, camera attitude information, and camera GPS information {V, W, G}, the output is depth map information, camera attitude information, and camera GPS information {D, W, G}. In this case, the camera attitude information W and camera GPS information G are used as input information, and the depth map information D is generated using a method that generates depth map information based on the input video sequence.

[0071] Examples of cases where external additional information is used in the additional information generation unit (120) include cases where the vision input device (100) according to the present invention is used in a mobile device or a robot, etc. In such cases, accurate camera attitude information and GPS information of the camera can be provided from an IMU (Inertial Measurement Unit) sensor and a GPS sensor that are generally mounted on the mobile device or a robot.

[0072] Figure 4 is a drawing referenced in the description of the sampler part illustrated in Figure 1.

[0073] Referring to FIG. 4, the sampler unit (130) receives the output signal of the analysis area detection unit (110) and the output signal of the additional information generation unit (120) to generate a selected analysis area image sequence and spatiotemporal location information of an object within the selected analysis area image. At this time, the purpose of selecting the analysis area images is to effectively remove redundant elements present in the image sequences that express the behavior and situation of an object within the sequence to be analyzed through a large vision language model, etc., so as to leave only a very small number of analysis area images and spatiotemporal location information of the object corresponding thereto.

[0074] The input and output signals of the sampler unit (130) can be expressed as follows:

[0075]

[0076] However, O, W, and G can be entered optionally.

[0077]

[0078] Here,

[0079] In the above equation, the analysis area image sequence R input to the sampler unit (130), the center coordinate sequence C of each frame within the analysis area image sequence, the optical flow information sequence O of each pixel within the analysis area, the depth map information sequence D of each pixel within the analysis area image, the camera attitude information sequence W that performs world coordinate transformation of each image within the analysis area image sequence, and the GPS information sequence G of the camera of each frame within the analysis area image sequence are as described above.

[0080] In [Mathematical Formula 3], the selected analysis area image sequence R output by the sampler unit (130) s and the spatiotemporal location information sequence P of the selected analysis area image s is determined by a set of time indices I selected by the sampler unit (130), and spatiotemporal location information sequence P S Each spacetime location information P t It can be generated by comprehensively utilizing the information input into the sampler unit (130).

[0081] Spatiotemporal location information P of the sampler unit (130) t It can be generated in various ways as follows.

[0082] First, if W is not input, the depth frame information D of the input depth map information sequence t From, the average distance Z of the front object t Calculate the calculated average distance and the center coordinate C having the same time information as the depth frame used in the average distance calculation within the input analysis area image sequence. t Combined with, spacetime location information Pt = (x t , y t , z t It can generate , t).

[0083] When W is input, each depth frame D of the input depth map information sequence t From, the average distance Z of the front object t Calculate the calculated average distance and the area center coordinate C having the same time information as the depth frame used in the average distance calculation within the input analysis area image sequence. t Combined with, spatial position S t = (x t , y t , z t Generate ) and camera pose information W having the same time information as the depth map frame used in the average distance calculation within the input analysis area image sequence. t By transforming the spatial location, the spatiotemporal location information P t = (x t , y t , z t It can generate , t).

[0084] When W and G are input (if G is input, W must be input), each depth frame D of the input depth map information sequence t From, the average distance Z of the front object t Calculate the calculated average distance, and the area center coordinate C having the same time information as the depth map frame used in the average distance calculation within the input analysis area image sequence. t Combined with, spatial position S t = (x t , y t , z t Generates ). And, camera pose information W having the same time information as the depth map frame used in the average distance calculation within the input analysis area image sequence. tBy transforming the spatial position to the integrated spatial position P c t = (x c t , y c t , z c t After generating ), GPS information G having the same time information as the depth map frame used in the average distance calculation within the input analysis area image sequence. t By performing a geometric coordinate system transformation using , the spacetime location information P t = (x t , y t , z t Generates , t).

[0085] The task of the sampler unit (130) to construct a selected set of time indices I can be implemented in various ways. For example, by modeling the spatial movement of each frame within the image sequence of the analysis area and detecting the change in that movement, the time before and after the time index where the change occurs can be included in the selected set of indices. Additionally, by modeling the movement of an object between two adjacent times in the selected set of indices based on spatial movement change and detecting the change in that object movement, the time before and after the time index where the change occurs can be added. Furthermore, the sampler unit (130) can be implemented to generate a selected set of time indices I through neural network learning from some input combinations.

[0086] At this time, spatial movement modeling of each frame within the analysis area image sequence can be implemented in various ways from the input of the sampler unit (130). For example, a change prediction device for the center coordinate sequence C of the frame within the analysis area image sequence can be configured to detect the time index at which the prediction error exceeds a reference value. Additionally, the spatiotemporal position P of the sampler unit (130) t Time index detection can also be performed by configuring a change prediction device based on .

[0087] Meanwhile, detection of changes in object movement within a specific time interval can be implemented in various ways using some input combinations of the sampler unit (130).

[0088] For example, depending on whether the optical flow information sequence O and the camera attitude information sequence W are input, if both O and W are input, O is corrected with W to obtain the optical flow information O in the world coordinate system. c Generates and corrected O c An optical flow prediction device can be configured from to detect a time index based on the prediction error. Here, the prediction device may use a traditional optical flow predictor, and O t c It can also be implemented by training a neural network that models the flow of.

[0089] In the case where the optical flow information sequence O is input but W is not input, the method can be implemented as in the case where O and W are input by generating camera pose information through a neural network that estimates W using R and C, or by using C to correct O to the viewpoint of the analysis area image sequence and then configuring an optical flow prediction device for the corrected O to detect the time index.

[0090] In cases where W is input but O is not, optical flow can be estimated between two adjacent frames of R, and after correcting the estimated optical flow using C, an implementation can be utilized for cases where both optical flow information O and camera pose information W are input, or a change detection neural network can be constructed that uses the corrected analysis target area frame as input after correcting R with W to detect the time index.

[0091] In addition, if W and O are not input, W can be estimated from R and C and O can be generated to utilize the implementation for cases where both optical flow information O and camera attitude information W are input, or a time index can be detected using a token compressor that selects key frames within a interval with R as input.

[0092] In the implementation of detecting changes in object movement within a specific time interval based on whether optical flow information sequence O and camera pose information sequence W are input, the implementation when specific information is input includes all implementations when such information is not input.

[0093] The generation of a set of time indices through the learning of a neural network can be implemented in various ways. For example, a video token compression neural network that directly detects selected time indices can be implemented by inputting an analysis area image sequence R input into a sampler (130). R t E P t Selective time index detection can also be implemented through a video token compression neural network that combines the position encoding results of R t E P t It is also possible to implement a video token compression neural network that combines. And, R t and D t or R t and D t , and O t It is also possible to implement a video token compression neural network that combines all of them.

[0094] Figure 5 is a drawing referenced in the description of the location information generation unit illustrated in Figure 1.

[0095] Referring to FIG. 5, the position information generation unit (140) generates position feature information so that the vision token finally generated by the vision input device (100) according to the present invention can include spatiotemporal position information.

[0096] The input and output signals of the location information generation unit (140) can be expressed as follows in mathematical formulas.

[0097]

[0098]

[0099] Here,

[0100] In the above equation, the output signal PF of the position information generating unit (140) at a specific time index t t It can be implemented in various ways.

[0101] For example, spatiotemporal location information P entered for each image token information. t It can also be generated and inserted as a text token. For each token per image, it can also be implemented by extending the language model's trigonometric function-based position encoding to four dimensions to generate a position vector. In addition, P t It can also be implemented by configuring a neural network to learn the generation of position vectors for each vision token based on position information for each token.

[0102] In this case, an example implemented with a text token is “image space location: Image time location: t It can be like ”, and the position feature information generated as a position vector is expressed as an N-dimensional real value.

[0103] Figure 6 is a drawing referenced in the description of the image encoding unit illustrated in Figure 1.

[0104] Referring to FIG. 6, the image encoding unit (150) analyzes the sequence R of the target analysis area selected by the sampler unit (130). s It performs the role of generating feature vectors that describe image features for each image frame.

[0105] The input and output of the image encoding unit (150) can be expressed as follows in a formula.

[0106]

[0107]

[0108] Here,

[0109] In the above equation, the output feature vector PF of the image encoding unit (150) at a specific time index t t It can be implemented by applying any one of the general vision processors, or it can be constructed by concatenating feature vectors generated by applying various implementations.

[0110] Figure 7 is a drawing referenced in the description of the tokenization unit illustrated in Figure 1.

[0111] Referring to FIG. 7, the tokenization unit (160) [receives] each image feature vector IF output from the image encoding unit (150). t Projecting and the location feature information PF generated by the location information generation unit (140). t Generates a vision token that can be input into a vision language model by embedding it.

[0112] Here, projection refers to the process of converting a single data representation into a representation space that a vision language model can understand, and embedding refers to the process of representing data as vectors so that the vision language model can understand it.

[0113] The projection of the tokenization unit (160) can be implemented in various ways. For example, it can be implemented by learning a simple linear neural network layer, or by learning a neural network composed of 2 to 3 layers. In addition, it can be implemented by learning a convolution-based video token compression neural network.

[0114] The position feature information embedding of the tokenization unit (160) can be implemented differently depending on the output format of the position information generation unit (140). That is, if the output format of the position information generation unit (140) is a text token, it is implemented to connect a text token after each projected image token, and if the output format of the position information generation unit (140) is a position vector, a vision token can be generated by connecting a position vector to each projected image token.

[0115] FIG. 8 is a flowchart provided to describe a vision input method according to an embodiment of the present invention.

[0116] Referring to FIG. 8, when a video sequence is input (S300), the analysis area detection unit (110) generates an analysis area image for analysis and image-related information related to the analysis area image in each frame of the input video sequence (S305). The image-related information includes at least one of the center coordinates of the analysis area image and optical flow information.

[0117] The additional information generation unit (120) generates additional information that can configure the spatial location of the analysis area image (S310).

[0118] The sampler unit (130) selects an analysis area image to be tokenized using image-related information and additional information from an analysis area image sequence, which is a set of analysis area images arranged continuously over time, and generates spatiotemporal location information of the selected analysis area image (S315).

[0119] The location information generation unit (140) generates location feature information that describes the location features for each of the spatiotemporal location information of the selected analysis area image (S320).

[0120] The image encoding unit (150) generates a feature vector describing image features for each analysis area image selected from the sampler unit (130) (S325).

[0121] The tokenization unit (160) projects a feature vector and embeds position feature information to generate a vision token (S330). The generated vision token is input into a vision-language model and used (S335).

[0122] Through this process, vision tokens generated by converting video sequences can be used as input to a vision language model.

[0123] Meanwhile, the vision input device and method according to the present invention are not limited to the configurations of the embodiments described above; rather, all or part of each embodiment may be selectively combined to allow for various modifications to the embodiments.

[0124] Furthermore, the present invention can be implemented as a computer program on a programmable computer. Such a computer may include a processor, a storage device, an input device, and an output device. To implement the contents described in the present invention, program code may be input via a mouse or keyboard input device. Such programs may be implemented in a high-level language or an object-oriented language. They may also be implemented as a computer system implemented in assembly or machine code.

[0125] The present invention may also be implemented as processor-readable code on a processor-readable recording medium. A processor-readable recording medium includes all types of recording devices in which data that can be read by a processor is stored. Examples of processor-readable recording media include ROM, RAM, CD-ROM, optical data storage devices, etc. Additionally, the processor-readable recording medium may be distributed across networked computer systems, allowing processor-readable code to be stored and executed in a distributed manner.

[0126] Furthermore, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. Various modifications are possible by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention. Explanation of the symbols

[0127] 110: Analysis area detection unit 120: Additional information generation unit 130 : Sampler Section 140 : Location Information Generation Section 150: Image encoding unit 160: Tokenization unit

Claims

Claim 1 An analysis area detection unit that generates image-related information for each frame of an input video sequence, including an analysis area image which is an area containing an object to be analyzed, and at least one of the center coordinates of the analysis area image and optical flow information; an additional information generation unit that generates additional information capable of configuring the spatial location of the analysis area image, wherein the additional information includes depth map information of each pixel of the analysis area image and may further include external additional information which is at least one of camera pose information and GPS information of the camera; a sampler unit that selects an analysis area image to be tokenized using the image-related information and the additional information in the analysis area image sequence, generates spatiotemporal location information of the selected analysis area image, wherein the spatiotemporal location information includes the center pixel coordinates of the selected analysis area image and time information of the frame to which the selected analysis area image belongs; an image encoding unit that generates a feature vector describing image features for each of the selected analysis area images; and a location information generation unit that generates location feature information describing location features for each of the spatiotemporal location information. A vision input device comprising a tokenization unit that projects the feature vector and embeds the position feature information to generate a vision token. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 A vision input device according to claim 1, characterized in that the position feature information is generated as a text token or generated as a position vector by extending trigonometric function-based position encoding. Claim 6 A vision input device according to claim 5, characterized in that time information of each image frame within the selected analysis area image sequence is added to the position feature information. Claim 7 A vision input device according to claim 5, wherein the vision token is configured by connecting the text token to each projected image token when the output format of the position feature information is a text token, and configured by connecting the position vector to each projected image token when the output format of the position feature information is a position vector. Claim 8 A vision input device according to claim 1, wherein the additional information generating unit receives and uses the external additional information from at least one of an IMS sensor and a GPS sensor installed on a mobile device or a robot. Claim 9 A step of generating image-related information for each frame of a video sequence input from an analysis area detection unit, wherein the analysis area image is an area containing an object to be analyzed, and at least one of the center coordinates of the analysis area image and optical flow information; a step of generating additional information capable of configuring the spatial location of the analysis area image in an additional information generation unit, wherein the additional information may include depth map information of each pixel of the analysis area image and further include external additional information which is at least one of camera pose information and GPS information of the camera; a step of selecting an analysis area image to be tokenized in the analysis area image sequence using the image-related information and the additional information in a sampler unit, and generating spatiotemporal location information of the selected analysis area image, wherein the spatiotemporal location information includes the center pixel coordinates of the selected analysis area image and time information of the frame to which the selected analysis area image belongs; a step of generating location feature information describing location features for each of the spatiotemporal location information in a location information generation unit; and a step of generating a feature vector describing image features for each of the selected analysis area images in an image encoding unit. A vision input method comprising the step of projecting the feature vector in the tokenization section and generating a vision token by embedding the position feature information. Claim 10 A vision input method according to claim 9, further comprising the step of inputting the vision token generated by the tokenization unit into a vision language model for use.

Images