Image data processing method, electronic device and system using the method

TWI946233BActive Publication Date: 2026-11-11IND TECH RES INST
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
TW114150005
Authority / Receiving Office
TW · TW
Patent Type
Patents
Current Assignee / Owner
Filing Date
2025-12-18
Publication Date
2026-11-11
Estimated Expiration
2045-12-17

Smart Images

  • Figure TWG2TB001920985_001
    Figure TWG2TB001920985_001
  • Figure TWG2TB001920985_002
    Figure TWG2TB001920985_002
  • Figure TWG2TB001920985_003
    Figure TWG2TB001920985_003
Patent Text Reader

Abstract

This disclosure provides a video data processing method, electronic device, and system. The method includes acquiring an image sequence and converting each image frame in the image sequence into a set of corresponding image tokens. Next, a semantic analysis step is performed to divide the set of image tokens into a key token subset and a background token subset based on the content of the image sequence. After generating a set of query tokens, a set of key tokens, and a set of value tokens, an asymmetric dimensionality reduction process is performed on the set of query tokens to compress multiple background query tokens belonging to the background token subset, thereby generating a set of filtered query tokens. Finally, an attention calculation procedure is performed using the set of filtered query tokens, key tokens, and value tokens, wherein the number of key tokens and value tokens remains the same as the original number of tokens.
Need to check novelty before this filing date? Find Prior Art

Claims

1. An image data processing method, applied in an electronic device including a transformer architecture, the method comprising: A sequence of images is obtained, comprising multiple image frames, and each of the multiple image frames is converted into a set of corresponding image tokens; a semantic analysis step is performed to classify the set of image tokens corresponding to each of the multiple image frames into a key token subset and a background token subset based on the content of the sequence of images; based on the set of image tokens, when an attention calculation procedure of the converter architecture is started, a set of query tokens, a set of key tokens, and a set of value tokens are generated; an asymmetric dimensionality reduction process is performed on the set of query tokens to obtain a set of filtered query tokens, wherein the asymmetric dimensionality reduction process includes: maintaining multiple key query tokens belonging to the key token subset to form a first set of query tokens; A compression operation is performed on multiple background query scepters belonging to the subset of background scepters to form a second query scepter set; and the first query scepter set and the second query scepter set are integrated to generate the set of filtered query scepters; and the attention calculation procedure is continued using the set of filtered query scepters, the set of key scepters, and the set of value scepters, wherein the set of key scepters and the set of value scepters each maintain their original scepter count, and the original scepter count is greater than the scepter count of the set of filtered query scepters.

2. The method of claim 1, wherein the semantic analysis step includes at least one selected from the group consisting of skeleton detection, pose estimation, object detection, object segmentation and edge detection.

3. The method as described in claim 2, wherein the subset of key scepters corresponds to at least one of human skeleton key points, facial features, and specific object regions.

4. The method as described in request item 1, wherein the compression operation is a pooling operation.

5. The method as described in claim 4, wherein the degree of compression of the pooling operation is preset according to an application context or a deployment environment.

6. The method as described in claim 4, wherein the background scepter subset is divided into multiple background sub-regions with different semantic importance, and the pooling operation applies different degrees of compression to each of the multiple background sub-regions.

7. The method as described in claim 1, wherein the method further comprises: Determine the maximum number of key scepters among the plurality of image frames; and based on the maximum number of key scepters, uniformly set the number of scepters for the subset of key scepters for each of the plurality of image frames.

8. The method of claim 1, wherein the converter architecture includes multiple stacked attention layers, the asymmetric dimensionality reduction process and the attention computation program are performed in the first attention layer of the multiple attention layers, wherein the set of filtered query wands, the set of key wands and the set of value wands are used, after being computed by the attention computation program, to generate an output wand set for input to the second attention layer of the multiple attention layers.

9. The method as described in claim 1, wherein the group of image tokens includes a category token (CLS Token) that is excluded from the compression operation and included in the group of filtered query tokens.

10. The method as described in claim 1, further comprising: The output of the attention calculation program is integrated with a sequence of text scepters; The integrated scepter sequence is then input into a language model to generate a text output corresponding to the image sequence.

11. An electronic device for processing video data, the electronic device comprising: A storage device that stores multiple code modules; and a processor coupled to the storage device, wherein the processor executes the plurality of code modules configured to: acquire an image sequence comprising a plurality of image frames and convert each of the plurality of image frames into a set of corresponding image tokens; perform a semantic analysis step to classify the set of image tokens corresponding to each of the plurality of image frames into a key token subset and a background token subset based on the content of the image sequence; generate a set of query tokens, a set of key tokens, and a set of value tokens based on the set of image tokens when an attention computation procedure of a converter architecture is initiated; perform an asymmetric dimensionality reduction on the set of query tokens to obtain a set of filtered query tokens, wherein the asymmetric dimensionality reduction includes: maintaining a plurality of key query tokens belonging to the key token subset to form a first query token set; A compression operation is performed on multiple background query scepters belonging to the subset of background scepters to form a second query scepter set; and the first query scepter set and the second query scepter set are integrated to generate the set of filtered query scepters; and the attention calculation procedure is continued using the set of filtered query scepters, the set of key scepters, and the set of value scepters, wherein the set of key scepters and the set of value scepters each maintain their original scepter count, and the original scepter count is greater than the scepter count of the set of filtered query scepters.

12. A system for processing video data, the system comprising: An image capturing device for capturing an image sequence, the image sequence comprising multiple image frames; The system also includes a server communicatively connected to the image capturing device, the server comprising: a storage device storing a plurality of instructions; and a processor coupled to the storage device, wherein the processor executes the plurality of instructions to perform the following operations: receiving the image sequence from the image capturing device and converting each of the plurality of image frames into a set of corresponding image tokens; performing a semantic analysis step to classify the set of image tokens corresponding to each of the plurality of image frames into a key token subset and a background token subset based on the content of the image sequence; generating a set of query tokens, a set of key tokens, and a set of value tokens based on the set of image tokens when an attention calculation procedure of a converter architecture is initiated; and performing an asymmetric dimensionality reduction process on the set of query tokens to obtain a set of filtered query tokens, wherein the asymmetric dimensionality reduction process includes: maintaining a plurality of key query tokens belonging to the key token subset to form a first query token set; A compression operation is performed on multiple background query scepters belonging to the subset of background scepters to form a second query scepter set; and the first query scepter set and the second query scepter set are integrated to generate the set of filtered query scepters; and the attention calculation procedure is continued using the set of filtered query scepters, the set of key scepters, and the set of value scepters, wherein the set of key scepters and the set of value scepters each maintain their original scepter count, and the original scepter count is greater than the scepter count of the set of filtered query scepters.

Citation Information

Patent Citations

  • Personnel situation awareness and safety monitoring method for hoisting machinery

    CN115690835A

  • Transform multi-modal image segmentation method based on semantic constraint

    CN116433898A

  • Mongolian multi-modal sentiment analysis method based on pre-training model and Transform

    CN118503774A

  • An attention-guided adversarial patch generation method for secure detection in visual tracking

    CN120164184B

  • Window service behavior specification detection method and system for video space-time attention

    CN120340130A