Two-dimensional hybrid position encoding system, method, medium, terminal and program product

Abstract

The application provides a two-dimensional hybrid position coding system, method, medium, terminal and program product. The system comprises: a data acquisition module that acquires an image block; an absolute coding module that adds row coding and column coding to the row position and column position of the image block, respectively, and fuses the added row coding and column coding to obtain an absolute position vector; a rotating relative coding module that applies two-dimensional rotating position mapping to a query vector and a key vector to obtain a rotating query vector and a rotating key vector; a local relative coding module that adds sparse relative bias to an attention score based on a sparse relative bias matrix to obtain a corrected attention score; and a fusion gating module that calculates dynamic weights and fuses based on context information, the absolute position vector, the rotating query vector and the rotating key vector, and the sparse relative bias matrix. The application can retain and strengthen the two-dimensional grid structure and the relative geometric relationship, and has good migration for resolution, image block size and window size.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a two-dimensional hybrid position coding system, method, medium, terminal and program product. Background Technology

[0002] Existing Vision Transformer (ViT) models typically divide the input image into fixed-size patches, then linearly map each patch into a vector and concatenate them into a one-dimensional sequence, and superimpose one-dimensional positional encodings, such as sinusoidal encoding or learnable absolute position vectors; finally, the sequence is input into a standard encoder for feature learning and classification.

[0003] Although the ViT model performs well in many image recognition tasks, the above processing method ignores the row and column relationships and relative position information of the patch on the two-dimensional grid, which leads to the following problems in image processing tasks:

[0004] (1) Long-range dependencies across rows and columns are difficult to model accurately: In two-dimensional images, the spatial relationship between pixels or image blocks has a clear row and column orientation. However, one-dimensional serialization makes it difficult for the model to effectively distinguish long-range dependencies in the horizontal and vertical directions, thus limiting the model's accurate understanding of the global context of the image.

[0005] (2) Insufficient robustness to geometric changes such as scale, translation, and rotation: Existing one-dimensional positional encoding is usually bound to absolute position. When the image undergoes geometric changes such as scale, translation, and rotation, the positional relationship of the encoding may fail, resulting in insufficient robustness of the model.

[0006] (3) Limited spatial structure representation in downstream tasks (detection, segmentation, key points, etc.): Since one-dimensional location encoding is difficult to provide two-dimensional structural guidance, the performance of the model is limited when reconstructing or locating the spatial layout;

[0007] (4) The generalization ability decreases when the image resolution is transferred to arbitrary resolution or the patch size is changed: When the image resolution or the patch size needs to be changed, the current one-dimensional absolute position encoding needs to be interpolated or retrained, which not only increases the computational cost, but also causes the position information to be distorted, thus affecting the use of the model in image processing tasks.

[0008] Therefore, it is necessary to provide a two-dimensional hybrid location coding system, method, medium, terminal, and program product to solve the above-mentioned problems existing in the prior art. Summary of the Invention

[0009] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a two-dimensional hybrid position coding system, method, medium, terminal and program product to solve the technical problem that the prior art ignores the row and column relationship and relative position information of image blocks on a two-dimensional grid.

[0010] To achieve the above and other related objectives, a first aspect of this application provides a two-dimensional hybrid position coding system, comprising:

[0011] The data acquisition module is used to acquire several image blocks into which the image to be processed is divided;

[0012] The absolute encoding module is used to decompose the two-dimensional position of an image patch into row position and column position, add row code and column code to the decomposed row position and column position respectively, and fuse the added row code and column code to obtain the absolute position vector;

[0013] The rotational relative encoding module is used to apply a two-dimensional rotational position mapping to the query vector and key vector calculated based on the feature vector transformed from the absolute position vector and the image patch, so as to obtain the rotational query vector and the rotational key vector.

[0014] The local relative coding module is used to construct a sparse relative bias matrix based on a dynamic sparsity mechanism, and add a sparse relative bias to the attention score calculated based on the query vector and key vector based on the sparse relative bias matrix to obtain the corrected attention score.

[0015] The fusion gating module is used to calculate dynamic weights based on context information for the absolute position vector generated by the absolute encoding module, the rotation query vector and rotation key vector generated by the rotational relative encoding module, and the sparse relative bias matrix generated by the local relative encoding module, and then executes them after fusion.

[0016] In some embodiments of the first aspect of this application, a multi-scale hierarchical sharing module is also included, which is used to set the parameter representation in the preset basic relative position offset table of each layer based on the cross-hierarchical sharing mechanism.

[0017] In some embodiments of the first aspect of this application, the local relative encoding module includes: a receiving unit for receiving an input attention score calculated based on a query vector and a key vector; a bias acquisition unit for acquiring a bias item corresponding to the two-dimensional relative position from a preset two-dimensional relative displacement bias table; and a sparse bias addition unit for adding bias items to the attention scores of the local neighborhood and the axial key direction based on a sparse masking mechanism to obtain a corrected attention score.

[0018] In some embodiments of the first aspect of this application, the axial key direction includes a horizontal direction, a vertical direction, and a diagonal direction.

[0019] In some embodiments of the first aspect of this application, the local neighborhood is a set of all neighboring image blocks centered on the currently queried image block and whose spatial distance from it on the image block grid is less than or equal to a preset threshold.

[0020] In some embodiments of the first aspect of this application, the specific way to fuse the added row codes and column codes is by addition and fusion or by splicing followed by linear transformation and fusion.

[0021] To achieve the above and other related objectives, a second aspect of this application provides a two-dimensional hybrid positional encoding method, comprising: acquiring several image blocks into which an image to be processed is divided; decomposing the two-dimensional position of the image blocks into row positions and column positions, adding row codes and column codes to the decomposed row positions and column positions respectively, and fusing the added row codes and column codes to obtain an absolute position vector; applying a two-dimensional rotational position mapping to a query vector and a key vector calculated based on the absolute position vector and the feature vector transformed from the image blocks to obtain a rotated query vector and a rotated key vector; constructing a sparse relative bias matrix based on a dynamic sparsity mechanism, and adding a sparse relative bias to the attention score calculated based on the query vector and the key vector based on the sparse relative bias matrix to obtain a corrected attention score; calculating dynamic weights based on context information for the absolute position vector generated by the absolute encoding module, the rotated query vector and the rotated key vector generated by the rotational relative encoding module, and the sparse relative bias matrix generated by the local relative encoding module, and performing the fusion before execution.

[0022] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method.

[0023] To achieve the above and other related objectives, a fourth aspect of this application provides a computer program product comprising computer program code that, when executed on a computer, causes the computer to implement the method.

[0024] To achieve the above and other related objectives, a fifth aspect of this application provides an electronic terminal, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the method.

[0025] As described above, the two-dimensional hybrid location coding system, method, medium, terminal, and program products of this application have the following beneficial effects:

[0026] The absolute encoding module decomposes the two-dimensional position of an image patch into row and column positions, adds row and column codes to each decomposed row and column position respectively, and then fuses the added row and column codes to obtain the absolute position vector of the image patch, setting an absolute coordinate reference. Meanwhile, the rotational relative encoding module applies a two-dimensional rotational position mapping to the query vector and key vector, resulting in rotated query vectors and rotated key vectors after rotation transformation, naturally possessing translational isomorphism and long-range relative perception capabilities. Simultaneously, the local relative encoding module adds a sparse relative bias to the attention score based on a sparse relative bias matrix, resulting in a corrected attention score, thus strengthening local key structures. Finally, dynamic weights are calculated based on context information for the absolute position vector generated by the absolute encoding module, the rotated query vector and rotated key vector generated by the rotational relative encoding module, and the sparse relative bias matrix generated by the local relative encoding module. These weights are then fused and position encoding is performed, preserving and strengthening the two-dimensional grid structure and relative geometric relationships, exhibiting good transferability across resolution, image patch size, and window size. Attached Figure Description

[0027] Figure 1 The diagram shown is a block diagram of a two-dimensional hybrid position coding system according to an embodiment of this application.

[0028] Figure 2 The diagram shown illustrates the working principle of a two-dimensional hybrid position coding system in one embodiment of this application.

[0029] Figure 3 The diagram shown is a flowchart of a two-dimensional hybrid position encoding method in one embodiment of this application.

[0030] Figure 4 The diagram shown is a structural schematic of an electronic terminal according to an embodiment of this application. Detailed Implementation

[0031] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0032] It should be noted that, in the embodiments of this application, the words "exemplary" or "for example" indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0033] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0034] Before providing a further detailed description of the present invention, the nouns and terms used in the embodiments of the present invention are explained, and the nouns and terms used in the embodiments of the present invention are subject to the following interpretations:

[0035] <1> The Vision Transformer (ViT) model is a deep learning model based on the Transformer architecture, designed to apply the Transformer to computer vision tasks. It divides an image into multiple fixed-size image patches and treats these patches as sequential data, using a self-attention mechanism to capture global and local information in the image.

[0036] To facilitate understanding of the embodiments of this application, in conjunction with Figure 1 and Figure 2 Detailed explanation. Figure 1 A block diagram of a two-dimensional hybrid position coding system according to an embodiment of this application is shown. Figure 2 A schematic diagram illustrating the working principle of a two-dimensional hybrid position coding system according to an embodiment of this application is shown. The two-dimensional hybrid position coding system 100 in this embodiment includes: a data acquisition module 101, an absolute coding module 102, a rotational relative coding module 103, a local relative coding module 104, and a fusion gating module 105.

[0037] It should be noted that the two-dimensional hybrid position coding system of this application is applicable to the VisionTransformer (ViT) model, solving the technical problem of prior art neglecting the row and column relationships and relative position information of image patches on a two-dimensional grid. Without significantly increasing computational overhead, it accurately models the two-dimensional spatial relationships of image patches and adapts to universal position coding at different resolutions or scales. The following section will provide a detailed description using the two-dimensional hybrid position coding system provided in this application for the VisionTransformer model.

[0038] The data acquisition module 101 is used to acquire several image blocks into which the image to be processed is divided.

[0039] Specifically, the input is a two-dimensional image to be processed, which is divided into several image blocks. Then, each image block is mapped to a feature vector based on linear projection. A flattening operation is then performed to form a feature matrix. However, the visual transformer model does not know the position information of the image blocks, so position encoding is required to clarify the position information.

[0040] The absolute encoding module 102 is used to decompose the two-dimensional position of the image block into row position and column position, add row code and column code to the decomposed row position and column position respectively, and fuse the added row code and column code to obtain the absolute position vector.

[0041] In some embodiments of this application, the specific method for fusing the added row codes and column codes is to sum and fuse or to fuse by concatenation followed by linear transformation.

[0042] Specifically, the two-dimensional coordinates of the image patch are first decomposed into row and column position indices, thereby learning independent decomposable vectors for each row and column position. For example, the row code is obtained from the row encoding table based on the row position index, and the column code is obtained from the column encoding table based on the column position index. Then, the obtained row and column codes are fused using a learnable fusion method based on addition or linear transformation after concatenation, thus obtaining the absolute position code. This yields the two-dimensional absolute position vector of the image patch, endowing it with absolute position information, while maintaining linear parameter growth and supporting interpolation at arbitrary resolutions.

[0043] The rotational relative encoding module 103 is used to apply a two-dimensional rotational position mapping to the query vector and key vector calculated based on the feature vector transformed from the absolute position vector and the image block, so as to obtain the rotational query vector and the rotational key vector.

[0044] Specifically, in the self-attention calculation stage, the query, key, and value projections are calculated based on the absolute position vector and feature vector output from the previous layer. Then, a two-dimensional rotational position mapping is applied to the calculated query and key vectors to obtain the rotated query and key vectors. That is, the rotational relative encoding module modifies the feature representations of the query and key vectors so that their dot product contains relative position information, while making the attention score explicitly dependent on the relative displacement in the row and column directions, naturally possessing translational equivariance and long-range relative perception capabilities.

[0045] The local relative coding module 104 is used to construct a sparse relative bias matrix based on a dynamic sparsity mechanism, and add a sparse relative bias to the attention score calculated based on the query vector and the key vector based on the sparse relative bias matrix to obtain the corrected attention score.

[0046] In some embodiments of this application, the local relative encoding module includes: a receiving unit for receiving the input attention score calculated based on the query vector and the key vector; a bias acquisition unit for acquiring the bias item corresponding to the two-dimensional relative position from a preset two-dimensional relative displacement bias table; and a sparse bias addition unit for adding bias items to the attention scores of the local neighborhood and the axial key direction based on a sparse masking mechanism to obtain the corrected attention score.

[0047] Specifically, the encoding stage of the local relative encoding module is located in the correction stage after the attention score calculation. The previous layer calculates and outputs the attention score based on the query vector and key vector. First, the local relative encoding module receives the input attention score; then, it calculates the relative position difference for each query-key pair, constructs a relative position index table, and retrieves the corresponding bias item from a preset two-dimensional relative displacement bias table based on the two-dimensional relative position; then, it organizes the bias items retrieved from the two-dimensional relative displacement bias table into a matrix with the same shape as the attention score, and sets a mask based on a dynamic sparsity mechanism so that the bias value of non-important positions is zero, thus obtaining a sparse relative bias matrix; then, the sparse relative bias matrix is ​​added to the received attention score to obtain the corrected attention score. The corrected attention score is then normalized to obtain the attention weight.

[0048] By introducing the learned two-dimensional relative position index table (containing relative rows) and relative columns This approach employs a sparse masking mechanism, applying bias terms only to local neighborhoods and key cross-axis directions (horizontal, vertical, and diagonal), thereby reducing computational cost while preserving key spatial topology. Specifically, while adding bias terms based on two-dimensional relative positions to each pair of image patches in the self-attention mechanism, a sparse strategy is used to reduce computational overhead. The purpose of the sparse strategy is to calculate and add biases only to local neighborhoods and key axial directions, focusing only on local key structures.

[0049] In some embodiments of this application, the axial key direction includes the horizontal direction, the vertical direction, and the diagonal direction.

[0050] Specifically, the horizontal direction is represented by image patches in the same row, i.e. , The vertical direction is represented by image blocks in the same column, i.e. , The diagonal direction is along the main diagonal and the anti-diagonal, that is... .

[0051] In some embodiments of this application, the local neighborhood is a set of all neighboring image blocks centered on the currently queried image block and whose spatial distance from it on the image block grid is less than or equal to a preset threshold.

[0052] Specifically, the set of all neighboring image blocks on the image block grid that are less than or equal to a preset threshold, centered on the currently queried image block, such as a 3×3 or 5×5 neighborhood.

[0053] Rotational relative coding modules possess natural translational equivariance (relative position invariance) and require no learning parameters, exhibiting strong generalization ability and making them suitable for modeling long-range dependencies. Local relative coding modules are learnable, capable of capturing task-specific spatial patterns, efficiently modeling local neighborhoods through sparse masks, and learning different biases for different directions (horizontal / vertical / diagonal). Rotational and local relative coding modules complement each other, enabling both global relative relationship modeling and preservation of local key structures.

[0054] The fusion gating module 105 is used to calculate dynamic weights based on context information for the absolute position vector generated by the absolute encoding module, the rotation query vector and rotation key vector generated by the rotational relative encoding module, and the sparse relative bias matrix generated by the local relative encoding module, and then executes them after fusion.

[0055] Specifically, the fusion gating module is executed after the absolute encoding module, the rotational relative encoding module, and the local relative encoding module have calculated all positional codes. It fuses the absolute position vector generated by the absolute encoding module, the rotation query vector and rotation key vector generated by the rotational relative encoding module, and the sparse relative bias matrix generated by the local relative encoding module through gating or attention. This enables the task to adaptively balance absolute and relative, local and global information, thereby significantly improving the quality of spatial relationship modeling and maintaining low overhead in tasks such as classification, detection, segmentation, pose estimation, and video understanding.

[0056] For example, for classification tasks, which tend to use global absolute encoding, the absolute position vector is given higher weight; for detection or segmentation tasks, which tend to use local relative encoding, the sparse relative bias matrix is ​​given higher weight; and for pose estimation tasks, which tend to use rotation-equal encoding, the rotation query vector and rotation key vector are given higher weight. Thus, the weights of the three position encodings are dynamically adjusted according to the task type based on the context information, which has the characteristics of task adaptation.

[0057] By incorporating the gating module design, a single model can adapt to multiple tasks, providing flexibility. Meanwhile, the gating weights reflect the task's preference for location information, offering interpretability. Furthermore, inactive branches can be dynamically pruned during inference, improving efficiency. Additionally, multi-path fusion provides redundancy, enhancing generalization ability and robustness.

[0058] In some embodiments of this application, a multi-scale hierarchical sharing module is also included, which is used to set the parameter representation in the preset basic relative position offset table of each layer based on the cross-level sharing mechanism.

[0059] By designing multi-scale hierarchical shared modules, the number of parameters can be reduced by 50%-75% (multi-layer sharing of the same set of tables), and the cross-resolution generalization ability can be improved (interpolation mechanism), making model training more stable (reducing the risk of overfitting) and supporting dynamic resolution inference.

[0060] Specifically, the multi-scale hierarchical sharing module is executed during the parameter management phase of model initialization and inference. In the pyramid structure or windowed structure, a two-dimensional basic relative position bias table is shared or interpolated to adapt to different resolutions and window sizes, thereby reducing parameters and improving cross-scale generalization. Among them, the pyramid structure imitates the design of CNNs (Convolutional Neural Networks) such as ResNet, progressively reducing spatial resolution and increasing the number of channels to form a multi-scale feature pyramid; the windowed structure divides the image into non-overlapping local windows, calculates self-attention within the window, and avoids the secondary complexity of global attention.

[0061] Cross-layer sharing mechanisms include full sharing, group sharing, and progressive fine-tuning. Full sharing uses the same basic relative position bias table with interpolation operations across all layers; group sharing allows adjacent layers to share a set of parameters; and progressive fine-tuning involves shallow layer sharing and deep layer independent learning.

[0062] The two-dimensional hybrid positional encoding system provided in this application is highly compatible with the existing ViT model architecture. It only requires replacing the positional encoding and attention scoring modules, and obtains better spatial inductive bias without additional convolution, making the model training stable and inference efficient. It is compatible with methods such as distillation, fine-tuning, or LoRA.

[0063] Figure 3 This is a flowchart illustrating the two-dimensional hybrid position encoding method provided in an embodiment of this application. Figure 3 As shown in the diagram, the flowchart of the two-dimensional hybrid position coding method includes the following steps.

[0064] Step S31: Obtain several image blocks into which the image to be processed is divided;

[0065] Step S32: Decompose the two-dimensional position of the image patch into row position and column position, add row code and column code to the decomposed row position and column position respectively, and fuse the added row code and column code to obtain the absolute position vector;

[0066] Step S33: Apply a two-dimensional rotation position mapping to the query vector and key vector calculated based on the absolute position vector and the feature vector transformed from the image patch, so as to obtain the rotated query vector and the rotated key vector;

[0067] Step S34: Construct a sparse relative bias matrix based on the dynamic sparsity mechanism, and add a sparse relative bias to the attention score calculated based on the query vector and key vector based on the sparse relative bias matrix to obtain the corrected attention score.

[0068] Step S35: Calculate dynamic weights for the absolute position vector generated by the absolute encoding module, the rotation query vector and rotation key vector generated by the rotational relative encoding module, and the sparse relative bias matrix generated by the local relative encoding module based on context information, and then perform the fusion.

[0069] It should be understood that the two-dimensional hybrid position coding method provided in this application embodiment can realize the functions of the above-mentioned two-dimensional hybrid position coding system, and for the sake of brevity, it will not be described in detail here.

[0070] It should also be understood that the module division in the embodiments of this application is illustrative and only represents a logical functional division; in actual implementation, there may be other division methods. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0071] Figure 4 This is a schematic block diagram of the electronic terminal provided in an embodiment of this application. Figure 4 As shown, the electronic terminal 400 includes at least one processor 401, a memory 402, at least one network interface 403, and a user interface 405. The various components in the electronic terminal 400 are coupled together via a bus system 404. It is understood that the bus system 404 is used to implement communication between these components. In addition to a data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 4 The general will label all buses as bus systems.

[0072] The user interface 405 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0073] It is understood that memory 402 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0074] In this embodiment of the invention, the memory 402 is used to store various types of data to support the operation of the electronic terminal 400. Examples of this data include: any executable program for operation on the electronic terminal 400, such as the operating system 4021 and application programs 4022; the operating system 4021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 4022 may contain various applications, such as a media player, browser, etc., for implementing various application services. The methods provided in this embodiment of the invention can be included in the application program 4022.

[0075] The methods disclosed in the above embodiments of the present invention can be applied to processor 401, or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 401 or by instructions in the form of software. The processor 401 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 401 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 401 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in a memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0076] In an exemplary embodiment, the electronic terminal 400 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to execute the aforementioned method.

[0077] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes: computer program code, which, when run on a computer, causes the computer to execute... Figure 3 The method in the illustrated embodiment.

[0078] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium storing program code, which, when executed on a computer, causes the computer to perform... Figure 3 The method in the illustrated embodiment.

[0079] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0080] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0081] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0082] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0083] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0084] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0085] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs, DVDs), or semiconductor media (e.g., solid-state disks, SSDs, etc.).

[0086] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0087] In summary, addressing the technical problem of existing technologies neglecting the row and column relationships and relative position information of image blocks on a two-dimensional grid, this application provides a two-dimensional hybrid position encoding system, method, medium, terminal, and program product. The system decomposes the two-dimensional position of an image block into row and column positions using an absolute encoding module, adds row and column codes to each decomposed row and column position respectively, and then fuses the added row and column codes to obtain the absolute position vector of the image block, establishing an absolute coordinate reference. Furthermore, a rotational relative encoding module applies a two-dimensional rotational position mapping to the query vector and key vector, thereby obtaining a rotated query vector and a rotated key vector after rotation transformation, naturally possessing translational equivalence. This invention enhances both spatial and long-range relative perception capabilities. Simultaneously, the local relative encoding module adds sparse relative biases to the attention score based on a sparse relative bias matrix, resulting in a corrected attention score that strengthens local key structures. Finally, dynamic weights are calculated based on contextual information for the absolute position vector generated by the absolute encoding module, the rotation query vector and rotation key vector generated by the rotational relative encoding module, and the sparse relative bias matrix generated by the local relative encoding module. These weights are then fused and position encoding is performed, preserving and strengthening the two-dimensional mesh structure and relative geometric relationships. This approach exhibits good transferability across resolutions, image patch sizes, and window sizes, while significantly improving spatial relationship modeling quality and maintaining low overhead. Therefore, this application effectively overcomes the shortcomings of existing technologies and possesses high industrial applicability.

[0088] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A two-dimensional hybrid position coding system, characterized in that, include: The data acquisition module is used to acquire several image blocks into which the image to be processed is divided; The absolute encoding module is used to decompose the two-dimensional position of an image patch into row position and column position, add row code and column code to the decomposed row position and column position respectively, and fuse the added row code and column code to obtain the absolute position vector; The rotational relative encoding module is used to apply a two-dimensional rotational position mapping to the query vector and key vector calculated based on the feature vector transformed from the absolute position vector and the image patch, so as to obtain the rotational query vector and the rotational key vector. The local relative coding module is used to construct a sparse relative bias matrix based on a dynamic sparsity mechanism, and add a sparse relative bias to the attention score calculated based on the query vector and key vector based on the sparse relative bias matrix to obtain the corrected attention score. The local relative encoding module includes: a receiving unit for receiving the input attention score calculated based on the query vector and the key vector; a bias acquisition unit for acquiring the bias item corresponding to the two-dimensional relative position from a preset two-dimensional relative displacement bias table; and a sparse bias addition unit for adding bias items to the attention score of the local neighborhood and the axial key direction based on the sparse masking mechanism to obtain the corrected attention score. The fusion gating module is used to calculate dynamic weights based on context information for the absolute position vector generated by the absolute encoding module, the rotation query vector and rotation key vector generated by the rotational relative encoding module, and the sparse relative bias matrix generated by the local relative encoding module, and then executes them after fusion.

2. The two-dimensional hybrid position coding system according to claim 1, characterized in that, It also includes a multi-scale hierarchical sharing module, which is used to set the parameter representation in the basic relative position offset table of each level sharing based on the cross-level sharing mechanism.

3. The two-dimensional hybrid position coding system according to claim 1, characterized in that, The key axial directions include the horizontal direction, the vertical direction, and the diagonal direction.

4. The two-dimensional hybrid position coding system according to claim 1, characterized in that, The local neighborhood is the set of all neighboring image blocks centered on the currently queried image block, whose spatial distance from it on the image block grid is less than or equal to a preset threshold.

5. The two-dimensional hybrid position coding system according to claim 1, characterized in that, The specific method for merging the added row and column codes is either summation and fusion or concatenation followed by linear transformation and fusion.

6. A two-dimensional hybrid position encoding method, characterized in that, include: Obtain the image to be processed into several image blocks; The two-dimensional position of the image patch is decomposed into row position and column position. Row code and column code are added to the decomposed row position and column position respectively. The added row code and column code are then fused to obtain the absolute position vector. A two-dimensional rotation position mapping is applied to the query vector and key vector calculated based on the absolute position vector and the feature vector transformed from the image patch to obtain the rotated query vector and the rotated key vector; A sparse relative bias matrix is ​​constructed based on a dynamic sparsity mechanism. Based on the sparse relative bias matrix, sparse relative biases are added to the attention scores calculated based on the query vector and key vector to obtain the corrected attention scores. Specifically, this includes: receiving the input attention scores calculated based on the query vector and key vector; obtaining the bias terms corresponding to the two-dimensional relative positions from a preset two-dimensional relative displacement bias table; and adding bias terms to the attention scores of the local neighborhood and the axial key directions based on a sparse masking mechanism to obtain the corrected attention scores. Dynamic weights are calculated for the generated absolute position vector, rotation query vector, rotation key vector, and sparse relative bias matrix based on context information, and then fused before execution.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in claim 6.

8. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer, causes the computer to implement the method as described in claim 6.

9. An electronic terminal, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method as described in claim 6.

Images