A visual encoding method and device for native structural axis combinations

The visual encoding method based on native structural axis combination solves the problem of difficult computation for sharing 2D and 3D visual data in existing technologies, realizes independent observation and depth axis definition, avoids information loss and invalid calculation, and improves the processing power and accuracy of the encoder.

CN123071669APending Publication Date: 2026-11-10ZHONGSHAN HOSPITAL FUDAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611294567.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-25
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing visual coding schemes cannot effectively share the main visual computations of 2D images and 3D volume data. Furthermore, 2D inputs introduce computations without corresponding physical meaning after dimensionality upgrades. The observation axis and depth axis lack independent definitions, tokens are prone to losing local information during axis updates, and the number of parameters grows unreasonably when depth is increased.

Method used

A visual encoding method based on native structural axis combinations is adopted. By receiving visual data and recording the number of observations and depth, the same two-dimensional patch projection is applied for round-robin updates. Tensor rearrangement and axial attention flow are used to ensure the independence of plane, depth and observation axes, and to perform precise bypass when an axis is missing, avoiding meaningless calculations.

Benefits of technology

This invention enables the same encoder to process 2D images, volume data, and multi-view images, while preserving local information, reducing invalid computations, improving encoding accuracy and efficiency, and enhancing the scalability of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123071669A_ABST
    Figure CN123071669A_ABST
Patent Text Reader

Abstract

This invention discloses a visual encoding method and device for native structural axis combinations, mainly used for processing two-dimensional images, volume data with continuous slices, and image sets containing multiple observations of the same object. The encoding results obtained by this invention have the following applications: generating comparable, language-aligned visual representations for image-text retrieval, sample retrieval, zero-sample or few-sample classification, and downstream model input; preserving terminal tokens of observation, depth, and local planar structure for downstream network reading; and enabling the same encoder to receive natural two-dimensional images, chest X-ray images, CT volume data, and multi-sequence MRI, reducing the workload of maintaining separate visual encoding towers for each input type.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to a visual encoding method and apparatus. Background Technology

[0002] Existing visual coding schemes typically employ the following technical approach:

[0003] Technical approach 1: Use dedicated encoders for 2D and 3D respectively. A common practice is to configure a 2D visual encoder for 2D images and a 3D convolutional network, 3D Transformer, or volumetric data-specific encoder for CT and MRI scans, then align them at the text space, projection layer, or downstream interface. Each encoder can adjust its structure based on the input, but visual computation is divided into multiple sets of parameters. Each additional input structure often requires additional training procedures and interfaces. The outputs of different encoder towers may also need to be aligned again.

[0004] Technical Route 2: Upgrade the 2D input and send it into a 3D pathway One approach involves copying a 2D image, filling in the depth dimension, or stitching together multiple 2D images into a pseudo-data structure before feeding it into the same 3D path. While this approach aligns the interface dimension, it introduces a depth axis that was not originally present in the 2D input. This depth axis can still trigger convolutions, attention, normalization, and random operations, increasing computations without corresponding physical meaning and potentially altering the 2D pre-trained representation.

[0005] Technical approach three: averaging, stitching, or late-stage aggregation of multiple observation images. Multiview chest radiographs, natural multiview images, and multisequence examinations often involve averaging, splicing, or shallow aggregation after image-by-image encoding. Averaging compresses complementary information between observations, direct splicing is affected by variations in the number of observations, and late aggregation can only read results that have already been independently encoded or pooled.

[0006] Technical Route 4: Increase fixed depth but do not reuse parameters Stacking Transformer layers increases computational depth, and the number of parameters typically grows with each layer, with each layer executing only once. Looping Transformers can repeatedly call the same transformation, but current solutions are mostly geared towards single token sequences. Whether the observation axis, depth axis, and planar axis are independent, and where missing axes should be bypassed, are usually not specified simultaneously. Summary of the Invention

[0007] This invention aims to solve the following technical problems existing in current visual coding schemes: 1) Native 2D images, volume data, and multi-view images should share the main visual computation, and 2D inputs should not rely on pseudo-depth copying; 2) The observation axis and the depth axis should be defined separately; they can exist independently or simultaneously. 3) The token should retain its original structure between updates on each axis to avoid local information loss caused by averaging before interaction; 4) When an axis is missing from the input, the parameters corresponding to that axis and random operations should not be invoked; 5) When increasing the execution depth, the entire module should not be copied according to the number of executions.

[0008] To address the aforementioned technical problems, this invention provides a visual encoding method for native structural axis combinations, characterized by comprising the following steps: Step 1: Receive visual data and record the number of observations. and depth number The color or grayscale channels contained in visual data are located in the input channel dimension; Step 2: Apply the same 2D patch projection to each preserved plane of each observation in the visual data, add position encoding and prefix token, and obtain the initial persistent token state. The dimension of the persistent state is... , For batch size, The length of the token sequence for each reserved plane, including M prefix tokens (M>=1) and A dense token, , For single token embedding feature dimensions; Step 3: Initial persistent token state conduct Cyclic updates, , any number The update cycle further includes the following steps: : Step 301, for the first The persistent token state obtained in the round Perform tensor rearrangement without losing tokens to make the length of the planar token sequence... Depth and observation number These become the attention sequence lengths, and the remaining structural dimensions are folded to the batch position. Step 302: Execute three plane operators, depth operator one, three plane operators, depth operator two, and one observation operator in sequence. The plane, depth, and observation operators operate sequentially on the same persistent token state. The output of the previous operator serves as the input of the next operator. The tensor rearrangement from step 301 is used to achieve the following: Length of planar token sequence As a sequence axis for information interaction in planar operators, planar tokens are mixed within each observation and slice; Depth As a sequence axis for information interaction in depth operators, tokens are used across the depth mixing plane at shared patch indices; Number of observations As a sequence axis for information exchange in the observation operator, information is exchanged across observations at the shared token index; Step 4: Perform pooling on the terminal persistent token state obtained in the previous step to obtain language-aligned representation, or retain the structured token for downstream network to read.

[0009] Preferably, in step 1, the visual data includes a single two-dimensional image, CT volume data, paired or multi-view images. Figure 2 Two-dimensional images and registered multi-sequence MRI, wherein: for the single two-dimensional image, the number of observations Depth ; For the CT body data, the number of observations Depth ; For the aforementioned pairing or multi-view Figure 2 3D image, number of observations Depth ; For the registered multi-sequence MRI, the number of observations Depth .

[0010] Preferably, in step 3, The same set of physical modules consisting of plane operators, depth operators, and observation operators is reused in a round-robin fashion to perform multi-step updates without copying the physical modules according to the number of executions.

[0011] Preferably, in step 3, the length of the shaft is checked to see if it is 1. If the detected shaft length is 1: If depth number If the value is 1, the depth operator does not perform the corresponding parameter calculation and dropout operation, and returns the input before the parameters and random operations of the depth operator. The depth operator becomes an execution-level identity mapping. If the number of observations If the value is 1, the observer operator does not perform the corresponding parameter calculation and dropout operation, and returns the input before the parameter and random operation of the observer operator. The observer operator becomes an execution-level identity mapping. If the corresponding axis length is detected to be greater than 1, the plane operator, depth operator, or observation operator performs an axial attention process on the persistent token state, including: The tensor rearrangement without losing tokens described in step 301; Perform axial attention calculations, or perform multi-layered sensory processing after performing axial attention calculations; Restore the original dimensional arrangement of the persistent token state.

[0012] The present invention also provides an electronic device, comprising: One or more processors; A storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the visual encoding method described above.

[0013] This invention is primarily used to process three types of visual data: two-dimensional images, volumetric data with continuous slices, and image sets containing multiple observations of the same object. The encoding results obtained through this invention have the following applications: 1) Generate comparable language-aligned visual representations for image-text retrieval, sample retrieval, zero-shot or few-shot classification, and input to downstream models; 2) Retain the terminal tokens containing the observation, depth, and local planar structure for downstream network access; 3) The same encoder can receive natural two-dimensional images, chest X-ray images, CT body data and multi-sequence MRI, reducing the work of maintaining visual coding towers separately according to input type.

[0014] In the technical solution disclosed in this invention: 1) The three axes have different physical meanings. The planar axis corresponds to the visual structure within a single observation or slice. The depth axis corresponds to the continuity of data along the slice direction within the same volume. The observation axis corresponds to multiple observations, projections, or sequences under the same object or examination. These three axes are independent of each other: multiview. Figure 2 3D MRI has an observation axis but no depth axis; CT has a depth axis but no multiple observation axes; multi-sequence MRI can have both depth and observation axes simultaneously.

[0015] 2) Structure-dependent computation graph After N and D are written into the input structure, the execution path can be directly determined by the axis length. Two-dimensional single-image bypass depth operator and observation operator with N=D=1; CT only executes the depth operator; multi-view... Figure 2 1D images only perform observation operators; multi-sequence MRI performs both. This computational graph selects operators based on the input structure, without needing to first transform various inputs into the same pseudo-dimensionality.

[0016] 3) Persistent states prevent premature information compression. The observation, depth, and plane token dimensions are preserved between ordered updates, allowing subsequent depth or observation operators to still access local patch locations. If slices or observations are averaged first, subsequent modules can only process information that has been lost in the localized areas. The persistent state thus provides a common data carrier for the continuous interaction of the three axis operators.

[0017] 4) Tensor rearrangement assigns a state to operators on different axes. The three operators share a single tokenized state. During rearrangement, the target axis is moved to the sequence position, and the remaining structural dimensions are folded to the batch position. Thus, attention can be performed along L, D, or N for the same state. After the operator finishes, the original shape is restored, and the result of the previous step is directly passed to the next step, with the overall state dimensions remaining unchanged.

[0018] 5) Orderly alternation and cyclical reuse Each unit executes three planar operators first, then the first depth operator; subsequently, it executes three more planar operators and the second depth operator, with the observation operator only executed at the end. Cross-slice and cross-observation computations are therefore inserted into the planar feature update process, not as a separate post-processing step after pooling. Subsequent loops continue to modify the same persistent state and repeatedly call the original parameters. As the depth increases, the number of independent parameters does not increase proportionally with the number of loops.

[0019] By adopting the above technical solution, the present invention has the following advantages compared with existing visual coding schemes: 1) Because this invention uses explicit and independent N, D, and L axes, the same representation can cover single images, volume data, multi-observation two-dimensional data, and multi-sequence volume data, without confusing observation, depth, and color channels; 2) This invention employs a technique of sharing two-dimensional projection plane by plane, which eliminates the need for pseudo-depth expansion of the two-dimensional input and tokenizes each retained slice using the same plane; 3) This invention has the structural feature of "persistent B×N×D×L×d state", which allows subsequent axis operators to still access local tokens and avoids averaging before axis interaction; 4) In the technical solution disclosed in this invention, tensor rearrangement forms three types of sequences, thereby allowing the same token content to interact along different structural axes without the need to maintain three separate states. 5) This invention uses fixed ordered axis updates, inserting depth and observation interactions into the plane update process, rather than just aggregating them after final pooling; 6) In the technical solution disclosed in this invention, the same module is reused three times in a loop, and multi-step updates are performed without copying the physical module according to the number of executions; 7) This invention employs a precise bypass technique with D=1 or N=1, where missing axes do not perform corresponding parameter calculations or dropout operations, reducing meaningless calculations and maintaining identity mapping; 8) In the comparative experiment, after enabling the Depth operator, the average AUROC of the complete CT and MRI models increased by 0.0734 and 0.0401 respectively compared with the Depth operator being disabled.

[0020] It should be noted that the technical solution disclosed in this invention provides a visual representation or structured token, and does not require the direct acquisition of disease diagnosis conclusions or treatment plans as a necessary step. Probabilities, similarities, or candidate tags in medical scenarios are all used as results of computer information processing and should be used by professionals in conjunction with other information. Attached Figure Description

[0021] Figure 1 The diagram illustrates the overall process of combining native structural axes. In the diagram, the input is projected into a persistent state through a shared two-dimensional projection, then updated and reused cyclically according to a fixed axis order, and finally outputs a representation or structured token.

[0022] Figure 2 The diagram illustrates the axis signatures for four types of inputs. In the diagram, XY is always executed, while Depth and Observation are only executed when the length of the corresponding axis is greater than 1. Figure 3 The diagram illustrates persistent state, where a shared two-dimensional projection is applied plane-by-plane, and the three structural dimensions of observation, depth, and plane token are preserved during the loop. Figure 4 The diagram illustrates an ordered loop unit and three parameter reuses. In the diagram, the output of the previous operator becomes the input of the next operator. Figure 5 The diagram illustrates three tensor rearrangements. In each rearrangement, only one structural dimension is exposed as an attention sequence, while the other dimensions are folded into the batch index. Figure 6 The diagram illustrates the precise bypass for missing axes. In the figure, when the axis length is 1, the operator becomes an execution-level identity mapping. Figure 7 The training process and loss composition are illustrated in the figure. When the joint training reaches the 8,000th update, 6,000 chest X-ray / report refinements are carried out from this checkpoint branch. Figure 8 The diagram illustrates computing devices and software modules. In the diagram, the same technical solutions can be described as devices, electronic devices, storage media, and computer program products. Detailed Implementation

[0023] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading this disclosure, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0024] like Figure 1 As shown, the visual encoding method for native structural axis combinations provided in this embodiment of the invention includes the following sequential steps: Step S1: Receive visual data and determine batch size. Observation number Number of input channels Depth and planar dimensions , ,in, Indicates the plane height. Indicates the width of the plane.

[0025] In one optional implementation of this invention, the visual data is represented as follows: ,in, Indicates the number of images or MRI sequences observed. Indicates the number of slices in each observation, such as Figure 2 As shown, the input visual data falls into the following four categories: 1) Single two-dimensional image: , ; 2) CT scan data: , ; 3) Pairing or multiple views Figure 2 3D image: , ; 4) Registration with multi-sequence MRI: , .

[0026] The three RGB color channels are located in Dimension, not to be broken down into three observations.

[0027] Step S2: Apply the same two-dimensional patch projection to each preserved plane of each observation in the visual data, add location-related information and prefix tokens, and obtain the initial persistent state. .

[0028] In one alternative implementation, the same two-dimensional patch projection is applied independently to each slice in each observation. This two-dimensional patch projection is... The 2D patch does not set the projection step size along the depth direction.

[0029] In one embodiment of the present invention, an optional implementation is to copy the grayscale image into three channels and then project it into the same two-dimensional patch.

[0030] In one embodiment of the present invention, an optional implementation is to add position embedding and two prefix tokens to the persistent state. ,in: The length of the token sequence for each reserved plane, including the two prefix tokens and A dense patch token, ; For each token's embedding feature dimension, in this embodiment of the invention, Each reserved slice (i.e., reserved plane) remains accessible within the loop unit, such as... Figure 3 As shown.

[0031] It should be noted that in the embodiments of the present invention, the input resolution, cropping and slice sampling can be configured according to the implementation scenario. "Retaining the complete depth" in the present invention means that the pre-processed and retained slices will not be subjected to additional depth stepping or pooling, and does not mean that all slices of the original scan are used unconditionally.

[0032] Step S3: Perform tensor rearrangement on the persistent state obtained in the previous step without losing any tokens, so that the length of the planar tokens is... ,depth or observation number These are respectively called the attention sequence lengths.

[0033] Step S4: Execute three plane operators, depth operator one, three plane operators, depth operator two, and one observation operator in sequence.

[0034] Step S5: Repeat the ordered update recorded in steps S3 and S4. Next, among them In this embodiment, to determine the total number of cycles, Choose 3, and reuse the same set of physical module parameters in multiple ordered updates.

[0035] In one embodiment of the present invention, an optional implementation is to denote the six planar operators as... The two depth operators (i.e., depth operator one and depth operator two) are The observation operator is Then the first step described in step S4 The update of the wheel can be expressed as follows:

[0036] The above compound formula is executed from right to left, with the output of the right-hand operator serving as the input of its left-hand adjacent operator, thus conforming to the fixed execution order described in step S4.

[0037] In the formula, Indicates the first The persistent state acquired by the round; This indicates that the functions are compounded and executed from right to left, with the output of the previous operator serving as the input of the next operator. This indicates that three planar operators are executed sequentially. ; This indicates that three planar operators are executed sequentially. ; Used to identify the current update's position within a fixed loop, and by generate; This represents a multilayer perceptron that maps normalized round information to cyclic position condition vectors. This indicates the predetermined total number of cycles, in this embodiment... The value is 3.

[0038] In one embodiment of the invention, an optional implementation is to perform three updates. The three updates call the same set of physical modules; that is, the number of executions increases, but module parameters are not copied according to the number of executions. For example... Figure 4 As shown, the three updates point to the same set of physical parameters. These three updates reuse module parameters without reusing intermediate states. Round-up output As the first Rotation Update Input.

[0039] In one embodiment of the present invention, an optional implementation is a planar operator. Depth Operator and observation operators All inputs are persistent states arranged in the same dimension; each operator receives the output of the previous operator in sequence, and selects the attention sequence through the corresponding tensor rearrangement in step S3 before execution:

[0040]

[0041]

[0042] in, This indicates that the persistent state will be rearranged as Each bar is independent and has a length of The token sequence, where the embedding dimension of each token is... The planar operator runs along each slice within each observation. The axis performs attention and mixes the planar tokens, and the output is still a sequence of the same shape, which is then restored to the dimensional arrangement of the original persistent state; This indicates that the persistent state will be rearranged as Each bar is independent and has a length of The token sequence, where the embedding dimension of each token is... The depth operator runs along the same observation and shared token index. Axis attention is performed to blend depth information across slices, and the output remains a sequence of the same shape, which is then restored to the original persistent dimensional arrangement; This indicates that the persistent state will be rearranged as Each bar is independent and has a length of The token sequence, where the embedding dimension of each token is... ;Observe the operator at the same depth position and shared token index along Axis attention is performed to exchange information across observations, and the output remains a sequence of the same shape, which is then restored to the original persistent dimensional arrangement.

[0043] The three rearrangements described above only change the axis that serves as the attention sequence, without averaging, deleting, or copying the tokens; after each operator is completed, the original persistent state's dimensional arrangement is restored, and the updated result is passed to the next operator.

[0044] In this embodiment of the invention, for registered MRI, the shared token index can preserve approximate anatomical correspondence; for different projections or natural viewpoints, the shared token index provides the communication location encoded in the same plane, but does not require geometric registration. The rearrangement process does not average, delete, or copy tokens, such as... Figure 5 As shown.

[0045] Step S6: Perform a bypass check each time the corresponding operator is called in step S4: when the depth number When, the input is returned before the parameters and random operations of depth operator one and depth operator two; when the number of observations... At that time, the same bypass is performed on the observation operator.

[0046] In one embodiment of the present invention, an optional implementation is as follows: Figure 6 As shown, when the depth number At this time, the depth operator returns the input directly before invoking attention, MLP, dropout, and related parameters. When the number of observations... When the observation operator is used, the same treatment is applied.

[0047] In this embodiment of the invention, since the judgment occurs before the parameter and random calculation, the missing axis branch does not change the tensor value, nor does it perform parameter calculation for that branch. The implementation code should first determine the axis length, and then perform rearrangement and module calls.

[0048] Step S7: Perform pooling on the terminal state to obtain language-aligned representations, or retain the structured token for downstream network reading.

[0049] In one embodiment of the invention, an optional implementation involves the terminal state comprising two prefix tokens and a dense token. For each of the two prefix tokens, an average is taken along the observation axis and the depth axis. A global average is then performed on the normalized dense tokens. Each language alignment header concatenates a prefix summary with the same dense summary and projects it as a 1024-dimensional vector. In this embodiment, the first output corresponds to the sampled report sentence, and the second output corresponds to the complete report. The complete report is divided into a maximum of eight SigLIP2 text windows, each containing 64 tokens. Adjacent windows overlap by eight tokens, and the summarization weight is calculated based on the number of valid content tokens after removing overlaps.

[0050] When training the model constructed using the above technical solutions, the normalized visual vectors are... and text vector Paired sigmoid loss is used :

[0051] In the formula, Indicates the batch size, and the summation covers all combinations of visual and text samples within the batch; This represents the logistic sigmoid function, i.e. ; Indicates the first The visual sample and the first The paired labels of each text sample are incremented by 1 when a pair is matched and decremented by 1 when no pair is matched. The scaling factor representing the similarity logit; This indicates the bias amount.

[0052] During training, the sentence text loss and the complete report text loss are weighted equally. An additional positive sample auxiliary loss with a weight of 0.1 is applied to the report window. Natural RGB batches also need to match frozen SigLIP2 image features and cross-modal affinity distributions, and text anchors are used to maintain the original language alignment coordinates. The training steps are detailed below. Figure 7 .

[0053] First, 2D checkpoints are obtained using SigLIP2 feature distillation. These checkpoints are then used to initialize six XY blocks (i.e., "plane operators"). Two Depth blocks (i.e., "depth operators") copy the parameters of the 3rd and 6th XY blocks, respectively. The Observation block (i.e., "observation operator") copies the parameters of the 6th XY block. The attention output projection and MLP output projection in the newly added residual branch are set to 10. -3 Scaling, thus approaching the identity mapping in the early stages of training. The technical solution disclosed in this invention was verified using two-dimensional images, volume data, and multi-observation image tasks. The results are as follows: ImageNet-1K zero-sample top-1 accuracy is 0.6122, CheXpert 5×200 accuracy is 0.4540, CT-RATE average AUROC is 0.7810, MR-RATE average AUROC is 0.7692; MIMIC-CXR image-report retrieval MRR is 0.2165, R@10 is 0.4181.

[0054] The present invention also provides an electronic device including a processor. The processor can perform corresponding actions and processes according to a software program stored in a read-only memory (ROM) or a software program loaded from a storage portion into a random access memory (RAM). The processor may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)). The processor may also include onboard memory for caching. The processor may include one or more processing units for executing the visual encoding method of native structural axis combinations described in the embodiments of the present invention. The processor executes a software program to implement the above-described visual encoding method of native structural axis combinations, such as... Figure 8 As shown, the software modules may include a data receiving module, a structure determination module, a visual encoding module, a representation reading module, and an output module, with each module corresponding to the steps of the above method.

[0055] The RAM stores various programs and data required for the operation of the electronic device. The processor, ROM, and RAM are interconnected via a bus. The processor implements the visual encoding method of the aforementioned native structural axis combination by executing programs in the ROM and / or RAM. The programs can also be stored in one or more memories other than ROM and RAM. The processor can also perform various operations of the aforementioned visual encoding method by executing programs stored in the one or more memories.

[0056] According to embodiments of the present invention, the electronic device may further include an input / output (I / O) interface, which is also connected to a bus. The I / O interface receives visual data and acquires metadata, and transmits language alignment representations or structured tokens. The electronic device may also include one or more of the following components connected to the I / O interface: an input section including a keyboard, mouse, etc.; an output section including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section including a hard disk, etc.; and a communication section including a network interface card such as a LAN card, modem, etc. The communication section performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface as needed. Removable media, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on the drive as needed so that computer programs read from them can be installed into the storage section as needed.

[0057] In the above technical solution, visual inputs with different structures share plane-by-plane calculation and terminal readout. During runtime, only the structural axes actually possessed by the input are called. Compared with existing visual encoding schemes, the technical solution disclosed in this embodiment of the invention achieves the following: 1) Record the number of observations N and the number of depths D separately in the input structure, and do not mix observation, depth and color channels into the same dimension; 2) Each reserved plane uses the same 2D patch projection, does not step along the depth direction, and establishes the view-depth-plane token state accordingly; 3) Perform tensor rearrangement on the same token state so that the plane, depth and view axes become the sequence axes for information interaction in sequence; 4) Axis operators are called in a predetermined order, with the output of the previous operator directly used as the input of the next operator; 5) Multiple updates call the same set of physical modules to achieve iterative calculation depth through parameter reuse; 6) Perform precise bypass on missing axes so that the 2D single-view input only travels through the planar path.

[0058] Those skilled in the art will understand that, without departing from the spirit of the present invention, the features described in the various embodiments and / or claims can be combined and / or combined, and all related technical solutions fall within the protection scope of the present invention.

[0059] The above embodiments are for illustrative purposes only and do not limit the scope of protection of the present invention. The scope of protection of the present invention is determined by the appended claims and their equivalents.

Claims

1. A visual encoding method for native structural axis combinations, characterized in that, Includes the following steps: Step 1: Receive visual data and record the number of observations. and depth number The color or grayscale channels contained in visual data are located in the input channel dimension; Step 2: Apply the same 2D patch projection to each preserved plane of each observation in the visual data, add position encoding and prefix token, and obtain the initial persistent token state. The dimension of the persistent state is... , For batch size, The length of the token sequence for each reserved plane, including M prefix tokens (M>=1) and A dense token, , For single token embedding feature dimensions; Step 3: Set the initial persistent token state conduct Cyclic updates, , any number The update cycle further includes the following steps: : Step 301, for the first The persistent token state obtained in the round Perform tensor rearrangement without losing tokens to make the length of the planar token sequence... Depth and observation number These become the attention sequence lengths, and the remaining structural dimensions are folded to the batch position. Step 302: Execute three plane operators, depth operator one, three plane operators, depth operator two, and one observation operator in sequence. The plane, depth, and observation operators operate sequentially on the same persistent token state. The output of the previous operator serves as the input of the next operator. The tensor rearrangement from step 301 is used to achieve the following: Length of planar token sequence As a sequence axis for information interaction in planar operators, planar tokens are mixed within each observation and slice; Depth As a sequence axis for information interaction in depth operators, tokens are used across the depth mixing plane at shared patch indices; Number of observations As a sequence axis for information exchange in the observation operator, information is exchanged across observations at the shared token index; Step 4: Perform pooling on the terminal persistent token state obtained in the previous step to obtain language-aligned representations, or retain the structured token for downstream network reading.

2. The visual encoding method for native structural axis combinations as described in claim 1, characterized in that, In step 1, the visual data includes single two-dimensional images, CT volume data, paired or multi-view two-dimensional images, and registered multi-sequence MRI, wherein: for the single two-dimensional image, the number of observations... Depth ; For the CT body data, the number of observations Depth ; For the paired or multi-view two-dimensional image, the number of observations Depth ; For the registered multi-sequence MRI, the number of observations Depth .

3. The visual encoding method for native structural axis combinations as described in claim 1, characterized in that, In step 3, The same set of physical modules consisting of plane operators, depth operators, and observation operators is reused in a round-robin fashion to perform multi-step updates without copying the physical modules according to the number of executions.

4. The visual encoding method for native structural axis combinations as described in claim 1, characterized in that, In step 3, the function of checking if the shaft length is 1 is performed: If the detected shaft length is 1: If depth number If the value is 1, the depth operator does not perform the corresponding parameter calculation and dropout operation, and returns the input before the parameters and random operations of the depth operator. The depth operator becomes an execution-level identity mapping. If the number of observations If the value is 1, the observer operator does not perform the corresponding parameter calculation and dropout operation, and returns the input before the parameter and random operation of the observer operator. The observer operator becomes an execution-level identity mapping. If the corresponding axis length is detected to be greater than 1, the plane operator, depth operator, or observation operator performs an axial attention process on the persistent token state, including: The tensor rearrangement without losing tokens described in step 301; Perform axial attention calculations, or perform multi-layered sensory processing after performing axial attention calculations; Restore the original dimensional arrangement of the persistent token state.

5. An electronic device, comprising: One or more processors; A storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the visual encoding method according to claim 1.