Multi-scale adaptive large model visual coding method and system based on convolution thought

Abstract

The invention discloses a multi-scale adaptive large-model visual coding method and system based on a convolution idea, and relates to the technical field of visual coding, and the method comprises the steps: obtaining a plurality of visual tokens corresponding to an input image, mapping the visual tokens to a two-dimensional visual square with a predetermined size, and obtaining a sparse point corresponding to each visual token on the two-dimensional visual square; respectively calculating a position correlation value between each sparse point and each sparse point in the corresponding neighborhood, then obtaining a convolution kernel corresponding to each sparse point and carrying out convolution calculation, inversely mapping each coded visual token from the two-dimensional visual square to an original two-dimensional space position corresponding to the input image, and finally obtaining a position correlation value between each sparse point and each sparse point in the corresponding neighborhood; the original two-dimensional arrangement of each visual token is recovered and flattened into a one-dimensional sequence which is used for being in butt joint with a large language model. According to the method, the perceptual ability of the large language model to the target layout and details in the visual scene can be greatly enhanced.

Description

Technical Field

[0001] This invention relates to the field of visual coding technology, and in particular to a large-scale visual coding method and system based on the concept of convolution and multi-scale adaptation. Background Technology

[0002] In the field of large-scale multimodal models, visual modality positional encoding is a crucial step, with its core task being to enable the model to understand the arrangement and relationships of visual elements in two-dimensional space. A current technical challenge is that when mainstream visual positional encoding methods flatten the visual token sequence formed by image segmentation into one dimension and perform positional encoding calculations, the inherent two-dimensional spatial structure information of the image is inevitably lost. This leads to a decline in the model's ability to perceive visual scenes, making it difficult to accurately identify objects and their relationships in directions such as up / down and left / right within the image. In simpler terms, the model lacks a "sense of direction" in the two-dimensional visual environment.

[0003] To address the problem of representing the position of visual sequences, researchers have proposed several positional encoding schemes. The first type is absolute positional encoding, which generates a fixed set of vectors using sine and cosine functions and directly adds them to the input embedding vector. This method assigns a unique code to each position, capturing absolute positional information, but it cannot directly model the relative relationships between positions. The second type is relative positional encoding, which does not explicitly represent the absolute coordinates of each position, but focuses on modeling the relative distance between two tokens. For example, if word A is at position 5 and word B is at position 8, their relative distance is 3. This method allows attention weights to consider both content similarity and positional relativity during calculation, directly capturing relative relationships, but its mathematical implementation is more complex, resulting in higher computational complexity. The third type is rotational positional encoding, which rotates each feature vector on a two-dimensional plane according to the angle corresponding to its position index, naturally integrating positional information into the vector's direction. Rotational positional encoding combines intuitiveness and efficiency, and has become the de facto standard used by many large models in text and vision tasks.

[0004] However, in-depth analysis reveals a common limitation among the aforementioned mainstream positional encoding methods: regardless of whether a sine function, relative distance bias, or vector rotation is used, the computational prerequisite is to first flatten the visual token grid generated by 2D image segmentation into a one-dimensional sequence. This flattening operation severs the 2D adjacency relationships between visual tokens in the original image, forcibly compressing the 2D spatial structure into a one-dimensional linear order, thus losing crucial 2D topological information from the image. Therefore, even if the model can perceive the order of elements in the sequence, it struggles to recover and understand the true spatial layout of elements on the 2D plane, ultimately resulting in poor ability to capture fine-grained features and spatial relationships of visual modalities.

[0005] To address the problem that existing visual position encoding methods lose two-dimensional spatial information due to flattening operations, a new method is needed that can effectively preserve and utilize the two-dimensional feature relationships of visual tokens. Summary of the Invention

[0006] The technical problem to be solved by this invention is to address the shortcomings of existing technologies, specifically by providing a multi-scale adaptive large-model visual encoding method and system based on the concept of convolution, as detailed below: 1) In a first aspect, the present invention provides a large-scale visual encoding method based on the idea of ​​convolution, the specific technical solution of which is as follows: The input image is acquired and visually encoded to obtain multiple visual tokens corresponding to the input image; Map all visual tokens onto a two-dimensional visual square of a predetermined size to obtain the sparse points on the two-dimensional visual square corresponding to each visual token; Calculate the positional correlation value between each sparse point and each sparse point in its corresponding neighborhood; The calculated positional correlation values ​​are normalized to obtain the convolution kernel corresponding to each sparse point; Using the convolution kernel corresponding to each sparse point, convolution calculation is performed on each sparse point in the two-dimensional visual square to obtain the encoded visual token. Each encoded visual token is inversely mapped from the two-dimensional visual square back to its original two-dimensional spatial position in the input image to restore the original two-dimensional arrangement of each visual token. Each restored visual token is then flattened into a one-dimensional sequence for interface with the large language model.

[0007] The beneficial effects of the large-scale visual encoding method based on convolutional thinking provided by this invention are as follows: This method effectively solves the problem of existing methods losing two-dimensional spatial information when visual tokens are flattened into a one-dimensional sequence. By mapping all visual tokens onto a two-dimensional visual square of a predetermined size, the model can perform positional encoding calculations while preserving the original two-dimensional topological relationships. The positional correlation values ​​between sparse points are calculated using a two-dimensional Gaussian distribution function, and convolutional kernels are constructed, allowing the features of each visual token to fuse information from other visual tokens in its two-dimensional neighborhood. This process is performed directly in two-dimensional space, avoiding the loss of directionality caused by one-dimensional flattening. After convolutional calculations, the encoded visual tokens are inversely mapped back to their original two-dimensional spatial positions and flattened, resulting in a final one-dimensional sequence that is not only rich in enhanced local features but also implicitly encodes precise two-dimensional spatial relationships. Therefore, this method can significantly enhance the ability of large language models to perceive the layout and details of targets in visual scenes, thereby generating more accurate and richer visual descriptions. Furthermore, the entire process is fully compatible with subsequent attention calculations, with minimal additional computational overhead.

[0008] Based on the above scheme, the large model visual encoding method based on the concept of convolution and multi-scale adaptation of the present invention can be further improved as follows.

[0009] Furthermore, all visual tokens are mapped onto a two-dimensional visual square of a predetermined size, resulting in sparse points on the two-dimensional visual square corresponding to each visual token, including: Calculate the scaling factors between the number of visual tokens in the height direction and the number of visual tokens in the width direction of the input image and the corresponding directional dimensions of the two-dimensional visual square; Based on all scaling factors, each visual token in the input image is uniformly mapped onto a two-dimensional visual square to obtain the sparse points corresponding to each visual token on the two-dimensional visual square.

[0010] The beneficial effects of adopting the above-mentioned further scheme are as follows: by introducing a two-dimensional visual square and a scaling factor, the problem of visual token spatial alignment caused by input images of different sizes is effectively solved. First, the scaling factors in the height and width directions are calculated to establish a linear proportional relationship between the number of visual tokens in an image of any size and a fixed-size two-dimensional visual square. Second, each visual token is uniformly mapped onto the two-dimensional visual square according to the scaling factor, generating corresponding sparse points. This operation ensures that all input images, regardless of their original size, can obtain a normalized positional representation of their visual tokens in a unified two-dimensional coordinate system. The sparse points on the two-dimensional visual square maintain the two-dimensional topological relationship between visual tokens in the original image, laying a solid foundation for subsequent feature relationship calculation based on two-dimensional neighborhoods in a unified space. The entire process achieves standardized preprocessing of multi-scale visual inputs and completes spatial normalization of features while preserving the two-dimensional spatial structure.

[0011] Furthermore, the neighborhood of any sparse point refers to a square region with a side length of 6σ centered on the sparse point, where σ is the standard deviation of the distance between the sparse point and other sparse points.

[0012] The beneficial effect of adopting the above-mentioned further scheme is that adaptive modeling of two-dimensional spatial relationships is achieved by dynamically defining a neighborhood range based on statistical distance for each sparse point. A square region with a side length of 6σ is defined as its neighborhood, centered on the sparse point itself and using the standard deviation σ of the distance between that point and all other sparse points as the unit of measurement. This definition allows the neighborhood range to be flexibly adjusted according to the actual distribution density of the sparse points in the two-dimensional visual square. In areas with dense visual token distribution, the standard deviation σ is smaller, and the neighborhood range shrinks accordingly, focusing on capturing local, close detail relationships; in sparsely distributed areas, the standard deviation σ is larger, and the neighborhood range expands accordingly to cover potentially related points at greater distances. Using a 6σ boundary follows the characteristics of a Gaussian distribution, ensuring that the neighborhood covers the main areas with significant spatial correlation with the center point. This data-driven neighborhood delineation method overcomes the limitations of fixed-size convolutional kernels in adapting to different image content and sparse point distribution patterns, providing a reasonable and adaptive spatial basis for subsequent accurate calculation of positional correlation values ​​and construction of effective convolutional kernels.

[0013] Furthermore, the positional correlation value between each sparse point and each sparse point in its corresponding neighborhood is calculated, including: using a two-dimensional Gaussian distribution function to calculate the positional correlation value between each sparse point and each sparse point in its corresponding neighborhood.

[0014] The beneficial effect of adopting the above-mentioned further scheme is that by introducing a two-dimensional Gaussian distribution function, the spatial distance between sparse points is transformed into a quantified positional correlation value. This method can assign a continuous and smooth weight to each sparse point in the neighborhood based on geometric proximity. The closer to the central sparse point, the higher the calculated positional correlation value; the farther away, the correlation value decays exponentially. This calculation method not only accurately models the local dependency strength between visual tokens in two-dimensional space, but also provides a direct numerical basis for the subsequent normalization and generation of convolutional kernels. The calculation based on the Gaussian function has a clear mathematical meaning, making the generation process of positional correlation values ​​differentiable and stable, which is convenient for integration into the end-to-end training framework of deep neural networks.

[0015] 2) Secondly, the present invention also provides a large-scale visual encoding system based on the concept of convolution, the specific technical solution of which is as follows: It includes an acquisition module, a mapping module, a position-related value calculation module, a normalization processing module, a convolution calculation module, and a recovery and flattening module; The acquisition module is used to: acquire the input image and perform visual encoding to obtain multiple visual tokens corresponding to the input image; The mapping module is used to: map all visual tokens onto a two-dimensional visual square of a predetermined size, and obtain the sparse points corresponding to each visual token on the two-dimensional visual square; The location correlation value calculation module is used to: calculate the location correlation value between each sparse point and each sparse point in its corresponding neighborhood; The normalization module is used to normalize all the calculated position-related values ​​to obtain the convolution kernel corresponding to each sparse point. The convolution calculation module is used to: use the convolution kernel corresponding to each sparse point to perform convolution calculation on each sparse point in the two-dimensional visual square to obtain the encoded visual token; The recovery and flattening module is used to: reverse map each encoded visual token from the two-dimensional visual square back to the original two-dimensional spatial position corresponding to the input image, so as to restore the original two-dimensional arrangement of each visual token, and flatten each restored visual token into a one-dimensional sequence for docking with the large language model.

[0016] Based on the above scheme, the large model visual coding system based on the concept of convolution and multi-scale adaptation of the present invention can be further improved as follows.

[0017] Furthermore, the mapping module is specifically used for: Calculate the scaling factors between the number of visual tokens in the height direction and the number of visual tokens in the width direction of the input image and the corresponding directional dimensions of the two-dimensional visual square; Based on all scaling factors, each visual token in the input image is uniformly mapped onto a two-dimensional visual square to obtain the sparse points corresponding to each visual token on the two-dimensional visual square.

[0018] Furthermore, the neighborhood of any sparse point refers to a square region with a side length of 6σ centered on the sparse point, where σ is the standard deviation of the distance between the sparse point and other sparse points.

[0019] Furthermore, the location correlation value calculation module is specifically used to: calculate the location correlation value between each sparse point and each sparse point in its corresponding neighborhood using a two-dimensional Gaussian distribution function.

[0020] 3) In a third aspect, the present invention also provides an electronic device, the electronic device including a processor coupled to a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor, so that the electronic device implements any of the above-mentioned large model visual encoding methods based on the idea of ​​convolution.

[0021] 4) In a fourth aspect, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-mentioned large model visual encoding methods based on the idea of ​​convolution.

[0022] It should be noted that the beneficial effects of the technical solutions of the second to fourth aspects of the present invention and their corresponding possible implementations can be found in the above description of the technical effects of the first aspect and its corresponding possible implementations, and will not be repeated here. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below: Figure 1 This is a flowchart illustrating a large-scale visual encoding method based on convolutional concepts according to an embodiment of the present invention. Figure 2 This is a schematic diagram of a visual coding graph; Figure 3 A flowchart illustrating the workflow of a large language model; Figure 4 For adaptive scaling of the image; Figure 5 This is a graph showing the convolution computation. Figure 6A three-dimensional Gaussian plot; Figure 7 This is a schematic diagram of the structure of a large-scale visual coding system based on the concept of convolution, according to an embodiment of the present invention. Detailed Implementation

[0024] The principles and features of the present invention are described below. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0025] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0026] like Figure 1 As shown in the figure, a large-scale visual encoding method based on the concept of convolution in this invention includes the following steps: S1. Obtain the input image and perform visual encoding to obtain multiple visual tokens corresponding to the input image. The specific implementation process is as follows: S10. The acquired input image may have arbitrary size and number of color channels. To adapt to the fixed input requirements of the subsequent visual encoder, the input image needs to be standardized preprocessed. Preprocessing operations mainly include resizing. The height and width of the input image are scaled to the fixed size required by the visual encoder, such as 224×224 pixels. Common scaling algorithms include bilinear interpolation to ensure the smoothness of the image content during scaling. Next, the adjusted image is converted from the original red, green, and blue three-channel pixel value matrix into the tensor format expected by the visual encoder. This conversion typically involves normalizing the pixel values ​​to a specific range and adjusting the data dimensionality to a format that conforms to the model input requirements.

[0027] S11. A visual encoder is a pre-trained deep neural network model whose core function is to extract high-level visual features from pixel data. The widely used visual encoder architecture is the Vision Transformer. The first stage of this model's image processing is to cut a two-dimensional image grid into a series of regularly arranged image patches. Each image patch has a fixed size, for example, 16 pixels × 16 pixels. For an input image of size 224 × 224, it will be cut into 196 image patches. Each image patch contains 16 × 16 × 3 = 768 pixel values. Next, all pixel values ​​of each image patch are flattened into a one-dimensional vector. Then, through a learnable linear projection layer, the one-dimensional pixel vector of each image patch is mapped to a higher-dimensional embedding space. This linear projection operation can be expressed as the following formula: In this formula, Indicates the first A one-dimensional pixel vector formed by flattening an image patch. This represents the weight matrix of the linear projection layer. The first term obtained after projection The embedding vectors of each image patch. This process transforms the raw pixel information into more expressive feature vectors.

[0028] S12. After linear projection, a series of image patch embedding vectors are obtained. To preserve the order information of these image patches in the original two-dimensional space, positional encoding needs to be added to each embedding vector. The positional encoding vector has the same dimension as the embedding vector, and the two are added element-wise. After addition, this series of vectors containing positional information is fed into the main body of the visual encoder, which is a module composed of multiple stacked Transformer encoding layers. In these layers, deep feature transformation and fusion are performed through self-attention mechanisms and feedforward neural networks. After processing by all layers, the visual encoder finally outputs a vector sequence. Each vector in this output sequence corresponds to a specific region in the original input image and encapsulates the semantic information of that region and its association with the global context of the image. These output vectors are the final visual tokens. Each visual token is a high-dimensional real-valued vector, which together constitute a discretized and structured feature representation of the input image.

[0029] In this context, a visual token is a structured vector representation output by the visual encoder after feature extraction and transformation of the input image. Each visual token corresponds to a local region or image patch in the original image. It is not simply a collection of pixels, but rather contains abstract semantic features learned through a deep neural network. These features can represent edges, textures, object parts, or higher-level semantic concepts. Visual tokens constitute the basic units for large models to understand and process visual information, similar to word tokens in natural language processing. By transforming an image into a sequence of visual tokens, complex two-dimensional visual information is converted into a serialized data form that can be understood and processed by subsequent language models, thereby achieving alignment and fusion of images and text at the representation level.

[0030] S2. Map all visual tokens onto a 2D visual plaza of a predetermined size to obtain the sparse points corresponding to each visual token on the 2D visual plaza, specifically including: S20. Calculate the scaling factors between the number of visual tokens in the height direction and the number of visual tokens in the width direction of the input image and the corresponding dimensions of the two-dimensional visual square. The specific implementation process is as follows: S200. After completing visual encoding and obtaining multiple visual tokens, these visual tokens logically maintain their two-dimensional spatial arrangement in the original input image. It is necessary to specify the number of rows in the height direction of this two-dimensional arrangement, that is, the number of visual tokens in the height direction, denoted as . At the same time, it is necessary to specify the number of columns in the width direction, that is, the number of visual tokens in the width direction, denoted as . A person with × The image of the nth visual token has its top-left corner's coordinates recorded as row 1, column 1, and its bottom-right corner's coordinates as row 1, column 1. Line 1 List.

[0031] S201, the two-dimensional visual square is a predefined, fixed-size abstract two-dimensional planar grid. Its dimensions are determined during the model design phase and remain unchanged throughout all subsequent training and inference processes. Assume the two-dimensional visual square has [dimensionality] in the height direction. A standard position unit, with [number] units in the width direction. A standard positional unit. Therefore, the dimensions of a two-dimensional visual square can be expressed as: × Here and It is a pre-defined positive integer, such as 64×64.

[0032] S202, the scaling factor is used to establish a linear mapping from the original visual token coordinates to the two-dimensional visual square coordinates. For the height direction, it is necessary to scale the original... The range to which each visual token is distributed is mapped to the height direction of a two-dimensional visual plaza. Within a unit range. To achieve uniform mapping and ensure that the coordinate distribution of the visual tokens on the 2D visual square is continuous and uniform after mapping, a scaling factor in the height direction is calculated. The formula is as follows: In this formula, The scaling factor in the height direction is a real number. This represents the number of visual tokens in the height direction of the input image. This represents the dimension of the two-dimensional visual square in the height direction. The formula uses... Instead The term "molecule" is used because it considers the length of the interval spanned from the center of the first visual token to the center of the last visual token. This calculation ensures that the first and last visual tokens in the original two-dimensional space can be accurately mapped to the corresponding beginning and end boundary positions in the two-dimensional visual square.

[0033] S203. The scaling relationship in the width direction is calculated using the same principle as in the height direction. For the width direction, the original... The range to which each visual token is distributed is mapped to the width direction of a two-dimensional visual plaza. Within a unit range, calculate the scaling factor in the width direction. The formula is as follows: In this formula, The scaling factor in the width direction is a real number. This represents the number of visual tokens in the width direction of the input image. This represents the dimension of a two-dimensional visual square in the width direction. Similarly, using... As molecules, they ensure the uniformity of the mapping and the alignment of the boundaries.

[0034] The scaling factor is calculated using the two formulas above. and Then, the key scaling parameters needed to map the original visual token's two-dimensional integer coordinate grid to the continuous coordinate space of the two-dimensional visual plaza are obtained. Subsequent steps will use these two scaling factors to perform a linear transformation on the two-dimensional coordinates of each visual token, thereby completing the projection onto the two-dimensional visual plaza.

[0035] The 2D visual square is a predefined, fixed-size abstract 2D coordinate plane within the algorithm. It serves as an intermediary, standardized computational space where all image visual tokens of different original sizes are mapped onto this fixed-size square according to geometric rules. The size of the 2D visual square is typically expressed as the product of the height (in units) and width (in units), for example... × Its core function is to unify the scale differences between different input images, providing a stable and consistent two-dimensional spatial basis for subsequent positional relationship calculations based on the convolution idea at a unified scale.

[0036] The number of visual tokens in the height direction refers to the number of rows along the vertical direction of the resulting visual tokens in a logical two-dimensional arrangement after visual encoding of the input image. This number directly depends on the aspect ratio of the original input image and the size of the image blocks segmented by the visual encoder. For example, if the visual encoder segments the image into fixed-size blocks, this number can be obtained by dividing the image height by the block height and rounding up. It represents the degree of discretization of visual features in the vertical dimension and is a fundamental input parameter for calculating the spatial mapping relationship in the vertical direction.

[0037] The number of visual tokens in the width direction refers to the number of columns along the horizontal direction of the resulting visual tokens in a logical two-dimensional arrangement after visual encoding of the input image. This number depends on the width of the original input image and the width of the image patches segmented by the visual encoder. It represents the degree of discretization of visual features in the horizontal dimension. Together with the number of visual tokens in the height direction, these two parameters fully define the two-dimensional grid shape of the visual token set, and are two key dimensional information required for subsequent multi-scale adaptive projection.

[0038] S21. Based on all scaling factors, uniformly map each visual token in the input image onto the two-dimensional visual square to obtain the sparse points corresponding to each visual token on the two-dimensional visual square. The specific implementation process is as follows: S210. The input for this step consists of three parts. The first part is the original two-dimensional coordinate set of all visual tokens in the input image. Assume there are a total of [number missing] visual tokens. One, of which It represents the number of visual tokens in the height direction. This refers to the number of visual tokens in the width direction. Each visual token can be indexed by the row it belongs to. and column indexes To locate, among which The value range is from 1 to , The value range is from 1 to Therefore, the first Line 1 The original coordinates of the visual token of the column can be represented as The second part consists of the two scaling factors calculated in the preceding steps, namely the scaling factor in the height direction. and scaling factor in the width direction The third part consists of the fixed dimensional parameters of the two-dimensional visual plaza, namely the dimensions in the height direction. and width dimension .

[0039] S211. For ease of calculation and uniform mapping, the origin of the original visual token grid is defined as being located at the first visual token, i.e. The starting position in the height and width directions of the corresponding two-dimensional visual square is usually defined as coordinate 0. The endpoint coordinate of the original visual token grid is defined as the last visual token, i.e. The endpoint in the height direction of the corresponding two-dimensional visual square End position in the width direction Thus, the goal of the mapping is to transform the original discrete integer coordinate interval [1, ... ] and [1, Linear transformation to a continuous real coordinate range [0, ... ] and [0, ].

[0040] S212, for the original coordinates are For any visual token, its corresponding coordinates on a two-dimensional visual plaza need to be calculated. The calculation follows the principle of linear mapping, and the formula is as follows: In this formula, Indicates the first Line 1 The coordinate value obtained by mapping the visual token of a column to a two-dimensional visual square in the height direction is a real number. This represents the coordinate value of the visual token in the width direction, which is also a real number. The row index representing the visual token in the original permutation is an integer starting from 1. The width column index of the visual token in the original permutation is an integer starting from 1. It is the scaling factor in the height direction. It is the scaling factor in the width direction.

[0041] in the formula and The operation shifts the origin of the coordinate system from the position of the first visual token (index 1) to point 0. This is then multiplied by a scaling factor. or This achieves linear scaling from the original index interval to the target continuous interval. For example, when hour, ;when hour, According to the scaling factor The definition can be derived from This ensures that the first and last visual tokens are accurately mapped to the boundary positions of the two-dimensional visual square.

[0042] S213. Through a loop process, for each visual token, that is, for each... From 1 to Each From 1 to The combination of these elements is used to calculate the mapping using the formula described above. After calculation, each visual token receives a unique pair of real coordinates on the two-dimensional visual plaza. Due to the size of the two-dimensional visual plaza It is fixed, while the number of original visual tokens is... Typically much smaller (For example, the original grid might be 14×14, while the plaza is 64×64), so these mapped points appear sparsely distributed on the vast two-dimensional visual plaza. These points with coordinate positions on the two-dimensional visual plaza are the sparse points corresponding to each visual token on the two-dimensional visual plaza. The set of all these sparse points constitutes the final output of this step.

[0043] S3. Calculate the positional correlation value between each sparse point and each sparse point in its corresponding neighborhood. Specifically, use the two-dimensional Gaussian distribution function to calculate the positional correlation value between each sparse point and each sparse point in its corresponding neighborhood. The neighborhood of any sparse point refers to a square region with a side length of 6σ centered on the sparse point, where σ is the standard deviation of the distance between the sparse point and other sparse points.

[0044] The specific implementation process of S3 is as follows: S30. Suppose that after mapping, a set containing N sparse points is obtained, where N equals the number of visual tokens in the input image, i.e. Each sparse point (Here, k is the index of the sparse point, ranging from 1 to N) Each of them has its real coordinates on the two-dimensional visual square. The computation needs to be performed sequentially for each sparse point in the set. For the target sparse point currently being computed, its index is denoted as... Its coordinates are .

[0045] S31. In order to calculate the standard deviation of the distance later, it is necessary to first calculate the sparse points of the target. With every other sparse point in the set (in From 1 to N, and The straight-line distance between two points. The Euclidean distance between two points. The calculation formula is as follows: In this formula, Represents the target sparse points With another sparse point The distance between them. and Represents the target sparse points The x and y coordinates. and Represents another sparse point The x and y coordinates. Iterate through all... From the sparse points, we can obtain a set containing N-1 distance values. (The following is omitted) ).

[0046] S32, Standard deviation σ is used to measure the sparsity of target points. The degree of dispersion of distances to all other sparse points. This will become a key parameter in defining the neighborhood range. First, calculate the average of these N-1 distances. : Next, the standard deviation of these distances is calculated. : In this formula, Represents the target sparse points The average distance to all other sparse points. This represents the standard deviation of these distances. It is a sparse point specifically belonging to the current target. One of the parameters, which usually has different values ​​at different sparse points. value.

[0047] S33. According to the definition of the final technical solution, with the target sparse points Let be the center, and determine a side with length . The square region is taken as its neighborhood. This means that, on the two-dimensional visual square, the horizontal coordinate is in the interval [missing information]. Within, and the ordinate is in the interval All sparse points within the range are considered as target sparse points. The points are within the neighborhood of the given sparse points. We need to select points whose coordinates satisfy this condition from the entire set of sparse points to form a subset of sparse neighborhood points.

[0048] S34. For each sparse point in the neighborhood selected in the previous step... (including target sparse points) (Itself), calculate its sparseness relative to the target point. Location-related values The two-dimensional Gaussian distribution function used in the calculation is as follows: In this formula, Represents sparse points within the neighborhood. Relative to target sparse points The position-related value is a non-negative real number. This represents the natural exponential function. It represents pi (π). and These represent the differences between two points along the x-coordinate and y-coordinate directions, respectively, and the sum of their squares is the square of the distance between the two points. It is the target point calculated earlier. The standard deviation.

[0049] This function has the following property: when and When they overlap (i.e., the distance is 0), the correlation value Get the maximum value ;along with and As distance increases, the correlation value decreases exponentially; when the distance reaches or exceeds... At this point, the function value has become very small. This aligns with the intuition that "the closer to the target point, the stronger the positional correlation." Performing the above calculation on every point in the neighborhood (including the target point itself) yields a set of positional correlation values.

[0050] S35. For each point in the sparse point set, repeat the operations from S30 to S34, and use them sequentially as target sparse points. Calculate its corresponding standard deviation. The system determines a unique neighborhood for each sparse point and calculates the positional correlation values ​​of all points within that neighborhood relative to it. Finally, the system generates a list for each sparse point, recording its positional correlation values ​​with each other in its neighborhood. These values ​​form the basis for constructing the convolutional kernel.

[0051] S4. Normalize all calculated positional correlation values ​​to obtain the convolution kernel corresponding to each sparse point. Specifically: For any sparse point, the position correlation value of that sparse point is set as the peak value. The sum of the correlation values ​​of all sparse points in the neighborhood of that sparse point is used as the denominator. The proportion of the position correlation value of each sparse point is calculated, and the proportion of each sparse point in the neighborhood of that sparse point is used as the contribution score of each sparse point to that sparse point. Each contribution score is used as the weight value of the corresponding position in the convolution kernel to form a convolution kernel used for convolution calculation of that sparse point. The process is repeated for each sparse point to obtain the convolution kernel corresponding to each sparse point. The specific implementation process is as follows: S40. For the target sparse points where convolutional kernels need to be constructed... Two pieces of information have been clarified. The first is the target sparse points. neighborhood point set This set contains all those located in Centered on, with side length as Sparse points within a square region, where yes The distance standard deviation. The second term is the set of pre-calculated location correlation values. ,here Representing the neighborhood Total number of internal sparse points (including) itself), Indicates the number of neighbors within the neighborhood. sparse point Relative to target point Location-related values.

[0052] S41. Calculate the sum of all location-related values ​​within the neighborhood; this sum will be used as the denominator for normalization. This calculation requires processing the neighborhood... The positional correlation values ​​corresponding to each sparse point within the range are summed. The calculation formula is as follows: In this formula, Represents the target sparse points The sum of the position correlation values ​​of all sparse points in the neighborhood of . Representing the neighborhood The number of internal sparse points. Indicates the number of neighbors within the neighborhood. sparse point The location-related values. This sum is a real number greater than zero.

[0053] S42. The contribution score is the normalized weight, representing the contribution of each sparse point in the neighborhood to the target sparse point. The relative importance when performing feature aggregation. For neighborhood The first sparse point Its contribution score Calculated by dividing its location-related values ​​by the sum: In this formula, Represents sparse points within the neighborhood. For target sparse points The contribution score. It is the location-related value used in step two. This is the sum of the relevant values ​​calculated in the previous step. This includes the positional relevant values ​​of the target point itself. Typically, it is the maximum value (peak value) within the neighborhood, and after this normalization, its contribution score is... It is also usually the largest weight value within the neighborhood. For all From 1 to Calculations were performed to obtain a set of contribution scores. This set of fractions satisfies .

[0054] S43. The convolution kernel is a discrete two-dimensional weight matrix, whose center point corresponds to the target sparse points. The other positions in the matrix correspond to other sparse points in their neighborhood. The construction process requires determining the size of the convolution kernel and the weights at each position. The size of the convolution kernel is determined by the neighborhood range. Since the neighborhood is a region with sides of length... For a square, a discretized grid can be defined to cover the region. A simple approach is to set both the height and width of the convolution kernel to... (Round up to ensure the boundary is included), and ensure the center point coordinates are located at... ,in , For the neighborhood Each sparse point within Its coordinates are Calculate its relative to the target point coordinate Integerized offset: , Then, the previously calculated contribution score... The value is assigned to the position in the convolution kernel matrix. The weight of a point in the convolution kernel matrix is ​​set to 0 for any point that has no sparse neighborhood.

[0055] S44. For each point in the sparse point set, repeat steps S40 to S43. Each sparse point... Each is based on its own unique neighborhood structure and distance standard deviation Based on the calculated contribution scores, a unique convolutional kernel with normalized weights is generated. Ultimately, the system will obtain a set of convolution kernels equal to the number of sparse points. Each convolution kernel is a two-dimensional matrix whose weight distribution reflects the local spatial relationship pattern centered on the corresponding sparse point and has been normalized, so it can be directly used for subsequent convolution calculations.

[0056] Each contribution score, corresponding to a specific position within the convolutional kernel, refers to the weight value contributed by a particular sparse point within the neighborhood of the two-dimensional convolutional kernel weight matrix constructed for the target sparse point. The exact row and column indices of the matrix where the point is placed. This position is not random, but determined by the physical coordinate offset of the neighborhood sparse point relative to the target sparse point. By converting the coordinate offsets into row and column index offsets of the matrix and superimposing them on the center position index of the convolution kernel, the "corresponding position" of the contribution score in the convolution kernel matrix is ​​determined. This process ensures that the weight distribution of the convolution kernel is consistent with the distribution pattern of the original sparse points in geometric space, allowing the convolution operation to correctly aggregate spatial neighborhood information.

[0057] In this invention, the convolution kernel is a dynamically generated, discrete two-dimensional weight matrix. It is not fixed but tailored to each visual token (mapped to a sparse point) in the image. The core function of the convolution kernel is to define how to aggregate the feature information of other sparse points in the neighborhood surrounding the target sparse point during subsequent convolution calculations. Each weight value in the matrix, i.e., the contribution score, quantifies the degree of influence of the corresponding point in the neighborhood on the feature update of the center point. The weight values ​​are normalized so that the sum of all non-zero weights is 1, which helps maintain the numerical stability after feature aggregation. This convolution kernel is the key data structure for transforming a Gaussian-based spatial correlation measure into an executable, standard discrete convolution operation.

[0058] S5. Using the convolution kernel corresponding to each sparse point, perform convolution calculation on each sparse point in the 2D visual square to obtain the encoded visual token. The specific implementation process is as follows: S50. Three sets of preprocessed data need to be collected. The first set of data is for each sparse point. The corresponding original visual token feature vector is denoted as This vector is the direct output of the visual encoder, representing the original semantic information of the image region corresponding to the sparse point. The second set of data is for each sparse point. Specially constructed convolutional kernels The convolution kernel is a two-dimensional weight matrix, whose center point corresponds to... The matrix itself, the position and weight of non-zero elements in the matrix. Accurately recorded Other sparse points in the neighborhood The contribution score. The third set of data is the coordinates of each sparse point on the two-dimensional visual square. and its neighborhood point set Information.

[0059] S51. Convolution calculation requires traversing all sparse points. For a sparse point that is currently the target of the calculation... The operations revolve around this. First, indexing to the convolutional kernel corresponding to that sparse point. At the same time, based on its coordinates and standard deviation Determine its neighborhood point set What other sparse points are included? Convolution kernel Each non-zero weight value All are related to the neighborhood point set A specific sparse point .

[0060] S52. Perform discrete convolution computation on the target sparse points. This computation is performed along the feature dimension, not in the spatial pixel domain. Specifically, the encoded visual token feature vector... It is obtained by weighted summation of the original feature vectors of all sparse points in its neighborhood (including itself). The calculation formula is as follows: In this formula, Representing sparse points The encoded visual token feature vector is obtained after convolution calculation. This indicates that the set of points belonging to the neighborhood is... All sparse point indexes Perform summation. Indicates from the convolution kernel Sparse points obtained from For target point The contribution score weights, which have been normalized in previous steps. Indicates the number of neighbors within the neighborhood. sparse point The corresponding original visual token feature vector. This represents the multiplication of a scalar and a vector.

[0061] S53. In actual calculations, it is necessary to ensure the weights With feature vectors The correct correspondence is achieved through coordinate mapping of sparse points. Convolution kernel. Each weight The position in the matrix encodes the corresponding neighborhood point. Relative to the center point The spatial offset. When performing summation, the system uses this offset or directly from the neighborhood point set. The list retrieves the correct original feature vector. Perform multiplication and summation. Since the convolution kernel weights have been normalized, this operation is essentially a convex combination of neighborhood features, ensuring the output features... Stability over a numerical range.

[0062] S54. For each sparse point on the two-dimensional visual square The steps S51 to S53 are repeated sequentially. Each sparse point uses its own unique convolution kernel to perform an aggregation calculation on its neighborhood features. After all sparse points have completed the calculation, each sparse point obtains a new feature vector. This new set of feature vectors is the encoded visual token. They not only retain the semantic information of the original visual token, but also encode the proximity information in two-dimensional space into the feature representation by weighted aggregation of features from surrounding visual tokens. These encoded visual tokens will be used in subsequent inverse mapping and serialization steps.

[0063] S6. Reverse map each encoded visual token from the two-dimensional visual square back to its original two-dimensional spatial position in the input image to restore the original two-dimensional arrangement of each visual token. Then, flatten each restored visual token into a one-dimensional sequence for interface with the large language model. The specific implementation process is as follows: S60. Establish the inverse mapping relationship from the 2D visual square coordinates to the original 2D spatial position. This step requires utilizing the key parameters recorded during the forward mapping. For any encoded visual token, it is currently associated with a sparse point on the 2D visual square. Its square coordinates are To place it back into its original space, the row index in the original visual token grid corresponding to that coordinate needs to be calculated. and column indexes This is achieved by applying a scaling factor. and This is done by inverse operation. The calculation formula is as follows: In this formula, This represents the row index (integer or rounded) in the height direction of the calculated, encoded visual token within the original visual token grid. This represents the calculated column index in the width direction. and It is the coordinates of the sparse points associated with the currently encoded visual token on the two-dimensional visual square. and These are the previously calculated scaling factors in the height and width directions. Since the forward mapping formula is... Therefore, the inverse operation is... Calculated and Typically, these are floating-point numbers that need to be converted to integers through rounding operations (e.g., to the nearest integer) to match discrete grid indices. The tuple is the original two-dimensional spatial position corresponding to the input image of the encoded visual token.

[0064] S61. Create an empty two-dimensional feature matrix. Its size is Line × Column, among which and These represent the number of visual tokens in the height and width directions of the original input image, respectively. (Matrix) Each element position This corresponds to an original two-dimensional spatial location. Then, iterate through all encoded visual tokens. For each visual token and its calculated original location... , and its feature vector Assignment to matrix The line, number The column position. Due to each original position In the forward mapping, there is one and only one corresponding visual token. Therefore, this assignment operation will accurately and without omission fill all the encoded visual tokens back into the matrix. In the middle. After filling, the matrix It completely restores the original two-dimensional arrangement of the visual token, where the features at each position are enhanced versions after convolutional computation.

[0065] S62. Large language models typically process input data in sequence form. Therefore, a two-dimensional feature matrix is ​​needed. Convert to a one-dimensional vector sequence. The flattening operation traverses the matrix in a specific order. All positions. The general order is row-major order, that is, starting from the first row, traversing all columns of the row from left to right, and extracting the feature vector of each position in turn; then processing the second row, and so on, until the last row. Assume Represents the first in the matrix Line 1 If the eigenvectors of a column are flattened, then the resulting one-dimensional sequence is... It can be represented as: This sequence The length is equal to the total number of visual tokens. The order of elements in the sequence strictly corresponds to the spatial topology of the visual tokens in the original image, i.e., from left to right, then from top to bottom.

[0066] S63. One-dimensional sequence obtained after flattening Each element in the matrix is ​​a feature vector. These vectors need to be fed into a projection layer, which is a trainable linear transformation whose purpose is to align the dimensions of the visual feature vectors to the word embedding space dimension of the large language model. Assume the word embedding dimension of the large language model is... The visual feature dimension is Then the projection layer is a The weight matrix. For the sequence After applying this linear projection to each visual feature vector, a final visual token sequence is obtained that can be directly concatenated with or cross-attention calculated from the text token sequence of the large language model. Thus, the multi-scale visual encoding information based on the convolutional concept is successfully injected into the large language model.

[0067] Here, the original two-dimensional spatial position corresponding to the input image refers to the coordinates of each visual token within the logical grid initially divided by the visual encoder, before any scaling or mapping operations. This position is a discrete binary index. , where i represents the index in the height direction and j represents the index in the width direction. It directly corresponds to a specific region block in the input image and is the fundamental basis for the spatial identity of the visual token. Throughout the entire method flow, regardless of the complex calculations performed on the two-dimensional visual square by mapping the visual tokens, they ultimately need to be repositioned based on this original two-dimensional spatial location to ensure that the processed features can be correctly associated with the image content they depict.

[0068] The original two-dimensional arrangement of visual tokens refers to a two-dimensional matrix-like data structure naturally formed by all visual tokens according to their original two-dimensional spatial positions in the input image, i.e., row index i and column index j. In this arrangement, the spatial adjacency relationships between visual tokens are consistent with their regional adjacency relationships in the original image. For example, position... In a two-dimensional arrangement, the visual token's neighbors above, below, to the left, and to the right correspond to the image regions above, below, to the left, and to the right, respectively. Restoring this original two-dimensional arrangement is the goal of the inverse mapping step. It is a crucial step in restoring the processed features from the abstract, standardized computational space (two-dimensional visual square) to the concrete semantic space bound to specific image content, and also provides a clear order basis for subsequent conversion to a sequence format.

[0069] An embodiment of the present invention provides a large-scale visual coding method based on convolutional thinking, which uses images containing vehicles captured on highways or roads as input images and ultimately achieves vehicle recognition. The specific implementation process is as follows: S101. Obtain an input image containing vehicles and perform visual encoding to obtain multiple visual tokens corresponding to the input image. Specifically: The input image is taken on a highway and contains one or more cars. This image is fed into a pre-trained visual encoder, such as the Vision Transformer model. The visual encoder segments the image into fixed-size patches. Each patch undergoes linear transformation and deep feature extraction, resulting in a high-dimensional feature vector. Each such feature vector is a visual token. For a vehicle image, these visual tokens represent features of different local regions in the image; for example, some visual tokens correspond to the vehicle's headlights, some to the tires, some to the body color, and others to the road or guardrail in the background. All these visual tokens together constitute a preliminary, discretized semantic representation of the vehicle image.

[0070] S102. Map all visual tokens onto a two-dimensional visual square of a predetermined size to obtain the sparse points corresponding to each visual token on the two-dimensional visual square. Specifically: First, calculate the scaling factor. Assume the visual tokens extracted from the vehicle image are logically arranged as follows: OK The dimensions of the two-dimensional visual plaza are pre-fixed. Multiply Calculate the scaling factor in the height direction. and width scaling factor Next, mapping is performed based on the scaling factor. For the position located at the... Line 1 Calculate the coordinates of a visual token representing a specific part of a vehicle (such as a door) on a two-dimensional visual square. ,in , This coordinate point is the sparse point corresponding to the visual token on a vast, fixed-size two-dimensional visual square. All visual tokens representing vehicles and background in the image are mapped in this way, forming a sparse point cloud on the two-dimensional visual square, where sparse points representing vehicle parts will be relatively clustered in a certain area.

[0071] S103. Calculate the positional correlation value between each sparse point and each sparse point in its corresponding neighborhood, specifically: A sparse point on a two-dimensional visual square corresponding to the front bumper of a vehicle. For example, first calculate the set of Euclidean distances between this point and all other sparse points in the square (such as points corresponding to car lights, grilles, and background trees), and then calculate the standard deviation of this set of distances. .according to Determine a Centered on, with side length as The square neighborhood of the given information. Other sparse points within this neighborhood, such as a sparse point representing a nearby car light. It is considered to be related to the target point. They are spatially correlated. The positional correlation between them can be calculated using a two-dimensional Gaussian distribution function. : in, and These are sparse points and Coordinates on a two-dimensional visual plaza. The value quantifies the spatial influence of the headlight point on the bumper point; the closer the distance, the greater the influence. This calculation is performed for every point in the neighborhood.

[0072] S104. Normalize all calculated positional correlation values ​​to obtain the convolution kernel corresponding to each sparse point. Specifically: Continuing with the sparse points corresponding to the aforementioned vehicle bumpers For example, summing the position correlation values ​​of all sparse points in its neighborhood yields... For the sparse point representing the headlights within the neighborhood... Its contribution score .this This is a weight value between 0 and 1, representing the proportion of the headlight features' contribution to the bumper feature update during feature aggregation. The contribution scores of all neighboring points (including themselves), according to the points' relative positions on the two-dimensional visual square. The coordinate offsets are organized into a weight matrix. This weight matrix is ​​used for the bumper points. The convolutional kernel is dynamically generated. It assigns high weights to nearby vehicle component points and low weights to distant background points. The system generates such a unique convolutional kernel for each sparse point in the image (i.e., each visual token).

[0073] S105. Using the convolution kernel corresponding to each sparse point, perform convolution calculation on each sparse point in the 2D visual square to obtain the encoded visual token. Specifically: Perform feature aggregation for each sparse point. For example, for bumper points... The convolution kernel is applied to the original visual token feature vectors carried by all sparse points in its neighborhood. The calculation method is a weighted summation: .in, Neighboring points The original feature vector of (e.g., headlight point). After calculation, the new feature vector of the bumper point. It no longer only includes the texture and color information of the bumper itself, but also incorporates feature information from spatially adjacent components such as headlights and grilles. This gives the visual token features of the "bumper" a stronger contextual awareness. After all sparse points have undergone this type of convolution calculation, a batch of encoded visual tokens is obtained, and their features are enhanced by incorporating surrounding information.

[0074] S106. Reverse map each encoded visual token from the two-dimensional visual square back to its original two-dimensional spatial position in the input image to restore the original two-dimensional arrangement of each visual token. Then, flatten each restored visual token into a one-dimensional sequence for interface with the large language model. Specifically: First, inverse mapping is performed. For each sparse point whose features have been enhanced in the 2D visual square, according to the formula... and Calculate its original row and column indexes. and put its feature vector back into a OK The corresponding positions of the columns in the two-dimensional matrix are then determined. In this way, all the enhanced features are arranged according to the original visual token space structure of the vehicle image. Then, the matrix is ​​flattened, and all feature vectors in this two-dimensional matrix are arranged sequentially in row-major order to form a one-dimensional sequence. This sequence fully preserves the enhanced features of each part of the vehicle and their spatial order. Finally, this visual token sequence is transformed through a projection layer to match the dimension of the word embedding space of the large language model, thus preparing it for docking with the large language model for subsequent joint understanding and reasoning.

[0075] After being processed by the above encoding method, the visual token sequence is input into a large language model, enabling various advanced vehicle recognition applications. The first type of application is fine-grained visual question answering. Users can ask the system text questions about images, such as "What color is the leftmost vehicle in the image?" or "What model is this car?" The large language model jointly analyzes the processed visual sequence and the question text sequence. Because the visual features have undergone convolutional encoding, the model has a more accurate grasp of the various parts of the vehicle and their relative positions, thus outputting accurate and detailed answers, such as "The leftmost vehicle is a red SUV." The second type of application is automated traffic scene description and report generation. The system can automatically analyze input highway surveillance images and generate structured natural language descriptions. For example, the output might be: "The image was taken during the day on a dry road. A blue truck is driving in the middle lane in the center of the image, and a white sedan is about 20 meters behind it on its left. No vehicles are seen in the emergency lane on the right." This description combines vehicle detection, attribute recognition (color, type), and spatial relationship judgment. This structured information can be automatically populated into traffic accident reports, traffic flow statistics tables, or used to trigger alerts for abnormal events (such as illegal parking).

[0076] The overall method of this invention is as follows: Given an image understanding dataset or a video understanding dataset, the dataset includes two parts: image or video data and text data describing the visual data. Preprocessing is performed on the image or video data; for images, preprocessing includes size scaling; for videos, preprocessing includes size scaling and second-level frame extraction, such as... Figure 3 As shown, the processed image or video frames are fed into a visual encoder for feature extraction. The visual encoder converts the input image into a series of visual tokens. For example... Figure 3 As shown, the process includes starting with the original image, resizing it, and then generating a visual token by a visual encoder. After obtaining the visual token, it needs to be fused with the multi-scale visual position encoding calculation based on the convolutional concept proposed in this invention. This calculation process is completed by the visual fusion module, and then the data is flattened into one-dimensional data by a flatten layer. A projection layer is used to align the flattened one-dimensional visual feature data to the word embedding space of the large language model. During training, the visual feature sequence and the corresponding text description sequence are input into the model. The cross-entropy loss function is used to calculate the difference between the predicted text and the true description, realize gradient backpropagation, update the model parameters, and complete the model training. The entire process is as follows. Figure 3 As shown, starting from video frame extraction, the video is processed by a visual encoder and the fusion module proposed in this invention, and then connected to a large language model through a projection layer. Finally, the large language model generates a description such as "There are two kangaroos in this video".

[0077] To better illustrate the large-model visual modality position encoding method based on convolutional concepts proposed in this invention, the invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0078] First, a two-dimensional visual square is pre-defined. The size of the two-dimensional visual square is fixed according to the model design and will not change during subsequent training and inference. Then, we map images of any scale onto this two-dimensional visual square. The specific implementation method is as follows: the scaling dimensions in the height and width dimensions are calculated sequentially, and the first and last visual tokens in each direction of the original image are mapped to the first and last positions in the corresponding directions of the two-dimensional visual square. Then, the visual tokens in each direction of the middle part are multiplied by the scaling factor to calculate the mapped coordinates. At this point, we obtain a uniform projection of an image of any scale onto this two-dimensional visual square. This is the "multi-scale adaptation" mentioned in this invention.

[0079] The specific implementation method is as follows: Assume that the four vertices of the original image are top left, top right, bottom left, and bottom right, respectively. Figure 4 As shown, both the original image image1 and the original images image2 of different sizes need to be mapped to the same 2D visual square suq, requiring the calculation of a scaling factor. The scaling factor Sh in the height direction of the original image image1 is the number of visual tokens Ktokens in the height direction minus 1, divided by the length Hsuq in the height direction of the 2D visual square, i.e.: In this formula, This represents the scaling factor in the height direction. This represents the number of visual tokens in the height direction of the original image. It represents the dimension of the two-dimensional visual square in the height direction.

[0080] Similarly, the scaling factor in the width direction Therefore, the formula is calculated as follows: ,in It is the number of visual tokens in the width direction. This refers to the width dimension of the two-dimensional visual square. The mapped coordinates can then be obtained by multiplying the visual token coordinates from the original image by a scaling factor and the length of the corresponding direction on the two-dimensional visual square. Clearly, all points are uniformly distributed. For example... Figure 4 As shown, points o, p, q, and s on the image are uniformly mapped onto the two-dimensional visual square.

[0081] Next, we need to perform crucial calculations. On this projected 2D visual square, many sparse points will be distributed; these points are the visual tokens corresponding to the image. We can observe that the sparsity of points projected onto the 2D visual square varies depending on the image size. So, how do we measure the positional relationships between visual tokens at different sparsity levels? This is where we introduce the Gaussian distribution function on the 2D plane: In this formula, Indicates coordinates The Gaussian distribution function value at that location. It represents pi (π). This represents the standard deviation of the Gaussian distribution. Represents the natural constant.

[0082] A correlation value is obtained by calculating the positional relationships within the neighborhood of a sparse point, where the correlation value is higher for neighbors closer to the central sparse point. Since the sparse points are uniformly distributed in this two-dimensional visual square, we only need to calculate the correlation values ​​of other points within the neighborhood of a single point, and then normalize them for use as the convolution kernel in subsequent convolution calculations.

[0083] Now that we have the mapped image and the corresponding convolution kernel values, we will perform convolutional encoding calculations. First, we need to determine the size of the convolution kernel. We adopt the 3σ principle, directly taking points in (μ-3σ, μ+3σ) as the convolution kernel size, where μ is the mean distance. Next, we perform a convolution operation on each sparse point, as follows: Figure 5 As shown, sparse points on the 2D visual square Suq are convolved using kernels ker1 and ker2. Note that some neighboring points near the boundary are replaced with 0, ensuring the final calculated size remains unchanged. Finally, all points are inversely mapped back to the original image, flattened, and then used to replace the text tokens. This incorporates feature information from the neighborhood of the image visual tokens, enhancing the model's ability to perceive visual images.

[0084] In this calculation, the peak value of the image represents the contribution of the current point, and the correlation values ​​of other points in the neighborhood are calculated using their absolute distances from the current point as the x and y values. Note that we only consider points within 3σ for calculation, where σ is the standard deviation of the average distance between the current point and all other points. Figure 6 The three-dimensional Gaussian distribution plot shown below has peaks representing the center point: In this formula, The standard deviation of the distance. This represents the absolute distance between the current point and the i-th other point. This represents the average of all distances. Indicates distance The number of.

[0085] The normalization operation is performed after obtaining the relevant values ​​of all points within 3σ, and a normalization calculation is performed on these values. Specifically, it assumes there is a set of points within 3σ. (n represents the number), where the points The contribution value to the current point is Then point Contribution score for: In this formula, Point The contribution score to the current center point. Point Location-related values. Represents a set of points Positional correlation values ​​of all n points Perform summation.

[0086] Then multiply the feature vectors of all visual tokens within 3σ by their corresponding contribution scores. Then, by summing the results, we obtain the new encoded value for the current point. At this point, the mathematical information of each point contains the features of the surrounding points.

[0087] A visual encoder is a deep neural network model used to convert input image or video frames into a series of feature vectors with semantic information, i.e., visual tokens. For example... Figure 2 and Figure 3 As shown, the visual encoder receives a resized image, extracts spatial and semantic features through multiple convolutional or Transformer layers, and outputs a two-dimensional grid or sequence of visual tokens. It serves as a bridge connecting raw pixel data with advanced semantic understanding, and the quality of its output directly impacts the performance of subsequent positional encoding and model understanding.

[0088] The projection layer is a trainable linear transformation layer or a small neural network. Its main function is to map the dimension of the visual token feature vector output by the visual encoder to the word embedding space dimension of the large language model. For example... Figure 3 As shown, the projection layer is located between the visual encoder and the large language model, serving as a modality alignment layer. Through the projection layer, features from the visual modality and word vectors from the text modality can interact and be fused in a common, comparable feature space, enabling the large language model to process and understand two different types of information simultaneously.

[0089] A tokenizer is a standard component in natural language processing used to segment the input raw text string into a series of discrete basic units, or text tokens. These basic units can be words, subwords, or characters. For example... Figure 3 As shown, in the text branch, the user's input question, such as "describe this video," first passes through a tokenizer and is converted into a sequence of text tokens that the model can process. The tokenizer's built-in vocabulary defines all possible text tokens, ensuring that text information is input into the model in a structured way and combined with the processed visual token sequence.

[0090] This invention solves the problem of calculating the relationships between visual features of different images or video frames in a unified space by projecting images or video frames of different sizes onto a fixed-size two-dimensional visual square. For example... Figure 4As shown, regardless of the input image size, by calculating scaling factors in the height and width directions, all visual tokens can be uniformly mapped onto a two-dimensional visual square of a predetermined size. This step achieves adaptive normalization of multi-scale inputs, laying the foundation for subsequent consistent, scale-independent calculation of positional relationships. The visual encoding method based on convolution proposed in this invention can efficiently capture the potential feature relationships between each visual token and its surrounding visual tokens. This method first uses a two-dimensional Gaussian distribution function to calculate the positional correlation value between each sparse point and other sparse points in its neighborhood. The neighborhood is defined by a square region with a side length of 6σ centered on that point. Then, these positional correlation values ​​are normalized to generate a convolution kernel with normalized weights corresponding to each sparse point. Next, using the convolution kernel corresponding to each sparse point, convolution calculations are performed on each sparse point on the two-dimensional visual square. This calculation process ensures that the new feature vector of each visual token incorporates the feature information of other visual tokens in its spatial neighborhood. Simultaneously, after encoding, each encoded visual token is inversely mapped from the two-dimensional visual square back to its original two-dimensional spatial position in the input image to restore the original two-dimensional arrangement of the visual tokens. After flattening, these visual tokens can continue to perform attention calculations with subsequent text tokens. This method can effectively capture the two-dimensional spatial correspondences contained in visual features and directly adapt to existing attention calculation architectures based on one-dimensional sequences. This invention can effectively fuse the spatial feature relationships between visual tokens. The multi-scale fusion method adopted can flexibly handle images or video frames of different sizes. The convolution idea combined with two-dimensional Gaussian distribution features can effectively capture the local contextual information of visual features, enhance the model's ability to capture details of targets within the image and perceive spatial structure, thereby enabling large language models to output visual descriptive information containing richer details. Compared with current positional encoding methods, this invention introduces a complete processing flow in the visual encoding process, including mapping to the two-dimensional visual square, fusion of neighborhood features based on Gaussian distribution and convolution kernels, and inverse mapping to restore the original arrangement. After fusing rich 2D spatial feature information, the visual tokens are mapped back to the original image size and flattened, perfectly adapting to the attention calculation of subsequent text tokens. Since the convolution calculation is performed on a fixed and relatively sparse 2D visual square, and the convolution kernel size is dynamically determined by the data-driven standard deviation, the additional computational cost is controllable and efficient, and its significant impact on the overall model computational efficiency can be basically ignored.

[0091] In the above embodiments, although the steps are numbered S1, S2, etc., they are only specific embodiments given by the present invention. Those skilled in the art can adjust the execution order of S1, S2, etc. according to the actual situation. The scheme after adjusting the order is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0092] like Figure 7 As shown, an embodiment of the present invention provides a large-scale visual coding system 200 based on the idea of ​​convolution, which includes an acquisition module 201, a mapping module 202, a position correlation value calculation module 203, a normalization processing module 204, a convolution calculation module 205, and a recovery and flattening module 206. The acquisition module 201 is used to: acquire the input image and perform visual encoding to obtain multiple visual tokens corresponding to the input image; The mapping module 202 is used to: map all visual tokens onto a two-dimensional visual square of a predetermined size, and obtain the sparse points on the two-dimensional visual square corresponding to each visual token; The location correlation value calculation module 203 is used to: calculate the location correlation value between each sparse point and each sparse point in its corresponding neighborhood; The normalization processing module 204 is used to: normalize all the calculated position-related values ​​to obtain the convolution kernel corresponding to each sparse point; The convolution calculation module 205 is used to: use the convolution kernel corresponding to each sparse point to perform convolution calculation on each sparse point in the two-dimensional visual square to obtain the encoded visual token; The recovery and flattening module 206 is used to: reverse map each encoded visual token from the two-dimensional visual square back to the original two-dimensional spatial position corresponding to the input image, so as to restore the original two-dimensional arrangement of each visual token, and flatten each restored visual token into a one-dimensional sequence for docking with the large language model.

[0093] Optionally, in the above technical solution, the mapping module 202 is specifically used for: Calculate the scaling factors between the number of visual tokens in the height direction and the number of visual tokens in the width direction of the input image and the corresponding directional dimensions of the two-dimensional visual square; Based on all scaling factors, each visual token in the input image is uniformly mapped onto a two-dimensional visual square to obtain the sparse points corresponding to each visual token on the two-dimensional visual square.

[0094] Optionally, in the above technical solution, the neighborhood of any sparse point refers to a square region with a side length of 6σ centered on the sparse point, where σ is the standard deviation of the distance between the sparse point and other sparse points.

[0095] Optionally, in the above technical solution, the location correlation value calculation module 203 is specifically used to: calculate the location correlation value between each sparse point and each sparse point in the corresponding neighborhood using a two-dimensional Gaussian distribution function.

[0096] It should be noted that the beneficial effects of the multi-scale adaptive large model visual coding system 200 based on convolutional thinking provided in the above embodiments are the same as those of the multi-scale adaptive large model visual coding method based on convolutional thinking, and will not be repeated here. Furthermore, the system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to the actual situation to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, and will not be repeated here.

[0097] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned large model visual encoding methods based on the concept of convolution.

[0098] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-mentioned large model visual encoding methods based on the concept of convolution.

[0099] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A large-scale visual encoding method based on convolutional concepts, characterized in that, include: The input image is acquired and visually encoded to obtain multiple visual tokens corresponding to the input image; Map all visual tokens onto a two-dimensional visual square of a predetermined size to obtain the sparse points on the two-dimensional visual square corresponding to each visual token; Calculate the positional correlation value between each sparse point and each sparse point in its corresponding neighborhood; The calculated positional correlation values ​​are normalized to obtain the convolution kernel corresponding to each sparse point; Using the convolution kernel corresponding to each sparse point, convolution calculation is performed on each sparse point on the two-dimensional visual square to obtain the encoded visual token; Each encoded visual token is inversely mapped from the two-dimensional visual square back to its original two-dimensional spatial position corresponding to the input image to restore the original two-dimensional arrangement of each visual token. Each restored visual token is then flattened into a one-dimensional sequence for interface with the large language model.

2. The large-scale visual encoding method based on convolutional thinking for multi-scale adaptation as described in claim 1, characterized in that, Map all visual tokens onto a two-dimensional visual square of a predetermined size to obtain the sparse points corresponding to each visual token on the two-dimensional visual square, including: Calculate the scaling factors between the number of visual tokens in the height direction and the number of visual tokens in the width direction of the input image and the corresponding directional dimensions of the two-dimensional visual square; Based on all scaling factors, each visual token in the input image is uniformly mapped onto the two-dimensional visual square to obtain sparse points on the two-dimensional visual square corresponding to each visual token.

3. A large-scale visual encoding method based on convolutional thinking for multi-scale adaptation, as described in claim 1 or 2, characterized in that... The neighborhood of any sparse point refers to a square region with a side length of 6σ centered on the sparse point, where σ is the standard deviation of the distance between the sparse point and other sparse points.

4. A large-scale visual encoding method based on convolutional thinking for multi-scale adaptation, as described in claim 1 or 2, characterized in that... Calculate the positional correlation between each sparse point and each sparse point in its corresponding neighborhood, including: The positional correlation between each sparse point and each sparse point in its corresponding neighborhood is calculated using a two-dimensional Gaussian distribution function.

5. A large-scale visual encoding system based on convolutional concepts, characterized in that, It includes an acquisition module, a mapping module, a position-related value calculation module, a normalization processing module, a convolution calculation module, and a recovery and flattening module; The acquisition module is used to: acquire an input image and perform visual encoding to obtain multiple visual tokens corresponding to the input image; The mapping module is used to: map all visual tokens onto a two-dimensional visual square of a predetermined size, and obtain the sparse points corresponding to each visual token on the two-dimensional visual square; The location correlation value calculation module is used to: calculate the location correlation value between each sparse point and each sparse point in its corresponding neighborhood; The normalization processing module is used to: normalize all the calculated position-related values ​​to obtain the convolution kernel corresponding to each sparse point; The convolution calculation module is used to: use the convolution kernel corresponding to each sparse point to perform convolution calculation on each sparse point on the two-dimensional visual square to obtain the encoded visual token; The recovery and flattening module is used to: reverse map each encoded visual token from the two-dimensional visual square back to the original two-dimensional spatial position corresponding to the input image, so as to restore the original two-dimensional arrangement of each visual token, and flatten each restored visual token into a one-dimensional sequence for docking with the large language model.

6. The large-scale visual coding system based on convolutional thinking according to claim 5, characterized in that, The mapping module is specifically used for: Calculate the scaling factors between the number of visual tokens in the height direction and the number of visual tokens in the width direction of the input image and the corresponding directional dimensions of the two-dimensional visual square; Based on all scaling factors, each visual token in the input image is uniformly mapped onto the two-dimensional visual square to obtain sparse points on the two-dimensional visual square corresponding to each visual token.

7. A large-scale visual coding system based on convolutional thinking for multi-scale adaptation, as described in claim 5 or 6, characterized in that, The neighborhood of any sparse point refers to a square region with a side length of 6σ centered on the sparse point, where σ is the standard deviation of the distance between the sparse point and other sparse points.

8. A large-scale visual coding system based on convolutional thinking for multi-scale adaptation, as described in claim 5 or 6, characterized in that... The location correlation value calculation module is specifically used to: calculate the location correlation value between each sparse point and each sparse point in its corresponding neighborhood using a two-dimensional Gaussian distribution function.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the large model visual encoding method based on the convolution concept of any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the large model visual encoding method based on the concept of convolution, as described in any one of claims 1 to 4.

Images