Multimodal large model low-resource modality adaptive learning method

Abstract

This invention provides a low-resource modality adaptive learning method for multimodal large models, relating to the field of data processing technology. The method includes: dividing polygons in a baseline topological unit into convex and concave regions based on convexity / concavity identification results; segmenting the baseline topological unit to generate multi-scale topological partitions; mapping low-resource modality data streams to the corresponding domains of the multi-scale topological partitions and extracting multi-resolution structural features; calculating geometric feature parameters based on each polygon in the multi-scale topological partitions; matching the multi-resolution structural features with high-resource modality data streams to generate heterogeneous feature topologies; optimizing the heterogeneous feature topologies to obtain deeply optimized model parameter configurations; validating the deeply optimized model parameter configurations; adjusting the model hyperparameter set to obtain a low-resource modality adaptive multimodal large model. This invention effectively improves the feature extraction accuracy and adaptive adaptation capability of multimodal large models for low-resource modalities.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a multimodal large-model low-resource modal adaptive learning method. Background Technology

[0002] With the development of information technology, it has become commonplace for electronic digital data processing systems to process modal data (such as text, images, 3D point clouds, etc.) from different sources and with different structures. How to enable computer systems to effectively and collaboratively utilize these multimodal data is the key to improving their intelligence level. However, existing data processing architectures often face problems such as low computational efficiency and insufficient model generalization ability when dealing with complex heterogeneous data with unbalanced resources between modalities (i.e., huge differences in data quality and scale between different modalities). Although some methods attempt to process low-resource modalities independently, their feature extraction is often designed for high-resource modalities and lacks the ability to adaptively model the inherent, fine-grained structural features of low-resource modalities. For example, for 3D point cloud or mesh data, the surface concavity, concavity, continuity, and other topological structures are core features, but general data processing cannot express these structures explicitly and computationally, resulting in insufficient discriminative power of the extracted features. Summary of the Invention

[0003] The technical problem to be solved by this invention is to provide an adaptive learning method for low-resource modalities in multimodal large models, which effectively improves the feature extraction accuracy and adaptive adaptation capability of multimodal large models for low-resource modalities.

[0004] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0005] Firstly, a multimodal large-model low-resource modality adaptive learning method, the method comprising:

[0006] Step 1: Obtain multimodal input data containing both high-resource modal data and low-resource modal data;

[0007] Step 2: Perform deep modal identification and decoupling operations on the multimodal input data to obtain modal decoupling results containing high-resource modal data streams and low-resource modal data streams;

[0008] Step 3: Analyze the low-resource modal data stream, construct a geometric reference datum, establish a reference topology unit on the geometric reference datum, calculate the cross product of the vectors of adjacent edges at each vertex based on the polygon vertex coordinate sequence of the reference topology unit, and obtain the concavity / convexity identification result of each vertex based on the sign of the cross product.

[0009] Step 4: Based on the concavity and convexity identification results, the polygons in the reference topological unit are divided into convex and concave regions, and the reference topological unit is segmented to generate multi-scale topological partitions. The low-resource modal data stream is mapped to the corresponding domain of the multi-scale topological partitions and multi-resolution structural features are extracted.

[0010] Step 5: Calculate the geometric feature parameters of each polygon in the multi-scale topological partition, match the multi-resolution structural features with the high-resource modal data stream, and generate a heterogeneous feature topology.

[0011] Step 6: Optimize the heterogeneous feature topology to obtain a deeply optimized model parameter configuration;

[0012] Step 7: Verify the deeply optimized model parameter configuration, adjust the model hyperparameter set, and obtain a low-resource modal adaptive multimodal large model.

[0013] In a second aspect, a computing device includes:

[0014] One or more processors;

[0015] A storage device for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the method.

[0016] Thirdly, a computer-readable storage medium storing a program that, when executed by a processor, implements the method.

[0017] The above-described solution of the present invention has at least the following beneficial effects:

[0018] By constructing geometric reference planes, baseline topological units, and multi-scale topological partitioning, and combining polygon vertex concavity / convexity judgment with multi-resolution structural feature extraction, low-resource modal data streams are transformed into structured, fine-grained geometric feature representations. This effectively compensates for the deficiencies of sparse and incomplete information in low-resource modal data, avoids problems such as fuzzy feature extraction and weak generalization ability, and significantly improves the accuracy and effectiveness of feature representation in low-resource modalities. Through structured segmentation of baseline topological units, accurate judgment of concavity / convexity identification, and targeted optimization of heterogeneous feature topology, model parameter adjustment becomes more directional and scientific, avoiding problems such as blind parameter optimization and slow convergence, shortening the model training cycle. At the same time, through hyperparameter adjustment in the validation phase, the stability and reliability of the model inference process are ensured. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the multimodal large-model low-resource modal adaptive learning method provided in an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram of the process of optimizing heterogeneous feature topology to obtain deeply optimized model parameter configuration, provided by an embodiment of the present invention. Detailed Implementation

[0021] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0022] like Figure 1 As shown, embodiments of the present invention propose a multimodal large-model low-resource modality adaptive learning method, the method comprising the following steps:

[0023] Step 1: Obtain multimodal input data containing both high-resource modal data and low-resource modal data;

[0024] Step 2: Perform deep modal identification and decoupling operations on the multimodal input data to obtain modal decoupling results containing high-resource modal data streams and low-resource modal data streams;

[0025] Step 3: Analyze the low-resource modal data stream, construct a geometric reference datum, establish a reference topology unit on the geometric reference datum, calculate the cross product of the vectors of adjacent edges at each vertex based on the polygon vertex coordinate sequence of the reference topology unit, and obtain the concavity / convexity identification result of each vertex based on the sign of the cross product.

[0026] Step 4: Based on the concavity and convexity identification results, the polygons in the reference topological unit are divided into convex and concave regions, and the reference topological unit is segmented to generate multi-scale topological partitions. The low-resource modal data stream is mapped to the corresponding domain of the multi-scale topological partitions and multi-resolution structural features are extracted.

[0027] Step 5: Calculate the geometric feature parameters of each polygon in the multi-scale topological partition, match the multi-resolution structural features with the high-resource modal data stream, and generate a heterogeneous feature topology.

[0028] Step 6: Optimize the heterogeneous feature topology to obtain a deeply optimized model parameter configuration;

[0029] Step 7: Verify the deeply optimized model parameter configuration, adjust the model hyperparameter set, and obtain a low-resource modal adaptive multimodal large model.

[0030] In this embodiment of the invention, by constructing a geometric reference datum, a benchmark topological unit, and multi-scale topological partitioning, and combining polygon vertex concavity / convexity judgment with multi-resolution structural feature extraction, the low-resource modal data stream is transformed into a structured, fine-grained geometric feature representation. This effectively compensates for the deficiencies of sparse and incomplete information in low-resource modal data, avoids problems such as fuzzy feature extraction and weak generalization ability, and significantly improves the accuracy and effectiveness of feature representation of low-resource modalities. Through the structured segmentation of the benchmark topological unit, the accurate judgment of concavity / convexity identification, and the targeted optimization of heterogeneous feature topology, the adjustment of model parameters becomes more directional and scientific, avoiding the problems of blind parameter optimization and slow convergence, shortening the model training cycle. At the same time, through hyperparameter adjustment in the verification stage, the stability and reliability of the model inference process are ensured.

[0031] In a preferred embodiment of the present invention, step 1, acquiring multimodal input data containing high-resource modal data and low-resource modal data, specifically includes: a data acquisition server pre-configured with a text data acquisition unit, an image data acquisition unit, a 3D point cloud data acquisition unit, and an audio data acquisition unit, each unit corresponding to a different data source; the text data acquisition unit accesses a preset text database via the Hypertext Transfer Protocol, acquiring text data such as news articles, product descriptions, and user comments stored in the database, filtering text with a single character count between ten and one thousand twenty-four characters during acquisition to ensure that the acquired text data is free of garbled characters and uniformly encoded in UTF-8 character encoding format, while using the Word2Vec model to pre-encode the text, with the vector dimension set to 500; the image data acquisition unit receives two-dimensional image data from a 2-megapixel high-definition camera or image database via a gigabit Ethernet interface, with the acquired image resolution range set to 256 x 256 pixels to 1024 x 1024 pixels, and the image format uniformly set to portable network graphics. The system employs a grayscale mean method to remove images with abnormal brightness (i.e., images with a grayscale mean not between 50 and 200), an edge detection algorithm to remove blurry images (edge ​​sharpness below 0.6), and an occlusion ratio calculation to remove images with an occlusion area exceeding 30%. The 3D point cloud data acquisition unit connects to the LiDAR device via a USB 3.0 interface to acquire 3D point cloud data of the scene or objects. The LiDAR sampling frequency is controlled at 30 frames per second, with the number of points per frame controlled between 1024 and 4096, using a distance threshold. The system filters out invalid point clouds that are more than 100 meters away from the lidar in real time, and removes isolated noise points (points with fewer than 5 neighboring points) through a statistical filtering algorithm. The audio data acquisition unit receives audio data from a 48kHz sampling rate microphone input or an audio database through a PCI-E audio acquisition card. The sampling rate is set to 44,100 Hz, the sampling bit depth is 16 bits, and the audio duration range is from one second to ten seconds. The sound pressure level detection method is used to remove audio data with background noise exceeding 40 decibels, and the clarity of high-frequency signals is improved through a pre-emphasis filter.

[0032] After data collection, all data were initially screened to identify high-resource modal data and low-resource modal data. The criteria for high-resource modal data were: a single modal data volume of no less than 100,000 records, a feature dimension of no less than 512 dimensions, and data integrity (i.e., no missing values) of no less than 95%. The criteria for low-resource modal data were: a single modal data volume of no more than 10,000 records, or a feature dimension of no more than 128 dimensions, or data integrity (i.e., no missing values) of no more than 80%. After screening, the high-resource modal data and low-resource modal data that met the requirements were integrated into multimodal input data.

[0033] This embodiment ensures that the acquired multimodal data has a unified format and quality standard by setting dedicated acquisition programs and parameters for different modal data, effectively reducing format conversion and noise interference problems in subsequent data processing.

[0034] In a preferred embodiment of the present invention, step 2, performing deep modal identification and decoupling operations on the multimodal input data to obtain a modal decoupling result containing high-resource modal data streams and low-resource modal data streams, may include:

[0035] Step 201: Perform modality type identification on the multimodal input data to generate modality type identification results; based on the modality type identification results, segment the multimodal input data into multiple modality data subsets corresponding to different modality types. Specifically, after the modality identification unit is started, it first reads the multimodal input data stored in the data acquisition server through the fiber optic channel, and performs a two-dimensional analysis on the format features and dimensional features of each data item; for text data, the format feature is a UTF-8 encoded stream containing characters and punctuation marks, and the dimensional feature is the number of characters (10 to 1024); the image data format feature is a matrix containing RGB three-channel pixels (pixel values ​​0 to 255), and the dimensional feature is width × height × 3 (256 × ...). The data format characteristics of 3D point cloud data are floating-point data sets containing X, Y, and Z three-dimensional coordinate values, and the dimensional characteristic is the number of points × 3 (1024×3 to 4096×3); the audio data format characteristics are integer data sequences containing time-domain waveforms, and the dimensional characteristic is the number of sampling points (44100 to 441000). The modal recognition unit compares the features of each data with the standard feature templates of each modality dimension by dimension through preset feature template matching rules. If the matching degree exceeds 95%, the corresponding modal type identification result is generated for the data. The identification results include four types: text, two-dimensional image, three-dimensional point cloud, and audio. Data that fails to match is marked as invalid data and stored separately.

[0036] After generating modal type identification results, the modal recognition unit triggers a data segmentation process, dividing the multimodal input data into multiple modal data subsets based on the identification results. During segmentation, it is ensured that each modal data subset contains only a single type of modal data. For example, all data identified as text is assigned to the text modal data subset, and all data identified as two-dimensional images is assigned to the image modal data subset. At the same time, a counter is used to count the amount of data in each subset. If the original data volume of a certain modality is less than one thousand, a low-resource identifier is directly added to that subset and it is retained. If the data volume exceeds one thousand, it is assigned to the corresponding modal data subset and a regular identifier is added.

[0037] Step 202: Perform deep feature extraction on multiple modal data subsets to generate corresponding multiple modal feature maps; classify the multiple modal feature maps according to a predefined resource level threshold to obtain a high-resource modal feature group and a low-resource modal feature group, specifically including: execute the deep feature extraction process, and use exclusive feature extraction models for different modal data subsets; for the text modal data subset, use a bidirectional long short-term memory network to extract semantic features. The construction process of this network is as follows: the input layer uses 500-dimensional neurons, corresponding to the dimension of the word embedding vector after encoding the corresponding text by the Word2Vec model; the hidden layer is set with two layers, the number of hidden units in the first layer is 256, and the second layer is 512, and the activation function uses the hyperbolic tangent function tanh(x); the output layer is a 256-dimensional fully connected layer, and uses the Sigmoid activation function to output semantic features; during network training, select the publicly available IMDB movie review dataset as the training data (including 50,000 labeled samples), set the training batch size to 32, set the initial learning rate to 0.001, set the number of iterations to 50 to 100 rounds, and use the cross-entropy loss function to calculate the loss value. The expression of the loss function is , where is the loss value, which is used to measure the difference between the model prediction result and the true label of the sample, N is the number of batch samples, is the true label of the sample, is the model prediction probability; adjust the network parameters through the Adam optimizer, calculate the validation set loss value after each round of iteration, and stop training when the validation set loss value fluctuates less than 0.001 for three consecutive rounds and stabilizes below 0.05; after training is completed, the text modal data subset is first segmented by the Jieba segmentation tool, stop words such as "de" and "le" are removed, and then converted into 500-dimensional word embedding vectors, input into the trained bidirectional long short-term memory network, and 256-dimensional semantic features of each data are obtained through forward propagation calculation. All semantic features are arranged in rows and integrated into a text modal feature mapping matrix (number of samples × 256 dimensions).

[0038] Visual features were extracted from a subset of image modality data using a convolutional neural network. The network construction process was as follows: the input layer was a 256×256×3 image tensor (RGB three channels); 3 to 8 convolutional layers were set, with the first layer having a kernel size of 3×3 and 32 kernels, the second layer having a kernel size of 5×5 and 64 kernels, and subsequent layers increasing in size from 7×7 with 128 to 256 kernels, and the stride of all convolutional layers was set to 1, with zero padding (0 pixels for edge padding); pooling layers were alternated with convolutional layers, using 2×2 max pooling, with a stride of 1. 2. Preserve feature texture information; the fully connected layer consists of two layers, the first layer is 1000-dimensional and the second layer is 512-dimensional, using the ReLU activation function; the output layer is 256-dimensional, using the Softmax activation function; during training, the publicly available CIFAR-10 dataset (containing 60,000 32×32 images, scaled to 256×256 before use) is used, with a batch size of 16, an initial learning rate of 0.001, and 80 to 120 iterations. The cross-entropy loss function is used, with the same expression as the text modality: The optimizer uses a stochastic gradient descent optimizer with a momentum coefficient of 0.9. After training, the image modal data subset is first normalized to 256×256 pixels by bilinear interpolation, converted into tensors and input into the convolutional neural network. The 512-dimensional features of the second fully connected layer are extracted and dimensionality reduced to obtain 256-dimensional visual features, which are then integrated into an image modal feature mapping matrix (number of samples × 256 dimensions).

[0039] Spatial features were extracted from a subset of 3D point cloud modal data using the PointNet network. The network construction process was as follows: the input layer received the 3D coordinates (X, Y, Z) of 1024 points; the sampling layer used the farthest point sampling algorithm to select 512 sampling points from the 1024 points, with a sampling interval of 0.5 units; the grouping layer used the ball query algorithm with a sphere radius of 0.5 units, selecting 36 sub-points around each sampling point to construct a local region; the feature extraction layer consisted of four 1D convolutional layers with kernel numbers of 64, 128, 256, and 512 respectively, and the activation function was ReLU; the output layer was a 256-dimensional fully connected layer; during training, the publicly available ModelNet40 dataset (containing 12311 3D model point clouds) was used, with a batch size of 8, an initial learning rate of 0.001, 100 to 150 iterations, and the cross-entropy loss function. The optimizer used was the Adam optimizer, with the weight decay coefficient set to 0.0001. After training, the 3D point cloud data subset was sampled and unified to 1024 points, which were then input into the PointNet network. The 512-dimensional features output by the feature extraction layer were extracted and reduced to 256-dimensional spatial features by PCA, and then integrated into a point cloud modal feature mapping matrix (number of samples × 256 dimensions).

[0040] The audio modal data subset was used to extract acoustic features using Mel frequency cepstral coefficients. The extraction process was as follows: First, the audio data was framed using a Hanning window with a frame length of 25ms and a frame shift of 10ms, converting it into a short-time Fourier transform spectrum. Then, the spectrum was converted into a Mel spectrum using 13 to 40 Mel filter banks, with the center frequency range of the Mel filters corresponding to 20 Hz to 20000 Hz and a sampling frequency of 44100 Hz. After taking the natural logarithm of the Mel spectrum, a discrete cosine transform was performed to extract the first 13 to 40 coefficients as Mel frequency cepstral coefficients, i.e., the acoustic features. The discrete cosine transform formula is: ,in Here, m is the cepstral coefficient of the m-th Mel frequency (m is the coefficient index, ranging from 1 to 40), and M is the number of spectral points. The value is the Mel spectrum value at the nth spectral point. The acoustic features of all audio data are integrated into an audio modal feature mapping matrix (sample number × 13 to 40 dimensions). Each modal feature mapping matrix has a dimension range of 128 to 1024 dimensions, with rows representing data samples and columns representing feature dimensions. Floating-point numbers are used for storage, with a precision of 6 decimal places. After feature extraction, a predefined set of resource level thresholds is established: a data volume threshold of 10,000 records, a feature dimension threshold of 128 dimensions, and a data integrity threshold of 80% (the percentage of samples without missing values). The classification unit obtains the dataset information corresponding to each modal feature mapping through the data reading interface, including sample counts, feature dimension statistics, and the calculation results of the percentage of missing values. Modal feature mappings that meet the criteria of ≥10,000 records, ≥128 dimensions, and ≥80% data integrity are assigned to the high-resource modal feature group; those that do not meet any of these conditions are assigned to the low-resource modal feature group. After classification, labels are added to the two groups of features: the high-resource group is labeled as a high-resource modal type, and the low-resource group is labeled as a low-resource modal type.

[0041] Step 203 involves performing feature normalization on the high-resource modality feature group and the low-resource modality feature group respectively to generate modality decoupling results. Specifically, the normalization processing unit first reads the high-resource modality feature group and processes it using the Z-Score normalization method. The calculation process is as follows: For each feature dimension, the mean μ and standard deviation σ of all sample feature values ​​under that dimension are calculated first. The mean calculation formula is: μ = Where N is the batch sample size. Here are the eigenvalues; the standard deviation is calculated using the following formula: Then, each feature value x is converted into a normalized value. The formula is: The normalized eigenvalues ​​are controlled to be within the range of -3 to 3; if they exceed this range, truncation is performed. When the value is greater than 3, set it to 3. When the value is less than -3, it is set to -3. For the low-resource modality feature group, the Min-Max normalization method is used. The calculation process is as follows: for each feature dimension, calculate the minimum value min and the maximum value max of all sample feature values ​​under that dimension, and then convert each feature value x into a normalized value. The formula is: Map to the interval between 0 and 1; if max=min (all feature values ​​of this dimension are the same), then set all feature values ​​of this dimension to 0.5; after the two sets of features are normalized, the modal decoupling result is generated.

[0042] This embodiment, based on format and dimensional features, improves the accuracy of modality type judgment and avoids subsequent processing errors caused by confusion of different modal data.

[0043] In a preferred embodiment of the present invention, step 3, analyzing the low-resource modal data stream, constructing a geometric reference datum, establishing a reference topological unit on the geometric reference datum, calculating the cross product of the vectors of adjacent edges at each vertex based on the polygon vertex coordinate sequence of the reference topological unit, and obtaining the concavity / convexity identification result of each vertex based on the sign determination of the cross product, may include:

[0044] Step 301: Analyze the low-resource modal data stream to generate construction parameters for the geometric reference datum. Based on these parameters, establish a reference topology unit composed of polygonal meshes on the geometric reference datum. This includes: starting the datum construction unit, reading the low-resource modal data stream via a high-speed serial bus interface, and adaptively determining the geometric reference datum type according to the low-resource modality type. A two-dimensional semantic coordinate system is constructed for text or audio modalities, a two-dimensional pixel coordinate system for image modalities, and a three-dimensional spatial coordinate system for three-dimensional point cloud modalities. The x-axis and y-axis of the two-dimensional coordinate system are both set to 0 to 100, with units of dimensionless normalized units. The x-axis, y-axis, and z-axis of the three-dimensional coordinate system are also set to 0 to 100, with units of dimensionless normalized units. The coordinate system resolution is set to 0.01 units per pixel, corresponding to each pixel covering a coordinate range of 0.01 x 0.01 in a two-dimensional scene and 0.01 x 0.01 x 0.01 in a three-dimensional scene. Generate a geometric reference datum plane construction parameter file, which includes information such as coordinate system type, axis range, resolution, and units. The file is stored in an initialization configuration format, and parameter values ​​are retained to two decimal places.

[0045] Based on the construction parameters, the topology element construction unit constructs a reference topology element on the geometric reference datum plane. The reference topology element is a rectangular region with the upper left corner coordinates of 10.00 and 10.00, the lower right corner coordinates of 90.00 and 90.00, and the size of 80 by 80 units. The coordinates of the lower left corner of the 3D base plane are 10.00, 10.00, 10.00, and the coordinates of the upper right corner are 90.00, 90.00, 90.00, with a size of 80 x 80 x 80 units. The reference topology unit is composed of a polygonal mesh. The mesh type selection requirements are: triangular mesh is preferred for 3D point cloud modalities, quadrilateral mesh is preferred for text and audio modalities, and triangular or quadrilateral mesh can be selected for image modalities according to actual needs. The side length of the mesh unit is set to 0.5 to 1.0 units. The specific construction process is as follows: The triangular mesh adopts the triangulation calculation process based on the empty circle criterion. First, the two vertices farthest apart on the boundary of the reference topology unit are selected as the initial edge. Among the remaining vertices, the vertex with the largest distance from the edge and satisfying the empty circle condition is selected, that is, the circumcircle of the triangle formed by this vertex and the initial edge has no other vertices, forming the first triangle. Then, based on the three sides of this triangle, the above point selection process is repeated until all vertices are included in the triangular mesh. The interior angles of each triangle are controlled to be between 30 and 120 degrees. The quadrilateral mesh is generated using an orthogonal partitioning process. First, the length and width of the baseline topological unit are partitioned with a side length step of 0.5 to 1.0 units. The coordinates of the partitioned nodes are calculated. In the 2D scene, the node coordinates in the x-direction are 10.00, 10.00 + step, ..., 90.00, and the y-direction is the same. In the 3D scene, the z-direction is partitioned simultaneously. Then, adjacent nodes are connected sequentially to form quadrilaterals, ensuring that the side length deviation of adjacent quadrilaterals does not exceed 0.1 units. Both mesh construction methods must ensure that the number of polygons contained in the baseline topological unit is between 1024 and 4096. After generation, the mesh is checked by a mesh verification unit. The checks include whether the vertex coordinate overlap reaches 99% or more, whether the face missing rate reaches 1% or more, and whether the side length exceeds the set range. If any problem exists, the mesh generation parameters are adjusted, the side length step is set to 0.05 units, and the mesh is regenerated until the verification passes. Verified baseline topology units are stored in polygon file format, recording information such as vertex coordinates and face indices for each polygon, with coordinate values ​​retained to two decimal places.

[0046] Step 302: Extract the vertex coordinate sequence of each polygon in the baseline topology unit to generate a set of polygon vertex coordinate sequences; based on the set of polygon vertex coordinate sequences, calculate the cross product of the vectors corresponding to the two adjacent edges at each vertex to generate the cross product calculation results for each vertex. Specifically, the coordinate extraction unit reads the polygon file format file of the baseline topology unit and extracts the vertex coordinates of each polygon in a clockwise direction to generate a vertex coordinate sequence; each polygon in a triangular mesh contains three vertices, and each polygon in a quadrilateral mesh contains four vertices; two-dimensional coordinates are recorded in the form of x and y values, and three-dimensional coordinates are recorded in the form of x, y, and z values, with all coordinate values ​​retained to two decimal places. Decimal values ​​are controlled within the range of 10.00 to 90.00; for example, the vertex coordinate sequence of a triangle polygon is 20.00 and 30.00, 20.50 and 30.00, 20.25 and 30.43, and the vertex coordinate sequence of a 3D polygon is 20.00, 30.00, 40.00, 20.50, 30.00, 40.00, 20.25, 30.43, 40.20; the vertex coordinate sequences of all polygons are integrated to generate a set of polygon vertex coordinate sequences, stored in comma-separated value format; the storage columns include polygon number, vertex number, x coordinate, y coordinate, and a z coordinate column is added in the 3D scene, and the encoding format adopts a general character encoding format.

[0047] The cross product calculation unit reads the set of polygon vertex coordinate sequences and calculates the cross product of the vectors of the two adjacent edges for each vertex. For vertex A, whose coordinates are x1 and y1 in the 2D scene or x1, y1, and z1 in the 3D scene, the preceding vertex B and the following vertex C are first determined. The determination rule is that when vertex A is the first vertex in the sequence, the preceding vertex B is the last vertex in the sequence; when vertex A is the last vertex in the sequence, the following vertex C is the first vertex in the sequence. The vector V1 of the first edge BA is calculated. In the 2D scene, the x-coordinate of this vector is the difference between x1 and the x-coordinate of the preceding vertex B, and the y-coordinate is the difference between y1 and the y-coordinate of the preceding vertex B. The difference in coordinates; in a 3D scene, the x-coordinate of this vector is the difference between x1 and the x-coordinate of the preceding vertex B, the y-coordinate is the difference between y1 and the y-coordinate of the preceding vertex B, and the z-coordinate is the difference between z1 and the z-coordinate of the preceding vertex B; calculate the vector V2 of the second edge AC. In a 2D scene, the x-coordinate of this vector is the difference between x1 and x1 of the following vertex C, and the y-coordinate is the difference between y1 and y1 of the following vertex C; in a 3D scene, the x-coordinate of this vector is the difference between x1 and x1 of the following vertex C, the y-coordinate is the difference between y1 and y1 of the following vertex C, and the z-coordinate is the difference between z1 and z1 of the following vertex C.

[0048] The vector cross product calculation process is as follows: In a 2D scene, let the x-coordinate of V1 be a, the y-coordinate be b, and the x-coordinate of V2 be c, the y-coordinate be d. First, calculate the product of a and d, then calculate the product of c and b. Subtract the second product from the first product to obtain the vector cross product of that vertex. The calculation expression is: the 2D cross product equals the x-component of V1 multiplied by the y-component of V2 minus the x-component of V2 multiplied by the y-component of V1. In a 3D scene, let the x-coordinate of V1 be a, the y-coordinate be b, and the y-coordinate be c, and the x-coordinate of V2 be d, the y-coordinate be d. Labeled as e and with vertical coordinate f, first calculate the three components of the cross product vector: the horizontal coordinate is the difference between b multiplied by f and c multiplied by e, the vertical coordinate is the difference between c multiplied by d and a multiplied by f, and the vertical coordinate is the difference between a multiplied by e and b multiplied by d. Take the vertical coordinate value of this three-dimensional vector as the result of the vector cross product calculation for that vertex. The calculation expression is that the z component of the three-dimensional cross product is equal to the x component of V1 multiplied by the y component of V2 minus the y component of V1 multiplied by the x component of V2. The cross product calculation results of all vertices are rounded to three decimal places and then integrated to generate the vector cross product calculation results for each vertex.

[0049] Step 303: Based on the sign characteristics of the vector cross product calculation results, determine the concavity / convexity attribute of each vertex and generate a concavity / convexity identification result. Specifically, this includes: the concavity / convexity judgment unit reads the set of vector cross product calculation results for each vertex. The preset cross product sign judgment standard is that a cross product result greater than 0.001 is judged as a convex vertex, a cross product result less than -0.001 is judged as a concave vertex, and a cross product result between -0.001 and 0.001 and containing boundary values ​​is judged as a collinear vertex. Collinear vertices are uniformly classified into the concave vertex set. According to the above standard, the judgment unit judges the cross product result of each vertex one by one and generates a convex or concave vertex identification label for each vertex. After the judgment is completed, the vertex number, vertex coordinates, cross product calculation result and identification label are integrated to generate a concavity / convexity identification result. This result is stored in the form of a relational database table. The table fields include the vertex number as the primary key, x coordinate, y coordinate, z coordinate, cross product result, and concavity / convexity label in the 3D scene.

[0050] This embodiment distinguishes between convex vertices, concave vertices, and collinear vertices by using a clear cross product sign judgment criterion, ensuring the objectivity and consistency of the concavity / convexity identification results.

[0051] In a preferred embodiment of the present invention, step 4, which involves dividing the polygons in the reference topological unit into convex and concave regions based on the convexity / concavity identification results, segmenting the reference topological unit to generate multi-scale topological partitions, mapping the low-resource modality data stream to the corresponding domain of the multi-scale topological partitions, and extracting multi-resolution structural features, may include:

[0052] Step 401: Classify the polygon vertices in the baseline topology unit according to the concavity / convexity identification results, generating a set of convex vertices and a set of concave vertices. Specifically, this includes: the vertex classification unit reading the concavity / convexity identification results stored in a relational database, which includes vertex number, vertex coordinates, cross product calculation results, and concavity / convexity labels; the vertex classification unit traversing all vertices in vertex number order, first filtering vertices labeled as convex vertices: checking the label field of each vertex one by one; if the label is convex vertex, extracting the vertex number, x and y coordinates in the 2D scene or x, y, and z coordinates in the 3D scene, and the cross product calculation results; and then... This information is sorted and integrated in ascending order by vertex number to generate a convex vertex set. Vertices labeled as concave are filtered, and their information is extracted in the same way as the convex vertex set. They are also sorted and integrated in ascending order by vertex number to generate a concave vertex set. During the filtering and integration process, coordinate verification is performed: the x, y, and z coordinates (in a 3D scene) of each vertex must be between 10.00 and 90.00. If the x coordinate of a vertex is less than 10.00 or greater than 90.00, or if the y and z coordinates (in 3D) are in the same condition, it is marked as an abnormal vertex. The vertex number and abnormal coordinate value are recorded and stored separately in the abnormal data directory. It is not included in the convex or concave vertex set.

[0053] Step 402: Based on the distribution characteristics of the convex vertex set and the concave vertex set, the baseline topological unit is divided into convex regions and concave regions. According to the geometric characteristics of the convex and concave regions, a multi-level subdivision operation is performed on the baseline topological unit to generate multi-scale topological partitions. Specifically, the region partitioning unit reads the convex vertex set and the concave vertex set, and first divides the baseline topological unit into basic sub-regions with a size of 10 by 10 units. In the two-dimensional scene, a total of 8 by 8 basic sub-regions are divided, and in the three-dimensional scene, a total of 8 by 8 basic sub-regions are divided. The boundary coordinates of each basic sub-region are equally distributed according to the range of the baseline topological unit. For example, in the two-dimensional scene, the upper left corner coordinates of the first basic sub-region are 10.00 and 10.00, and the lower right corner coordinates are 20.00 and 20.00.

[0054] Calculate the vertex distribution density for each basic sub-region: Count the number of convex and concave vertices within each basic sub-region. In a 2D scene, the region area is 10 x 10 = 100 square units; in a 3D scene, the region volume is 10 x 10 x 10 = 1000 cubic units. The convex vertex distribution density is the number of convex vertices in the sub-region divided by the region area or volume. The concave vertex distribution density is calculated in the same way. When the convex vertex distribution density within a basic sub-region is greater than the concave vertex distribution density, and the proportion of convex vertices to the total number of vertices (convex vertices plus concave vertices) in the sub-region exceeds 70%, the basic sub-region is designated as a convex region. When the concave vertex distribution density is greater than the convex vertex distribution density, and the proportion of concave vertices exceeds 70%, it is designated as a concave region. For basic sub-regions where the difference between the two types of vertex distribution densities is less than 10%, the following calculations are performed: The mean of the cross products of all vertices within a subregion is calculated by summing the cross products of all vertices and then dividing by the total number of vertices in that subregion. If the mean is greater than 0, the region is classified as a convex region; if the mean is less than 0, it is classified as a concave region. After defining the convex and concave regions, the topology subdivision unit performs a three-level subdivision operation on the base topology unit. The first level of subdivision divides both the convex and concave regions by halving their side lengths, dividing the original 80x80 unit region into four 40x40 unit sub-regions. The boundary coordinates of each sub-region are equally divided according to the original region's range. The second level of subdivision, based on the first level, further divides each 40x40 unit sub-region into four 20x20 unit sub-regions. The third level of subdivision continues to divide the 20x20 unit sub-regions into four 10x10 unit sub-regions. Each level of subdivision maintains that the sub-region boundaries are parallel to the coordinate axes of the base topology unit.

[0055] During the subdivision process, the number of vertices in each newly generated sub-partition is counted to ensure that each sub-partition has at least 10 vertices. If a sub-partition has fewer than 10 vertices, the nearest neighboring sub-partition (defined as sub-partitions sharing an edge or a vertex) is found, and the sub-partition with the fewest vertices is selected for merging. The area or volume of the merged sub-partition is the sum of the two sub-partitions. The number of vertices and region information of the merged sub-partition are counted again. After completing the three-level subdivision, all sub-partitions of different scales are integrated, and each topological partition is labeled with its corresponding subdivision level (first level, second level, third level), region type (convex, concave), number of vertices, area or volume, boundary coordinates, and other information.

[0056] Step 403: Map the low-resource modal data stream according to the spatial correspondence between feature distribution and multi-scale topological partitions to generate a distribution mapping of modal data in the topological partitions. Specifically, this includes: the data mapping unit reads the low-resource modal data stream and multi-scale topological partition information. The low-resource modal data stream contains normalized feature data of types such as text, image, audio, and 3D point cloud (the normalized feature values ​​range from 0 to 1); for different types of low-resource modal data, feature distribution values ​​are calculated: each semantic feature of the text modal data is assigned a weight value between 0.1 and 0.5 (the higher the importance, the larger the weight value). The feature distribution value is the weight value of each semantic feature multiplied by the frequency of occurrence of that feature in the text (the frequency of occurrence is the number of times that feature appears in the text), and then... The product results are summed. For image modal data, the pixel percentage of each visual feature is the number of pixels corresponding to that feature divided by the total number of pixels in the image. The feature distribution value is the pixel percentage multiplied by the gray value of that feature (gray value range 0 to 255). All product results are then summed. For audio modal data, the duration percentage of each acoustic feature is the duration corresponding to that feature divided by the total duration of the audio. The feature distribution value is the duration percentage multiplied by the sound pressure level of that feature (range 0 to 120 dB). All product results are then summed. For 3D point cloud modal data, the point cloud quantity percentage of each spatial feature is the number of points corresponding to that feature divided by the total number of points. The feature distribution value is the point cloud quantity percentage multiplied by the spatial distance value of that feature (range 0 to 100 units). All product results are then summed.

[0057] Calculating the spatial characteristic values ​​of multi-scale topological partitions involves calculating the center coordinates of each partition. For a two-dimensional partition, the x-axis center coordinate is the sum of the x-coordinates of the top-left and bottom-right corners, divided by 2; the y-axis center coordinate is calculated in the same way. The x-axis, y-axis, and z-axis center coordinates of a three-dimensional partition are all calculated in the same manner. The spatial characteristic value of a two-dimensional topological partition is the sum of its x-axis and y-axis center coordinates, multiplied by the partition area (length x width). The spatial characteristic value of a three-dimensional topological partition is the sum of its x-axis, y-axis, and z-axis center coordinates, multiplied by the partition volume (length x width x height).

[0058] Establishing the correspondence between feature distribution values ​​and spatial feature values ​​involves iterating through each data point in the low-resource modal data stream line by line, calculating its feature distribution value, then iterating through all multi-scale topological partitions to calculate their spatial feature values. The absolute value of the difference between the feature distribution value of each data point and the spatial feature value of each partition is calculated, and the topological partition with the smallest absolute difference is determined as the mapping target partition for that data point. During the mapping process, the data point's ID, modality type, feature distribution value, and corresponding topological partition ID, level, spatial feature value, and absolute difference value are recorded to ensure that each modal data point is mapped to only one topological partition. After mapping, the mapping coverage rate is calculated, which is the number of successfully mapped modal data points divided by the total data volume of the low-resource modal data stream. If the coverage rate is below 95%, the original weight values ​​(the initial weight values ​​for text, images, audio, and 3D point clouds are all 0.5) are increased by 0.05, the feature distribution values ​​and spatial feature values ​​are recalculated, and the mapping operation is performed again until the coverage rate reaches 95% or higher. Finally, all related information is integrated to generate the distribution mapping of modal data in the topological partitions.

[0059] Step 404: Based on the distribution mapping of modal data in topological partitions, extract local structural features from topological partitions at different scales, and fuse these local structural features to generate multi-resolution structural features. Specifically, this includes: a feature extraction unit reading the distribution mapping of modal data in topological partitions and multi-scale topological partition information, and extracting local structural features for topological partitions at different scales; extracting global local structural features from the first-level subdivided topological partitions: first, calculate the sum of the feature distribution values ​​of all modal data within the partition, that is, add the feature distribution values ​​of each data point in the partition one by one, then divide by the number of modal data points in the partition to obtain the mean of the feature distribution values, and finally multiply the mean of the feature distribution values ​​by the area or volume of the partition to obtain the global local structural feature value; extracting from the second-level subdivided topological partitions... Intermediate-level local structural features: First, calculate the mean of the feature distribution values ​​of all modal data within the partition (calculation method is the same as the first level). Then, subtract the mean from the feature distribution value of each data point to obtain the difference for each data point. Take the absolute value of the difference and sum them to obtain the total difference. Divide the total difference by the number of modal data points in the partition to obtain the average difference. Multiply the average difference by the area or volume of the partition to obtain the intermediate-level local structural feature value. Fine-grained local structural features are extracted from the topological partitions of the third level: Multiply the feature distribution value of each modal data point in the partition by the cross product of the corresponding vertex. Sum all the product results to obtain the total product. Divide the total product by the number of modal data points in the partition to obtain the average product. Multiply the average product by the area or volume of the partition to obtain the fine-grained local structural feature value.

[0060] During feature extraction, extraction operations are performed on each topological partition at each scale one by one, recording information such as local structural feature values, partition level, number of modal data, area or volume of each partition. The feature fusion unit reads the local structural features of the topological partitions at each scale and generates multi-resolution structural features using a weighted fusion method. The weight allocation rule is as follows: the first-level subdivision topological partition has a weight of 0.5 (because it contains global information, its weight ratio is the highest), the second level has 0.3, and the third level has 0.2 (fine-grained features have the lowest weight ratio). The fusion calculation method is to multiply the first-level local structural feature value by 0.5, add the second-level local structural feature value by 0.3, and add the third-level local structural feature value by 0.2. The three products are then added to obtain the multi-resolution structural feature value corresponding to each topological partition. After fusion, the partitions are sorted according to their topological partition level and number, and the multi-resolution structural feature values, global local structural feature values, mid-level local structural feature values, and fine-grained local structural feature values ​​of all topological partitions are integrated to generate multi-resolution structural features.

[0061] This embodiment ensures the accuracy of convex and concave vertex sets through vertex classification and coordinate verification; it delineates regions based on the distribution density of basic sub-regions and the mean of cross products, making the division of convex and concave regions more consistent with vertex distribution characteristics.

[0062] In a preferred embodiment of the present invention, step 5, calculating the geometric feature parameters of each polygon in the multi-scale topological partition, matching the multi-resolution structural features with the high-resource modal data stream to generate a heterogeneous feature topology, may include:

[0063] Step 501: Perform geometric feature calculations on each polygon in the multi-scale topological partition to generate a set of geometric feature parameters; based on the set of geometric feature parameters, associate the multi-resolution structural features with the corresponding geometric feature parameters to generate a multi-resolution topological feature descriptor. Specifically, this includes: the geometric feature calculation unit reading the stored information of the multi-scale topological partition, which includes the vertex coordinates, partition level, region type, etc. of each polygon, and performing geometric feature calculations on each polygon; calculating the perimeter of the polygon: traversing each vertex of the polygon in clockwise order, for two adjacent vertices, calculating their x-coordinate difference and y-coordinate difference (3D scene plus z-coordinate difference), the x-coordinate difference is the x-coordinate of the latter vertex minus the x-coordinate of the former vertex, the y-coordinate difference and z-coordinate difference are calculated in the same way; squaring each difference, the square of the x-coordinate difference plus the square of the y-coordinate difference (3D plus z-coordinate difference), to obtain the sum of squares; taking the square root of the sum of squares to obtain the length of the edge; adding the lengths of all edges of the polygon sequentially to obtain the perimeter of the polygon.

[0064] Calculating the area of ​​a polygon involves several steps. For a 2D polygon, the coordinates of each vertex are taken in clockwise order. The x-coordinate of each vertex is multiplied by the y-coordinate of the next vertex, and the sum of these products is obtained. Then, the y-coordinate of each vertex is multiplied by the x-coordinate of the next vertex, and the sum of these products is obtained. The area of ​​the 2D polygon is then calculated by subtracting the second sum from the first sum and dividing the absolute value of the difference by 2. For a 3D polygon, the area is first projected onto the xoy plane, the 2D area of ​​the projected polygon is calculated, and then the angle between the normal vector of the polygon and the normal vector of the xoy plane is calculated. The cosine of the included angle is calculated by multiplying the x-coordinates, y-coordinates, and y-coordinates of the two normal vectors, and then dividing by the product of the magnitudes of the two normal vectors (the magnitude of the normal vector is the square root of the sum of the squares of the x-coordinates, y-coordinates, and y-coordinates). The area of ​​a 3D polygon is the projected area multiplied by the cosine of the included angle. To calculate the compactness of a polygon: square the perimeter of the polygon to get the square of the perimeter; divide the square of the perimeter by the area of ​​the polygon to get the first ratio; multiply 4 by pi (3.1416) to get the product; divide the first ratio by the product to get the compactness.

[0065] Calculate the vertex density of the polygon: In a 2D scene, the vertex density is the number of vertices divided by the area of ​​the polygon; in a 3D scene, it is the number of vertices divided by the volume (the volume of a 3D polygon is calculated by multiplying the base area by the height, where the height is the average distance between vertices along the z-axis). Integrate geometric feature parameters such as perimeter, area, compactness, and vertex density, keeping two decimal places for each parameter. Associate the set with polygon ID, topological partition ID, partition level, and other information to generate a geometric feature parameter set. The feature association unit reads the geometric feature parameter set and multi-resolution structural features, groups them by topological partition ID, and calculates the mean of the geometric feature parameters for all polygons within each topological partition: the mean of each geometric feature parameter (such as perimeter) is the sum of the parameter values ​​for all polygons within that partition, divided by the number of polygons within that partition. Associate the multi-resolution structural feature value of each partition with the corresponding mean of the geometric feature parameter to generate a multi-resolution topological feature descriptor.

[0066] Step 502 involves performing feature-level matching between the multi-resolution topological feature descriptor and the high-resource modality data stream to establish a cross-modal feature correspondence. Specifically, this includes: the feature matching unit reading the multi-resolution topological feature descriptor and the high-resource modality data stream. The high-resource modality data stream contains normalized feature data of various types, such as text, images, audio, and 3D point clouds (deep feature extraction and normalization have been completed, with feature values ​​ranging from 0 to 1); global feature values ​​are extracted from the high-resource modality data stream, using the same extraction method as the feature distribution value calculation method for low-resource modality data: for text modality, the semantic feature weight value (0.1 to 0.5) is multiplied by the frequency of occurrence and then summed; for image modality, the visual feature weight value is multiplied by the frequency of occurrence and then summed. The similarity is calculated as follows: the proportion of pixel values ​​is multiplied by the grayscale value and then summed; the proportion of acoustic feature duration is multiplied by the sound pressure level and then summed; the proportion of spatial feature point cloud quantity is multiplied by the spatial distance value and then summed. The similarity between the multi-resolution topological feature descriptor and the global feature value of the high-resource modality data stream is calculated: feature values ​​greater than 0.5 are considered valid features. The number of valid features in the multi-resolution topological feature descriptor and the number of valid features in the global features of the high-resource modality data stream are counted. Then, the number of valid features shared by both (intersection count) and the number of valid features of both (union count) are counted. The intersection count is divided by the union count and then multiplied by 100 to obtain the similarity percentage value.

[0067] A similarity threshold of 80% is set. When the similarity between a multi-resolution topological feature descriptor and a high-resource modality data is greater than or equal to 80%, a feature-level correspondence between the two is established. When the similarity is less than 80%, feature enhancement processing is performed on the high-resource modality data. Each feature value of the data is multiplied by 1.1. If the enhanced feature value is greater than 1.0, it is adjusted to 1.0, and the similarity is recalculated. If the similarity is still less than 80% after enhancement, the high-resource modality data is marked as unmatched data and stored separately in the unmatched data directory. The feature matching unit matches each multi-resolution topological feature descriptor and each high-resource modality data one by one, recording the descriptor number, the corresponding high-resource modality data number, the similarity value, and whether feature enhancement has been performed for each successfully matched descriptor. After all matches are completed, all correspondence information is integrated to establish a cross-modal feature correspondence.

[0068] Step 503: Based on the cross-modal feature correspondence, the multi-resolution topological feature descriptor is fused with the high-resource modal data stream to generate a heterogeneous feature topology. Specifically, this includes: a feature fusion unit reading the cross-modal feature correspondence, the multi-resolution topological feature descriptor, and the high-resource modal data stream; and fusing the successfully matched multi-resolution topological feature descriptor with the high-resource modal data stream based on the correspondence. The fusion process is divided into three stages; the first stage, feature alignment, determines the feature dimensions (e.g., 512 dimensions) of the high-resource modal data stream. If the multi-resolution topological feature descriptor... If the dimension (e.g., 256 dimensions) is smaller than the dimension of the high-resource modality data, zero values ​​are added to the feature vector of the descriptor until the dimension matches that of the high-resource modality data. If the dimension of the descriptor (e.g., 1024 dimensions) is larger than that of the high-resource modality data, the first part of the dimension of the descriptor feature vector is truncated, retaining the dimension consistent with that of the high-resource modality data. The second stage is feature weighted summation: the weights are assigned as follows: 0.4 for the multi-resolution topology feature descriptor and 0.6 for the high-resource modality data stream. The weighted summation method is: each feature value of the descriptor is multiplied by 0.4 to obtain the weighted descriptor summation. Eigenvalues; Each eigenvalue corresponding to the high-resource modality data is multiplied by 0.6 to obtain the high-resource weighted eigenvalue; The descriptor-weighted eigenvalue is added to the high-resource weighted eigenvalue one by one to obtain the fused eigenvalue (range 0 to 1); The third stage, topology construction, arranges all fused eigenvalues ​​according to the matching order of cross-modal feature correspondences, with each fused eigenvalue as a feature node and each node corresponding to a matching pair; Calculate the connection strength between any two nodes: Multiply the fused eigenvalues ​​of the two nodes to obtain the feature product; Add the fused eigenvalues ​​of the two nodes to obtain the eigenvalues. The feature product is divided by the feature sum, and then multiplied by 100 to obtain the connection strength value (range 0 to 100). The connection strength threshold is set to 50. When the connection strength between two nodes is greater than or equal to 50, a connection edge is established between the nodes, and the weight of the connection edge is the connection strength value. After the construction of all nodes and connection edges is completed, a heterogeneous feature topology is generated. This topology contains information such as the feature node set, connection edge set, node feature values, and edge weight values. It is stored in the topology storage directory of the model training server in a graphical data format to ensure the integrity and resolvability of the topology structure.

[0069] This embodiment calculates the multi-dimensional geometric feature parameters of polygons in detail, so that structural features also have geometric attributes, enriching the dimensions of feature description; feature matching combines effective feature statistics and feature enhancement to improve the accuracy of cross-modal feature correspondence.

[0070] like Figure 2 As shown, in another preferred embodiment of the present invention, step 6, optimizing the heterogeneous feature topology to obtain a deeply optimized model parameter configuration, may include:

[0071] Step 601a: Based on the heterogeneous feature topology, the average node connectivity is calculated by statistically analyzing the number of connecting edges between nodes and the total number of nodes in the topology; the average edge weight is calculated by statistically analyzing the weights of connecting edges; and the percentage of topology connectivity is calculated by statistically analyzing the number of interconnected subgraphs and the total number of nodes. This generates a topology integrity assessment result. Specifically, the topology analysis unit reads the stored information of the heterogeneous feature topology, which includes the set of feature nodes, the set of connecting edges, node feature values, edge weight values, etc., and calculates the connectivity integrity index between nodes in the topology, including three dimensions: node connectivity, average edge weight, and topology connectivity. The node connectivity is calculated by counting the number of connecting edges for each node (bidirectional connections are considered as one edge, i.e., a connection between node A and node B is counted only once), dividing the number of connecting edges for each node by the total number of nodes in the topology, and then multiplying by 100 to obtain the percentage connectivity value for each node. The average node connectivity is calculated by summing the connectivity percentages of all nodes and dividing by the total number of nodes. The mean edge weight is calculated by summing the weights of all connected edges and dividing by the total number of edges (range 0 to 100). Topological connectivity is calculated by traversing all feature nodes, counting the number of interconnected subgraphs (connected components), dividing the number of connected components by the total number of nodes, and multiplying by 100 to obtain the topological connectivity percentage. These three dimensions are integrated to generate a topological integrity assessment result. An excellent assessment is defined as an average node connectivity greater than or equal to 50%, an average edge weight greater than or equal to 60, and a topological connectivity percentage less than or equal to 10%. A good assessment is defined as an average node connectivity between 30% and 50%, an average edge weight between 40 and 60, and a topological connectivity percentage between 10% and 20%. All other cases are considered average.

[0072] Step 601b: Based on the topology integrity assessment results, a multi-objective optimization function is constructed by setting target values ​​for each indicator and calculating the difference between the target values ​​and the corresponding actual values ​​in the assessment results. The partial derivatives of the multi-objective optimization function with respect to the model parameters are calculated to generate the parameter gradient matrix. Specifically, the optimization function construction unit reads the topology integrity assessment results and constructs a multi-objective optimization function based on these results. The optimization direction is to increase the average node connectivity, increase the average edge weight, and reduce the percentage of topology connectivity. The construction process of this multi-objective optimization function is as follows: First, set the target value of average node connectivity to 90%, the target value of average edge weight to 80%, and the target value of topological connectivity percentage to 5%. Then, calculate and square the difference between the target value and the actual value of average node connectivity to obtain the first squared term. Calculate and square the difference between the target value and the actual value of average edge weight to obtain the second squared term. Calculate and square the difference between the actual value and the target value of topological connectivity percentage to obtain the third squared term. Finally, sum the three squared terms to obtain the total optimization loss value. The gradient calculation unit reads the multi-objective optimization function and model parameters. The model parameters include feature node weights, connection edge weights, topological dimension parameters, etc., with a quantity ranging from 100,000 to 1 million, and are sorted in the order of feature node weights, connection edge weights, and dimension parameters. The gradient calculation unit calculates the partial derivative for each model parameter. The specific process is as follows: all parameters in the optimization function except the current parameter are treated as constants, the initial value of the current parameter is recorded, and the parameter is increased by 0.001 (the selection of this small perturbation value is based on not affecting the essential characteristics of the parameter and being able to reflect the parameter change trend). The difference between the optimized loss value after parameter perturbation and the initial optimized loss value is calculated to obtain the loss change. The loss change is divided by 0.001 to obtain the partial derivative value of the parameter and is retained to four decimal places. After the gradient calculation unit completes the partial derivative calculation for all model parameters one by one, all partial derivative values ​​are arranged in the parameter sorting order to generate the parameter gradient matrix.

[0073] Step 601c involves performing multi-round iterative optimization based on the parameter gradient matrix to generate the gradient update amounts of the model parameters. Specifically, this includes: calculating the decay coefficient for the current round: the decay coefficient is 1 minus the current round divided by the total number of iterations. For example, in the 10th iteration, the decay coefficient is 1 minus 10 divided by 100, which equals 0.9, ensuring the decay coefficient decreases linearly from 1 to 0; calculating the gradient update amount for the current round: each partial derivative value in the parameter gradient matrix is ​​multiplied by the learning rate of the current round, and then multiplied by the decay coefficient to obtain the gradient update amount for each parameter in this round; updating the parameter gradient matrix involves subtracting the gradient update amount for the current round from each partial derivative value in the original parameter gradient matrix to obtain a new parameter gradient matrix for the next iteration; each iteration... After each iteration, the optimization loss value is calculated (the calculation method is consistent with the multi-objective optimization function), and the loss value for that round is recorded. When the change in the optimization loss value for five consecutive iterations is less than 0.001 (e.g., the loss values ​​for rounds 10 to 14 are 0.852, 0.851, 0.850, 0.849, and 0.849 respectively, and the change is less than 0.001), the iteration is terminated early. After completing all iterations or terminating early, the gradient update amounts for each round are integrated, and the gradient update amount for the last round is taken as the final gradient update amount for the model parameters. This update amount contains information such as the update value corresponding to each model parameter, the number of iterations, the final optimization loss value, and whether it was terminated early, and is stored in the update amount storage directory of the model training server.

[0074] Step 602: Update the model parameters according to the gradient update amount to generate a deeply optimized model parameter configuration. Specifically, this includes: the parameter update unit reading the gradient update amount and the current model parameter configuration, which includes all feature node weights, connection edge weights, topological dimension parameters, etc., with initial parameter values ​​ranging from 0 to 1; the parameter update method involves reading the current value and corresponding gradient update amount of each model parameter one by one, adding the gradient update amount to the current value to obtain the updated parameter value; a reasonableness check is performed during the update process: if the updated parameter value is less than 0, it is adjusted to 0; if the updated parameter value is greater than 1, it is adjusted to 1. If the value is between 0 and 1, it remains unchanged. For example, if the current value of a parameter is 0.8, the gradient update amount is 0.3, and the updated value is 1.1, it is adjusted to 1; if the current value of a parameter is 0.2, the gradient update amount is -0.1, and the updated value is 0.1, it remains unchanged; if the current value of a parameter is 0.1, the gradient update amount is -0.2, and the updated value is -0.1, it is adjusted to 0. The parameter update unit performs update and verification operations on each model parameter one by one. After all parameters are updated, they are classified and organized according to parameter type and number to generate a deeply optimized model parameter configuration. This configuration includes information such as parameter type, parameter number, value before update, value after update, whether the value range has been adjusted, and the reason for adjustment.

[0075] In this embodiment, a multi-objective optimization function is constructed based on the evaluation results, which precisely focuses on the core requirements of topology and feature matching; gradient calculation and iterative optimization enable fine-tuning of model parameters, while decay coefficient and early stopping mechanism ensure optimization efficiency and effectiveness.

[0076] In a preferred embodiment of the present invention, step 7, verifying the deeply optimized model parameter configuration and adjusting the model hyperparameter set to obtain a low-resource modal adaptive multimodal large model, may include:

[0077] Step 701: Input the deeply optimized model parameter configuration into the multimodal large model, verify the model performance, and generate model performance metrics; based on the model performance metrics, evaluate the model's adaptation performance on low-resource modal data and generate modal adaptation performance evaluation results. Specifically, this includes: inputting the deeply optimized model parameter configuration into the multimodal large model that has been constructed and initially trained, verifying the model performance, generating model performance metrics, evaluating the model's adaptation performance on low-resource modal data based on the model performance metrics, and generating modal adaptation performance evaluation results. Specifically, this includes: preparing a validation dataset, which must cover all modal data supported by the model, wherein the number of low-resource modal data must be consistent with the number of low-resource modal data in the actual application scenario, and the number of high-resource modal data must be more than ten times the number of low-resource modal data, while ensuring that the data in the validation dataset has not participated in the initial training of the model. The optimized model parameter configuration, including feature node weights, edge weights, topology dimension parameters, etc., is assigned to the corresponding parameter positions of each sub-network and modality fusion layer of the multimodal large model, completing the loading of model parameters. Each modality of data from the validation dataset is input into its corresponding sub-network: text data into the text sub-network to obtain text feature vectors, image data into the image sub-network to obtain image feature vectors, and speech data into the speech sub-network to obtain speech feature vectors. These modal feature vectors are then input into the modality fusion layer to obtain fused feature vectors, which are then input into the output layer to obtain the model's prediction results. The model performance evaluator calculates the performance metrics for each modality. For classification tasks, [further details are needed]. For the text modality, text classification accuracy is calculated by dividing the number of correctly classified samples in the text modality validation data by the total number of samples in the text modality validation data. For the image modality, image recognition accuracy is calculated by dividing the number of correctly recognized samples in the image modality validation data by the total number of samples in the image modality validation data. For the speech modality, speech recognition accuracy is calculated by dividing the number of correctly recognized samples in the speech modality validation data by the total number of samples in the speech modality validation data. For regression tasks, the mean absolute error is calculated for each modality by dividing the sum of the absolute values ​​of the differences between the model's predicted values ​​and the true values ​​in the validation data for each modality by the total number of samples in the validation data for that modality.

[0078] Cross-modal task performance metrics are calculated. Taking cross-modal retrieval as an example, retrieval accuracy is calculated by dividing the number of results matching the query data among the top five retrieved results when retrieving data from one modality as a query condition for another modality by the total number of retrievals. The single-task performance metrics for each modality and the cross-modal task performance metrics calculated above together constitute the model performance metrics. The model's adaptability to low-resource modal data is evaluated based on the model performance metrics. This evaluation is performed by an adaptability performance evaluator. During the evaluation, the performance metrics for low-resource modal data are compared with those for high-resource modal data to analyze the gap between them. At the same time, considering the characteristics of low-resource modal data, it is determined whether the model can maintain good performance under conditions of limited data volume or low data quality. If the classification accuracy of low-resource modal data is close to that of high-resource modal data and meets the actual application requirements, the model is considered to have good adaptability to low-resource modal data. Otherwise, the adaptability is considered insufficient. These judgments are combined to generate the modality adaptability performance evaluation results.

[0079] Step 702: Based on the modal adaptation performance evaluation results, adjust the model hyperparameter set to generate an optimized hyperparameter configuration; based on the optimized hyperparameter configuration, update the parameter settings of the multimodal large model to generate a low-resource modal adaptive multimodal large model. Specifically, this includes: clarifying the composition of the model hyperparameter set, including the learning rate, batch size, number of hidden layer neurons, and number of network layers for each modal sub-network; the dimension of the attention vector in the modal fusion layer; and the number of iterations during model training. The hyperparameter tuner determines the adjustment targets and adjustments based on the modal adaptation performance evaluation results. If the evaluation results show that the performance indicators of the low-resource modality are low and the model underfits on the low-resource modality data, then the hyperparameters of the sub-network of that modality are adjusted. The learning rate of the sub-network is multiplied by 0.8 to reduce the learning rate and avoid excessively fast parameter updates. The batch size is divided by 2 to increase the frequency of parameter updates. The number of hidden layer neurons is increased by 200 to improve the feature extraction capability of the sub-network. The number of network layers is increased by one to enhance the modeling capability of the sub-network. At the same time, the number of model training iterations is increased by 500 to allow the model to have a more thorough training process. If the evaluation results show that the performance metrics of the low-resource modality are low, but the model is overfitting on the low-resource modality data, then the learning rate of the sub-network of that modality is multiplied by 1.2 to speed up the parameter update speed, the batch size is multiplied by 2 to reduce the parameter update frequency, a dropout layer is added to the sub-network with a dropout rate of 0.3 to reduce overfitting by randomly dropping some neurons, and the number of network layers is reduced by one to reduce model complexity. For the modality fusion layer, if the cross-modal task performance metrics are low, then the dimension of the attention vector is increased by fifty to improve the ability to distinguish the attention to features of each modality. After adjusting all hyperparameters, the optimized hyperparameter configuration is obtained. Based on the optimized hyperparameter configuration, the parameter settings of the multimodal large model are updated. The adjusted hyperparameters such as learning rate and batch size are assigned to the corresponding modules of the model. Then, the updated model is trained a second time using the training dataset and validation dataset used in step 701. The training process is the same as the initial training process, using the same optimizer and loss function. Iterative training is performed until the loss function value of the model is stable and the performance index on the validation dataset reaches the expected value. After training, the model is saved to generate a low-resource modality adaptive multimodal large model.

[0080] This embodiment, by inputting deeply optimized parameter configurations into the model and combining them with a dataset containing low-resource modal data for performance verification, can comprehensively and accurately obtain the model's performance.

[0081] Embodiments of the present invention also provide a computing device, including: a processor and a memory storing a computer program, wherein the computer program, when executed by the processor, performs the method described above. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0082] Embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described above. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0083] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multi-modal large model low-resource modality self-adaptive learning method, characterized in that, The method comprises: Step 1, obtaining multi-modal input data containing high-resource modal data and low-resource modal data, the multi-modal input data including four types of text, image, three-dimensional point cloud and audio; Step 2, performing deep modal recognition and decoupling operation on the multi-modal input data to obtain modal decoupling results containing high-resource modal data stream and low-resource modal data stream; Step 3, analyzing the low-resource modal data stream, constructing a geometric reference base surface, establishing a reference topological unit on the geometric reference base surface, calculating the vector cross product of adjacent edges at each vertex based on the polygon vertex coordinate sequence of the reference topological unit, and judging the concave-convex identification result of each vertex based on the sign of the vector cross product, specifically including: adaptively determining the type of the geometric reference base surface according to the type of the low-resource modal, wherein the text modal or the audio modal corresponds to constructing a two-dimensional semantic coordinate system, the two-dimensional image modal corresponds to constructing a two-dimensional pixel coordinate system, and the three-dimensional point cloud modal corresponds to constructing a three-dimensional space coordinate system; Step 4, dividing the polygons in the reference topological unit into convex regions and concave regions based on the concave-convex identification result, and segmenting the reference topological unit to generate multi-scale topological partitions, and mapping the low-resource modal data stream to the corresponding domain of the multi-scale topological partitions and extracting multi-resolution structural features, including: classifying the polygon vertices in the reference topological unit according to the concave-convex identification result to generate a convex vertex set and a concave vertex set; dividing the reference topological unit into convex regions and concave regions based on the distribution characteristics of the convex vertex set and the concave vertex set; performing multi-level subdivision operation on the reference topological unit according to the geometric characteristics of the convex region and the concave region to generate multi-scale topological partitions; mapping the low-resource modal data stream according to the spatial correspondence between the feature distribution and the multi-scale topological partitions to generate the distribution mapping of the modal data in the topological partitions; extracting local structural features from topological partitions of different scales based on the distribution mapping of the modal data in the topological partitions, and fusing the local structural features to generate multi-resolution structural features; Step 5, calculating geometric feature parameters based on each polygon in the multi-scale topological partition, matching the multi-resolution structural features with the high-resource modal data stream, and generating heterogeneous feature topology; Step 6, optimizing the heterogeneous feature topology to obtain a deep optimized model parameter configuration; Step 7, verifying the deep optimized model parameter configuration, adjusting the model hyperparameter set, and obtaining a low-resource modal adaptive multi-modal large model.

2. The multi-modal large model low-resource modality adaptation learning method according to claim 1, characterized in that, Performing deep modal recognition and decoupling operation on the multi-modal input data to obtain modal decoupling results containing high-resource modal data stream and low-resource modal data stream, including: Performing modal type identification on the multi-modal input data to generate a modal type identification result; based on the modal type identification result, the multi-modal input data is segmented into a plurality of modal data subsets corresponding to different modal types; Performing deep feature extraction on the plurality of modal data subsets to generate a plurality of corresponding modal feature mappings; classifying the plurality of modal feature mappings according to a predefined resource level threshold to obtain a high-resource modal feature group and a low-resource modal feature group; The high-resource modality feature group and the low-resource modality feature group are subjected to feature normalization processing respectively to generate a modality decoupling result.

3. The multi-modal large model low-resource modality adaptation learning method according to claim 2, characterized in that, The low-resource modality data stream is analyzed to construct a geometric reference base surface, and a reference topological unit is established on the geometric reference base surface. The vector cross product of adjacent edges at each vertex is calculated based on the polygon vertex coordinate sequence of the reference topological unit. The concave-convex property of each vertex is determined based on the sign of the vector cross product to obtain a concave-convex identification result, including: The low-resource modality data stream is analyzed to generate construction parameters of a geometric reference base surface. Based on the construction parameters of the geometric reference base surface, a reference topological unit composed of polygon meshes is established on the geometric reference base surface. The vertex coordinate sequence of each polygon in the reference topological unit is extracted to generate a polygon vertex coordinate sequence set. Based on the polygon vertex coordinate sequence set, the vector cross product of adjacent edges corresponding to each vertex is calculated to generate a vector cross product calculation result for each vertex. The concave-convex property of each vertex is determined based on the sign of the vector cross product calculation result to generate a concave-convex identification result.

4. The multi-modal large model low-resource modality adaptation learning method according to claim 3, characterized in that, Based on each polygon in the multi-scale topological partition, a geometric feature parameter is calculated. The multi-resolution structural feature is matched with the high-resource modality data stream to generate a heterogeneous feature topology, including: Geometric feature calculation is performed on each polygon in the multi-scale topological partition to generate a geometric feature parameter set. Based on the geometric feature parameter set, the multi-resolution structural feature is associated with the corresponding geometric feature parameter to generate a multi-resolution topological feature descriptor. Feature-level matching is performed between the multi-resolution topological feature descriptor and the high-resource modality data stream to establish a cross-modal feature correspondence. Based on the cross-modal feature correspondence, the multi-resolution topological feature descriptor is fused with the high-resource modality data stream to generate a heterogeneous feature topology.

5. The multi-modal large model low-resource modality adaptation learning method according to claim 4, characterized in that, The heterogeneous feature topology is optimized to obtain a deeply optimized model parameter configuration, including: The heterogeneous feature topology is analyzed to generate a topological structure integrity evaluation result. Based on the topological structure integrity evaluation result, a multi-objective optimization function is constructed. The multi-objective optimization function is iteratively solved to generate a gradient update amount of the model parameter. The model parameter is updated based on the gradient update amount of the model parameter to generate a deeply optimized model parameter configuration.

6. The multi-modal large model low-resource modal adaptation learning method according to claim 5, characterized in that, The heterogeneous feature topology is analyzed to generate a topological structure integrity evaluation result. Based on the topological structure integrity evaluation result, a multi-objective optimization function is constructed. The multi-objective optimization function is iteratively solved to generate a gradient update amount of the model parameter, including: Based on the heterogeneous feature topology, the average node connectivity is calculated by counting the number of node connection edges and the total number of nodes, the edge weight mean is calculated by counting the connection edge weight, and the topology connectivity percentage value is calculated by counting the number of mutually connected subgraphs and the total number of nodes to generate a topological structure integrity evaluation result. Based on the topological structure integrity evaluation result, a multi-objective optimization function is constructed by setting the target value of each index and calculating the difference between the corresponding actual value in the evaluation result. The partial derivative of the multi-objective optimization function with respect to the model parameter is calculated to generate a parameter gradient matrix. Based on the parameter gradient matrix, a plurality of rounds of iterative optimization are performed to generate a gradient update amount of the model parameter.

7. The multi-modal large model low-resource modality adaptation learning method according to claim 6, characterized in that, The deep-optimized model parameter configuration is verified, the model hyperparameter set is adjusted, and a low-resource-modal-adaptive multi-modal large model is obtained, including: The deep-optimized model parameter configuration is input into the multi-modal large model, the model performance is verified, and a model performance index is generated; based on the model performance index, the adaptive performance of the model on the low-resource-modal data is evaluated, and a modal adaptive performance evaluation result is generated; According to the modal adaptive performance evaluation result, the model hyperparameter set is adjusted, and an optimized hyperparameter configuration is generated; based on the optimized hyperparameter configuration, the parameter setting of the multi-modal large model is updated, and a low-resource-modal-adaptive multi-modal large model is generated.

8. A computing device, comprising: Including: One or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, so that the one or more processors implement the method of any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a program which is executed by the processor to implement the method of any one of claims 1 to 7.

Images