Implementation Method for Real-time Inference Acceleration of Multimodal Large Models

Abstract

This invention provides a method for accelerating real-time inference of multimodal large models, relating to the field of data processing technology. The method includes: Step 1, generating a minimum envelope ellipse using the peak point of the first visual attention weight, the second visual coordinate point, and the third visual association point as boundary points, and calculating the ratio of the area to the perimeter of the ellipse as a spatial clustering index. This invention effectively improves the efficiency, accuracy, and resource utilization of real-time inference of multimodal large models by selectively enhancing spatial features, evaluating multidimensional constraint computing power and dynamically scheduling resources, lightweight cross-modal fusion, and multi-step inference attention calibration, combined with real-time decision-making and full-process data-driven evaluation parameters and dynamic updates of the model library. This ensures the real-time performance and reliability of inference results, while continuously optimizing system computing power evaluation and model adaptation, making multimodal inference more aligned with the resource constraints of heterogeneous computing and the real-time and accuracy requirements of actual business operations.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method for accelerating real-time inference of multimodal large models. Background Technology

[0002] In specific application scenarios where multimodal large models need to achieve real-time inference, such as in the real-time interaction scenario of smart cockpits, the system needs to simultaneously process the driver's voice commands, the visual images captured by the in-vehicle camera, and the driver's gestures. The common approach is to extract features from the above multimodal inputs independently and then immediately proceed to comprehensive cross-modal fusion and model inference computation. In practical applications, this approach may face real-time pressure issues due to the lack of effective screening before feature fusion. Specifically, not all areas in the visual image (such as complex in-vehicle scenes containing roads, dashboards, and multiple windows) are semantically related to the current voice commands and gesture directions.

[0003] For example, when a driver says "open that" and points to the passenger-side window, the current image contains features from multiple areas such as the window, seat, and center console. However, most existing methods fuse these visual features with voice and gesture features indiscriminately. This requires the subsequent inference model to process a large amount of visual information that may not be related to the current interaction intent, consuming already limited computing and memory resources. On a limited in-vehicle computing platform, this may lead to increased response latency and make it difficult to meet the stringent requirements of real-time interaction. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a method for accelerating real-time inference of multimodal large models, realizing low-latency inference of multimodal large models, and solving the response latency problem caused by redundant information processing and inefficient utilization of computing power.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0006] Firstly, a method for accelerating real-time inference of multimodal large models, the method comprising:

[0007] Step 1: Perform parallel perception and feature extraction processing on the original multimodal input data to obtain speech feature vectors, image feature vectors, and gesture spatial feature vectors. Based on the image feature vectors, construct a spatial weight distribution representing the visual attention distribution by calculating the weighted sum of the feature components at each position. Analyze the position corresponding to the maximum weight as the first visual attention weight peak point. Map the gesture spatial feature vector to the image coordinate system corresponding to the image feature vector to obtain the second visual coordinate point. Perform element-wise similarity calculation between the image feature vector and the visual feature vectors stored in the historical interaction sequence. Analyze the spatial position corresponding to the feature with the highest similarity to the current image feature vector and exceeding a preset threshold as the third visual association point. Using the first visual attention weight peak point, the second visual coordinate point, and the third visual association point as boundary points, generate a minimum envelope ellipse and calculate the ratio of the area to the perimeter of the ellipse as a spatial clustering index.

[0008] Step 2: Selectively enhance the features located within the minimum envelope ellipse region in the image feature vector based on the spatial clustering index to generate spatially enhanced image feature vectors, and perform a multi-dimensional constrained computing power assessment covering computing power requirements, memory capacity requirements, and communication bandwidth requirements to generate a dynamic resource scheduling strategy.

[0009] Step 3: Based on the dynamic resource scheduling strategy, the speech feature vector, gesture spatial feature vector and spatially enhanced image feature vector are subjected to lightweight cross-modal fusion processing to obtain a unified multimodal joint semantic vector.

[0010] Step 4: Based on multimodal joint semantic vectors and dynamic resource scheduling strategies, the target lightweight inference model is matched from the pre-built lightweight inference model library to perform multi-step inference and attention calibration processing, and generate preliminary inference results and corresponding confidence scores.

[0011] Step 5: Make a real-time judgment on the preliminary reasoning results and corresponding confidence levels. When the preset time delay threshold and confidence level threshold are met, the final multimodal understanding result is obtained.

[0012] Step 6: Collect resource consumption data, performance index data, geometric parameters of the minimum envelope ellipse and spatial clustering index throughout the process, and update the adaptation relationship between the evaluation parameters used in the multidimensional constrained computing power evaluation and the lightweight inference model library.

[0013] The above-described solution of the present invention has at least the following beneficial effects:

[0014] By generating a minimum envelope ellipse from three visual feature points and quantifying its spatial clustering, and using this as a basis for selective enhancement and attenuation of image feature vectors in a regionalized manner, the system can lock in visually effective feature regions related to interactive intent. This avoids irrelevant features from entering subsequent fusion and inference stages and consuming computational and memory resources, thus reducing the basic computational power consumption of multimodal feature processing. Furthermore, a multi-dimensional constrained computational power evaluation system covering computing power, memory capacity, and communication bandwidth is constructed to estimate the resource requirements of each stage of multimodal feature processing and generate dynamic resource scheduling strategies accordingly. This effectively avoids single resource bottlenecks and resource mismatch problems, improving efficiency and efficiency. This approach improves resource utilization on platforms with limited computing power and solves the problem of wasted computing power caused by untargeted resource scheduling. It employs a lightweight cross-modal fusion processing strategy based on dynamic resource scheduling, unifying feature dimensions through linear projection and capturing semantic relationships between multimodal features using cross-attention, thus avoiding the loss of multimodal semantic relationships caused by lightweight processing. Furthermore, by matching multimodal joint semantic vectors with computing power requirements to a target lightweight inference model, and combining multi-step inference with iterative attention calibration, it compensates for the potential shortcomings of lightweight models in inference accuracy, achieving a balance between inference efficiency and accuracy.

[0015] By setting dual real-time judgment criteria of latency threshold and confidence threshold, the initial inference results are rigorously judged. This ensures that the generation speed of inference results meets the latency requirements of real-time interaction, while effectively filtering out inference results with low confidence, avoiding misunderstandings caused by simply pursuing inference speed. By collecting resource consumption, performance indicators, elliptic geometric parameters, and spatial clustering indicators throughout the entire process, the system dynamically updates the adaptation relationship between the parameters of multidimensional constraint computing power evaluation and the lightweight inference model library, enabling the system to have adaptive iterative upgrade capabilities and ensuring the stability and continuity of inference acceleration. Through end-to-end inference acceleration design, the system optimizes computing power consumption and improves inference speed at every stage, from feature processing, resource scheduling, fusion inference to result verification. This enables low-latency inference of multimodal large models, solves the response latency problem caused by redundant information processing and inefficient computing power utilization, and improves the smoothness and user experience of real-time human-computer interaction. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the method for accelerating real-time inference of multimodal large models provided in an embodiment of the present invention.

[0017] Figure 2 This is a flowchart illustrating step 3 provided in an embodiment of the present invention. Detailed Implementation

[0018] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0019] like Figure 1 As shown, embodiments of the present invention propose a method for accelerating real-time inference of multimodal large models, the method comprising the following steps:

[0020] Step 1: Perform parallel perception and feature extraction processing on the original multimodal input data to obtain speech feature vectors, image feature vectors, and gesture spatial feature vectors. Based on the image feature vectors, construct a spatial weight distribution representing the visual attention distribution by calculating the weighted sum of the feature components at each position. Analyze the position corresponding to the maximum weight as the first visual attention weight peak point. Map the gesture spatial feature vector to the image coordinate system corresponding to the image feature vector to obtain the second visual coordinate point. Perform element-wise similarity calculation between the image feature vector and the visual feature vectors stored in the historical interaction sequence. Analyze the spatial position corresponding to the feature with the highest similarity to the current image feature vector and exceeding a preset threshold as the third visual association point. Using the first visual attention weight peak point, the second visual coordinate point, and the third visual association point as boundary points, generate a minimum envelope ellipse and calculate the ratio of the area to the perimeter of the ellipse as a spatial clustering index.

[0021] Step 2: Selectively enhance the features located within the minimum envelope ellipse region in the image feature vector based on the spatial clustering index to generate spatially enhanced image feature vectors, and perform a multi-dimensional constrained computing power assessment covering computing power requirements, memory capacity requirements, and communication bandwidth requirements to generate a dynamic resource scheduling strategy.

[0022] Step 3: Based on the dynamic resource scheduling strategy, the speech feature vector, gesture spatial feature vector and spatially enhanced image feature vector are subjected to lightweight cross-modal fusion processing to obtain a unified multimodal joint semantic vector.

[0023] Step 4: Based on multimodal joint semantic vectors and dynamic resource scheduling strategies, the target lightweight inference model is matched from the pre-built lightweight inference model library to perform multi-step inference and attention calibration processing, and generate preliminary inference results and corresponding confidence scores.

[0024] Step 5: Make a real-time judgment on the preliminary reasoning results and corresponding confidence levels. When the preset time delay threshold and confidence level threshold are met, the final multimodal understanding result is obtained.

[0025] Step 6: Collect resource consumption data, performance index data, geometric parameters of the minimum envelope ellipse and spatial clustering index throughout the process, and update the adaptation relationship between the evaluation parameters used in the multidimensional constrained computing power evaluation and the lightweight inference model library.

[0026] In this embodiment of the invention, a minimum envelope ellipse is generated by three visual feature points and the spatial clustering is quantified. Based on this, selective enhancement and attenuation of image feature vectors are performed in a regionalized manner. This can lock the visually effective feature regions related to the interaction intent, avoiding the problem of irrelevant features occupying computing and memory resources in subsequent fusion and inference stages, and reducing the basic computing power consumption of multimodal feature processing. A multi-dimensional constrained computing power evaluation system covering computing power, memory capacity, and communication bandwidth is constructed to estimate the resource requirements of each stage of multimodal feature processing and generate dynamic resource scheduling strategies accordingly, effectively avoiding single resource bottlenecks and resource mismatches. The problem is that it improves the resource utilization of limited computing power platforms and solves the problem of computing power waste caused by untargeted resource scheduling. It carries out lightweight cross-modal fusion processing based on dynamic resource scheduling strategy. It unifies feature dimensions through linear projection and captures semantic associations between multimodal features by combining cross attention, avoiding the loss of multimodal semantic associations caused by lightweight processing. On the other hand, it matches the target lightweight inference model with multimodal joint semantic vectors and computing power requirements, and combines multi-step inference and iterative attention calibration processing to make up for the potential deficiency of lightweight model in inference accuracy, and achieves a dual balance of inference efficiency and inference accuracy.

[0027] By setting dual real-time judgment criteria of latency threshold and confidence threshold, the initial inference results are rigorously judged. This ensures that the generation speed of inference results meets the latency requirements of real-time interaction, while effectively filtering out inference results with low confidence, avoiding misunderstandings caused by simply pursuing inference speed. By collecting resource consumption, performance indicators, elliptic geometric parameters, and spatial clustering indicators throughout the entire process, the system dynamically updates the adaptation relationship between the parameters of multidimensional constraint computing power evaluation and the lightweight inference model library, enabling the system to have adaptive iterative upgrade capabilities and ensuring the stability and continuity of inference acceleration. Through end-to-end inference acceleration design, the system optimizes computing power consumption and improves inference speed at every stage, from feature processing, resource scheduling, fusion inference to result verification. This enables low-latency inference of multimodal large models, solves the response latency problem caused by redundant information processing and inefficient computing power utilization, and improves the smoothness and user experience of real-time human-computer interaction.

[0028] In a preferred embodiment of the present invention, in step 1:

[0029] Step 001 involves parallel perception and feature extraction processing of the raw multimodal input data to obtain speech feature vectors, image feature vectors, and gesture spatial feature vectors. Specifically, this includes: first, classifying the raw multimodal input data received by the large multimodal model. The raw multimodal input data includes three categories: speech input data, visual image input data, and gesture action input data. A parallel perception acquisition mechanism is initiated for these three types of data, simultaneously performing perception acquisition and preprocessing on each type of data to avoid time loss caused by sequential processing. Then, feature extraction is performed separately for each type of preprocessed data. Finally, the corresponding feature vectors are obtained. The specific processing steps are as follows: Preprocessing the speech input data involves sequentially sampling, denoising, and framing the speech signal. Then, acoustic and semantic features are extracted from the framed speech signal. Acoustic feature extraction requires first converting the framed signal to a Mel spectrum using a Mel filter bank, then performing a discrete cosine transform on the Mel spectrum to extract the Mel frequency cepstral coefficients as the core acoustic features. Simultaneously, the fundamental frequency, speech rate, and spectral entropy of each frame are calculated to supplement auxiliary acoustic features. The specific calculation process is as follows: the fundamental frequency is calculated using the autocorrelation method, and the autocorrelation function formula is... ,in These are discrete sampled values ​​of a single frame of speech signal. For sampling point index, For delay amount, The number of sampling points for a single frame of speech signal is used to calculate the autocorrelation function value under different delays using this formula. The frequency value corresponding to the peak value of the function is found, and noise interference peaks below the preset amplitude threshold (set to 0.02, based on the amplitude range after normalization of the speech signal, which can effectively filter weak noise peaks) are removed to obtain the fundamental frequency of the frame. The speech rate is calculated by counting the number of syllables corresponding to the effective speech frames per unit time. First, the effective speech and silent segments in the frame signal are distinguished, and the number of syllables corresponding to the effective speech frames per unit time is counted and converted into syllables per minute as the speech rate. The spectral entropy is obtained by first performing a fast Fourier transform on the single frame of speech signal to obtain the spectrum, calculating the power ratio of each frequency component to form a frequency probability distribution, and then calculating the uncertainty of this distribution using the entropy formula to obtain the spectrum. Entropy; integrating core and auxiliary acoustic features to form an acoustic feature set; semantic feature extraction requires first constructing and training a pre-trained speech semantic model. The model construction adopts a three-layer structure of input layer, feature encoding layer, and semantic representation layer. The input layer receives the speech feature sequence after frame segmentation. The feature encoding layer adopts the Transformer encoder architecture and sets a multi-head attention mechanism to capture temporal correlation. The semantic representation layer outputs frame-level semantic vectors through a fully connected network. The model training uses a large-scale speech-text alignment dataset as training samples. First, the samples are preprocessed by format unification, noise cleaning, and label annotation. An adaptive moment estimation optimizer is used with the cross-entropy loss function as the optimization objective. Iterative training is carried out until the loss value tends to stabilize and the semantic matching accuracy on the validation set reaches the standard, thus completing the model training.

[0030] The framed speech signal is input into the pre-trained speech semantic model. Frame-level semantic representations are extracted through a feature encoding layer. Then, the semantic representations of all frames are aggregated using temporal average pooling to obtain global semantic features. The acoustic feature set and global semantic features are then integrated in an orderly manner according to the signal temporal sequence to construct a speech feature vector. Visual image input data is preprocessed, including image normalization, deblurring, and pixel calibration. Multi-layer visual feature extraction is then performed on the preprocessed image to extract texture, contour, and semantic features at different spatial locations. All spatial feature information is then integrated in an orderly manner according to the image's spatial coordinates to construct an image feature vector. Gesture input data is preprocessed, including the detection, tracking, and position calibration of key points in the gesture. Spatial feature information such as the position, trajectory, and posture of the gesture in three-dimensional space is extracted. All extracted spatial feature information is then integrated in an orderly manner according to the temporal sequence and spatial location of the gesture to construct a gesture spatial feature vector. The perception acquisition, preprocessing, and feature extraction of these three types of data are executed in parallel. After all processing steps are completed, the speech feature vector, image feature vector, and gesture spatial feature vector are obtained simultaneously.

[0031] Step 002: Based on the image feature vector, a spatial weight distribution representing the visual attention distribution is constructed by calculating the weighted sum of its positional feature components. The position corresponding to the maximum weight is resolved as the first visual attention weight peak point. The gesture spatial feature vector is mapped to the image coordinate system corresponding to the image feature vector to obtain the second visual coordinate point. Specifically, this includes: based on the image feature vector obtained in step 001, extracting all spatial positional feature components contained in the vector, and configuring a corresponding positional weight for each spatial positional feature component. This positional weight is pre-set according to the design requirements of the visual attention mechanism, with a value range of 0.01 to 1.0. This avoids both excessively low weights that would mask weak and effective features and excessively high weights that would cause feature distortion. At the same time, the positional weight of the core visual region is not less than 0.5. This means that the core visual region is more strongly associated with multimodal interaction intent, and its feature influence needs to be strengthened with higher weights. The positional weights correspond one-to-one with the spatial positions of the image feature components. Multiplication is performed on the feature components of each spatial position and the corresponding positional weight to obtain the weighted sum of the feature components of a single spatial position. This process is repeated for all spatial positions of the image feature vector to complete the weighted sum calculation of all positional feature components. Using the spatial position of the image as the horizontal axis and the weighted sum value corresponding to each spatial position as the vertical axis, the weighted sum values ​​of all spatial positions are arranged according to the spatial coordinate distribution to construct a spatial weight distribution that represents the distribution of visual attention. All weighted sum values ​​in the spatial weight distribution are compared, and the weighted sum with the largest value is selected. The image spatial position corresponding to this maximum value is the first visual attention weight peak point.

[0032] A corresponding image coordinate system is established for the image feature vector obtained in step 001. The origin, horizontal and vertical axes, and scale of this coordinate system are determined. The scale of the image coordinate system is precisely mapped to the pixel coordinates of the image, achieving a one-to-one correspondence between the image spatial position and the coordinate system coordinates. Parameters representing the spatial position of the gesture core are extracted from the gesture spatial feature vector obtained in step 001. These parameters are the three-dimensional Cartesian coordinates (X3D, Y3D, Z3D) of the gesture core point, where X3D and Y3D represent the horizontal and vertical spatial positions, and Z3D represents the distance to the image acquisition device. The preset coordinate transformation rule is as follows: First, a perspective projection transformation algorithm is used, with the optical center of the image acquisition device as the projection center. The process involves combining intrinsic device parameters (focal length, pixel equivalent) and extrinsic parameters (installation posture angle). The X3D and Y3D coordinates in the three-dimensional coordinate system are multiplied by the focal length and then divided by Z3D to eliminate distance influence, yielding preliminary two-dimensional coordinates. Subsequently, the preliminary two-dimensional coordinates are offset by subtracting the three-dimensional projected coordinates corresponding to the origin of the image coordinate system to compensate for device installation position deviations. Finally, the calibrated coordinates are quantized and adjusted to ensure their scale perfectly matches the established image coordinate system, completing the conversion from three-dimensional spatial position parameters of the gesture to two-dimensional coordinate parameters. The converted two-dimensional coordinate parameters are then mapped to the image coordinate system corresponding to the established image feature vector; the resulting coordinate point is the second visual coordinate point.

[0033] Step 003 involves calculating the element-wise similarity between the image feature vector and the visual feature vectors stored in the historical interaction sequence. The spatial location corresponding to the feature with the highest similarity to the current image feature vector and exceeding a preset threshold is parsed as a third visual association point. Specifically, this includes: retrieving all stored visual feature vectors from the historical interaction sequence of the multimodal large model. The visual feature vectors in the historical interaction sequence are all visual feature vectors that the model has previously processed and that are related to the interaction intent, and each historical visual feature vector corresponds to a unique image spatial location information; and calculating the element-wise similarity between the current image feature vector obtained in Step 001 and each retrieved historical visual feature vector. The specific calculation process is as follows: first, calculate the similarity between the current image feature vector and a single historical visual feature vector. The dot product of the spatial location feature components corresponding to the visual feature vector is calculated by performing multiplication operations on the feature components at the same position in the two vectors, and then adding the results of all multiplication operations to obtain the sum of the dot products of the two vectors. Next, the magnitude of the current image feature vector is calculated by squaring all feature components of the current image feature vector, adding the results of all squaring operations, and then taking the square root of the addition results to obtain the magnitude of the current image feature vector. Then, the same calculation method is used to calculate the magnitude of a single historical visual feature vector. Finally, the sum of the dot products of the two vectors is divided by the product of the magnitude of the current image feature vector and the magnitude of the historical visual feature vector to obtain the element-wise similarity value between the current image feature vector and the historical visual feature vector.

[0034] Following the above calculation process, the element-wise similarity calculation between the current image feature vector and all historical visual feature vectors is performed sequentially to obtain a set of similarity values. These similarity values ​​are then filtered. First, all values ​​below a preset similarity threshold of 0.7 are removed. This threshold effectively eliminates historical visual features with low semantic relevance while retaining a sufficient number of potentially relevant features, balancing filtering accuracy and feature coverage. Next, the highest similarity value is selected from the remaining similarity values ​​to determine the corresponding historical visual feature vector. Finally, the image spatial location corresponding to the selected historical visual feature vector in the historical interaction sequence is extracted; this spatial location is the third visual association point.

[0035] This embodiment employs a parallel perception and feature extraction processing method for the original multimodal input data. Compared to serial processing, this shortens the extraction time of multimodal features, improves the efficiency of early data processing, and meets the low latency requirements of real-time inference. By constructing a visual attention spatial weight distribution through multiplication and addition operations of feature components and positional weights, the visual attention spatial weight distribution can be accurately quantified at each spatial location in the image, thereby accurately locating the peak point of the first visual attention weight and effectively capturing the most attention-grabbing core visual region in the image. The gesture spatial feature vector is accurately mapped to the image coordinate system corresponding to the image feature vector, realizing the spatial association between gesture actions and visual images. This allows the second visual coordinate point to accurately correspond to the image region pointed to by the gesture, effectively associating the spatial semantics of gesture features and visual features, and avoiding spatial decoupling between gestures and visual images. An element-wise similarity calculation method is used. By combining historical interaction sequences, the system can locate the historical visual spatial position with the highest semantic correlation to the current visual features. The resulting third visual correlation point effectively connects the visual semantics of historical interactions, allowing visual feature analysis to incorporate historical interaction information and improving the rationality and accuracy of visual core region positioning. The precise analysis and determination of the three visual feature points provide clear three-dimensional boundary points for the subsequent generation of the minimum envelope ellipse, giving the generation of the minimum envelope ellipse a quantitative spatial basis and enabling precise delineation of visual regions related to interaction intentions from a spatial dimension. Precise feature extraction is performed on voice, vision, and gesture data respectively to form standardized feature vectors. At the same time, spatial correlation between gestures, historical interaction information, and visual images is realized, allowing the three types of multimodal features to form an effective connection at the spatial semantic level, avoiding the loss of fusion accuracy caused by semantic disconnection during cross-modal fusion.

[0036] In a preferred embodiment of the present invention, step 1 further includes:

[0037] Step 100: Based on the coordinate set consisting of the first visual attention weight peak point, the second visual coordinate point, and the third visual associated point, calculate an ellipse that completely contains the coordinate set and has the smallest area. This ellipse is defined as the minimum envelope ellipse. Specifically, this includes: First, extracting the specific coordinates of the three visual feature points in the image coordinate system, denoted as the first point coordinate, the second point coordinate, and the third point coordinate, respectively. Recording the x and y coordinate values ​​of each point one by one. By calculating the straight-line distance between each pair of points and the angle between the connecting line and the horizontal axis of the image coordinate system, clarifying the relative spatial relationship between the three points, i.e., determining whether the three points are distributed in an isosceles triangle, an acute triangle, or an obtuse triangle, and marking the closest and farthest point pairs to avoid affecting subsequent calculations due to coordinate confusion. Next, using the coordinate set consisting of these three points as the core basis, gradually calculate the four core parameters of the ellipse: the ellipse center coordinates, the length of the major semi-axis, the length of the minor semi-axis, and the rotation angle. All parameter calculations are carried out around the core objective of completely containing the three points and having the smallest area, ensuring that the ellipse neither misses key feature points nor expands the coverage area additionally.

[0038] When calculating the coordinates of the ellipse center, the three-point coordinate component mean method is used. First, the x-coordinates of the three points are added together, and the sum is divided by three; the result is the x-coordinate of the ellipse center. Then, the y-coordinates of the three points are added together, and the sum is divided by three; the result is the y-coordinate of the ellipse center. This method ensures that the ellipse center evenly covers the area containing the three points. Next, the major and minor axes and rotation angle of the ellipse are calculated. First, vectors are constructed relative to the ellipse center from the ellipse center, with the ellipse center as the origin. The x-coordinate component of each vector is the x-coordinate of the corresponding feature point minus the ellipse angle. The x-coordinate of the center is calculated, and the y-coordinate component is the y-coordinate of the corresponding feature point minus the y-coordinate of the ellipse center. The magnitude of each vector is then calculated by adding the squares of the x-coordinate and y-coordinate components, and then taking the square root of the sum. This yields the distances from the three points to the ellipse center. The two largest distances are selected, and these two points are the key points determining the direction of the ellipse's major axis. The dot product of the vectors corresponding to these two points is used to calculate the angle between them, thus determining the rotation angle of the ellipse and ensuring that the major axis accurately aligns with the spatial distribution of the three points. Finally, iterative calculations are performed using geometric constraints. The lengths of the major and minor semi-axes are determined as follows: the major semi-axe is 1.05 times the distance from the point with the largest distance to the center of the ellipse. This small margin is reserved to prevent feature points from exceeding the ellipse's range due to coordinate calculation errors, ensuring complete coverage of the target point. The minor semi-axe length is determined through repeated verification and adjustment. Initially, the minor semi-axe length is preset to 0.6 times the major semi-axe length (based on the geometric characteristics of the three-point distribution, balancing initial coverage with subsequent adjustment space). Then, the distance from the third point to the ellipse boundary is calculated. If the point is outside the ellipse, the minor semi-axe length is increased by 0.05 times the major semi-axe length each time until the point enters the ellipse. If a point is inside the ellipse and has significant redundant space (the criterion is that the distance from the point to the ellipse boundary is greater than 0.2 times the length of the minor semi-axis), then the length of the minor semi-axis is reduced by 0.03 times the length of the major semi-axis each time. This adjustment is iterated until the ellipse can just enclose the third point (the distance from the point to the ellipse boundary is between 0 and 0.02 times the length of the minor semi-axis). At this point, the area of ​​the ellipse reaches its minimum. Finally, based on the calculated coordinates of the ellipse center, the length of the major semi-axis, the length of the minor semi-axis, and the rotation angle, a complete ellipse outline is generated. This ellipse that can completely contain the three visual feature points and has the smallest area is defined as the minimum envelope ellipse.

[0039] Step 101: Calculate the ratio between the area and perimeter of the minimum enveloping ellipse, using this ratio as a spatial aggregation index. Specifically, this includes: First, calculating the area of ​​the minimum enveloping ellipse using the standard ellipse area formula, which states that the area of ​​an ellipse equals pi multiplied by the length of its major semi-axis multiplied by the length of its minor semi-axis. The calculation involves multiplying the lengths of the major and minor semi-axis obtained in Step 100 to obtain their product, and then multiplying this product by pi to obtain the area of ​​the minimum enveloping ellipse. Next, calculating the perimeter of the minimum enveloping ellipse. Since there is no precise analytical formula for the ellipse perimeter, the Ramanujan approximation formula is commonly used in engineering. This formula achieves a precise approximation of the ellipse perimeter through fixed coefficient correction. The specific calculation process is as follows: First, calculate the difference between the lengths of the major and minor semi-axis, and then square this difference to obtain the squared result. Second, calculate the sum of the lengths of the major and minor semi-axis, and then square this sum to obtain the squared result. Third, use... The squared difference is divided by the squared sum to obtain the proportionality coefficient, which characterizes the flattening of the ellipse. The fourth step involves multiplying the proportionality coefficient by 0.25, the first correction factor preset in the Ramanujan formula, to weaken the excessive influence of flattening on the perimeter calculation. Then, 1 is added to the product, the second correction factor preset in the formula, used to calibrate the basic calculation benchmark. Combining these two factors yields the correction coefficient, ensuring the perimeter calculation result is closer to the actual value. The fifth step involves multiplying the sum of the major and minor semi-axis lengths by pi to obtain a basic perimeter reference value. This reference value is then multiplied by the correction coefficient to obtain the approximate perimeter value of the minimum envelope ellipse. Finally, the spatial clustering index is calculated by dividing the area of ​​the minimum envelope ellipse obtained in this step by the ellipse's perimeter. The resulting ratio is the spatial clustering index, which directly reflects the degree of spatial clustering of visual features in the region where the three visual feature points are located. A larger ratio indicates stronger feature clustering, while a smaller ratio indicates more dispersed feature distribution.

[0040] This embodiment delineates the core visual region, effectively avoiding irrelevant visual features from entering subsequent processing, reducing redundant calculations from a spatial perspective, and meeting the real-time inference needs of scenarios with limited computing power. The calculation process of the minimum envelope ellipse balances complete envelopment and area minimization, ensuring coverage of all key visual feature points while avoiding effective feature dilution due to an excessively large ellipse range, thus improving the targeting of visual feature processing. The spatial clustering index quantifies the degree of feature clustering through a concrete ratio, making the feature enhancement strategy more scientific rather than subjective. All calculation processes are based on geometric principles and approximate formulas, with simple and efficient operational logic, no complex redundant steps, and no additional computing power burden. It adapts to the low latency requirements of multimodal real-time inference, realizing the transformation of visual feature points from scattered coordinates to structured elliptical regions.

[0041] In a preferred embodiment of the present invention, step 2 includes:

[0042] Step 200: Determine a feature gain coefficient based on the spatial clustering index value. Multiply the values ​​of all feature components whose spatial coordinates lie within the minimum envelope ellipse region in the image feature vector by the feature gain coefficient, and attenuate the values ​​of feature components outside the region to generate a spatially enhanced image feature vector. Specifically, this includes: First, establishing the correspondence between the spatial clustering index and the feature gain coefficient, dividing the interval according to the clustering index value to determine the coefficient size, ensuring that the gain strategy fits the spatial distribution characteristics of the features. If the spatial clustering index is greater than or equal to 0.8, it indicates that the visual feature clustering is extremely strong, and the feature gain coefficient is set to 1.5, moderately enhancing the core region features; if the spatial clustering index is between 0.4 and 0.8, it indicates that the feature clustering is moderate, and the feature gain coefficient is set to 1.2, balancing the intensity of core and surrounding features; if the spatial clustering index is less than 0.4, it indicates that the feature distribution is scattered, and the feature gain coefficient is set to 1.0, without additional enhancement to avoid noise amplification. Next, traverse the original image feature vector obtained in step 001, and determine one by one whether the spatial coordinates corresponding to each feature component lie within the minimum envelope ellipse region. The determination method is to use the standard equation of the rotated ellipse, the formula is...

[0043] ,in( , () represents the coordinates of the center of the ellipse. The length of the major semi-axis For the length of the shorter half-axis, Let be the angle of rotation of the ellipse, ( Let be the spatial coordinates of the feature component to be judged. After substituting the coordinates into the equation, if the result is less than or equal to 1, it is determined to be a feature within the region; otherwise, it is a feature outside the region. For all feature components within the region, their values ​​are multiplied by a determined feature gain coefficient to complete the feature enhancement process. The feature gain coefficient is precisely selected according to the spatial clustering index range. When the spatial clustering index is ≥0.8, the gain coefficient is 1.5, because the core region features are extremely clustered at this time, and this value can moderately enhance the intensity of the core features while avoiding excessive gain that could lead to feature value saturation and distortion. When the clustering index is between 0.4 and 0.8, the gain coefficient is 1.2 to balance the intensity difference between the core region features and the surrounding weakly correlated features, taking into account both feature specificity and completeness. When the clustering index is <0.4, the gain coefficient is 1.0, because the feature distribution is dispersed, and no additional enhancement can avoid amplifying noise features and preserve the feature strength. To improve the semantic purity of the features, all feature components outside the region are attenuated by multiplying their values ​​by 0.3. This coefficient, between 0 and 1, represents the attenuation ratio, meaning that the values ​​of feature components outside the region are reduced to 30% of their original values. The value is determined based on thousands of samples from multimodal interaction scenarios (including typical scenarios such as human-computer interaction and smart cockpits). This value can accurately balance the weakening of irrelevant features and the preservation of potential information. It significantly reduces the intensity of irrelevant features, minimizing interference with subsequent cross-modal fusion and inference calculations, without excessive attenuation (e.g., a coefficient below 0.2 easily loses weak semantic associations, while a coefficient above 0.4 results in insufficient weakening). It can retain potential weak semantic associations in the features and avoid semantic breaks. Finally, all feature components after gain and attenuation processing are re-integrated according to the spatial coordinate order of the original image feature vector to generate a spatially enhanced image feature vector.

[0044] Step 201 involves estimating the computational power and memory requirements for processing speech feature vectors, gesture spatial feature vectors, and spatially enhanced image feature vectors, as well as the communication bandwidth requirements for transmitting these feature vectors between heterogeneous computing units. Specifically, this includes estimating the computational power required for processing speech feature vectors, gesture spatial feature vectors, and spatially enhanced image feature vectors. The calculation logic is as follows: The total number of feature components in each feature vector is counted. The number of single-component operations is determined based on the computational complexity of each feature processing step. Specifically, the number of single-component operations for speech features is 32 (including core operations such as Mel-spectrum conversion and cepstral coefficient extraction), the number of single-component operations for gesture spatial features is 24 (including operations such as 3D coordinate calibration and trajectory fitting), and the number of single-component operations for spatially enhanced image features is 48 (including operations such as feature gain / attenuation and spatial semantic calibration). The number of single-component operations is broken down and counted according to the core steps of each feature processing step to ensure coverage of the entire computation process. Then, the total number of feature components is multiplied by the number of single-component operations to obtain the total number of operations required for processing each feature vector. Finally, the total number of operations is divided by a preset unit-time computation benchmark value, which is taken as per second. The floating-point operations (based on the single-core computing performance setting of mainstream heterogeneous computing cores, adapted to the computing power level of multimodal real-time inference) are used to obtain the computing power requirements for processing the corresponding feature vectors. The unit is uniformly the number of floating-point operations per second to ensure that the statistical standard of computing power requirements for the three types of features is consistent.

[0045] The memory capacity required to store the three types of feature vectors and temporary data during processing is estimated separately. The calculation logic is as follows: First, determine the memory size occupied by a single component based on the data type of the feature component. Multiply the memory size of a single component by the total number of feature components to obtain the storage capacity requirement of the feature vector itself. Then, reserve temporary data storage space at 0.2 times the storage capacity of the feature vector (to adapt to the storage requirements of intermediate results during feature processing). Add the two together to obtain the total memory capacity requirement of the corresponding feature vector. The unit is unified as bytes.

[0046] The communication bandwidth required for transmitting three types of feature vectors between heterogeneous computing units is estimated. The calculation logic is as follows: first, calculate the total data volume of a single feature vector (consistent with the storage capacity of the feature vector itself in the memory capacity requirement); then, based on the time constraint of feature transmission, divide the total data volume by the maximum allowable transmission time to obtain the communication bandwidth requirement for transmitting the corresponding feature vector, with the unit being bytes per second. At the same time, considering the transmission requirements of the three types of feature vectors and the concurrent interference of the transmission link, reserve redundant bandwidth at 1.1 times the total bandwidth requirement, and finally determine the overall communication bandwidth requirement.

[0047] Step 202a: Based on computing power requirements, assess whether available computing resources meet the computation completion constraint to obtain a first assessment result; based on memory capacity requirements, assess whether available memory resources meet the storage capacity constraint to obtain a second assessment result; based on communication bandwidth requirements, assess whether available data transmission paths meet the transmission completion constraint to obtain a third assessment result. Specifically, this includes: obtaining the available computing power of the current heterogeneous computing unit, i.e., the total number of floating-point operations per second of all idle computing cores; comparing the available computing power with the total computing power requirement estimated in step 201; if the available computing power is greater than or equal to the total computing power requirement, it indicates that the computation completion constraint is met, and the first assessment result is qualified; if the available computing power is less than or equal to the total computing power requirement, it indicates that the computation completion constraint is met, and the first assessment result is qualified. If the available memory is less than the total computing power requirement, it means that the computation cannot be completed within the specified time, and the first evaluation result is unqualified. Next, obtain the current available memory capacity of the system, i.e., the remaining memory space after deducting the memory already occupied. Compare the available memory capacity with the total memory capacity requirement estimated in step 201. If the available memory capacity is greater than or equal to the total memory capacity requirement, and at least 0.1 times the total memory requirement is reserved as free memory (to cope with sudden temporary data growth), the second evaluation result is qualified. If the available memory capacity is insufficient, or the reserved free memory is inadequate, the second evaluation result is unqualified. Then, obtain the available communication bandwidth of the current data transmission path, i.e., the remaining bandwidth after deducting the occupied bandwidth from the maximum actual transmission capacity of the transmission link. Compare the available communication bandwidth with the overall communication bandwidth requirement determined in step 201. If the available communication bandwidth is greater than or equal to the overall communication bandwidth requirement, it means that the real-time data transmission constraints can be met, and the third evaluation result is qualified. If the available communication bandwidth is less than the overall communication bandwidth requirement, it will lead to data transmission delay, and the third evaluation result is unqualified.

[0048] Step 202b: Based on the combined results of the first, second, and third evaluations, a dynamic resource scheduling strategy for allocating computing cores, memory regions, and data transmission channels is generated. Specifically, this includes: First, a comprehensive judgment is made on the first, second, and third evaluation results. If all three results are satisfactory, a scheduling strategy is generated based on the principle of optimal resource utilization. This principle specifically includes on-demand allocation, load balancing, and core priority. On-demand allocation means that resource supply strictly matches the needs of each feature processing task, avoiding over-allocation and waste. Load balancing avoids excessive load on individual computing cores, memory blocks, or transmission channels, ensuring stable resource operation. Core priority prioritizes resource supply for core feature processing tasks; if one or more tasks are unsatisfactory, core feature processing needs are prioritized, and resource allocation priority is adjusted. Regarding computing core allocation, idle computing cores are precisely allocated as a percentage based on the proportion of computing power demand for each feature vector. The proportional allocation formula is: Number of cores allocated to a feature = Total number of idle cores × (Computing power demand for that feature ÷ Total computing power demand for the three types of features). After rounding down, the remaining cores are allocated to the highest proportion of computing demand. The system prioritizes high-performance computing cores (such as GPU cores in heterogeneous computing units) to improve processing efficiency for spatially augmented image feature vectors with high computational demands, defined as those exceeding 40% of the total computational requirements. For memory allocation, the system divides memory into contiguous and independent blocks based on the total memory capacity requirements of each feature vector. Each block is designed as a binary structure of original feature data area + temporary data area, with dedicated address isolation markers at their boundaries to prevent memory address conflicts and data tampering. The original feature data area is allocated in full according to the corresponding feature vector's storage capacity, while the temporary data area is fixed at 0.2 times the capacity of the original data area, with an additional 5% reserved for dynamic expansion to handle sudden increases in intermediate results. For frequently accessed feature data, the system monitors and counts the number of accesses per second for each feature component. Feature data accessed more than 50 times per second is classified as high-frequency data, and the heterogeneous computing unit's L2 cache space is allocated at 1.2 times the total amount of high-frequency data. This cache space is exclusively allocated to avoid competing for resources with other low-frequency data, significantly reducing memory read latency.

[0049] In terms of data transmission channel allocation, based on the bus architecture of heterogeneous computing units, logically independent transmission channels are allocated for the three types of feature transmission tasks: voice, gesture, and vision. Each channel is configured with a unique communication identifier and bandwidth threshold. The bandwidth threshold for the voice feature transmission channel is set at 2GB / s to meet its high real-time requirements and ensure that voice data is transmitted without delay. The bandwidth threshold for the gesture spatial feature transmission channel is set at 1.2GB / s to match the requirements of moderate gesture data volume and moderate transmission timeliness. Due to the large data volume, the bandwidth threshold for the spatially enhanced image feature vector is set at 3GB / s to avoid transmission stuttering due to insufficient bandwidth. Concurrency isolation between channels is achieved through a bus scheduling mechanism to avoid data transmission interference. At the same time, a three-level bandwidth priority system is established. Voice feature vectors are classified as the highest priority due to real-time requirements, followed by gesture spatial feature vectors at the second level, and spatially enhanced image feature vectors at the third level. The first-priority channel exclusively occupies 20% of the total system bandwidth and enjoys priority response rights in bus scheduling. The channel transmission rate is monitored in real time, and if a delay occurs, the bandwidth expansion mechanism is immediately triggered to ensure that voice feature data transmission is delayed, ultimately forming a complete dynamic resource scheduling strategy.

[0050] This embodiment dynamically determines the feature gain coefficient through spatial clustering index, achieving differentiated enhancement and attenuation of visual features. It accurately strengthens core region features and weakens irrelevant features, reducing redundant computation from the source and adapting to the real-time inference needs of scenarios with limited computing power. Multi-dimensional resource requirement estimation covers three core dimensions: computing, memory, and communication, and includes redundancy reservation design to ensure that the requirement estimation is accurate and fits the actual processing scenario, avoiding inference delays caused by insufficient resource estimation. Dimensional computing power constraint assessment clarifies the supply capacity and demand gap of each resource, providing a quantitative basis for resource scheduling and avoiding low utilization or bottleneck problems caused by blindly allocating resources. The dynamic resource scheduling strategy combines the evaluation results and task priorities to achieve fine-grained allocation of computing cores, memory, and transmission channels, maximizing resource utilization while ensuring the real-time performance and stability of core feature processing. Feature enhancement, resource evaluation, and scheduling form a closed loop, which not only strengthens the semantic correlation of multimodal features but also reduces inference latency through reasonable resource allocation, balancing inference accuracy and speed. All processing follows a unified terminology system and quantitative calculation logic, with no subjective judgment steps, ensuring the objectivity and consistency of processing results.

[0051] like Figure 2 As shown, in another preferred embodiment of the present invention, step 3 includes:

[0052] Step 300: Based on the computing core allocation and data path planning information contained in the dynamic resource scheduling strategy, the speech feature vector, gesture spatial feature vector, and spatially enhanced image feature vector are transmitted to the corresponding processing units. Specifically, this includes: First, parsing the computing core allocation information in the dynamic resource scheduling strategy to clarify the precise matching relationship between various feature vectors and processing units. That is, the speech feature vector corresponds to a high-performance CPU core in the heterogeneous computing unit (adapting to the low latency requirements of speech signal timing processing, with a single core frequency of not less than 2.8GHz), and the gesture spatial feature vector corresponds to a dedicated DSP core (adapting to the lightweight operation of gesture coordinates). It supports single-cycle multi-dimensional vector operations, spatially enhanced image feature vectors correspond to GPU cores (adapted for parallel processing of high-dimensional image features, with a memory bandwidth of no less than 512GB / s), and simultaneously records the physical core number and memory address mapping table of each target processing unit. It also extracts data path planning information from the strategy, clearly defining the core parameters of each transmission path. The communication identifier uses a modality type-priority-link number format (e.g., Y-1-001 for voice, S-2-002 for gestures, T-3-003 for images) to ensure that each channel is uniquely identifiable. Bandwidth priority strictly follows the order: Level 1 (voice) > Level 2 (gestures) > ... The scheduling rules for Level 3 (image) transmission links are bound to dedicated PCIe 4.0 channels to avoid sharing links with other non-feature transmission tasks. Subsequently, the feature vector transmission process is initiated, using a flow control algorithm to adapt the bandwidth thresholds of each channel. Specifically, the voice channel transmits at a rate of 2GB / s, dynamically adjusting the data transmission frame size (128KB per frame) by sampling the transmission rate every 10ms. The gesture channel transmits at a rate of 1.2GB / s, with a single frame size of 64KB, balancing rate and transmission stability. Due to the larger data volume, the image channel transmits at a rate of 3GB / s, with a single frame size of 256KB, while simultaneously enabling direct DMA. Memory access technology reduces CPU intervention. During transmission, the bus scheduling mechanism uses a time-division multiplexing strategy to achieve channel concurrency isolation, allocating 40% of the bus time slice to the first-priority channel, 30% to the second-priority channel, and 30% to the third-priority channel to avoid cross-channel data interference. Layered data integrity checks are performed during transmission, triggering a check every 1MB of data transmitted. The first check verifies the total number of feature components by comparing the vector length flags at the sending and receiving ends to ensure no component loss. The second check verifies key values ​​by selecting the average of the first 10 and last 10 feature components of the vector and calculating the deviation between the sending and receiving ends, with the deviation not exceeding 0.001 is regarded as qualified; if any re-verification fails, the retransmission mechanism is immediately triggered. The upper limit of the number of retries is set to 3 times. The first two retransmissions are carried out at the original rate, and the rate is reduced to 50% of the original rate for the third time to improve stability. If the 3 retries fail, the system alarm is triggered. After all feature vectors are successfully transmitted to the corresponding processing units and the receiving end returns the signal of receiving completion - data is correct, each processing unit loads the feature data into the exclusive memory block and enters the subsequent feature projection transformation link.

[0053] Step 301 involves performing linear projection transformations on the speech feature vector, gesture space feature vector, and spatially enhanced image feature vector to obtain projected feature vectors of the same dimension. Specifically, this includes: determining a unified target dimension of 2048. This dimension is set based on the multimodal fusion semantic carrying requirements and computational power balance, covering the core semantic features of speech, gestures, and images while avoiding increased attention computation time due to excessively high dimensions; constructing a dedicated linear projection matrix for each type of feature vector, clarifying the matrix parameters and initialization rules, i.e., the original dimension of the speech feature vector is 512, corresponding to a projection matrix dimension of 2048×512; the gesture space feature vector... The original dimension is 256, corresponding to a projection matrix dimension of 2048×256; the original dimension of the spatially augmented image feature vector is 1024, corresponding to a projection matrix dimension of 2048×1024. All elements in the matrix are initialized using a uniform random distribution within the range [-0.01, 0.01], with a data type of 32-bit floating-point. After initialization, L2 regularization is applied (constraining the sum of squares of matrix elements to within 1) to avoid excessively large initial values ​​leading to feature distortion after projection. Subsequent model iterations optimize the matrix element values. A linear projection transformation is performed on the speech feature vector. Specifically, the original 512-dimensional speech feature vector is used as... The column vector is multiplied by a 2048×512 speech projection matrix. Each element of the projection matrix is ​​multiplied by the corresponding element of the speech feature vector, i.e., the element in the first row and first column of the matrix is ​​multiplied by the first component of the vector, the element in the first row and second column of the matrix is ​​multiplied by the second component of the vector, and so on, up to the element in the first row and 512th column of the matrix multiplied by the 512th component of the vector. These 512 products are then added together to obtain the first feature component of the projected vector. The same logic is used to traverse the 2048 rows of the projection matrix, calculating all projected feature components one by one, finally generating a 2048-dimensional speech projected feature vector, which is stored in row-major format. A completely consistent projection calculation method is used. The formula processes the gesture spatial feature vector and the spatially augmented image feature vector separately. Specifically, the gesture feature vector (256-dimensional column vector) is multiplied and accumulated by a 2048×256 gesture projection matrix, and the image feature vector (1024-dimensional column vector) is multiplied and accumulated by a 2048×1024 image projection matrix. Both generate 2048-dimensional projected feature vectors. After projection, a dimension consistency check is performed. By reading the dimension identifier field at the head of the feature vector, it is confirmed that all three types of vectors are 2048-dimensional. At the same time, the numerical range of the feature components of each vector is checked (constrained to the interval [-1, 1], and normalization adjustment is performed if it exceeds the range) to ensure that there are no dimension mismatches or numerical abnormalities.

[0054] Step 302: Calculate the cross-attention weights between the projected feature vectors, and perform a weighted summation of the projected feature vectors based on the cross-attention weights to generate a unified multimodal joint semantic vector. Specifically, this includes: First, constructing an attention calculation matrix. A 3×2048 dimension matrix is ​​built based on the three classes of 2048-dimensional projected feature vectors. The first row of the matrix corresponds to the 2048 components of the speech projected feature vector, the second row corresponds to the 2048 components of the gesture projected feature vector, and the third row corresponds to the 2048 components of the image projected feature vector. All matrix elements are rounded to six decimal places to ensure calculation accuracy. Next, calculate the initial cross-attention weights, focusing on the pairwise semantic associations between the three classes of vectors. Three sets of initial weights are generated (speech-gesture, speech-image, gesture-image), each weight being a single value. The calculation method is as follows: [The text abruptly ends here, likely due to an incomplete sentence or missing information.] Taking sound and gesture vectors as an example, the first step is to calculate the sum of their dot products, that is, to multiply the components of the corresponding dimensions in the first and second rows of the attention matrix in sequence (first-dimensional speech component × first-dimensional gesture component, second-dimensional speech component × second-dimensional gesture component, ... up to the 2048th dimension), and then add the 2048 product results to get the sum of the dot products; the second step is to calculate the magnitude of the speech vector, to square each of the 2048 components of the speech vector, to add all the square results and take the square root of the sum to get the magnitude of the speech vector; the third step is to calculate the magnitude of the gesture vector in the same way (square root of the sum of the squared components); the fourth step is to divide the sum of the dot products by the product of the magnitude of the speech vector and the magnitude of the gesture vector to get the initial cross-attention weights of speech and gesture. Following this logic, the initial cross-attention weights of speech-image and gesture-image are calculated in sequence to form three initial weight values.

[0055] The initial cross-attention weights are normalized by summing the three initial weights. Each initial weight is then divided by this sum to obtain the normalized cross-attention weights (the sum of the three weights equals 1). For example, if the speech-gesture weight is 0.3, speech-image weight is 0.4, and gesture-image weight is 0.3, the normalized weights can be adjusted to 0.33, 0.44, and 0.23 (rounded to two decimal places, with errors adjusted by the last decimal place). Simultaneously, the self-attention weight for each vector class is calculated, taking the average of the three normalized cross-attention weights to reinforce the core semantics of each modality. Finally, a joint semantic vector is generated through weighted summation, generating components dimension by dimension. The first step calculates the sum of weights for each vector class: Speech vector weight sum = Speech self-attention weight + Speech-gesture cross-attention weight + Speech-image cross-attention weight; Gesture vector weight sum = Gesture self-attention weight + Speech-gesture cross-attention weight. The first step involves weighting the speech vector by combining the attention weights and the gesture-image cross-attention weights. The total weights of the image vectors are calculated as: image self-attention weights + speech-image cross-attention weights + gesture-image cross-attention weights. The second step performs weighted processing on each type of vector, multiplying each dimension of the speech vector by the total speech weights. The gesture vectors and image vectors are processed in the same way to obtain the three-class weighted feature vectors. The third step accumulates the components dimension by dimension, adding the first-dimensional components of the three-class weighted vectors to obtain the first-dimensional component of the joint vector. The second-dimensional components of the three-class weighted vectors are added to obtain the second-dimensional component of the joint vector, and so on, until all 2048 dimensions are traversed to complete the calculation of all components. The fourth step performs normalization calibration on the joint vector (mapping all component values ​​to the interval [-1, 1]), finally generating a 2048-dimensional unified multimodal joint semantic vector. After generation, the vector integrity is verified (total number of components is 2048, no null values), and then it is stored in the GPU cache area.

[0056] In this embodiment, the directional transmission mechanism strictly follows a dynamic resource scheduling strategy to achieve precise matching between feature vectors and processing units. Combined with hierarchical bandwidth control and data verification, it ensures the real-time performance, integrity, and accuracy of feature transmission, avoiding semantic loss or processing delays caused by transmission. Linear projection transformation, through a unified dimension and dedicated projection matrix, preserves the core semantics of each modality's features while eliminating the obstacles to cross-modal fusion caused by dimensional differences. The optimizable nature of the projection matrix reserves space for subsequent improvements in semantic fusion accuracy. Cross-attention weight calculation, combined with similarity quantification and normalization, captures the semantic relationships between features of each modality, making the weighted summation process... It prioritizes strengthening highly correlated feature components, improving the semantic consistency of multimodal joint semantic vectors; the generated joint semantic vectors can comprehensively integrate speech, vision, and gesture information, providing high-quality fusion data for multimodal large-scale model inference; transmission rate control, projection dimension setting, and attention weight calculation all adopt quantitative logic, with no subjective judgment steps, ensuring the consistency and reproducibility of the processing process and improving the engineering feasibility of the technical solution; it adapts to the processing characteristics of heterogeneous computing units to allocate tasks, with CPU, DSP, and GPU each performing their respective tasks, and combined with the efficient computing logic of linear projection and attention fusion, it controls computing power consumption while ensuring fusion accuracy.

[0057] In a preferred embodiment of the present invention, step 4 includes:

[0058] Step 400: Based on the vector dimension of the unified multimodal joint semantic vector and the computational power and memory capacity requirements defined in the dynamic resource scheduling strategy, a matching model is selected from the pre-built lightweight inference model library as the target lightweight inference model. Specifically, this includes: First, clarifying the construction process of the pre-built lightweight inference model library to ensure that the models in the library adapt to the multimodal semantic inference requirements and possess lightweight characteristics. The first step is data preparation: collecting a large-scale multimodal interaction dataset covering feature data and corresponding semantic labels for three modalities: speech, gesture, and image. The data is cleaned, normalized, and labeled, and divided into training, validation, and test sets in an 8:1:1 ratio. The second step is basic model construction: based on the Transformer architecture, building basic inference models with different input dimensions (512-dimensional, 1024-dimensional, and 2048-dimensional). The number of neurons in each layer is configured according to the input dimension ratio to ensure the model's semantic capacity. The first step involves matching capabilities with input dimensions. The second step is lightweight optimization, which performs dual optimization of pruning and quantization on the base model. Pruning removes redundant parameters with absolute weight values ​​less than 0.0001, reducing the number of model parameters by 30% to 50%. Quantization converts the model parameters from 32-bit floating-point to 16-bit floating-point, reducing memory usage and computational consumption with minimal loss of precision. The third step is model training and validation, where the optimized model is trained using the training set. An adaptive moment estimation optimizer is used, with the cross-entropy loss function as the optimization objective. Iterative training continues until the loss value stabilizes. Hyperparameters are then adjusted using the validation set. Finally, the model's inference accuracy, computational requirements, and memory usage are evaluated using the test set, and the core parameters of each model are labeled. The fourth step is classification and archiving, where models are classified and stored according to three dimensions: input dimension, computational requirement level, and memory usage level. A model index table is established to record the accuracy indicators, optimization methods, and applicable scenarios of each model, forming a complete pre-built lightweight inference model library.

[0059] Subsequently, the core matching criteria were extracted: first, the dimension of the unified multimodal joint semantic vector was fixed at 2048 dimensions, serving as a hard matching condition for the model input dimension; second, the computing power and memory capacity requirements defined in the dynamic resource scheduling strategy, namely, the number of floating-point operations per second required to process the vector, and the total memory usage for storing model parameters and intermediate results; the pre-built lightweight inference model library was categorized and archived according to three dimensions: input dimension, computing power requirement level, and memory usage level. Each model was labeled with core parameters, namely, the input dimensions covered three categories: 512 dimensions, 1024 dimensions, and 2048 dimensions, and the computing power requirement level was divided into low (less than or equal to 5×) (times / second), medium (5×) Up to 2× (times / second), high (greater than 2×) The memory usage is categorized into small (less than or equal to 2GB), medium (2 to 4GB), and large (greater than 4GB). All models undergo pruning and quantization optimization to adapt to heterogeneous computing unit deployments. The model matching process proceeds in a priority order: First, models with an input dimension of 2048 are selected, eliminating candidate models with mismatched dimensions. Second, models whose computing power requirement level matches the available computing power are selected based on the computing power requirements in the dynamic resource scheduling strategy. If the available computing power is ≥2× Hours per second, a high-performance computing model can be selected; if the available computing power is 5× Up to 2× For speeds between [number] times / second, select a medium-level computing power model; if available computing power is <5× [number] times / second, [further options are needed]. The first step is to select a low-computing-power model based on the speed of calculations per second. The second step is to further filter models based on memory capacity requirements, ensuring that the model's memory usage is less than or equal to the available memory capacity, and reserving at least 0.1 times the model's memory usage as free memory to handle the storage needs of intermediate results during inference. If multiple models meet the criteria, they are sorted by inference accuracy priority, and the model with the highest verification accuracy in the multimodal semantic inference task is selected as the target lightweight inference model. If no perfectly matching model is found, the model with consistent input dimensions, the closest computing power and memory requirements, and not exceeding the constraint limit is selected. At the same time, the model dynamic compression mechanism is enabled to quantize the model parameters into 16-bit floating-point type to reduce memory usage and computing power consumption, ensuring adaptation to resource constraints. After selection, the target model is loaded into the corresponding computing unit (using the GPU core allocated in step 300). During the loading process, the integrity of the model parameters is verified, and the total parameters before and after loading are compared. A deviation of no more than 0.001 is considered a successful loading.

[0060] Step 401: Input the multimodal joint semantic vector into the target lightweight inference model and execute a multi-step inference process. The multi-step inference process includes: generating a primary semantic representation based on the multimodal joint semantic vector; performing a first round of semantic inference based on the primary semantic representation to obtain a first-round hidden state containing semantic features and contextual information, and performing a first round of attention weight distribution calibration to obtain calibrated attention weights; performing a second round of semantic inference based on the calibrated attention weights and the first-round hidden state, iteratively performing semantic inference and attention weight distribution calibration until a preset number of iterations is reached to obtain the final hidden state. Specifically, this includes: first, generating a primary semantic representation based on the unified multimodal joint semantic vector and constructing a semantic mapping matrix. The matrix has a dimension of 2048×2048, and its elements are initialized with a uniform random distribution in the interval [-0.01, 0.01]. It is then processed using L2 regularization. Specifically, L2 regularization calculates the sum of squares of all elements in the matrix, and then divides each element of the matrix by this sum of squares to ensure that the sum of squares of all elements in the processed matrix is ​​strictly constrained to within 1. This suppresses projection distortion caused by excessively large matrix elements and improves the stability of the semantic representation. The 2048-dimensional joint semantic vector is used as a column vector and multiplied with the semantic mapping matrix. Each row element of the matrix is ​​multiplied by its corresponding component in the vector, and the 2048 product results are summed sequentially to obtain a single primary semantic representation component. This process is repeated across all rows of the matrix to generate the 2048-dimensional... The initial semantic representation is then superimposed with a bias term (0.005 added to each component to offset initialization bias). Next, the first round of semantic inference is performed, inputting the initial semantic representation into the Transformer decoder layer of the target model. The decoder layer includes a multi-head attention mechanism and a fully connected network. The multi-head attention mechanism has eight heads, each with attention weights of 0.15, 0.14, 0.13, 0.13, 0.12, 0.12, 0.10, and 0.09, with a total weight of 1. This differentiated weight allocation strengthens the semantic capture capability of the core attention heads while also considering the auxiliary association information of secondary attention heads, avoiding semantic bias caused by excessively high weights for a single head. This multi-head attention mechanism is then used to... Semantic associations are captured, and an attention weight distribution is generated. The primary semantic representation and the attention weight distribution are then weighted and fused together and input into a fully connected network (containing two hidden layers, each with 4096 neurons). After ReLU activation, the first round of hidden states in 2048 dimensions are output. Subsequently, the first round of attention weight distribution calibration is performed, calculating the mean and standard deviation of the attention weight distribution. The mean is the sum of all weight values ​​divided by the total number of weights, and the standard deviation is the square root of the sum of the squares of the differences between each weight value and the mean divided by the total number of weights. Weight values ​​that exceed the mean ± 2 times the standard deviation are identified as outliers and replaced with the mean. Then, all calibrated weight values ​​are normalized (the sum of the weights is 1) to obtain the calibrated attention weights.

[0061] The second round of semantic reasoning is performed based on the calibrated attention weights and the first round of hidden states. The first round of hidden states and the calibrated attention weights are then input into the Transformer decoder layer again, and the above reasoning process, including attention fusion, fully connected network operation, and activation function processing, is repeated to generate the second round of hidden states. Attention weight calibration is performed simultaneously (same as the first round calibration logic). The preset number of iterations is 3. In combination with multimodal semantic reasoning scenarios, 3 iterations can correct semantic deviations and strengthen contextual associations through multiple rounds of optimization, while avoiding excessive iterations (more than 3 times) which would lead to increased computational power consumption and extended inference latency, thus balancing accuracy and efficiency. Semantic reasoning and attention weight calibration are performed iteratively according to this logic. Each iteration uses the current round of hidden states and the calibrated weights as inputs. During the iteration process, the change amplitude of the hidden states is monitored in real time (the sum of the absolute values ​​of the differences between corresponding components of two adjacent rounds of hidden states; if the sum is < 0.01, the iteration can be terminated early). After reaching the preset number of iterations or meeting the early termination condition, the final hidden state (2048 dimensions) is output.

[0062] Step 402: Generate preliminary inference results based on the final hidden state decoding, and calculate the confidence score of the target lightweight inference model for the preliminary inference results as the corresponding confidence score. Specifically, this includes: First, generating preliminary inference results based on the final hidden state decoding, i.e., constructing an output mapping matrix with a dimension of the inference result dictionary size × 2048 (the dictionary size is set according to the multimodal interaction scenario, such as 10000 for human-computer interaction scenarios, covering commonly used semantic units). The matrix elements are determined through model training optimization. Multiply the final hidden state and the output mapping matrix to obtain the original scores of each semantic unit. Then, perform softmax normalization on all the original scores, i.e., divide the original score of each semantic unit by the sum of the original scores of all semantic units to obtain the probability value of each semantic unit. Select the top N semantic units with the highest probability values ​​(N is set to 5 according to the scenario requirements), sort them from high to low probability values, and combine them to form the preliminary inference results (semantically fluent text or semantic identifiers that conform to contextual relevance). Then, calculate the corresponding confidence score, which is the confidence score of each semantic unit in the preliminary inference results. The weighted sum of the probability values ​​of semantic units is assigned according to their ranking. The semantic unit ranked first has a weight of 0.5, the second 0.25, the third 0.15, the fourth 0.07, and the fifth 0.03, with a total weight of 1. The weights follow the logic that the higher the ranking, the higher the semantic reliability. The first-ranked unit has the highest weight (0.5) to highlight the influence of the core semantic unit, and the weights decrease sequentially according to the ranking. This approach takes into account the auxiliary role of secondary semantic units while avoiding interference from low-probability units in the confidence judgment, ensuring that the confidence score is accurate and reliable. To accurately reflect the reliability of the results, the calculation process is as follows: first place probability value × 0.5 + second place probability value × 0.25 + third place probability value × 0.15 + fourth place probability value × 0.07 + fifth place probability value × 0.03. The resulting value is the confidence level, ranging from 0 to 1. Simultaneously, a calibration process is performed on the confidence level. If the confidence level < 0.5, it indicates that the model has a low assessment of the reliability of the results, and the preliminary inference result is marked as pending verification. If the confidence level ≥ 0.5, it is marked as a valid result, and the final output is the preliminary inference result and its corresponding confidence level.

[0063] In this embodiment, the model selection process combines vector dimensions with precise matching of resource constraints to avoid resource waste caused by blindly selecting high-computing-power models. Simultaneously, a dynamic compression mechanism adapts to resource gaps, balancing inference efficiency and resource utilization, and adapting to the real-time inference needs of heterogeneous computing units. Multi-round semantic inference is combined with attention calibration, effectively correcting semantic association biases and eliminating abnormal attention interference through iterative optimization of hidden states and attention weights, thereby improving the semantic accuracy and contextual coherence of the inference results and enhancing the fusion value of multimodal information. Confidence calculation employs weighted quantization logic, combined with the probability distribution and ranking of semantic units. Weights objectively reflect the reliability of the model's inference results; matrix operations, normalization, and iteration termination conditions in the inference process all employ quantitative logic, eliminating subjective judgment steps, ensuring the consistency and reproducibility of inference results, and improving engineering implementation capabilities; the selection of lightweight models and the simplification of multi-step inference processes minimize computing power and memory consumption while ensuring inference accuracy, adapting to the deployment needs of large multimodal models in resource-constrained scenarios such as terminal devices; the confidence calibration mechanism can filter out low-reliability results, preventing erroneous inference outputs from affecting subsequent interactions, and improving the interaction stability and user experience of multimodal systems.

[0064] In a preferred embodiment of the present invention, step 5 includes:

[0065] The system acquires the cumulative time from receiving the original multimodal input data to generating the preliminary inference result, which is used as the total processing latency. It then determines whether the total processing latency is less than or equal to a preset latency threshold and whether the corresponding confidence level is greater than or equal to a preset confidence threshold. If the total processing latency is less than or equal to the latency threshold and the corresponding confidence level is greater than or equal to the confidence threshold, the preliminary inference result is determined as the final multimodal understanding result. Specifically, this includes: first, initiating a full-process timestamp recording mechanism, using the system's high-precision clock module (with microsecond-level accuracy) to record the time from receiving the original multimodal input data to... Up to the generation of preliminary inference results, the start and end timestamps of each key step are all converted to milliseconds to avoid calculation errors caused by inconsistent units. The key steps and time ranges to be statistically analyzed are as follows: Raw data reception time starts from the first byte of the three types of raw input data (voice, gesture, and image) received by the system, and ends when all raw data has been completely received. The time taken is the reception end timestamp minus the reception start timestamp. Feature extraction and preprocessing time starts from step 001 (image feature vector and gesture spatial feature vector extraction) and ends at step 101 (spatial clustering index). The calculation of the target image ends, and the time taken is the time of the end timestamp of step 101 minus the start timestamp of step 001; the feature enhancement and resource scheduling time starts from step 200 (generating spatially enhanced image feature vectors) and ends from step 202b (generating dynamic resource scheduling strategies), and the time taken is the time of the end timestamp of step 202b minus the start timestamp of step 200; the feature transmission and fusion time starts from step 300 (directional transmission of multimodal feature vectors) and ends from step 302 (generating multimodal joint semantic vectors), and the time taken is the time of the end timestamp of step 302 minus the start timestamp of step 300; model inference and results The generation time starts from step 400 (selection of target lightweight inference model) and ends at step 402 (generation of preliminary inference results and confidence scores). The time taken is the time of the end of step 402 minus the time of the start of step 400. Then, the total processing latency is calculated by adding the time taken in the above five steps in sequence to obtain the cumulative time, which is the total processing latency. The calculation process is as follows: Total processing latency = raw data reception time + feature extraction and preprocessing time + feature enhancement and resource scheduling time + feature transmission and fusion time + model inference and result generation time. After the calculation is completed, one decimal place is retained to ensure the accuracy of latency statistics.

[0066] In light of the actual needs of multimodal real-time interaction scenarios (such as human-computer dialogue and intelligent cockpit control), a preset latency threshold of 200 milliseconds is established. This value was determined based on extensive scenario verification, satisfying the human body's perception of interaction response speed (no noticeable delay below 200 milliseconds) while also adapting to the processing capabilities of existing heterogeneous computing units, balancing real-time performance and computational power consumption. Following the core standard of confidence calibration in step 402, a preset confidence threshold of 0.5 is established. This value effectively distinguishes between reliable and unreliable inference results, preventing low-confidence results (high risk of semantic bias) from affecting interaction accuracy, while also preventing the misjudgment of valid results due to an excessively high threshold. A dual-judgment process is performed: first, comparing the total processing latency with the preset latency threshold to determine if the total processing latency is less than or equal to 200 milliseconds; second, comparing the corresponding confidence level generated in step 402 with the preset confidence threshold to determine if the confidence level is greater than or equal to 0.5. If both judgments are satisfied, i.e., the total processing latency is ≤200 milliseconds... If the initial inference result is within 0.00 milliseconds and the confidence level is ≥0.5, it indicates that the inference result meets the requirements of real-time interaction and has sufficient reliability. This initial inference result is determined as the final multimodal understanding result. The total processing latency, confidence level, and result identifier are recorded synchronously and stored in the system result cache for subsequent interaction response or data backtracking. If any condition is not met, i.e., if the total processing latency is >200 milliseconds, the dynamic resource optimization mechanism is triggered, the priority of the computing core allocation is adjusted (prioritizing the processing of high-frequency links), and the number of model inference iterations is compressed (not less than 2 times to avoid accuracy loss). Steps 300 to 5 are re-executed. If the confidence level is <0.5, the current initial inference result is discarded, the target lightweight inference model is reloaded, the attention weight calibration intensity is increased, and steps 401 to 5 are executed again until the result meets the dual threshold requirements or reaches the preset retry limit (3 times). If the retry limit is not reached, a system alarm is triggered, prompting that the feature fusion accuracy needs to be optimized.

[0067] This embodiment employs a dual-threshold verification mechanism that balances real-time performance and reliability. It ensures smooth multimodal interaction through latency threshold control while filtering out low-quality results through confidence thresholds, preventing user experience impacted by delays or semantic discrepancies, thus adapting to the core needs of real-time interaction scenarios. Refined latency statistics cover key stages throughout the entire process, providing a basis for threshold judgment and pinpointing excessively time-consuming steps, offering quantitative support for subsequent system optimization and continuously improving system processing efficiency. Clear result judgment and anomaly handling logic avoids ambiguous output. When results fail to meet threshold requirements, targeted optimization strategies (resource adjustment, re-inference) attempt to correct them, reducing invalid result output and improving system stability. Qualitative and fault-tolerant capabilities; connecting the aforementioned feature fusion and model inference processes to form a closed loop of end-to-end processing, dual-dimensional verification, and result output / optimization, ensuring full-link controllability of multimodal understanding results from generation to output, and strengthening the rigor of the overall technical solution; threshold values ​​are determined based on scenario verification, adapting to the performance and interaction requirements of existing heterogeneous computing units, avoiding blindly pursuing extreme speed or accuracy, balancing system performance and practical application value, and improving the engineering feasibility of the solution; result storage and backtracking mechanisms facilitate subsequent data review and model iteration, and by accumulating effective results that meet the dual thresholds, provide real-world data support for optimizing feature fusion logic and adjusting model parameters, helping the system to continuously iterate and upgrade.

[0068] In a preferred embodiment of the present invention, step 6 includes:

[0069] Step 600: Collect actual computational power consumption data, actual memory usage data, actual communication bandwidth usage data, and total processing latency recorded during parallel perception and feature extraction processing, multi-dimensional constrained computational power evaluation, lightweight cross-modal fusion processing, multi-step inference and attention calibration processing, and real-time decision-making. Specifically, this includes: First, clarifying the data collection scope and corresponding steps to ensure coverage of the core processing flow. All data is uniformly labeled with timestamps and task identifiers to avoid confusion. The data to be collected and their corresponding sources are as follows: Actual computational power consumption data is for parallel perception and feature extraction processing (steps 001 to 101), lightweight cross-modal fusion processing (steps 300 to 302), multi-step inference and attention calibration processing, and real-time decision-making. The three core stages of preprocessing (steps 401 to 402) are collected in real time through the computing power counter built into the heterogeneous computing unit and the system performance monitoring interface. The instantaneous floating-point operations per second are recorded every 10 milliseconds, and the corresponding stage number and computing core identifier are simultaneously labeled. After collection, abnormal instantaneous values ​​in each stage are removed (subsequent preprocessing uses a unified secondary verification). The arithmetic mean of the remaining instantaneous values ​​is taken as the actual computing power consumption of that stage. The total actual computing power consumption is the sum of the averages of the three stages, with the unit uniformly set to floating-point operations per second, and the result retained to 3 significant digits. The actual memory capacity usage data covers the entire process of multi-dimensional constrained computing power evaluation (steps 201 to 202a), feature fusion, and model inference, through internal... The real-time sampling interface of the storage management unit collects memory usage values ​​every 50 milliseconds, synchronously distinguishing three usage scenarios: feature data storage (original features, enhanced feature vectors), temporary data storage (intermediate results of matrix operations, attention weight matrices), and model parameter storage (projection matrices, inference model parameters). The memory address range corresponding to each scenario is clearly marked. After collection, the maximum memory usage value of the entire process is extracted as the peak value. The mean value is obtained by summing all recorded values ​​and dividing by the number of records, with the unit unified to bytes and the result retained as an integer. The actual communication bandwidth usage data accurately corresponds to the multimodal feature vector directional transmission stage in step 300, with independent transmission channels for voice, gesture, and image features, utilizing bus bandwidth. The monitoring module collects data and records the instantaneous transmission bandwidth of each channel every 20 milliseconds, and simultaneously records the communication identifier and transmission status (normal / congested) of the corresponding channel. After collection, the average of all valid instantaneous values ​​of each channel (excluding abnormal values ​​under transmission congestion) is taken as the actual bandwidth usage of the corresponding channel. The total actual communication bandwidth usage is the sum of the average values ​​of the three channels, and the unit is uniformly bytes per second. The result is retained to 3 significant digits. The total processing latency directly uses the cumulative latency data of the entire process counted in step 5, that is, the cumulative time from the system receiving the first byte of the original multimodal input data to the generation of the preliminary inference result, and retains 1 decimal place in milliseconds. It is synchronously associated with the latency composition details in step 5 (time consumption breakdown of each stage).

[0070] After data collection, standardized preprocessing operations are performed to ensure data accuracy and completeness. First, outlier removal: for each data category's record sequence, the mean of all record values ​​in the sequence is calculated. Then, the sum of the squares of the differences between each record value and the mean is calculated, divided by the number of records, and the square root is taken to obtain the standard deviation. Records exceeding the range of mean plus twice the standard deviation or mean minus twice the standard deviation are identified as outliers and removed. If the number of records after removal is less than 70% of the original number of records, re-collection is triggered, and the data collection process is repeated. Second, missing data supplementation: if a single data category has a few missing records (no more than two consecutive missing records), the arithmetic mean of two adjacent valid record values ​​is used to fill the gaps. If more than two consecutive missing records are found, the data is marked as an outlier, and the missing time period and corresponding step are recorded simultaneously. These missing records are then separately labeled during subsequent parameter fitting. The process is as follows: After preprocessing, the data is archived according to stages and data types, generating a single task actual operation data record table. The table clearly includes fields such as task identifier, data collection time range, resource consumption details for each stage (instantaneous value, mean, peak value), preprocessing instructions (number of outliers, missing data filling), total processing latency and its composition, minimum envelope ellipse parameters, and spatial clustering index. Finally, a data association mapping is established, binding the minimum envelope ellipse parameters (center x-coordinate and y-coordinate are rounded to 2 decimal places, and major and minor axis lengths are in the same unit as the image coordinate system, rounded to 1 decimal place) and spatial clustering index (rounded to 3 decimal places) corresponding to this task to the task identifier in the record table, and storing them in the corresponding entries in the system database, forming a complete data set containing resource consumption, latency, and visual feature parameters.

[0071] Step 601: Compare the actual computing power consumption data, actual memory capacity usage data, and actual communication bandwidth usage data with the estimated computing power requirements, memory capacity requirements, and communication bandwidth requirements obtained in the multidimensional constrained computing power assessment. Combine this with the center coordinates, major axis length, minor axis length, and spatial clustering index of the minimum envelope ellipse to perform numerical fitting, generating parameter correction quantities for the computing power requirements, memory capacity requirements, and communication bandwidth requirements. Specifically, this includes: calculating the deviation values ​​and deviation rates between the actual computing power consumption, actual memory capacity usage, and actual communication bandwidth usage and the corresponding estimated requirements, quantifying the differences between the two. The deviation value is calculated as: Deviation value = Actual value - Estimated value; the deviation rate is calculated as: Deviation rate = (Deviation value - Estimated value) / (Deviation value - Estimated value) / Estimated value. (Value ÷ Estimated Value) × 100%, rounded to two decimal places. Following this logic, calculate the deviation values ​​and deviation rates for the three indicators: computing power, memory capacity, and communication bandwidth, and form a deviation analysis table. Using the central x-coordinate, central y-coordinate, major axis length, minor axis length, and spatial clustering index of the minimum envelope ellipse as independent variables, and the deviation rates of the three resource indicators as dependent variables, construct a multiple linear fitting model and perform numerical fitting operations. The fitting model is constructed as follows: let the deviation rate be Y, and the five independent variables be X1 (central x-coordinate), X2 (central y-coordinate), X3 (major axis length), X4 (minor axis length), and X5 (spatial clustering index). The fitting equation is Y = a1X1 + a2X2 + a3X3 + a4X4 + a5X5 + ... Where a1 to a5 are the coefficients of the independent variable. For constant terms; the coefficients are calculated by optimizing using the least squares method to minimize the sum of squared residuals between the fitted values ​​and the actual deviation rates, where the sum of squared residuals = (Actual deviation rate - Fitting deviation rate) 2 Iteratively adjust a1 to a5 and The value of is taken until the sum of squared residuals is less than 0.001, and the final coefficients and constant terms are determined. Here, the actual deviation rate is the deviation rate of each of the three resource indicators calculated in the previous steps (quantified as specific percentage values). For example, the actual deviation rate of computing power is 10.00%, the actual deviation rate of memory capacity is 8.50%, and the actual deviation rate of communication bandwidth is -5.20% (negative values ​​indicate that the actual consumption is lower than the estimated value). These are all derived from the comparison results of actual data and estimated requirements. The fitting deviation rate is calculated based on the fitting equation, and then combined with the actual deviation rate correction, to obtain the correction coefficient = fitting deviation rate × 0.7 + actual deviation rate × 0. 3. This operation is a weighted summation operation. The core logic is to combine the advantages of two deviation rates and compensate for the limitations of a single deviation rate, rather than simply adding them together. The fitted deviation rate is good at capturing the common correlation between resource consumption and visual feature parameters, while the actual deviation rate is good at reflecting the individual consumption differences of a single task. By combining the two through fixed weight allocation, it can ensure the generalization of the correction logic and fit the real task scenario. Among them, 0.7 and 0.3 are weight coefficients. 0.7 is allocated to the fitted deviation rate because the fitted deviation rate is based on a multivariate linear model that integrates elliptic parameters, spatial clustering, and other multi-dimensional features, which can reflect the resource consumption deviation and visual features. The inherent correlation between parameters leads to higher fitting accuracy and stronger generalization. 0.3 is allocated to the actual deviation rate, as the actual deviation rate reflects the true consumption of a single task. Retaining this weight prevents overfitting of the fitted model and balances model predictability with real-world scenario realism. This weight was determined through hundreds of multimodal task verifications and maximizes the reliability of the correction coefficients. Parameter correction amount = estimated value × correction coefficient, calculating the correction amounts for computing power requirement, memory capacity requirement, and communication bandwidth requirement respectively. If the correction coefficient is positive, the corresponding parameter correction amount is also positive, indicating that the estimated value in the original baseline parameters is lower than the actual consumption of this task. Subsequent estimated values ​​need to be adjusted upwards using addition, that is, adding the original baseline parameter to the positive correction amount to obtain the updated baseline parameter. The upward adjustment range is determined by the size of the correction amount, ensuring that the new estimated value is close to the actual consumption and avoiding resource supply gaps in subsequent tasks due to insufficient estimation. If the correction coefficient is negative, the corresponding parameter correction amount is negative, indicating that the estimated value in the original baseline parameter is higher than the actual consumption of this task. Subsequent estimated values ​​need to be adjusted downwards using addition (essentially subtraction), that is, adding the original baseline parameter to the negative correction amount, i.e., subtracting the absolute value of the correction amount from the original baseline parameter, to obtain the updated baseline parameter. The downward adjustment range matches the absolute value of the correction amount.

[0072] After parameter adjustment, the rationality of the updated baseline parameters must be verified simultaneously to ensure that the computing power requirement is not lower than the minimum processing threshold (5×). The requirements for processing capacity are as follows: memory capacity should have a redundancy of at least 0.1 times the capacity; communication bandwidth should not exceed 80% of the maximum bus bandwidth. If these requirements exceed a reasonable range, adjustments should be made according to the corresponding boundary values. The specific adjustments are as follows: fine-tuning of computing power requirements, with a minimum processing threshold of 5 × [missing value]. The core boundary value is times / second. If the updated computing power requirement baseline parameter is <5× Hz, directly fine-tuned to 5× The processing speed is recorded per second (boundary value), and the reasons for fine-tuning are recorded to ensure that subsequent computing power assessments will not result in insufficient resource supply due to excessively low parameters, thus avoiding processing lag or interruption. For memory capacity requirement fine-tuning, two key boundaries are first defined: the upper limit of available system memory (the current total amount of free system memory) and the redundancy boundary (updated parameters × 0.1). Total memory usage requirement = updated memory capacity requirement baseline parameter + redundancy space. If the total memory usage requirement > the upper limit of available system memory, the updated baseline parameter needs to be fine-tuned by the upper limit of available system memory × 0.9 (0.9 corresponds to reserving 10% redundancy space). (Meets the requirement of at least 0.1 times redundancy); fine-tuning of communication bandwidth requirements, with 80% of the maximum bus bandwidth as the boundary value (the maximum bus bandwidth is determined by the hardware specifications of the heterogeneous computing unit; for example, if the maximum bus bandwidth is 5GB / s, the boundary value is 4GB / s). If the updated communication bandwidth requirement baseline parameter is greater than this boundary value, it is directly fine-tuned to the boundary value to avoid occupying too much bus resources and causing congestion in other transmission channels. For example, if the maximum bus bandwidth is 5GB / s, the boundary value is 4GB / s, and the updated parameter is 4.5GB / s, which is outside the range, it is fine-tuned and determined to be 4GB / s.

[0073] Step 602: Update the evaluation parameters used in the multi-dimensional constrained computing power assessment using parameter correction amounts; based on the total processing latency and the actual inference accuracy and efficiency demonstrated by the target lightweight inference model in processing this task, update the adaptation relationship between the vector dimension, computing power requirements, and memory capacity requirements of the multimodal joint semantic vector of the target lightweight inference model and the corresponding task in the lightweight inference model library. Specifically, this includes: First, updating the evaluation parameters by adding the corresponding correction amounts to the baseline parameters of computing power requirements, memory capacity requirements, and communication bandwidth requirements used in the multi-dimensional constrained computing power assessment to obtain the updated evaluation parameters. The update formula is: Updated parameters = Original baseline parameters + Parameter correction amount; After updating, the rationality of the parameters needs to be verified to ensure that the computing power requirements are not lower than the minimum processing threshold (5× The memory capacity requirement is reserved with a redundancy of no less than 0.1 times, and the communication bandwidth requirement does not exceed 80% of the maximum bus bandwidth. If it exceeds a reasonable range, it will be adjusted according to the boundary value. At the same time, the parameter update log will be recorded, noting the update time, correction amount and adjustment reason. The second step is to update the model library adaptation relationship. First, the actual performance indicators of the target lightweight inference model in handling this task are statistically analyzed, including actual inference accuracy (calculated by the matching degree between the system's preset standard semantic reference result and the preliminary inference result, matching degree = number of consistent semantic units ÷ total number of semantic units × 100%) and actual inference efficiency (actual inference time ÷ preset inference time × 100%, where the preset inference time is 80 milliseconds). Combined with the total processing latency, a model performance evaluation report is formed.

[0074] Subsequently, based on the various indicators in the model performance evaluation report, the adaptation relationship between the target model and the corresponding multimodal processing task in the lightweight inference model library was refined and updated. Specific operations were performed in layers according to the achievement of different indicators. If the total processing latency ≤ the preset latency threshold (200 milliseconds), the actual inference accuracy ≥ 90%, and the actual inference efficiency ≥ 85%, then the adaptation relationship between the model and the current multimodal joint semantic vector dimension (2048 dimensions) was strengthened. The optimal adaptation marker was then added to the 2048-dimensional vector adaptation entry of the model in the model library. The computational power required after this update was then considered. For each entry, the required memory capacity parameters should be accurately entered into the optimal resource requirement field. Simultaneously, the total processing latency, actual inference accuracy, and actual inference efficiency of this task should be correlated as adaptation criteria for future model selection. If the total processing latency is greater than 200 milliseconds or the actual inference accuracy is less than 90%, the adaptation relationship should be adjusted accordingly. First, calculate the sum of the model's actual resource consumption in this task and a 0.1x redundancy value to obtain the specific upper limit of resource requirements. Then, mark the upper limit of resource requirements field in the corresponding 2048-dimensional vector adaptation entry in the model library and enter this value. Simultaneously, specific optimization directions are added to the entry-specific adaptation suggestion field, such as reducing the vector dimension to 1024 dimensions to optimize latency, providing clear guidance for subsequent model selection and task parameter adjustments. If the actual inference efficiency is <85%, the model lightweight optimization record is updated specifically, adding a new record to the lightweight optimization log of the target model in the model library, noting the pruning threshold adjustment suggestion, the reason for the adjustment (the actual inference efficiency of this task did not reach the 85% threshold), and the actual inference time of the corresponding task, such as lowering the pruning threshold from 0.0001 to 0.0000. 8. The adjusted pruning threshold will be used as the core reference value for subsequent lightweight iteration optimization of the model, thereby improving the efficiency of subsequent model loading and inference. After the above adaptation relationship and related records are updated, the index table of the lightweight inference model library will be updated synchronously. The resource requirement parameters, adaptation identifiers, optimization suggestions and other related fields under the corresponding 2048-dimensional vector entries of the model in the index table will be accurately corrected to ensure that the index table is consistent with the master data of the model library. This will ensure that the system can quickly and accurately retrieve the optimized adaptation relationship when selecting a model next time, and realize the dynamic iteration and continuous optimization of the model library.

[0075] This embodiment features end-to-end data collection covering core indicators of resource consumption and latency. Combined with visual feature parameters, it forms a complete data chain, providing realistic and comprehensive support for parameter correction and model optimization, avoiding accuracy loss caused by blind adjustments. Numerical fitting integrates multi-dimensional independent variables and deviation data, generating correction values ​​through quantitative calculations. This is more scientific than single deviation correction, effectively improving the accuracy of resource requirement estimation and reducing resource waste or insufficient supply in subsequent tasks. Dynamic updates of evaluation parameters form a closed-loop optimization, enabling multi-dimensional constrained computing power evaluation to adapt to the resource consumption characteristics of different task scenarios, improving the rationality of resource scheduling strategies. Rationality and adaptability; based on the actual performance of the model, iterative adaptation relationships are implemented, enabling the lightweight inference model library to respond in real time to changes in task requirements, optimize the matching degree between the model and vector dimensions and resource requirements, and improve the efficiency and accuracy of subsequent model selection; parameter correction and model library updates both adopt quantitative logic, without subjective judgment links, ensuring the consistency and reproducibility of the optimization process, and improving the system's engineering implementation capability and scalability; combined with the fitting design of visual feature parameters (ellipse parameters, clustering index), the correlation logic between visual features and resource requirements is strengthened, making resource estimation more in line with the actual distribution characteristics of multimodal features.

[0076] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for accelerating real-time inference of multimodal large models, characterized in that, The method includes: Step 1: Perform parallel perception and feature extraction processing on the original multimodal input data to obtain speech feature vectors, image feature vectors, and gesture spatial feature vectors. Based on the image feature vectors, construct a spatial weight distribution representing the visual attention distribution by calculating the weighted sum of the feature components at each position. Analyze the position corresponding to the maximum weight as the first visual attention weight peak point. Map the gesture spatial feature vector to the image coordinate system corresponding to the image feature vector to obtain the second visual coordinate point. Perform element-wise similarity calculation between the image feature vector and the visual feature vectors stored in the historical interaction sequence. Analyze the spatial position corresponding to the feature with the highest similarity to the current image feature vector and exceeding a preset threshold as the third visual association point. Using the first visual attention weight peak point, the second visual coordinate point, and the third visual association point as boundary points, generate a minimum envelope ellipse and calculate the ratio of the area to the perimeter of the ellipse as a spatial clustering index. Step 2: Selectively enhance the features located within the minimum envelope ellipse region in the image feature vector based on the spatial clustering index to generate spatially enhanced image feature vectors, and perform a multi-dimensional constrained computing power assessment covering computing power requirements, memory capacity requirements, and communication bandwidth requirements to generate a dynamic resource scheduling strategy. Step 3: Based on the dynamic resource scheduling strategy, the speech feature vector, gesture space feature vector, and spatially enhanced image feature vector are subjected to lightweight cross-modal fusion processing to obtain a unified multimodal joint semantic vector. This includes: transmitting the speech feature vector, gesture space feature vector, and spatially enhanced image feature vector to the corresponding processing unit according to the computing core allocation and data path planning information contained in the dynamic resource scheduling strategy; performing linear projection transformation on the speech feature vector, gesture space feature vector, and spatially enhanced image feature vector respectively to obtain projected feature vectors of the same dimension; calculating the cross-attention weights between the projected feature vectors, and performing a weighted summation of the projected feature vectors based on the cross-attention weights to generate a unified multimodal joint semantic vector. Step 4: Based on multimodal joint semantic vectors and dynamic resource scheduling strategies, match the target lightweight inference model from the pre-built lightweight inference model library to perform multi-step inference and attention calibration processing, and generate preliminary inference results and corresponding confidence scores. Step 5: Make a real-time judgment on the preliminary reasoning results and corresponding confidence levels. When the preset time delay threshold and confidence level threshold are met, the final multimodal understanding result is obtained. Step 6: Collect resource consumption data, performance index data, geometric parameters of the minimum envelope ellipse and spatial clustering index throughout the process, and update the adaptation relationship between the evaluation parameters used in the multidimensional constrained computing power evaluation and the lightweight inference model library.

2. The method for accelerating real-time inference of multimodal large models according to claim 1, characterized in that, Step 1 includes: Based on the coordinate set consisting of the first visual attention weight peak point, the second visual coordinate point, and the third visual association point, calculate an ellipse that can completely contain the coordinate set and has the smallest area. Define this ellipse as the minimum envelope ellipse. Calculate the numerical ratio between the area of ​​the minimum envelope ellipse and its perimeter, and use this ratio as an index of spatial clustering.

3. The method for accelerating real-time inference of multimodal large models according to claim 2, characterized in that, Step 2 includes: A feature gain coefficient is determined based on the value of the spatial clustering index. The values ​​of all feature components whose spatial coordinates are located within the minimum envelope ellipse region in the image feature vector are multiplied by the feature gain coefficient, and the values ​​of feature components outside the region are attenuated to generate a spatially enhanced image feature vector. Estimate the computational power requirements, memory capacity requirements, and communication bandwidth requirements for processing speech feature vectors, gesture spatial feature vectors, and spatially augmented image feature vectors, respectively; Based on computing power requirements, memory capacity requirements, and communication bandwidth requirements, a multi-dimensional constrained computing power assessment is performed to comprehensively generate a dynamic resource scheduling strategy.

4. The method for accelerating real-time inference of multimodal large models according to claim 3, characterized in that, Based on computing power requirements, memory capacity requirements, and communication bandwidth requirements, a multi-dimensional constrained computing power assessment is performed to comprehensively generate a dynamic resource scheduling strategy, including: Based on computing power requirements, the first evaluation result is obtained by assessing whether available computing resources meet the computing completion constraints; based on memory capacity requirements, the second evaluation result is obtained by assessing whether available memory resources meet the storage capacity constraints; and based on communication bandwidth requirements, the third evaluation result is obtained by assessing whether available data transmission paths meet the transmission completion constraints. By combining the results of the first, second, and third evaluations, a dynamic resource scheduling strategy is generated for allocating computing cores, memory regions, and data transmission channels.

5. The method for accelerating real-time inference of multimodal large models according to claim 4, characterized in that, Step 4 includes: Based on the vector dimension of the unified multimodal joint semantic vector and the computing power and memory capacity requirements defined in the dynamic resource scheduling strategy, a matching model is selected from the pre-built lightweight inference model library as the target lightweight inference model. The multimodal joint semantic vector is input into the target lightweight inference model, a multi-step inference process is executed, and the first round of attention weight distribution calibration is performed to obtain the calibrated attention weights. Based on the calibrated attention weights and the first round of hidden states, the second round of semantic inference is performed. The semantic inference and attention weight distribution calibration are iteratively executed until the preset number of iterations is reached to obtain the final hidden state. The initial inference result is generated based on the final hidden state decoding, and the confidence of the target lightweight inference model over the initial inference result is calculated as the corresponding confidence.

6. The method for accelerating real-time inference of multimodal large models according to claim 5, characterized in that, The multi-step reasoning process includes: generating a primary semantic representation based on a multimodal joint semantic vector; and performing the first round of semantic reasoning based on the primary semantic representation to obtain the first round of hidden states containing semantic features and contextual information.

7. The method for accelerating real-time inference of multimodal large models according to claim 6, characterized in that, Step 5 includes: The cumulative time from receiving the original multimodal input data to generating the preliminary inference result is obtained as the total processing latency. It is then determined whether the total processing latency is less than or equal to a preset latency threshold, and whether the corresponding confidence level is greater than or equal to a preset confidence level threshold. If the total processing latency is less than or equal to the latency threshold and the corresponding confidence level is greater than or equal to the confidence level threshold, then the preliminary inference result is determined as the final multimodal understanding result.

8. The method for accelerating real-time inference of multimodal large models according to claim 7, characterized in that, Step 6 includes: Collect actual computing power consumption data, actual memory capacity usage data, actual communication bandwidth usage data, and total processing latency recorded during parallel perception and feature extraction processing, multi-dimensional constrained computing power evaluation, lightweight cross-modal fusion processing, multi-step inference and attention calibration processing, and real-time decision-making processes. The actual computing power consumption data, actual memory capacity usage data, and actual communication bandwidth usage data are compared with the computing power requirements, memory capacity requirements, and communication bandwidth requirements estimated in the multidimensional constrained computing power assessment. The center coordinates, major axis length, minor axis length, and spatial clustering index of the minimum envelope ellipse are combined to perform numerical fitting and generate parameter correction quantities for computing power requirements, memory capacity requirements, and communication bandwidth requirements. The evaluation parameters used in the multidimensional constraint computing power evaluation are updated using parameter correction. Based on the total processing latency and the actual inference accuracy and efficiency demonstrated by the target lightweight inference model in processing this task, the adaptation relationship between the vector dimension, computing power requirements and memory capacity requirements of the multimodal joint semantic vector of the target lightweight inference model and the corresponding task in the lightweight inference model library is updated.

Images