Parallel computing system based on multi-modal perceptual fusion and computing unit partitioning

Abstract

The invention provides a parallel computing system based on multi-modal sensing fusion and computing unit partitioning, which belongs to the technical field of computers and comprises a multi-modal sensing data acquisition module for acquiring data and performing primary processing; the computing unit partition management module is used for starting work according to task requirements; the multi-modal data fusion module is used for processing data from different partitions; and the task scheduling and executing module is responsible for scheduling and executing tasks of the whole system. The execution sequence can be dynamically adjusted according to the real-time task load, task backlog and response delay are avoided, and efficient operation of the system is achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a parallel computing system based on multimodal perception fusion and computing unit partitioning. Background Art

[0002] Although significant progress has been made in the field of parallel computing system technology at home and abroad, there are still many drawbacks in the current development. From the perspective of hardware resource utilization, the existing parallel computing system lacks fine management and efficient allocation of computing units. In most cases, computing units uniformly process a single type of task and cannot be flexibly partitioned and configured according to the characteristics and requirements of different tasks. This results in some computing units being idle and some being overloaded when processing diversified tasks, resulting in a waste of hardware resources and reduced computing efficiency. In terms of data processing, with the development of multimodal perception technology, many applications need to simultaneously process multiple types of data such as visual images, audio, and lidar point clouds, but the existing system lacks an effective fusion mechanism, and different modal data are processed independently, making it difficult to mine the correlation information between data, resulting in limited system perception and understanding of complex scenes. In addition, the current task scheduling strategy is single, only considering simple priority or first-come-first-served principles, and not fully considering factors such as task type, computational complexity, and real-time requirements. When faced with complex and variable task loads, problems such as task backlogs and response delays are prone to occur. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides a parallel computing system based on multimodal perception fusion and computing unit partitioning.

[0004] The technical solution of the present invention is: A parallel computing system based on multimodal perception fusion and computing unit partitioning, comprising: Multimodal perception data acquisition module, which collects data and performs preliminary processing; The computing unit partition management module starts work according to task requirements; Multimodal data fusion module, processing data from different partitions; The task scheduling and execution module is responsible for the scheduling and execution of the tasks of the entire system.

[0005] Furthermore, Multimodal perception data acquisition module: The data acquisition unit collects multimodal perception data such as visual images, lidar point clouds, millimeter-wave radar data, and audio signals from a variety of professional sensors, such as visual cameras, lidars, millimeter-wave radars, microphones, etc. These acquisition devices have high-precision and high-frame-rate data acquisition capabilities to ensure that the collected data is comprehensive and accurate.

[0006] The collected raw data will undergo preliminary data processing in the data conversion unit, such as signal amplification, noise reduction, filtering, etc., and then converted into digital signals for transmission to subsequent modules.

[0007] Computing unit partition management module: In this module, through in-depth analysis of various computing tasks, the parallel computing units are divided into multiple functional partitions according to the characteristics of the tasks. For example, in view of the characteristics of visual processing tasks, which involve many matrix operations and large amounts of data, a visual processing partition is set up; based on the requirements of point cloud processing tasks for three-dimensional spatial data processing, point cloud processing partitions are divided; considering the characteristics of audio processing tasks in signal time domain and frequency domain analysis, an audio processing partition is constructed; and in order to handle the requirements of multimodal data fusion, a fusion computing partition is set up.

[0008] In order to meet the different requirements of different partition tasks for computing resources, different partitions are allocated different computing cores and occupied spaces. Due to the high demand for parallel computing, the visual processing partition is allocated with more computing cores and larger spaces; the audio processing partition has a relatively small amount of computing, so it is allocated with fewer computing cores and smaller spaces. At the same time, during the operation of the system, the load status of each partition is monitored in real time, and the allocation of computing units is dynamically adjusted using the hardware configuration interface and software control instructions. When the task volume of the visual processing partition increases dramatically and the resources of the audio processing partition are idle, some computing units of the audio processing partition can be quickly allocated to the visual processing partition to achieve efficient use of resources. When the system detects that some tasks have closely related functional requirements for multiple partitions and synergy can improve efficiency, a merge operation is performed. For example, in augmented reality (AR) applications, in order to achieve more accurate environmental perception and object recognition, the visual processing partition and the point cloud processing partition can be merged into a comprehensive perception partition. By adjusting the hardware connection and software configuration, the computing cores and resources are uniformly allocated, the data transmission delay is reduced, and the processing efficiency is improved. If the task requirements are refined and some partition functions are too complex to affect efficiency, the system performs a split operation. For example, in complex multimedia processing tasks, the original heavy fusion computing partition is split into image and audio fusion partition and video processing partition, and the computing resources and data flow are re-planned so that each partition can focus on specific tasks, improving the processing targeting and efficiency.

[0009] This module is based on a parallel computing system. With its powerful parallel computing capabilities, it can process tasks of multiple partitions at the same time, greatly improving the overall computing efficiency. Moreover, its rich hardware resources can meet the diverse resource requirements of partitions and achieve efficient resource utilization. Its hardware and software collaborative features provide strong support for the dynamic adjustment, merging and splitting of partitions, allowing the system to flexibly adapt to complex and changing task scenarios and improve the overall performance and adaptability of the system.

[0010] Multimodal data fusion module: The data preprocessing unit can use corresponding feature extraction algorithms for different modal data. Convolutional neural networks (CNN) are used to extract image features from visual image data; audio features are extracted through Mel-frequency cepstral coefficients (MFCC) from audio data; and point cloud features are extracted from point cloud data with the help of point cloud convolutional networks (PointNet). After feature extraction, the features extracted from different modalities may differ in dimension, scale, time or space. Feature alignment is required to resolve these differences so that features from different modalities can be compared and fused under the same standards. The feature similarity matrix can be calculated based on the attention mechanism , select the features with strong correlation for alignment. Then, normalize the aligned features of different modalities to make them have the same scale and range, eliminate the influence of different dimensions and numerical ranges between features, and facilitate subsequent fusion operations.

[0011] The data fusion unit is used to fuse model data from different modalities or data sources, and can adopt a parallel or serial fusion structure. In the parallel fusion structure, different modal data are processed by their own sub-networks, and then the processed features are spliced. The weighted fusion formula can be used. Dynamically adjusted by task type, weighted fusion of different modal data is performed; in the serial fusion structure, one modal data is processed first and then fused with another modal data. Among them, the fusion algorithm component determines the best fusion algorithm through a large number of experiments and parameter tuning, such as weighted average, attention mechanism, etc. The attention mechanism can dynamically adjust the fusion weight according to the importance of different modal data. The fusion strategy component is used to determine the appropriate fusion strategy, such as early fusion, late fusion and mid-term fusion. Early fusion is to fuse immediately after data preprocessing, merge data of different modalities into a feature vector, and then input it into the subsequent model. Late fusion is to fuse the output results of the model after the data of each modality is processed by an independent model. Mid-term fusion is between the two, and data fusion is performed at an intermediate layer in the model processing process. Selecting the appropriate fusion strategy according to the specific application scenario and data characteristics can effectively improve the fusion effect. Finally, the result evaluation component evaluates the fused data to determine whether the fusion effect meets expectations. The fusion effect of classification tasks is evaluated using indicators such as accuracy, recall, and F1 value. For regression tasks, indicators such as mean square error (MSE) and mean absolute error (MAE) are used. By continuously evaluating and adjusting the fusion algorithm and strategy, the fusion results are optimized and the performance of the system is improved.

[0012] Task scheduling and execution module: When the system receives a computing task, the task analysis unit will first conduct a comprehensive analysis of the task. This includes extracting key features of the task, such as the task type (visual processing, audio processing, or multimodal fusion tasks, etc.), computational complexity (simple matrix operations or complex deep learning model reasoning), real-time requirements (whether it needs to be completed in a very short time, such as real-time target detection in autonomous driving), and data size. Based on these features and the current state of the system, the priority evaluation model is used to evaluate the task priority to provide a basis for subsequent scheduling and execution. For example, the target detection task in autonomous driving is given a higher priority due to its high real-time requirements.

[0013] The task scheduling unit adopts a multi-strategy scheduling algorithm, integrating priority-based scheduling, time slice-based scheduling, resource demand-based scheduling and other strategies. According to the task priority and system resource conditions, the appropriate scheduling strategy is intelligently selected. During the task execution process, the task execution progress and system resource usage are monitored in real time. If it is found that a task execution time is too long or the resource utilization rate of a partition is too low, the task scheduling strategy is adjusted in time through the task migration algorithm to migrate the task to other partitions for execution.

[0014] The execution monitoring unit monitors the task status in real time and updates the running status register through hardware counters and software process tracking. In terms of anomaly detection, the resource dynamic monitoring model is used to determine resource shortages, and verification algorithms and gradient monitoring are used to identify calculation errors. Once an anomaly is detected, the resource allocation table is queried for insufficient resources and resources are allocated from free partitions; in the case of calculation errors, the task status is set to "retrying", the environment is repaired, resources are reallocated, and then retry is performed. At the same time, the monitoring frequency is increased to ensure smooth execution of the task.

[0015] The entire system utilizes the parallel computing capabilities of the parallel computing system to achieve efficient parallel processing in various links such as data collection, processing, fusion, and task scheduling and execution, greatly improving the overall performance and intelligent computing level of the system.

[0016] The beneficial effects of the present invention are Compared with the traditional parallel computing system architecture, the partitioned system has significant advantages in resource allocation, task scheduling and system stability. In terms of resource allocation, the traditional parallel computing system has extensive resource allocation, and the competition for the same resources by various tasks can easily lead to unreasonable allocation. However, after partitioning, it can accurately allocate resources according to the characteristics of the tasks, such as allocating more resources to the visual processing partition for image data processing, and allocating more computing cores to the partition where the encryption execution module is located, effectively improving the efficiency of task execution and the overall performance of the system; in terms of task scheduling, traditional scheduling is based on a general mechanism and lacks optimization for specific task types. When there are many types of tasks and their characteristics vary greatly, the scheduling difficulty increases, and it is easy to have task backlogs or idle resources. After partitioning, different partitions can set different scheduling strategies, and tasks with high real-time requirements are scheduled first, and tasks that are not time-sensitive are flexibly scheduled to improve resource utilization; in terms of stability, task abnormalities in the traditional mode may affect the operation of the entire parallel computing system, while partitioning realizes functional isolation, and task abnormalities in one partition usually do not affect other partitions. For example, problems in the partition where the key storage module is located will not affect other partitions, which greatly enhances the stability and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Attached Figure 1 It is a schematic diagram of the newly added module architecture of the present invention; Attached Figure 2 It is an execution flow chart of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0019] The present invention provides a parallel computing system based on multimodal perception fusion and computing unit partitioning, which realizes refined management and flexible configuration of parallel computing units by introducing a computing unit partition management module. According to different computing task requirements, multiple functional partitions are divided, and different tasks are optimized for processing to improve hardware resource utilization and computing efficiency; deep learning algorithms and multi-sensor fusion technology are used to extract and match features of different modal data, integrate features through fusion strategies, fully mine data correlation information, and enhance the system's perception and understanding of complex scenes; a multi-strategy task scheduling algorithm is used to reasonably schedule and allocate tasks, and the execution order is dynamically adjusted according to the real-time task load to avoid task backlogs and response delays, so as to achieve efficient operation of the system.

[0020] 1. Multimodal data collection and preprocessing In the autonomous driving scenario, the multimodal perception data acquisition module starts working. The data acquisition unit uses visual cameras to capture visual images of vehicles, pedestrians, traffic signs, etc. on the road, the laser radar obtains 3D point cloud data of the surrounding environment, the millimeter wave radar collects information such as the distance and speed of the target object, and the microphone collects various sound signals during the vehicle's driving. These acquisition devices continuously collect data with high precision and high frame rate to ensure that the data is comprehensive and accurate.

[0021] The collected raw data is transmitted to the data conversion unit for preliminary processing. The signal amplification circuit enhances weak signals and improves signal strength; an adaptive noise reduction algorithm is used to remove environmental noise, such as vehicle driving noise, wind and rain noise, etc.; a digital filter is used to filter the signal to remove high-frequency or low-frequency interference. The processed analog signal is converted into a digital signal through a high-precision analog-to-digital converter and transmitted to subsequent modules.

[0022] 2. Computing unit partition management and task processing The computing unit partition management module starts working according to the task requirements. For visual image data, because it contains a large number of matrix operations and a large amount of data, it is assigned to the visual processing partition. This partition is equipped with more computing cores and a larger video memory space. It uses the parallel computing capabilities of the parallel computing system to perform target detection and recognition on visual images through convolutional neural networks (CNN). For example, it can identify traffic signs, determine vehicle types and driving directions.

[0023] The LiDAR point cloud data is sent to the point cloud processing partition. This partition is optimized for the needs of three-dimensional spatial data processing, and uses point cloud processing algorithms to analyze point cloud data to obtain information such as the spatial position and shape of objects, and combines it with the visual processing results to more accurately locate the target object.

[0024] The audio signal enters the audio processing partition. This partition has a relatively small amount of computation, and is allocated with fewer computing cores and smaller video memory. Digital signal processing technology is used to analyze the audio signal in the time domain and frequency domain, such as identifying vehicle horns and emergency brakes, to assist the autonomous driving system in making decisions.

[0025] If a multimodal data fusion task is encountered, such as analyzing visual, point cloud and audio data at the same time to determine complex road conditions, the data will be sent to the fusion computing partition. During operation, the system monitors the load of each partition in real time. When the traffic flow is large, the task volume of the visual processing partition increases dramatically, and the resources of the audio processing partition are idle, some computing units of the audio processing partition are allocated to the visual processing partition through the hardware configuration interface and software control instructions to improve the overall processing efficiency. In augmented reality navigation applications, in order to achieve more accurate environmental perception, the visual processing partition and the point cloud processing partition are merged into a comprehensive perception partition, and the computing cores and resources are uniformly allocated to reduce data transmission delays and improve processing efficiency. If the multimedia processing task requirements are refined, the fusion computing partition is split into an image and audio fusion partition and a video processing partition. Each partition focuses on a specific task to improve processing targeting and efficiency.

[0026] 3. Multimodal Data Fusion The multimodal data fusion module starts to process data from different partitions. In the data preprocessing unit, CNN is used to extract edge, texture, shape and other features from visual image data; audio features are extracted through Mel-frequency cepstral coefficients (MFCC) from audio data; and point cloud features are extracted from point cloud data with the help of point cloud convolutional network (PointNet). Due to the differences in dimension and scale of different modal features, the feature similarity matrix is ​​calculated based on the attention mechanism, and the features with strong correlation are selected for alignment. Subsequently, the aligned features are normalized so that the features of different modalities have the same scale and range, which is convenient for subsequent fusion.

[0027] In the data fusion unit, a parallel fusion structure is selected according to the characteristics of the autonomous driving scenario. Different modal data are processed by their respective sub-networks, and then the processed features are spliced, and the different modal data are weighted fused according to the weighted fusion formula. Through a large number of experiments and parameter tuning, the attention mechanism is determined as the fusion algorithm, and the fusion weight is dynamically adjusted according to the importance of different modal data in judging road conditions. A mid-term fusion strategy is selected to perform data fusion in the middle layer of the model processing process to improve the fusion effect. Finally, the result evaluation component uses indicators such as accuracy, recall rate, and F1 value to evaluate the fusion effect, continuously adjust the fusion algorithm and strategy, and optimize the fusion result.

[0028] 4. Task Scheduling and Execution The task scheduling and execution module is responsible for the scheduling and execution of the entire system tasks. During the operation of the autonomous driving system, new tasks are constantly generated, such as real-time target detection, path planning, decision making, etc. After receiving the task, the task analysis unit extracts the key features of the task, including task type, computational complexity, real-time requirements, and data size. For example, the real-time target detection task is given a higher priority because it needs to be completed in a short time, has high computational complexity and large data volume.

[0029] The task scheduling unit adopts a multi-strategy scheduling algorithm based on task priority and system resource conditions. For high-priority real-time target detection tasks, a priority-based scheduling strategy is adopted to prioritize the allocation of computing resources; for path planning tasks with high computational complexity but low real-time requirements, a scheduling strategy based on resource requirements is combined to arrange execution when system resources are sufficient. During task execution, the task execution progress and system resource usage are monitored in real time. If a task execution time is too long or the resource utilization rate of a partition is too low, such as a backlog of tasks in the visual processing partition and idle resources in the audio processing partition, some tasks are migrated to the audio processing partition through the task migration algorithm to ensure efficient operation of the system.

[0030] The execution monitoring unit monitors the task status in real time and updates the running status register through hardware counters and software process tracking. It uses the resource dynamic monitoring model to determine resource shortages and uses verification algorithms and gradient monitoring to identify calculation errors. If resources are insufficient, query the resource allocation table and allocate resources from free partitions; if there is a calculation error, set the task status to "retrying", repair the environment, reallocate resources, and try again. At the same time, increase the monitoring frequency to ensure smooth execution of tasks.

[0031] Through the above specific implementation methods, the entire system utilizes the parallel computing capabilities of the parallel computing system to achieve efficient parallel processing in various links such as data collection, processing, fusion, and task scheduling and execution, greatly improving the overall performance and intelligent computing level of the autonomous driving system, and ensuring driving safety and efficient operation.

[0032] The above description is only a preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, and is not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A parallel computing system based on multimodal perception fusion and computing unit partitioning, characterized in that: include: Multimodal perception data acquisition module, which collects data and performs preliminary processing; The computing unit partition management module starts work according to task requirements; Multimodal data fusion module, processing data from different partitions; The task scheduling and execution module is responsible for the scheduling and execution of the tasks of the entire system.

2. The system according to claim 1, characterized in that The multimodal perception data acquisition module includes: The data acquisition unit collects visual images, lidar point clouds, millimeter-wave radar data, and audio signals from visual cameras, lidars, millimeter-wave radars, and microphones; The data conversion unit performs preliminary data processing on the collected raw data and then converts it into digital signals for transmission to subsequent modules.

3. The system according to claim 1, characterized in that The computing unit partition management module divides the parallel computing units into several functional partitions according to the characteristics of the tasks through in-depth analysis of various computing tasks.

4. The system according to claim 3, characterized in that In view of the characteristics of visual processing tasks, which involve many matrix operations and large amounts of data, visual processing partitions are set up; Based on the requirements of point cloud processing tasks for 3D spatial data processing, point cloud processing partitions are divided; Construct audio processing partitions based on the characteristics of audio processing tasks in signal time domain and frequency domain analysis; In order to handle the needs of multimodal data fusion, a fusion computing partition is set up.

5. The system according to claim 4, characterized in that The visual processing partition is allocated with the most computing cores and the largest space; the audio processing partition has the smallest amount of computing, and is allocated with the least computing cores and a smaller space; at the same time, during the operation of the system, the load status of each partition is monitored in real time, and the allocation of computing units is dynamically adjusted using the hardware configuration interface and software control instructions; When the task load of the visual processing partition increases dramatically and the resources of the audio processing partition are idle, some computing units of the audio processing partition are allocated to the visual processing partition; When the system detects that some tasks have closely related functional requirements for several partitions and that synergy can improve efficiency, a merge operation is performed; If the task requirements are detailed and some partition functions are complex and affect efficiency, the system will perform a split operation.

6. The system according to claim 1, characterized in that The multimodal data fusion module includes: The data preprocessing unit uses corresponding feature extraction algorithms for different modal data; Visual image data uses convolutional neural network CNN to extract image features; Audio data uses Mel frequency cepstral coefficients (MFCC) to extract audio features; Point cloud data uses point cloud convolutional network PointNet to extract point cloud features; After feature extraction, the feature similarity matrix is ​​calculated based on the attention mechanism , select the features with strong correlation for alignment; then, normalize the aligned features of different modalities to make them have the same scale and range, eliminating the influence of different dimensions and numerical ranges between features; A data fusion unit is used to fuse model data from different modalities or data sources, using a parallel or serial fusion structure; In the parallel fusion structure, different modal data are processed by their respective sub-networks, and then the processed features are spliced ​​together according to the weighted fusion formula Dynamically adjust according to the task type and perform weighted fusion on data of different modalities; In the serial fusion structure, one modality data is processed first and then fused with another modality data; the fusion algorithm component is used to determine the best fusion algorithm through experiments and parameter tuning; the fusion strategy component is used to determine the appropriate fusion strategy, including early fusion, late fusion and mid-term fusion; early fusion is performed immediately after data preprocessing, merging data of different modalities into a feature vector, which is then input into the subsequent model; late fusion is the data of each modality after being processed by an independent model, and then the output results of the model are fused; mid-term fusion is between the two, and data fusion is performed at an intermediate layer in the model processing process; finally, the result evaluation component evaluates the fused data to determine whether the fusion effect meets expectations.

7. The system according to claim 6, characterized in that The accuracy, recall rate, and F1 value are used to evaluate the fusion effect of classification tasks. For regression tasks, mean square error (MSE) and mean absolute error (MAE) are used. By continuously evaluating and adjusting the fusion algorithm and strategy, the fusion results are optimized and the performance of the system is improved.

8. The system according to claim 1, characterized in that The task scheduling and execution modules include: Task analysis unit, when the system receives a computing task, the task analysis unit first conducts a comprehensive analysis of the task, including extracting the key features of the task, and based on these features and the current state of the system, uses the priority evaluation model to evaluate the task priority, providing a basis for subsequent scheduling and execution; The task scheduling unit adopts a multi-strategy scheduling algorithm, integrating priority-based scheduling, time-slice-based scheduling, and resource demand-based scheduling. It intelligently selects the appropriate scheduling strategy based on task priority and system resource conditions. During task execution, it monitors the task execution progress and system resource usage in real time. If it is found that the execution time of a task is higher than the set value or the resource utilization rate of a partition is lower than the set value, the task scheduling strategy is adjusted in time through the task migration algorithm to migrate the task to other partitions for execution. The execution monitoring unit monitors the task status in real time and updates the operation status register through hardware counters and software process tracking.

9. The system according to claim 8, characterized in that In terms of anomaly detection, the resource dynamic monitoring model is used to determine resource shortage, and the verification algorithm and gradient monitoring are used to identify calculation errors. Once an anomaly is detected, the resource allocation table is queried for resource shortage and resources are allocated from idle partitions. In the event of a calculation error, the task status is set to retrying, and the environment is repaired, resources are reallocated, and then retry is performed, while the monitoring frequency is increased.

Images