A cloud intelligent image processing system and an application method thereof

Abstract

The application relates to the field of image processing, and provides a cloud intelligent image processing system and an application method thereof, which comprises the following steps: determining the preloading demand of an AR model according to a high-frequency feature set, transmitting corresponding model data from external storage to special memory through a GPU calculation unit to generate a preloading data set; acquiring a voice instruction data stream; a DSP calculation unit directly calls an FPGA shared cache to generate a scheduling instruction set for an instruction sequence with a higher real-time requirement than a preset threshold; for a multi-modal feature group, a GPU calculation unit adjusts memory allocation by using a data layout optimization technology, processes the feature group by using a matrix transformation algorithm, and generates an optimized data block; the data update frequency of a local cache and a shared cache is adjusted through a matching result; if the matching degree is higher than a preset threshold, a sliding window algorithm is used to update cache content to generate a new cache set.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a cloud intelligent image processing system and an application method thereof. BACKGROUND

[0002] With the gradual evolution of outdoor advertising machines towards intelligent interactive services, their status as emerging commercial display and user connection carriers is increasingly prominent. The development of this field not only concerns the digital transformation of the advertising industry, but also directly affects the improvement of user experience and the realization of commercial value. However, the integration of multi-modal interaction functions puts higher requirements on the hardware architecture, and traditional design cannot meet the dual demands of real-time and efficiency, so innovative resource management solutions are needed to break through technical bottlenecks and promote the industry to a wider range of application scenarios.

[0003] Existing methods have significant limitations in dealing with multi-modal tasks. Traditional reliance on general-purpose CPU and GPU architectures cannot support concurrent processing of tasks such as object recognition, AR try-on, and voice interaction, resulting in performance bottlenecks. Even with the introduction of special-purpose acceleration chips such as NPU and FPGA, static resource allocation strategies cannot flexibly match dynamic task requirements, resulting in waste or insufficient computing resources. Dynamic scheduling based on the software layer can optimize some scenarios, but is limited by the delay of the hardware abstraction layer, making it difficult to achieve fast response. Advanced algorithms such as reinforcement learning have potential, but are difficult to adapt to the real-time requirements of outdoor advertising machines due to long decision-making cycles and high deployment costs. In addition, hardware stacking can improve computing power, but the surge in power consumption and cost limits its practical promotion.

[0004] The core technical difficulty lies in how to achieve dynamic matching of data flow and heterogeneous computing units. The real-time changes in task load require the system to have microsecond-level resource scheduling capabilities, while the current solutions lack efficiency in memory bandwidth contention and hardware coordination, resulting in response delay fluctuations and performance degradation. In particular, in multi-modal interaction, the highly heterogeneous data access patterns and static allocation of computing units highlight the contradiction, making it difficult to fully utilize the collaborative potential of NPU, GPU, FPGA, and other hardware.

[0005] Therefore, how to drive resource allocation through data flow, optimize multi-level cache architecture and cross-unit scheduling to solve the matching problem of data and computing units in multi-modal interaction has become a key issue in improving the performance and energy efficiency of outdoor advertising machines. The solution to this problem will directly promote the optimization of interactive experience and the expansion of industry applications. SUMMARY

[0006] The present application provides a cloud intelligent image processing system and an application method thereof, mainly comprising:

[0007] The high-frequency access mode is extracted from the object feature data, the data segment with the access frequency higher than a preset threshold is analyzed by the NPU calculation unit, and the high-frequency feature set is stored in the local cache to obtain the high-frequency feature set; the preloading demand of the AR model is determined according to the high-frequency feature set, the corresponding model data is transmitted from the external storage to the special memory by the GPU calculation unit to generate the preloading data set; the speech instruction data stream is obtained, and the DSP calculation unit directly calls the FPGA shared cache to generate the scheduling instruction set for the instruction sequence with the real-time requirement higher than a preset threshold; the feature extraction is performed on the high-frequency feature set by the NPU calculation unit, if the extraction result matches the preloading data set of the GPU calculation unit, the feature is transmitted to the GPU to generate the multi-modal feature group; for the multi-modal feature group, the GPU calculation unit adjusts the memory allocation by using the data layout optimization technology, and processes the feature group by using the matrix transformation algorithm to generate the optimized data block; after the optimized data block is obtained, the FPGA scheduling unit analyzes the cross-unit data stream demand, if the data transmission delay is lower than a preset threshold, the transmission order is adjusted by using the scheduling algorithm to generate the data stream sequence; according to the data stream sequence, the FPGA scheduling unit allocates the optimized data block to the corresponding calculation unit, and the matching result is obtained by judging the multi-modal interaction data and the calculation matching degree by using the consistency checking algorithm; the data update frequency of the local cache and the shared cache is adjusted by using the matching result, if the matching degree is higher than a preset threshold, the cache content is updated by using the sliding window algorithm to generate a new cache set; after the new cache set is obtained, the NPU calculation unit and the GPU calculation unit re-extract the features and process according to the data stream sequence, if the feature change trend exceeds a preset range, the final optimization scheme is generated by optimizing the scheduling strategy by using the feedback adjustment algorithm.

[0008] The technical scheme provided by the embodiment of the application can include the following beneficial effects:

[0009] The object feature high-frequency access mode is extracted by the NPU, the AR model is preloaded by the GPU, and the real-time speech instruction is processed by the DSP, so that the multi-modal data is efficiently processed. The NPU and the GPU work cooperatively to extract and match the features to generate the multi-modal feature group. The GPU processes the feature group by using the data layout optimization technology, and the FPGA scheduling unit analyzes the cross-unit data stream demand and optimizes the transmission order. The consistency checking algorithm is used to judge the multi-modal interaction data and the calculation matching degree, and the cache update strategy is dynamically adjusted. Finally, the NPU and the GPU re-extract the features according to the data stream sequence, and the scheduling strategy is optimized by using the feedback adjustment algorithm. The application realizes the efficient processing and intelligent interaction optimization of the multi-modal data, and improves the system performance and user experience. BRIEF DESCRIPTION OF DRAWINGS

[0010] Fig. 1 The flowchart of the cloud intelligent image processing system and the application method of the application.

[0011] Fig. 2A schematic diagram of a cloud intelligent image processing system and an application method thereof.

[0012] Fig. 3 Another schematic diagram of a cloud intelligent image processing system and an application method thereof. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be described in detail below with reference to the drawings and specific embodiments.

[0014] Embodiment one,

[0015] As Figs. 1-3 , the cloud intelligent image processing system and the application method thereof can specifically include:

[0016] Step S101, extracting a high-frequency access mode from the object feature data, analyzing the data segment with an access frequency higher than a preset threshold through an NPU calculation unit, and storing it to a local cache to obtain a high-frequency feature set.

[0017] The access mode is obtained from the object feature through data extraction, the change trend of the access mode is analyzed by using a statistical method to obtain a preliminary access frequency. The preliminary access frequency is compared with the preset threshold, if it is higher than the preset threshold, the access mode is analyzed in depth through the NPU calculation unit to obtain a high-frequency access data segment. The feature information is extracted from the high-frequency access data segment, the distribution rule of the high-frequency feature is determined through frequency analysis to obtain a feature distribution set. According to the feature distribution set, the high-frequency features are grouped by using a clustering algorithm to determine the feature subsets after grouping. The feature subsets are optimized by the NPU calculation unit to obtain an optimized high-frequency feature set, which is stored in the local cache. The high-frequency feature set is obtained from the local cache, the integrity of the feature set is judged by using a verification algorithm to obtain a final high-frequency feature set. The data segment in the local cache is updated by using a storage process for the final high-frequency feature set to obtain a real-time updated feature database.

[0018] For example, the access mode is obtained from the object feature through data extraction, which can be understood as extracting key information such as eye distance, nose height, etc. from the object image to form an access sequence.

[0019] For example, 10 key point coordinates are extracted from an object photo to form an access mode data set containing a time stamp. When the change trend is analyzed by using a statistical method, the change frequency of these key points in multiple accesses can be counted, for example, the object is identified 100 times a day, the average value of the coordinate offset of a certain key point is recorded, and the preliminary access frequency is obtained, such as 50 times a day. The preliminary access frequency is compared with the preset threshold, assuming that the threshold is set to 30 times / day, if it is higher than this value, the NPU calculation unit is used for in-depth analysis.

[0020] For example, the NPU can perform convolution processing on the time series of access patterns to extract high-frequency access data segments, such as identifying that the period from 8:00 to 9:00 every day is a peak period with an access frequency of 20 times / hour. By extracting feature information from the high-frequency access data segments, the focus can be on the key point changes during the peak period, for example, the displacement of detailed features during this period is larger, and by frequency analysis to determine the distribution rule, it is possible to find that 80% of the accesses are concentrated on certain specific features, and obtain a feature distribution set. According to the feature distribution set, a clustering algorithm is used to group high-frequency features.

[0021] In one possible implementation, the K-means algorithm is used to divide the features into 3 groups, such as detail, nose and mouth feature subsets, which respectively reflect the tendency of different access patterns.

[0022] Preferably, the feature subsets are optimized by the NPU, such as reducing redundant features, from 10 features to 5 in each group, to form an optimized high-frequency feature set, and stored in the local cache.

[0023] It should be noted that the local cache can be set as a fast access memory area to ensure that the data reading efficiency is improved by about 30%. After obtaining the high-frequency feature set from the local cache, the integrity is judged by a verification algorithm.

[0024] Specifically, the hash verification can be used to confirm that the data is not lost, and the final high-frequency feature set is obtained.

[0025] For example, after verification, it is found that the detail feature set is complete and contains 5 optimized features. For the final high-frequency feature set, a stored procedure is used to update the data segment in the local cache.

[0026] In one embodiment, the stored procedure can overwrite the old data in timestamp order to form a real-time updated feature database, such as the updated database reflecting the feature change trend in the past 24 hours.

[0027] For example, this method can significantly improve the efficiency of the object recognition system, and the focus of high-frequency features reduces the processing time of low-value data, and the deep analysis of the NPU enhances the recognition accuracy during the peak period.

[0028] It can be understood that the real-time updated feature database provides support for dynamic scenarios, such as quickly adapting to the face feature changes of new users in security monitoring.

[0029] In one embodiment, the optimized feature set can also reduce the storage requirement by about 20%, effectively improving the system resource utilization.

[0030] For example, the combination of clustering grouping and feature optimization allows the system to maintain high accuracy under complex light conditions, demonstrating the practical value of the technology.

[0031] In step S102, the preloading requirement of the AR model is determined according to the high-frequency feature set, and the corresponding model data is transmitted from the external storage to the special memory by the GPU computing unit to generate a preloading data set.

[0032] The preloading requirement of the AR model is determined by the high-frequency feature, the model data is obtained from the external storage by the GPU computing unit, and an initial transmission set is generated. The data stream is obtained from the initial transmission set, and the segmented processing is adopted for the data stream. The segmented data segments are stored in the special memory to obtain a memory temporary set. According to the memory temporary set, the data check is adopted to judge the integrity, and if the integrity meets the preset threshold, it is transmitted to the preloading set. For the preloading set, the data sorting is performed by the GPU computing unit to obtain the sorted feature sequence. The key data segment is extracted from the feature sequence, the K-means algorithm is adopted for grouping the key data segment to generate a grouped feature set. The data is compressed by the GPU computing unit through the grouped feature set to obtain a compressed data packet. According to the compressed data packet, it is stored in the special memory to generate a final preloading data set.

[0033] For example, when the preloading requirement of the AR model is determined by the high-frequency feature, it can be understood that according to the frequently occurring feature data in object recognition, the model part that needs to be preferentially loaded in the augmented reality (AR) system is inferred.

[0034] For example, it is assumed that the system identifies that the detail feature is accessed 80 times a day, which is higher than other parts, which prompts the AR model to preferentially load the rendering data related to the details. In one possible implementation, the GPU computing unit obtains the model data from the external storage, for example, extracts the 3D model file containing the detail texture from the cloud database to generate an initial transmission set. The initial transmission set can contain 100MB of model data, covering multiple facial features. After obtaining the data stream from the initial transmission set, the segmented processing of the data stream is a key link.

[0035] Specifically, the 100MB data can be divided into 10 10MB segments according to the feature type, such as details, nose, etc., and stored in the special memory to generate a memory temporary set.

[0036] In one embodiment, the special memory can be selected as a high-speed SRAM to ensure the improvement of the segmented data reading speed. For the integrity check of the memory temporary set, a simple counting check can be adopted, for example, to confirm whether all 10 segments are in place, and if less than 10, to reacquire to ensure that there is no data loss. After transmission to the preloading set, the data sorting by the GPU computing unit becomes important.

[0037] Preferably, the features can be ranked in descending order of access frequency, for example, the detail data is ranked first, followed by the mouth, and then the generated ordered feature sequence is generated. When extracting the key data segment from the feature sequence, it is assumed that the detail data in the first 20% of the sequence is identified as a high frequency segment. When grouping using the K-means algorithm, the data can be divided into 3 groups, for example, the detail group contains 5 key point coordinates, reflecting the trend in the access set. Compression processing by grouping feature sets is another technical focus.

[0038] For example, the GPU can perform lossless compression on the detail group data, reducing 5MB data to 3MB, and obtaining a compressed data packet.

[0039] It can be understood that storing in a dedicated memory after compression can reduce the occupied space. After the final preloading data set is generated, for example, containing 3 groups of compressed data, the total size is reduced from 50MB to 30MB. This way significantly optimizes the loading efficiency of the AR system.

[0040] It should be noted that the data stream segmentation processing can allow the system to load different features in parallel, reducing the waiting time.

[0041] In one possible implementation, the ordered feature sequence can also be dynamically adjusted, for example, when the access frequency of the nose feature rises, the system automatically promotes its priority.

[0042] Exemplarily, the combination of grouping and compression allows the AR model to focus more on high-frequency requirements when loading, avoiding redundant data occupying resources.

[0043] Specifically, checking the integrity ensures the reliability of the preloading data, for example, the hash check confirms that the detail data is not damaged.

[0044] In one embodiment, the compressed data packet can also support fast decompression, for example, the GPU completes decompression within 2 seconds before rendering, ensuring real-time performance.

[0045] Preferably, the final preloading data set can be updated by timestamp, for example, refreshed every hour, to ensure that the data reflects the latest access pattern.

[0046] It can be understood that this method allows the AR system to still run efficiently when resources are limited, and is particularly suitable for real-time object rendering scenarios on mobile devices.

[0047] Step S103, acquire the voice instruction data stream, and the DSP calculation unit directly calls the FPGA shared cache to generate a scheduling instruction set for instruction sequences with a real-time requirement higher than a preset threshold.

[0048] The data stream generated by the voice instruction is obtained, and the DSP computing unit extracts the instruction sequence by analyzing the data stream. For the extracted instruction sequence, whether it is higher than a preset threshold is determined by real-time detection, and an instruction set that meets the condition is obtained. The instruction set that meets the condition is stored in the FPGA shared cache to generate a cache data segment. The scheduling requirement is obtained from the cache data segment, and the DSP computing unit generates a scheduling instruction set by priority sorting. For the scheduling instruction set, it is transmitted to the computing unit through the shared cache to obtain an execution sequence. The K-means algorithm is used to group the execution sequence to generate a grouped instruction set. According to the grouped instruction set, it is stored in the FPGA shared cache to generate a final scheduling set.

[0049] Exemplarily, obtaining the data stream generated by the voice instruction is a core link in real-time voice processing.

[0050] Exemplarily, in an augmented reality system, a user inputs "enlarge detail view" through voice, and a microphone captures to generate continuous audio data stream, which may be 5MB per second. When the DSP computing unit extracts the instruction sequence by analyzing the data stream, it can be understood as converting the audio stream into recognizable text instructions. In a possible implementation manner, the DSP uses a pre-trained voice recognition model to analyze "enlarge detail view" into a specific instruction sequence, such as "enlarge-detail-view" three independent units. For real-time detection, whether the instruction sequence is higher than the preset threshold is the key to ensure system response.

[0051] Specifically, assuming that the preset threshold is 50 milliseconds, if the time consumed by analyzing and extracting "enlarge-detail-view" is 30 milliseconds, it meets the condition and enters the instruction set.

[0052] In an embodiment, if the time consumed by analyzing a certain instruction is 70 milliseconds, such as "adjust light brightness", it is excluded to avoid affecting the real-time experience. When the FPGA shared cache stores the instruction set that meets the condition, the cache data segment is generated, which embodies the efficient storage characteristics.

[0053] Preferably, the FPGA cache capacity is 10MB, which can store 200 instructions, each occupying 50KB.

[0054] Exemplarily, after "enlarge-detail-view" is stored as an instruction, the DSP obtains the scheduling requirement from the cache data segment, such as detail-related instructions that need to be processed in priority. When the DSP computing unit generates a scheduling instruction set by priority sorting, it can assign weights according to the application scenarios of the instructions.

[0055] For example, "enlarge-detail-view" has a weight of 90 because it involves core rendering functions, while "adjust background color" has a weight of 60. After the scheduling instruction set is transmitted to the computing unit through the shared cache, the execution sequence is generated, which is the starting point of actual operation.

[0056] In a possible implementation, the execution sequence is arranged in descending order of weight, with the "zoom-in-detail-view" first, ensuring that the rendering module responds preferentially. When grouping the execution sequence using the K-means algorithm, the instructions can be divided into three groups, such as rendering, adjustment, and auxiliary. The "zoom-in-detail-view" is classified into the rendering group, and contains 5 similar instructions, reflecting a centralized trend. The grouped instruction set is stored in the FPGA shared cache, and the generation of the final scheduling set is the final link of the process.

[0057] It should be noted that the rendering group can occupy 4MB of cache, and contains 10 high-priority instructions.

[0058] It can be understood that such grouped storage allows the system to process different categories of tasks in parallel, for example, the rendering group is preferentially loaded to the GPU, while the adjustment group waits for idle resources, thereby improving efficiency.

[0059] In an embodiment, if it is detected that the "zoom-in-detail-view" appears frequently, the system can dynamically adjust the cache allocation, increase the rendering group space to 6MB, and ensure fast response.

[0060] Specifically, the combination of real-time detection and priority sorting allows the voice instruction to be quickly converted into visual output, for example, the detail view is zoomed in within 1 second after the user finishes speaking.

[0061] Exemplarily, the high-speed read-write characteristics of the FPGA shared cache support seamless transmission of the instruction set, avoiding data bottlenecks.

[0062] Preferably, the grouped instruction set can also be dynamically updated according to the frequency of use, for example, refreshed every 5 minutes, keeping the scheduling set consistent with user habits. This significantly improves the fluency of voice control, especially when resources are scarce, while still ensuring the execution of core functions.

[0063] In step S104, the high-frequency feature set is subjected to feature extraction by the NPU calculation unit, and if the extraction result matches the preloaded data set of the GPU calculation unit, the features are transmitted to the GPU to generate a multi-modal feature group.

[0064] The NPU computing unit performs feature extraction on the high-frequency feature set to obtain an extraction result. If the extraction result matches a preloaded data set in the GPU computing unit, a multi-modal feature group is generated through feature transmission. The multi-modal feature group is used to perform calculation processing through the GPU computing unit to obtain a calculation execution sequence. For the calculation execution sequence, data processing requirements are obtained, and the unit cooperation is stored in the shared cache to obtain a processed data segment. The feature distribution characteristics are extracted from the processed data segment, and the K-means algorithm is used to group the feature distribution to obtain a grouped feature set. According to the grouped feature set, the feature is transmitted to the NPU computing unit to generate an optimized feature sequence. For the optimized feature sequence, the unit cooperation is transmitted to the GPU computing unit to obtain a final execution set.

[0065] For example, the NPU computing unit performs feature extraction on the high-frequency feature set to obtain an extraction result.

[0066] For example, in a voice-driven augmented reality system, a microphone captures the user's high-frequency voice signal, such as a 10MB audio stream per second. The NPU computing unit can use a dedicated neural network accelerator to extract key features such as tone changes or keyword spectrograms from it.

[0067] Specifically, the NPU can focus on the high-frequency band in the audio stream and extract the feature of the phrase "open menu", resulting in a feature vector set. If the extraction result matches a preloaded data set in the GPU computing unit, a multi-modal feature group is generated through feature transmission.

[0068] For example, the preloaded data set can contain feature vectors of common voice command templates in the system, such as the phrases "open" and "close". The GPU computing unit compares the feature vector transmitted by the NPU with the template and confirms the match. After confirming the match, the voice feature is integrated with other modal data, such as the user's gaze focus data, into a multi-modal feature group.

[0069] In one possible implementation, the voice "open menu" is combined with the gaze pointing to the upper left corner of the screen to form a feature group containing voice and visual information. The multi-modal feature group is used to perform calculation processing through the GPU computing unit to obtain a calculation execution sequence.

[0070] Preferably, the GPU uses its parallel computing capability to perform in-depth analysis on the multi-modal feature group, for example, to determine whether the user's intention is to open the main menu or the sub-menu.

[0071] It can be understood that the calculation execution sequence can be a time-ordered instruction list, such as "verify voice - analyze gaze - generate menu instruction". For the calculation execution sequence, data processing requirements are obtained, and the unit cooperation is stored in the shared cache to obtain a processed data segment.

[0072] Specifically, the data processing requirement can be fast response to core instructions, and the cooperative unit stores the high-priority part in the sequence to a shared cache with a capacity of 15 MB.

[0073] In an embodiment, a data segment of 50 instructions is stored in the cache, each occupying about 300 KB, ensuring that the subsequent module can be quickly read. The feature distribution characteristics are extracted from the processed data segment, and the K-means algorithm is used to group the feature distribution to obtain a grouped feature set.

[0074] For example, the feature distribution characteristics can be the frequency and type distribution of instructions, and the K-means algorithm divides them into three groups: a high-frequency instruction group, an adjustment type instruction group, and a low-frequency auxiliary group.

[0075] In an embodiment, “open menu” is classified into the high-frequency group due to frequent use, containing 8 similar instructions, reflecting user habits. According to the grouped feature set, the optimized feature sequence is generated by transmitting the features to the NPU calculation unit.

[0076] It should be noted that the NPU can adjust the weight of the feature vector according to the characteristics of the grouped feature set, for example, the feature weight of the high-frequency group is increased to 80, and the low-frequency group is reduced to 30.

[0077] Exemplarily, the optimized feature sequence can be a list of instruction features sorted by importance, highlighting core functional requirements. For the optimized feature sequence, the unit is cooperatively transmitted to the GPU calculation unit to obtain the final execution set.

[0078] In a possible implementation, cooperative transmission ensures that the optimized sequence is seamlessly delivered to the GPU, and after generating the final execution set, the system executes according to the priority.

[0079] For example, “open menu” is ranked first due to high weight, and the menu pops up within 0.8 seconds after the user's voice input.

[0080] Preferably, this cooperative approach can significantly improve the smoothness of instruction processing, especially when multiple tasks are parallel, while still ensuring fast response of core functions.

[0081] Step S105, for the multi-modal feature group, the GPU calculation unit uses data layout optimization technology to adjust memory allocation, and generates an optimized data block by processing the feature group through a matrix transformation algorithm.

[0082] The feature group is obtained through multi-modal feature acquisition, and a GPU computing unit performs matrix transformation on the feature group to obtain optimized data. For the optimized data, the GPU computing unit adjusts the data layout and determines a memory allocation scheme. The adjusted data is obtained from the memory allocation scheme, and the computing unit processes the adjusted data through matrix transformation to obtain a transformed data block. If the transformed data block meets a preset threshold, a processed feature set is generated through data processing. According to the processed feature set, an optimization technique is used to adjust the memory allocation to obtain an allocation optimization set. For the allocation optimization set, the computing unit generates a final data group through data layout adjustment. The feature distribution characteristics are obtained from the final data group, and the K-means algorithm is used to group the characteristics to obtain a grouping result set.

[0083] For example, the process of obtaining the feature group through multi-modal feature acquisition can be understood as extracting key information from multiple input sources.

[0084] For example, in a voice and gesture combined interaction system, a microphone captures user voice signals, and a camera records gesture actions at the same time, generating voice features and gesture features respectively, and finally fusing into a feature group.

[0085] For example, the voice may be "enlarge the screen", and the gesture is a double-hand expansion action. The feature group contains the representation data of the two. When the GPU computing unit performs matrix transformation on the feature group to obtain optimized data, the core is to use the parallel computing capability of the GPU to process high-dimensional data.

[0086] Specifically, the feature group may be a matrix containing voice spectrum and gesture coordinates, and the GPU compresses or reconstructs it into a more compact form through matrix transformation.

[0087] For example, a 10MB feature group data may be reduced to 2MB optimized data after transformation, which is convenient for subsequent processing. For the optimized data, the GPU computing unit adjusts the data layout and determines a memory allocation scheme.

[0088] It should be noted that this process aims to improve data access efficiency.

[0089] In one possible implementation, the optimized data is rearranged according to the frequency of use, and the high-frequency access part is allocated to the high-speed memory area of the GPU.

[0090] For example, voice features are preferentially allocated to a 5MB fast cache due to real-time requirements, while gesture data is placed in ordinary memory. After obtaining the adjusted data from the memory allocation scheme, the computing unit processes the adjusted data through matrix transformation to obtain a transformed data block.

[0091] It can be understood that this step may further decompose the data into independent units.

[0092] For example, the adjusted data is divided into voice blocks and gesture blocks, occupying 1MB and 0.5MB respectively, and transformed to generate transformed data blocks more suitable for analysis. If the transformed data blocks meet the preset threshold, a processed feature set is generated through data processing.

[0093] Preferably, the threshold can be that the correlation score of the data block needs to be higher than 80 points.

[0094] In an embodiment, the correlation of the voice block and the gesture block is detected to reach 85 points, indicating that both point to "zoom in the screen", and a feature set containing this intention is generated. According to the processed feature set, an optimization technique is used to adjust the memory allocation, and an allocation optimization set is obtained.

[0095] Specifically, it can be dynamically adjusted according to the priority of the feature set.

[0096] For example, high-priority features such as core instructions "zoom in" are allocated to a 10MB cache, while secondary features are allocated to a normal area, ensuring fast response to critical tasks. For the allocation optimization set, the calculation unit generates a final data group through data layout adjustment.

[0097] In a possible implementation, the data layout is arranged in time sequence to generate an ordered 2MB data group containing the fusion information of voice and gesture.

[0098] Exemplarily, the first 500KB of the final data group stores voice instructions, and the last 1500KB stores gesture details. From the final data group, the feature distribution characteristics are obtained, and the K-means algorithm is used to group the characteristics to obtain a grouping result set.

[0099] For example, the feature distribution characteristics can be instruction type and frequency distribution, and the K-means divides them into three groups: a core instruction group such as "zoom in" accounting for 60%, an auxiliary instruction group accounting for 30%, and others accounting for 10%.

[0100] In an embodiment, the core instruction group contains 5 high-frequency instructions, reflecting the main needs of the user. This grouping method helps the system to identify key patterns and improve the pertinence of subsequent processing.

[0101] After obtaining the optimized data blocks, the FPGA scheduling unit analyzes the cross-unit data flow requirements in step S106, and if the data transmission delay is lower than the preset threshold, the transmission order is adjusted through a scheduling algorithm to generate a data flow sequence.

[0102] After obtaining the optimized data block, the FPGA scheduling unit determines the data flow requirement through cross-unit analysis to obtain a unit requirement distribution. Transmission delay characteristics are extracted from the unit requirement distribution. If the transmission delay is lower than a preset threshold, a scheduling algorithm is used to adjust the transmission sequence to obtain an adjusted sequence. The FPGA scheduling unit analyzes the data flow sequence characteristics for the adjusted sequence to determine a delay judgment result. According to the delay judgment result, an optimized transmission flow is generated through sequence adjustment to obtain a streaming data group. Cross-unit analysis indicators are obtained from the streaming data group, and the indicators are grouped using a K-means algorithm to obtain a grouping feature set. The FPGA scheduling unit generates a final sequence group through data flow requirement adjustment for the grouping feature set to obtain a sequence optimization set. Transmission sequence characteristics are extracted from the sequence optimization set, and sequence integrity is determined using a consistency check to obtain a checked data group.

[0103] For example, after obtaining the optimized data block, the FPGA scheduling unit determines the data flow requirement through cross-unit analysis to obtain a unit requirement distribution.

[0104] For example, in a voice and gesture fusion interactive system, cross-unit analysis may involve multiple processing units respectively responsible for the preliminary processing of voice signals and gesture coordinates, and the data flow requirement reflects the load of each unit.

[0105] For example, the voice unit may process 100,000 instructions per day, and the gesture unit may process 50,000 instructions. The FPGA determines the requirement distribution by analysis and shows that the voice unit has a higher load. When transmission delay characteristics are extracted from the unit requirement distribution.

[0106] Specifically, the time for data to be transmitted from one unit to another unit can be focused on.

[0107] In one possible implementation, the delay of the voice unit to the fusion unit is 5 milliseconds, and the delay of the gesture unit is 3 milliseconds. The preset threshold is set to 4 milliseconds.

[0108] It should be noted that if the transmission delay is lower than the threshold, it indicates that the current sequence may not be efficient enough. Therefore, the transmission sequence is adjusted through a scheduling algorithm, for example, the gesture data is preferentially transmitted, and the adjusted sequence is generated. The voice data delay is increased to 4.5 milliseconds, and the gesture delay is decreased to 2 milliseconds. The FPGA scheduling unit analyzes the data flow sequence characteristics for the adjusted sequence to determine a delay judgment result.

[0109] Preferably, the characteristics can include data packet size and transmission interval.

[0110] For example, the gesture data packet is 200 KB with an interval of 1 millisecond, and the voice data packet is 500 KB with an interval of 2 milliseconds. The judgment result shows that the gesture data is more suitable for preferential processing. According to the delay judgment result, an optimized transmission flow is generated through sequence adjustment to obtain a streaming data group.

[0111] In an embodiment, the streaming data group is sorted by timestamp, the total size is compressed to 1 MB, and the ordered information containing voice and gestures. The cross-cell analysis indicators are obtained from the streaming data group, and the K-means algorithm is used to group the indicators to obtain a grouping feature set.

[0112] For example, the indicators can be transmission rate and inter-cell dependency, and the K-means algorithm divides them into a high dependency group and a low dependency group, and the high dependency group accounts for 70%, containing strongly correlated data of voice and gestures. For the grouping feature set, the FPGA scheduling unit generates a final sequence group by adjusting the data stream demand to generate a sequence optimization set.

[0113] Specifically, the high dependency group is prioritized, and the total size is reduced to 800 KB to ensure smooth transmission of critical data. The transmission order characteristics are extracted from the sequence optimization set, and the consistency check is used to judge the sequence integrity to obtain the checked data group.

[0114] It can be understood that the check can check whether the data packet is lost.

[0115] For example, the check finds that the gesture data packet is complete and the voice data packet loss rate is less than 1%, and generates a checked data group with a size of 750 KB. This way ensures the stable transmission of data streams between multiple cells, providing a reliable basis for subsequent processing.

[0116] In step S107, according to the data stream sequence, the FPGA scheduling unit allocates the optimized data block to the corresponding computing unit, and judges the matching degree of the multi-modal interaction data and the calculation through the consistency check algorithm to obtain the matching result.

[0117] The FPGA scheduling unit extracts the optimized data block from the data stream sequence by the allocation algorithm, transmits the optimized data block to the computing unit by the distribution rule to obtain the distributed data group. The multi-modal interaction characteristics are obtained from the distributed data group, the consistency check algorithm is used to analyze the correspondence between the interaction data and the computing unit, and the matching degree distribution is determined. For the matching degree distribution, the allocation order of the computing unit is sorted by the adjustment rule to obtain the sorted unit group. The interaction data features are extracted from the sorted unit group, and the K-means algorithm is used to group the features to obtain a grouping feature set. For the grouping feature set, an optimized allocation scheme is generated by dynamically adjusting the data stream sequence to obtain an adjusted data group. The calculation matching degree indicators are obtained from the adjusted data group, and if the matching degree is lower than the preset threshold, the allocation order is adjusted by the check algorithm to obtain the final data group. According to the final data group, the consistency check is used to judge the integrity of the data stream sequence and the computing unit to obtain the checked sequence group.

[0118] Exemplarily, when the FPGA scheduling unit extracts the optimized data blocks from the data stream sequence through the allocation algorithm, the core lies in how to efficiently screen the data units suitable for transmission.

[0119] Exemplarily, in a voice and gesture fusion interactive system, the data stream sequence can contain original information of voice instructions and gesture coordinates, and the allocation algorithm can screen according to the data packet size and real-time requirement, for example, the 200 KB gesture data block is extracted in priority to the 500 KB voice data block.

[0120] It should be noted that this screening is based on the priority of the data stream and the processing capacity of the computing unit, to ensure that the key information reaches in priority.

[0121] In a possible implementation, after the optimized data blocks are transmitted to the computing unit through the distribution rule, a post-distribution data group is formed.

[0122] Specifically, the distribution rule can be set to send the gesture data block to the low-delay computing unit, and the voice data block is allocated to the high-throughput unit.

[0123] For example, the gesture data transmission takes 2 milliseconds, and the voice data takes 4 milliseconds, and the post-distribution data group reflects the initial load distribution of the computing unit. When the multimodal interaction characteristics are obtained, the synchronization of voice and gesture data can be focused on.

[0124] Preferably, the consistency verification algorithm analyzes the correspondence between the two, finds that the gesture data matching degree reaches 90%, and the voice data is slightly lower to 85% due to delay, and the matching degree distribution is formed accordingly. When adjusting the allocation order of the computing unit according to the matching degree distribution, the generation of the sorted unit group depends on the dynamic adjustment rule.

[0125] In an embodiment, if the priority of the gesture computing unit is improved, the processing sequence of the sorted unit group is in the front, and the voice unit is in the back, and the overall transmission efficiency is optimized. When extracting the interaction data features from the sorted unit group, for example, the transmission interval of the gesture data is 1 millisecond, and the transmission interval of the voice data is 2 milliseconds. After grouping by using the K-means algorithm, the grouped feature set can be divided into a high real-time group and a low real-time group, and the high real-time group contains gesture data, accounting for 60%.

[0126] It can be understood that when the data stream sequence is dynamically adjusted according to the grouped feature set, the optimized allocation scheme can rebalance the computing resources.

[0127] For example, the data stream of the high real-time group is given priority, and the transmission proportion of gesture data in the adjusted data group increases by 10%. When calculating the matching degree index from the adjusted data group, if it is found that the matching degree of the gesture unit increases to 95% but the matching degree of the voice unit is lower than the preset threshold value 80%, the verification algorithm will re-adjust the allocation order, and finally the matching degrees of the two in the final data group are both stable above 90%. According to the consistency verification of the final data group, the integrity of the data stream sequence and the calculation unit is focused on.

[0128] For example, the verification finds that the gesture data packet completeness rate is 100% and the voice data packet loss rate is lower than 0.5%, and the sequence group size is compressed to 700KB after verification. This way ensures the efficient allocation of multi-modal interaction data and provides stable support for the system.

[0129] In one embodiment, the adjusted sequence group can also reduce the waiting time of the calculation unit and improve the real-time response capability.

[0130] Step S108, adjust the data update frequency of the local cache and the shared cache through the matching result, if the matching degree is higher than the preset threshold value, use the sliding window algorithm to update the cache content to generate a new cache set.

[0131] By comparing the matching degree with the preset threshold value through the matching result, if the matching degree is higher than the preset threshold value, the cache content is processed by the sliding window algorithm to obtain a new cache set. The frequency distribution characteristics are obtained from the new cache set, and the update frequency of the local cache is adjusted according to the frequency distribution to obtain an adjusted frequency group. The cache content of the shared cache is updated through the adjusted frequency group, and the synchronization of the shared cache and the local cache is judged by consistency verification to obtain a synchronization data group. Data adjustment parameters are extracted from the synchronization data group, and the processing range of the sliding window is optimized according to the data adjustment parameters to obtain an optimized window set. The new cache set is reprocessed through the optimized window set to obtain updated cache content, and the integrity of the cache content is determined. The change trend of the frequency distribution is analyzed from the updated cache content, and the update frequency of the local cache and the shared cache is adjusted through the change trend to obtain a final frequency group. According to the final frequency group, the data consistency of the local cache and the shared cache is judged by consistency verification to obtain a verified data group.

[0132] For example, when analyzing the matching degree and the preset threshold value through the matching result, it can be understood as a key step of judging the data processing direction.

[0133] For example, in a voice instruction system, assuming that the preset threshold value is 85% and the matching degree result is 90%. Since it is higher than the threshold value, the system will trigger the subsequent sliding window algorithm.

[0134] In a possible implementation, the sliding window algorithm processes the cache content by extracting continuous data segments from the original cache, such as processing 500 KB of cache blocks each time, to form a new cache set. The principle here is to dynamically filter out high-frequency data by window sliding.

[0135] Specifically, the concentration of data access can be observed.

[0136] For example, analysis finds that 80% of requests are concentrated on 20% of data blocks, which prompts the system to preferentially cache these high-frequency content. The update frequency of the local cache is adjusted according to the frequency distribution.

[0137] In an embodiment, if the access frequency of a data block is 10 times per second, the adjusted frequency group can shorten its update period to 0.5 seconds. This adjustment ensures that the local cache can respond to frequent requests more quickly. The cache content of the shared cache is updated by adjusting the frequency group.

[0138] It should be noted that the shared cache usually serves multiple nodes. Consistency checking is used to determine synchronization here, for example, checking finds that the synchronization rate of the shared cache is 95% after the local cache is updated, forming a synchronized data group. This ensures the consistency of data among multiple nodes. The data adjustment parameter is extracted from the synchronized data group.

[0139] Preferably, the adjustment value of the window size can be extracted, such as expanding the sliding window range from 500 KB to 600 KB, to obtain an optimized window set. This optimization can cover more high-frequency data. The new cache set is reprocessed by the optimized window set.

[0140] For example, the system re-scans the cache content and finds that the updated cache integrity is improved, and the packet loss rate is reduced from 1% to 0.2%. The trend of the change in the frequency distribution is analyzed from the updated cache content.

[0141] In a possible implementation, the frequency distribution shows that the proportion of high-frequency data increases from 60% to 70%. The update frequency of the local cache and the shared cache is adjusted according to this trend, such as shortening the update period of the shared cache from 1 second to 0.8 seconds, to form a final frequency group. This adjustment improves the response speed of the system. Consistency checking is used for the final frequency group.

[0142] Specifically, the checking may find that the data consistency of the local cache and the shared cache is 98%, generating a checked data group. This step ensures the reliability of data among multiple ends.

[0143] It can be understood that this way significantly improves the cache efficiency through dynamic adjustment and checking.

[0144] For example, in a voice instruction system, the response delay experienced by the user can be shortened from 200 milliseconds to 150 milliseconds. This optimization is particularly important for scenarios with high real-time requirements.

[0145] In an embodiment, the window size adjustment of the sliding window algorithm can also be dynamically changed according to the data type.

[0146] For example, for short voice instructions, the window is reduced to 300 KB, while long instructions are expanded to 700 KB. This flexibility enriches the applicability of the solution. From multiple perspectives, frequency adjustment and consistency checking complement each other, ensuring data synchronization and optimizing resource allocation.

[0147] In an embodiment, the system can also adjust the frequency group according to the peak load, for example, increase the update frequency by 20% during high load to ensure stable performance. These measures collectively support efficient data processing capabilities.

[0148] Step S109, after obtaining the new cache set, the NPU computing unit and the GPU computing unit re-extract features and process according to the data stream sequence, and if the feature change trend exceeds the preset range, the feedback adjustment algorithm is used to optimize the scheduling strategy to generate the final optimization scheme.

[0149] After obtaining the new cache set, the NPU computing unit and the GPU computing unit extract features and process for the data stream sequence to obtain a feature set. From the feature set, analyze the change trend, and determine if the change trend exceeds the preset range. If so, adjust the algorithm group through feedback adjustment. Optimize the scheduling strategy for the adjusted algorithm group, and use consistency checking to determine the integrity of the scheduling strategy to obtain an optimized strategy set. Re-process the data stream sequence through the optimized strategy set to obtain an updated feature set, and determine the distribution characteristics of the feature set. According to the distribution characteristics, adjust the processing unit allocation of the NPU computing unit and the GPU computing unit to obtain an allocated computing group. Process the new cache set for the allocated computing group, use the K-means algorithm to cluster the feature distribution, and obtain a clustered data group. Extract the scheduling parameters from the clustered data group, adjust the optimization scheme through the scheduling parameters, and obtain the final optimization scheme.

[0150] For example, after obtaining the new cache set, the NPU computing unit and the GPU computing unit extract features and process for the data stream sequence to obtain a feature set.

[0151] For example, in a voice recognition system, the NPU may focus on low-power timing feature extraction, while the GPU is responsible for parallel processing of frequency domain features, and the two work together to generate a feature set containing time and frequency information.

[0152] It can be understood that this division of labor takes advantage of hardware characteristics and improves processing efficiency. Analyzing the trend of changes in the feature set, if it is determined that the change rate exceeds the preset range, the algorithm is adjusted.

[0153] Specifically, assuming that the preset range is that the feature change rate does not exceed 20%, if the analysis finds that the change rate of a certain data stream reaches 30%, the system triggers the feedback mechanism.

[0154] In a possible implementation, the feedback adjustment algorithm can smooth feature extraction by increasing the size of the convolution kernel to obtain an adjusted algorithm set.

[0155] Preferably, this adjustment can make the system more adaptive to data fluctuations. The scheduling strategy is optimized for the adjusted algorithm set, and the consistency check is used to determine the integrity to obtain an optimized strategy set.

[0156] Illustratively, the scheduling strategy can involve the allocation ratio of tasks between the NPU and the GPU. When performing consistency checking, if it is found that the task allocation deviation of a certain strategy is less than 5% in multiple rounds of testing, it is considered complete and forms the optimized strategy set.

[0157] It should be noted that this step ensures the stability of resource allocation. The data stream sequence is reprocessed through the optimized strategy set, the updated feature set is obtained, and the distribution characteristics are determined.

[0158] In an embodiment, after reprocessing, it is found that the proportion of high-frequency features in the feature set increases from 50% to 65%, indicating that the distribution is more concentrated. According to the distribution characteristics, the allocation of processing units of NPU and GPU is adjusted, for example, the task proportion of NPU is increased from 40% to 50%, to obtain an allocated calculation group. This adjustment can better match the data characteristics. The new cache set is processed for the allocated calculation group, and the K-means algorithm is used to cluster the feature distribution to obtain a clustered data group.

[0159] For example, the system can divide the features into three categories: high-frequency, medium-frequency, and low-frequency access data, with high-frequency data accounting for 60%.

[0160] In a possible implementation, K-means achieves clustering by iteratively calculating the Euclidean distance between features to ensure classification accuracy. This step helps to clarify the usage patterns of data. The scheduling parameters are extracted from the clustered data group, and the optimization scheme is adjusted through the parameters to obtain the final optimization scheme.

[0161] Preferably, the extracted parameters can include the processing priority of each type of data, for example, the priority of high-frequency data is set to 1, and the priority of medium-frequency data is set to 2.

[0162] In an embodiment, the system adjusts the task queue according to the priority to ensure that high-frequency data is processed faster.

[0163] It can be understood that this solution improves the system's response to critical data.

[0164] In one possible implementation, if the system detects that the processing delay of a certain type of feature is high, for example, the high-frequency data increases from 100 milliseconds to 150 milliseconds, the system will further optimize the thread allocation of the NPU. This flexibility ensures the adaptability of the solution.

[0165] For example, the adjusted delay may decrease back to 120 milliseconds. From multiple perspectives, feature extraction, algorithm adjustment, and scheduling optimization complement each other, and together improve the processing efficiency of the data stream.

[0166] In one embodiment, the system can also dynamically adjust the number of clustering categories based on the load, for example, from three categories to four categories, covering more data patterns. This scalability enriches the scope of application of the solution.

[0167] Embodiment two,

[0168] In embodiment one, step S105 mainly focuses on "for multi-modal feature groups, the GPU computing unit uses data layout optimization techniques to adjust memory allocation, and processes the feature groups to generate optimized data blocks through matrix transformation algorithms". In this embodiment, in order to further improve the parallelism of data on the GPU and reduce memory bandwidth consumption, a dynamically adjustable parameter T (TileSize, block size) is introduced to this step, and adaptive scheduling is performed, aiming to meet the different needs of throughput and delay when different sizes of feature groups are parallel computed. Because the size and access pattern of multi-modal feature groups differ in different scenarios (object features + voice features + AR model features), a fixed matrix block size sometimes cannot balance high throughput and low delay.

[0169] When the feature matrix size is large and the computational load is high, too small blocks will cause frequent memory access; on the contrary, if the feature matrix size is small and the block is too large, it may cause resource waste and processing delay. Therefore, based on step S105 of the original embodiment one, this embodiment increases the dynamically adjustable TileSize parameter T, and cooperates with a set of adaptive algorithms to achieve higher performance optimization.

[0170] Parameter T represents that when performing data layout optimization and matrix transformation, the multi-modal feature group (or the intermediate generated optimized data) is divided into sub-blocks with a size of T x T for parallel computation or processing.

[0171] The value range of T can be dynamically fine-tuned according to the hardware specifications of the GPU (such as thread block size, register limit) and data features, and is usually selected within the interval of 8 x 8, 16 x 16, 32 x 32, etc.

[0172] Real-time frame rate, memory bandwidth utilization, kernel function execution time of GPU when processing multi-modal feature groups. If the kernel function execution time of the GPU is detected to be significantly increased, the cache hit rate is decreased, or the real-time frame rate is lower than the preset threshold within a certain time window, the adjustment of T is triggered. When the bandwidth bottleneck (memory access time ratio significantly increased) is detected, the value of T is reduced to reduce the reading amount of a single data block, so that each reading is more dispersed and the cache utilization is higher. When the parallelism is limited or the kernel scheduling time is too short, the value of T is increased so that a single thread block can process more elements, thereby reducing the scheduling overhead of kernel function startup.

[0173] Minimize the calculation delay and bandwidth contention within a certain time window; if the delay and bandwidth utilization are detected to be within a reasonable range, the existing T value is kept unchanged.

[0174] The following describes a specific improvement to step S105 in Embodiment One. Other parts that are the same or similar to Embodiment One are not described again.

[0175] Read the feature group from the previous step (such as the multi-modal features generated after interaction with the NPU and GPU) to prepare for matrix transformation and memory allocation optimization.

[0176] Initially, a default block size (such as T=16) can be set according to system configuration or empirical value, and the value is stored in the global constant or shared memory of the GPU.

[0177] The GPU computing unit performs preliminary matrix transformation on the feature group, splits the data into T×T sub-blocks, and forms the initial layout structure.

[0178] Collect key indicators during GPU runtime, such as kernel function execution time, memory bandwidth utilization, cache (L2 Cache, shared cache) hit rate, real-time frame rate, etc., and calculate the average value within a certain time window.

[0179] If one of the following conditions is observed, the automatic adjustment of T is triggered: the kernel function execution time is significantly increased (such as more than 30% higher than the historical average).

[0180] The cache hit rate drops below the threshold (such as below 70%).

[0181] The real-time frame rate is lower than expected (such as lower than 30 FPS).

[0182] The scheduling module determines the bandwidth bottleneck or parallelism bottleneck according to the currently measured indicator type and dynamically modifies the T value (for example, from 16 to 8, or from 16 to 32).

[0183] Record the new T value and update the scheduling table in the GPU to continue iterative monitoring and adjust as necessary in the future.

[0184] When the T value changes, the current feature group or the subsequent data block is re-blocked and transformed to generate a new "optimized data block".

[0185] The following formula is used to calculate the optimal block size under the given delay and bandwidth conditions:

[0186]

[0187] Where Topt represents the optimal block size, L t represents the calculation delay time, B represents the memory bandwidth, and M represents the feature matrix size.

[0188] The formula simply adds the "start-up delay / bandwidth" and "block number" which are the two most common main costs in parallel computing and data transmission to form an optimization index for block size T. It helps to find a balance between too small blocks and too large blocks, improving overall processing efficiency.

[0189] Due to the discrete values, hardware constraints or other additional delay items in reality, the optimal solution is usually approximated through discrete search or heuristic adaptive methods, rather than simply analytically solving the formula. However, this formula provides an important theoretical guidance, enabling us to better balance throughput and delay when processing feature matrices on heterogeneous architectures such as GPUs / NPUs (such as object recognition, AR superposition, and speech fusion), ultimately improving the performance of the entire multi-modal processing flow.

[0190] The physical meaning of each term in the formula and its trend with T reflects the basic contradiction between "fixed (or approximately fixed) delay + parallelization benefits brought by blocking" when performing high-performance computing or parallel processing. By combining parameter measurements from specific application scenarios and performing constrained optimization, a more reasonable block size can be set, further improving the efficiency and real-time performance of the system in the image / feature processing stage.

[0191] If the performance indicators (such as delay reduction to a predetermined range, hit rate recovery to over 80%) are met, this state is retained as the new optimal configuration.

[0192] When the transformation is completed and the "optimized data block" that meets the preset threshold conditions (such as delay, hit rate, frame rate, etc.) is obtained, it can be marked as the processed feature set.

[0193] Further, the memory allocation scheme is stored or transmitted according to the original method of Example One, and the subsequent steps are completed.

[0194] On the basis of step S105 of the embodiment one, a dynamically adjustable TileSize parameter T is added, and a set of adaptive algorithms is used to achieve higher performance optimization.

[0195] The parameter T represents that when data layout optimization and matrix transformation are performed, the multi-modal feature group (or the intermediate generated optimized data) is divided into sub-blocks with a size of TxT for parallel calculation or processing.

[0196] The value range of T can be dynamically fine-tuned according to the hardware specifications of the GPU (such as thread block size, register limit) and data characteristics, and is usually selected within the interval of 8x8, 16x16, 32x32, etc.

[0197] The real-time frame rate, memory bandwidth utilization, and kernel function execution time of the GPU when processing the multi-modal feature group. If the kernel function execution time of the GPU is detected to be significantly increased, the cache hit rate is decreased, or the real-time frame rate is lower than the preset threshold within a certain time window, the adjustment of T is triggered. When the bandwidth bottleneck (the memory access time proportion is obviously increased) is detected, the value of T is reduced, the reading amount of a single data block is reduced, the reading is more dispersed each time, and the cache utilization is higher. When the parallelism is limited or the kernel scheduling time is too short, the value of T is increased, so that a single thread block can process more elements, thereby reducing the scheduling overhead of the kernel function startup.

[0198] The calculation delay and bandwidth contention are minimized within a certain time window; if the delay and bandwidth utilization are detected to be within a reasonable interval, the existing T value is kept unchanged.

[0199] If the performance indicators (such as the delay is reduced to a predetermined range, and the hit rate is increased to more than 80%) are met, this state is retained as the new optimal configuration.

[0200] When the transformation is completed, the “optimized data block” that meets the preset threshold conditions (such as delay, hit rate, frame rate, etc.) is obtained, and the processed feature set can be marked.

[0201] Further, the memory allocation scheme is stored or transmitted according to the original method of embodiment one, and the subsequent steps are completed.

[0202] The following improvements are achieved by dynamically adjusting the value of T:

[0203] When the GPU kernel function processes multi-modal data in batches, the overall execution efficiency can be improved by about 10% to 20%.

[0204] Especially in high-load scenarios (such as simultaneously processing multi-object detection, AR superposition, and voice command analysis), adaptive blocking can reduce repeated memory access and better utilize shared cache.

[0205] When the data dimension changes or the access mode fluctuates, the system can automatically adjust the TileSize, avoid GPU thread starvation or cache jitter caused by improper block size, and make the overall frame rate more stable at most times.

[0206] In scenarios with high real-time requirements, such as outdoor advertising machines that need to respond to voice + object interaction instructions within 50ms, the average delay of the GPU processing link in this embodiment is reduced by about 15%.

[0207] The utilization of hardware resources (threads, registers, shared memory) within the GPU is more balanced, reducing waste or tedious scheduling caused by mismatched data sizes.

[0208] Compared to embodiment one, embodiment two mainly adds a dynamically adjustable block size (TileSize, T) and an adaptive algorithm to the data layout optimization step of the GPU, to further improve overall performance and real-time performance during multi-modal feature processing. In this way, in high-load scenarios of multi-modal interaction (object recognition, AR display, voice instructions), the memory bandwidth utilization and parallelism of the GPU can be automatically optimized according to the data characteristics, thereby achieving better processing efficiency in a wider range of application environments. This approach can also be coordinated with other steps in the patent (such as high-frequency feature extraction by NPU and scheduling optimization by FPGA) to further improve the stability and response speed of the overall system, addressing the need for high performance and high real-time performance in the background technology.

[0209] Based on the existing block size T determination scheme, further optimization of resource priority and allocation strategy is carried out for multi-task scenarios. While considering the parallelism and cache occupancy brought by the block size, a "weighting factor" is also introduced to adapt to the different latency sensitivity and bandwidth requirements of multi-modal tasks (such as voice instructions, AR model rendering, object recognition, etc.).

[0210] In a scheduling cycle, the following objective function is defined to comprehensively consider the scheduling cost of multiple types of tasks:

[0211]

[0212] Where, α i is the weighting factor of the i-th task (such as voice instruction processing, AR rendering, object recognition, etc.), representing the task urgency or real-time requirement;

[0213] D i is the data size required by the i-th task (which can be similar to M or a subset thereof);

[0214] T is the same block size parameter as in the previous formula;

[0215] represents the weighted scheduling cost when all k tasks are allocated resources for parallel processing at the block size T;

[0216] Approximately represents the number of tasks or processing batches after blocking, α i represents the priority, urgency or weight;

[0217] β×T is a penalty term, when T is too large, the overall throughput may increase, but the cache locality and scheduling flexibility decrease; β can be used to balance this effect. β×T is a penalty term, when T is too large, the overall throughput may increase, but the cache locality and scheduling flexibility decrease; β can be used to balance this effect.

[0218] It can be defined in the scheduling period, select:

[0219]

[0220] Here, T2 is the optimal block size or scheduling granularity obtained after integrating the weights of multiple tasks.

[0221] According to the number of concurrent multi-modal tasks k in the system and the real-time / weight α of each task i , data size D i , set or update parameters.

[0222] Select a discrete candidate set (such as {8, 16, 32, 64} or do iterative search within a certain range) to determine a candidate block size.

[0223] Calculate the objective function value FAWS(T) for each candidate T according to the formula.

[0224] Determine T2 Select the T that makes F AWS (T) minimum as T2.

[0225] Parallel processing and resource allocation of each task according to the block scheme T2, if necessary, combine the optimal block size to configure the GPU / NPU in practice.

[0226] By introducing the weighting factor α i and the additional penalty coefficient β, the contradiction between the real-time demand of high-priority tasks and the overall throughput of the system can be better balanced in the case of multiple tasks coexisting.

[0227] Further improve the system resource utilization, meet the individual needs of different types of tasks.

Claims

1. A cloud intelligent image processing method, characterized in that, The method comprises: extracting high-frequency access patterns from object feature data, analyzing data segments with access frequencies higher than a preset threshold by an NPU computing unit, storing them in a local cache to obtain a high-frequency feature set; determining the preloading demand of an AR model according to the high-frequency feature set, and transferring corresponding model parameters from external storage to a dedicated memory by a GPU computing unit to generate a preloading data set; obtaining a target instruction data stream, and directly calling an FPGA shared cache by a DSP computing unit to generate a scheduling instruction set for an instruction sequence with a real-time requirement higher than a preset threshold; extracting features from the high-frequency feature set by the NPU computing unit, and if the extraction result matches the preloading data set of the GPU computing unit, transmitting the features to the GPU to generate a multi-modal feature group; for the multi-modal feature group, the GPU computing unit adjusts memory distribution and allocation by using a data localization technology, and processes the feature group by a matrix transformation algorithm to generate an optimized data block; after obtaining the optimized data block, the FPGA scheduling unit analyzes cross-unit data stream requirements, and if the data transmission delay is lower than a preset threshold, adjusts the transmission order by a scheduling algorithm to generate a data stream sequence; according to the data stream sequence, the FPGA scheduling unit allocates the optimized data block to the corresponding computing unit, and judges the matching degree of multi-modal interactive data and calculation by a consistency checking algorithm to obtain a matching result; adjusting the data update frequency of the local cache and the shared cache according to the matching result, and if the matching degree is higher than a preset threshold, updating the prediction window to the cache storage inner window to generate a new cache set; after obtaining the new cache set, the NPU computing unit and the GPU computing unit re-extract features and process them according to the data stream sequence, and if the feature optimization trend exceeds the prediction range, optimize the scheduling strategy by a feedback adjustment algorithm to generate a final optimization scheme.

2. The method of claim 1, wherein, determining the preloading demand of an AR model according to the high-frequency feature set, and transferring corresponding model data from external storage to a dedicated memory by a GPU computing unit to generate a preloading data set, comprising: determining the preloading demand of an AR model according to the high-frequency feature set, and transferring corresponding model data from external storage to a dedicated memory by a GPU computing unit to generate a preloading data set, comprising: determining the preloading demand of an AR model according to the high-frequency feature set, and transferring corresponding model data from external storage to a dedicated memory by a GPU computing unit to generate a preloading data set, comprising: determining the preloading demand of an AR model according to the high-frequency feature set, and transferring corresponding model data from external storage to a dedicated memory by a GPU computing unit to generate a preloading data set, comprising: determining the preloading demand of an AR model according to the high-frequency feature set, and transferring corresponding model data from external storage to a dedicated memory by a GPU computing unit to generate a preloading data set, comprising: determining the preloading demand of an AR model according to the high-frequency feature set, and transferring corresponding model data from external storage to a dedicated memory by a GPU computing unit to generate a preloading data set, comprising: obtaining a voice instruction data stream, and directly calling an FPGA shared cache by a DSP computing unit to generate a scheduling instruction set for an instruction sequence with a real-time requirement higher than a preset threshold, comprising: obtaining a data stream generated by a voice instruction, and extracting an instruction sequence by a DSP computing unit by analyzing the data stream; 3. The method of claim 2, wherein, ​ ​ For the extracted instruction sequence, real-time detection is used to judge whether it is higher than the preset threshold to obtain an instruction set meeting the conditions; The instruction set meeting the conditions is stored in the shared cache of the FPGA to generate a cache data segment; The scheduling requirements are obtained from the cache data segment, and the DSP calculation unit generates a scheduling instruction set by priority sorting; For the scheduling instruction set, it is transmitted to the calculation unit through the shared cache to obtain an execution sequence; The K-means algorithm is used to group the execution sequence to generate a grouped instruction set; According to the grouped instruction set, it is stored in the FPGA shared cache to generate a final scheduling set.

4. The method of claim 3, wherein, The NPU calculation unit performs feature extraction on the high-frequency feature set, and if the extraction result matches the preloaded data set in the GPU calculation unit, the feature is transmitted to the GPU to generate a multi-modal feature group, including: The NPU calculation unit performs feature extraction on the high-frequency feature set to obtain an extraction result; If the extraction result matches the preloaded data set in the GPU calculation unit, a multi-modal feature group is generated through feature transmission; The multi-modal feature group is used to perform calculation and processing by the GPU calculation unit to obtain a calculation execution sequence; For the calculation execution sequence, data processing requirements are obtained, which are stored in the shared cache through unit cooperation to obtain a processed data segment; From the processed data segment, the feature distribution characteristics are extracted, and the K-means algorithm is used to group the feature distribution to obtain a grouped feature set; According to the grouped feature set, the feature is transmitted to the NPU calculation unit to generate an optimized feature sequence; For the optimized feature sequence, it is transmitted to the GPU calculation unit through unit cooperation to obtain a final execution set.

5. The method of claim 4, wherein, For the multi-modal feature group, the GPU calculation unit uses data layout optimization technology to adjust memory allocation, and processes the feature group through a matrix transformation algorithm to generate an optimized data block, including: The multi-modal feature group is obtained, and the GPU calculation unit performs matrix transformation on the feature group to obtain optimized data; For the optimized data, the GPU calculation unit adjusts the data layout to determine a memory allocation scheme; From the memory allocation scheme, the adjusted data is obtained, and the calculation unit processes the adjusted data through matrix transformation to obtain a transformed data block; If the transformed data block meets the preset threshold, a processed feature set is generated through data processing; According to the processed feature set, the memory allocation is adjusted using the optimization technique to obtain an allocation optimization set; For the allocation optimization set, the calculation unit generates a final data group through data layout adjustment; From the final data group, the feature distribution characteristics are obtained, and the K-means algorithm is used to group the characteristics to obtain a grouping result set.

6. The method of claim 5, wherein, After obtaining the optimized data block, the FPGA scheduling unit analyzes the cross-unit data stream requirements, and if the data transmission delay is lower than the preset threshold, the transmission order is adjusted through the scheduling algorithm to generate a data stream sequence, including: After obtaining the optimized data block, the FPGA scheduling unit determines the data stream requirements through cross-unit analysis to obtain a unit requirement distribution; From the transmission delay characteristics of the unit requirement distribution, if the transmission delay is lower than the preset threshold, the transmission order is adjusted through the scheduling algorithm to obtain an adjusted sequence; For the adjusted sequence, the FPGA scheduling unit analyzes the data stream sequence characteristics, determines the delay judgment result; According to the delay judgment result, the optimized transmission stream is generated by sequential adjustment, and the stream data group is obtained; From the stream data group, cross-unit analysis indicators are obtained, and the K-means algorithm is used for grouping the indicators to obtain a grouping feature set; For the grouping feature set, the FPGA scheduling unit generates the final sequence group by adjusting the data stream requirements to obtain a sequence optimization set; From the sequence optimization set, the transmission sequence characteristics are extracted, and the sequence integrity is judged by consistency checking to obtain the checked data group.

7. The method of claim 6, wherein, According to the data stream sequence, the FPGA scheduling unit allocates the optimized data block to the corresponding computing unit, and judges the matching degree of the multi-modal interaction data and the calculation by the consistency checking algorithm to obtain the matching result, including: The FPGA scheduling unit extracts the optimized data block from the data stream sequence by the allocation algorithm, and transmits the optimized data block to the computing unit by the distribution rule to obtain the distributed data group; From the distributed data group, the multi-modal interaction characteristics are obtained, and the correspondence between the interaction data and the computing unit is analyzed by the consistency checking algorithm to determine the matching degree distribution; For the matching degree distribution, the allocation sequence of the computing unit is sorted by the adjustment rule to obtain the sorted unit group; From the sorted unit group, the interaction data features are extracted, and the K-means algorithm is used for grouping the features to obtain a grouping feature set; For the grouping feature set, the optimized allocation scheme is generated by dynamic adjustment of the data stream sequence to obtain the adjusted data group; From the adjusted data group, the calculation matching degree index is obtained, and if the matching degree is lower than the preset threshold, the allocation sequence is adjusted by the checking algorithm to obtain the final data group; According to the final data group, the integrity of the data stream sequence and the computing unit is judged by the consistency checking to obtain the checked sequence group.

8. The method of claim 7, wherein, Adjust the data update frequency of the local cache and the shared cache through the matching result, and if the matching degree is higher than the preset threshold, use the sliding window algorithm to update the cache content to generate a new cache set, including: Through the matching result, the comparison relationship between the matching degree and the preset threshold is analyzed, and if the matching degree is higher than the preset threshold, the cache content is processed by the sliding window algorithm to obtain a new cache set; From the new cache set, the frequency distribution characteristics are obtained, and the update frequency of the local cache is adjusted according to the frequency distribution to obtain an adjusted frequency group; The cache content of the shared cache is updated through the adjusted frequency group, and the synchronization of the shared cache and the local cache is judged by the consistency checking to obtain a synchronization data group; From the synchronization data group, the data adjustment parameters are extracted, and the processing range of the sliding window is optimized according to the data adjustment parameters to obtain an optimized window set; The updated cache content is obtained by reprocessing the new cache set through the optimized window set, and the integrity of the cache content is determined; From the updated cache content, the trend of the frequency distribution is analyzed, and the update frequency of the local cache and the shared cache is adjusted according to the trend to obtain a final frequency group; For the final frequency group, the data consistency of the local cache and the shared cache is judged by the consistency checking to obtain a checked data group; After the new cache set is acquired, the NPU computing unit and the GPU computing unit re-extract features and process according to the data stream sequence, and if the feature variation trend exceeds the preset range, the final optimization scheme is generated by adjusting the algorithm through feedback to optimize the scheduling strategy, including: After the new cache set is acquired, the NPU computing unit and the GPU computing unit re-extract features and process according to the data stream sequence, and if the feature variation trend exceeds the preset range, the final optimization scheme is generated by adjusting the algorithm through feedback to optimize the scheduling strategy, including: After the new cache set is acquired, the NPU computing unit and the GPU computing unit re-extract features and process according to the data stream sequence, and if the feature variation trend exceeds the preset range, the final optimization scheme is generated by adjusting the algorithm through feedback to optimize the scheduling strategy, including: After the new cache set is acquired, the NPU computing unit and the GPU computing unit re-extract features and process according to the data stream sequence, and if the feature variation trend exceeds the preset range, the final optimization scheme is generated by adjusting the algorithm through feedback to optimize the scheduling strategy, including: After the new cache set is acquired, the NPU computing unit and the GPU computing unit re-extract features and process according to the data stream sequence, and if the feature variation trend exceeds the preset range, the final optimization scheme is generated by adjusting the algorithm through feedback to optimize the scheduling strategy, including: After the new cache set is acquired, the NPU computing unit and the GPU computing unit re-extract features and process according to the data stream sequence, and if the feature variation trend exceeds the preset range, the final optimization scheme is generated by adjusting the algorithm through feedback to optimize the scheduling strategy, including: After the new cache set is acquired, the NPU computing unit and the GPU computing unit re-extract features and process according to the data stream sequence, and if the feature variation trend exceeds the preset range, the final optimization scheme is generated by adjusting the algorithm through feedback to optimize the scheduling strategy, including: Using the cloud intelligent image processing method as claimed in any one of claims 1-8 through the controller.

9. A cloud intelligent image processing system, characterized in that, ​

Images